Compare commits

..
Author SHA1 Message Date
grenade b8098c4096 test(upstream): unique fixtures so web_pg can run twice
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m55s
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 5m4s
CI / Test (push) Successful in 8m17s
CI / CUDA type-check (push) Successful in 29m13s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Two tests inserted hard-coded secrets under unique indexes:

    verify_endpoint_consumes_token_once      "verify-raw-token-xyz"
      → email_tokens.token_hash (primary key)
    fingerprint_abuse_silently_deactivates…  "sk-helexa-deactivated-probe"
      → api_keys.key_hash (unique on active keys)

They pass against a virgin database and fail with `23505 duplicate key`
on every run afterwards. That looked like flakiness or parallel
contention — it is neither. CI always provisions a fresh database, so
the defect was invisible there and only appears when a developer runs
the suite twice against the same one, which is exactly when a fast local
loop matters most.

`unique_secret()` mirrors the existing `unique_email()`. Verified by
running the whole workspace three times against one database: all pass,
where runs 2 and 3 previously failed.

Neither test was removed. Single-use email-verification tokens and
silent deactivation on multi-account abuse are both real properties
worth holding; the fixtures were the problem, not the coverage.
2026-08-17 15:52:05 +03:00
153 changed files with 1874 additions and 13284 deletions
+2 -28
View File
@@ -105,11 +105,6 @@ jobs:
echo "commit_timestamp=${COMMIT_TIMESTAMP}" >> "$GITHUB_OUTPUT"
- id: changes
env:
# The push's own range. Empty on workflow_dispatch and all
# zeroes for a new branch; `decide` treats both as unusable and
# falls back to the full gate.
PUSH_BEFORE: ${{ github.event.before }}
run: |
set -ux
# Default: build everything. Detection only ever narrows
@@ -194,30 +189,9 @@ jobs:
BUILD_TOOLS=$(decide "$TOOLS_BASE" "$TOOLS_RE")
BUILD_ANGELS=$(decide "$ANGELS_BASE" "$ANGELS_RE")
if [ "$BUILD_CORTEX" = "true" ] || [ "$BUILD_NEURON" = "true" ] || [ "$BUILD_BENCH" = "true" ] || [ "$BUILD_UPSTREAM" = "true" ] || [ "$BUILD_ROUTER" = "true" ] || [ "$BUILD_TOOLS" = "true" ] || [ "$BUILD_ANGELS" = "true" ]; then
# Anything we are about to build gets linted and tested.
# This is the invariant that matters: no artefact reaches
# the fleet unchecked, however the branch below behaves.
CHECK_RUST=true
else
# Nothing is being built, so nothing will be published.
# The only question left is whether THIS PUSH touched
# Rust — and the honest base for that is the push's own
# range, not a package's publication lag.
#
# This used to ask "has anything Rust changed since
# cortex was last published". Cortex is only rebuilt when
# cortex paths change, so after a run of neuron-only work
# its base sits days behind and the diff from it sweeps in
# every unrelated Rust commit since. A one-character
# whitespace fix to a doc then re-ran fmt, clippy and the
# full test suite over a delta that had already been
# linted on the branch that introduced it (2026-08-18,
# base fec5b33).
#
# `decide` returns true for an unusable base, so a new
# branch (all-zero before-sha) or a force-push still gets
# the full gate.
CHECK_RUST=$(decide "$PUSH_BEFORE" "$RUST_RE")
CHECK_RUST=$(decide "$CORTEX_BASE" "$RUST_RE")
fi
fi
fi
@@ -262,7 +236,7 @@ jobs:
# fast instead of triggering a slower uncached rebuild; only a real
# sccache fault drops the cache. See script/ci-cargo-escalate.sh.
- name: Clippy (sccache escalation)
run: script/ci-cargo-escalate.sh cargo clippy --workspace --all-targets -- -D warnings
run: script/ci-cargo-escalate.sh cargo clippy --workspace -- -D warnings
test:
name: Test
+10 -105
View File
@@ -39,34 +39,24 @@ env:
CARGO_TARGET_DIR: target-${{ github.job }}
jobs:
# Two independent questions: does this change need the Rust gate, and
# does it need the web gate.
#
# The SPA under helexa.ai/ is built and shipped by its own deploy job and
# is not read by any crate, build script or spec file — so a change
# confined to it cannot alter a Rust result. Waiting up to 70 minutes for
# a CUDA type-check to prove that is pure latency, and the wait is what
# tempts people to merge front-end work unchecked. Documentation is inert
# the same way, and more obviously: no crate `include_str!`s a markdown
# file, no spec packages `doc/`, no build script reads one.
# tempts people to merge front-end work unchecked.
#
# Conversely a Rust-only change cannot break the SPA, so it need not
# spend three minutes on `npm ci` + lint + typecheck + build.
#
# The classification stays deliberately one-directional: it only ever
# narrows the gate when *every* changed path is known-inert, and every
# The classification is deliberately one-directional: it only ever
# narrows the gate when *every* changed path is front-end, and every
# other outcome — a tag, an unknown base, an empty diff, a mixed
# changeset — falls through to the full gate. A new top-level directory
# is therefore covered on the day it appears, without anyone
# remembering to add it here. Adding an inert pattern is the only way to
# widen what gets skipped, which is the property worth keeping.
# remembering to add it here.
changes:
name: Classify changes
timeout-minutes: 10
runs-on: fedora-43
outputs:
rust: ${{ steps.classify.outputs.rust }}
web: ${{ steps.classify.outputs.web }}
steps:
- uses: actions/checkout@v4
with:
@@ -74,12 +64,7 @@ jobs:
- id: classify
run: |
set -u
full() {
echo "→ full gate: $1"
echo "rust=true" >> "$GITHUB_OUTPUT"
echo "web=true" >> "$GITHUB_OUTPUT"
exit 0
}
full() { echo "→ full gate: $1"; echo "rust=true" >> "$GITHUB_OUTPUT"; exit 0; }
# Tags build the SRPMs from the whole tree, and the srpm jobs
# gate on fmt/clippy/test/cuda-check. Never narrow there.
@@ -101,51 +86,11 @@ jobs:
echo "changed files:"
printf '%s\n' "${files}" | sed 's/^/ /'
# Inert for Rust: the SPA, documentation, and fleet config.
# Markdown is matched only at the repo root and under doc/ — a
# .md inside crates/ is left to fall through, since
# `include_str!` could make one load-bearing without this file
# knowing.
#
# Fleet config (models.toml, asset/neuron/<host>.toml,
# asset/helexa-bench/<host>.toml) is read by a *running*
# daemon, never by a build script or an `include_str!`, so it
# cannot change a Rust result. Deploying one is a config rsync
# plus a service restart — neuron re-reads it and reloads the
# model — so spending ~25 minutes on fmt, clippy, test and a
# CUDA type-check to prove a one-line TOML edit compiles is
# pure latency (#291).
#
# The `config` job below stays ungated and still validates
# these files, so narrowing here removes the meritless work
# without removing the check that matters. build-prerelease
# already reaches the same conclusion on main: none of its
# per-package path regexes match these files, so a config-only
# push builds no binaries.
rust_relevant="$(printf '%s\n' "${files}" \
| grep -v '^helexa\.ai/' \
| grep -v '^doc/' \
| grep -vE '^[^/]+\.md$' \
| grep -vE '^asset/neuron/[^/]+\.toml$' \
| grep -vE '^asset/helexa-bench/[^/]+\.toml$' \
| grep -vE '^models\.toml$' \
|| true)"
if [ -n "${rust_relevant}" ]; then
echo "rust=true" >> "$GITHUB_OUTPUT"
echo "→ Rust gate: required"
else
echo "rust=false" >> "$GITHUB_OUTPUT"
echo "→ Rust gate: skipped (front-end and/or docs only)"
fi
other="$(printf '%s\n' "${files}" | grep -v '^helexa\.ai/' || true)"
[ -z "${other}" ] || full "changes outside helexa.ai/"
# The web gate is needed only when the SPA itself changed.
if printf '%s\n' "${files}" | grep -q '^helexa\.ai/'; then
echo "web=true" >> "$GITHUB_OUTPUT"
echo "→ web gate: required"
else
echo "web=false" >> "$GITHUB_OUTPUT"
echo "→ web gate: skipped (nothing under helexa.ai/)"
fi
echo "→ front-end only: skipping the Rust gate"
echo "rust=false" >> "$GITHUB_OUTPUT"
fmt:
name: Format
@@ -157,39 +102,10 @@ jobs:
- uses: actions/checkout@v4
- run: cargo fmt --check --all
# models.toml and asset/neuron/<host>.toml both describe how a model is
# served, and until #283 nothing forced them to agree. #252 is the bill
# for that: models.toml omitted `quant`, so a cortex cold-load served
# bf16 and could not prefill, while the same model was fine when the
# host loaded it from its own config. The sampling override (#283) is a
# second field with the same hazard, so the gate lands with it.
#
# Deliberately ungated by the `changes` classifier: it is a
# sub-second Python run with no build, and the whole point is that it
# cannot be skipped on the push that introduces the drift. A config
# change is neither `rust` nor `web`, so gating it on either would let
# a pure-TOML push through unchecked — precisely the push that breaks
# this.
config:
name: Config consistency
timeout-minutes: 5
runs-on: fedora-43
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
web:
name: Web (lint + typecheck + i18n + build)
timeout-minutes: 20
runs-on: fedora-43
needs: changes
if: needs.changes.outputs.web == 'true'
defaults:
run:
working-directory: helexa.ai
@@ -205,12 +121,6 @@ jobs:
# reached main exactly that way.
- run: npm run lint
- run: npm run typecheck
# Behaviour, not just types. Gating from the first commit was the
# deliberate call (#304): with no existing suite there is nothing to
# stabilise, so a warn-first period would only create a window in
# which a vacuous test lands and gets normalised — and a suite
# nobody trusts is worse than none, because it gets cited.
- run: npm test
# 42 locales with strict key parity is a promise that breaks
# quietly. Adding an English string without translating it must fail
# here, not in review, and certainly not in front of somebody
@@ -232,13 +142,8 @@ jobs:
# in sync with build-prerelease.yml): a signal death (rustc SIGSEGV
# / OOM-kill) keeps the cache and fails fast instead of an uncached
# rebuild; only a real sccache fault drops the cache.
# --all-targets so `#[cfg(test)]` modules, tests/, benches/ and
# examples/ are linted too (#286). Without it clippy sees only lib
# and bin targets, so `-D warnings` never gated a single line of
# test code — which is how two tests lost their `#[test]` attribute
# and stopped running unnoticed.
- name: Clippy (sccache escalation)
run: script/ci-cargo-escalate.sh cargo clippy --workspace --all-targets -- -D warnings
run: script/ci-cargo-escalate.sh cargo clippy --workspace -- -D warnings
test:
name: Test
+18 -426
View File
@@ -37,37 +37,7 @@ env:
${{ secrets.RSYNC_SSH_KEY }}
jobs:
# Gate the whole rollout on the two config files agreeing about every
# model they both describe. models.toml (cortex, cold loads) and
# asset/neuron/<host>.toml (resident models) each say how a model is
# served — quant, and since #283 the sampling override. When they
# disagree the model is served one way cold and another way resident,
# which is #252.
#
# Deliberately a *gate*, not a step inside deploy-cortex: an
# inconsistent pair must stop the neuron rollout too, and stopping it
# before anything restarts is much cheaper than after (a beast restart
# costs ~5-6 min of 27B cold-load).
config-check:
name: Config consistency
runs-on: fedora-43
if: >-
${{
github.event_name == 'workflow_dispatch'
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
deploy-cortex:
needs: [config-check]
runs-on: fedora-43
# Two trigger paths: manual dispatch always runs; workflow_run
# only runs if the upstream `build-prerelease` actually succeeded.
@@ -77,7 +47,6 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -96,33 +65,6 @@ jobs:
echo "inv=$(ssh gitea_ci@hanzalova.internal \
'systemctl show -P InvocationID cortex.service' || true)" >> "$GITHUB_OUTPUT"
# The catalogue is tracked (#283) precisely so it can be rolled out
# here rather than hand-copied. Sync it *before* the package step,
# so the restart that step performs is the one that picks it up —
# and report whether it changed, because a catalogue edit with no
# new RPM must still restart cortex or the change never takes
# effect.
#
# Compared by digest rather than rsync's own change detection so the
# answer is available to the shell that decides on the restart.
- name: Sync gateway catalogue (models.toml)
id: catalogue
run: |
local_sum=$(sha256sum models.toml | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@hanzalova.internal \
'sha256sum /etc/cortex/models.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "changed=0" >> "$GITHUB_OUTPUT"
echo "models.toml unchanged (${local_sum})"
else
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
models.toml gitea_ci@hanzalova.internal:/etc/cortex/models.toml
echo "changed=1" >> "$GITHUB_OUTPUT"
echo "models.toml updated: ${remote_sum:-<absent>} -> ${local_sum}"
fi
# Gating compares `rpm -q` against the packages.json manifest the
# publish job maintains — NOT unprivileged `dnf check-update`,
# which proved unreliable as the gitea_ci user (hung on metadata
@@ -133,11 +75,6 @@ jobs:
ssh gitea_ci@hanzalova.internal 'bash -s' <<'DEPLOY'
set -eu
pkg=cortex
# Default to 0 if the sync step somehow produced no output:
# `set -u` plus `[ "" -eq 0 ]` would abort the deploy on an
# arithmetic error rather than on anything meaningful.
config_changed="${{ steps.catalogue.outputs.changed }}"
config_changed="${config_changed:-0}"
# `rpm -q` prints "package X is not installed" on STDOUT and exits
# 1, so `|| echo not-installed` never fires — it captures the
# message text instead. Test the exit code.
@@ -153,31 +90,18 @@ jobs:
p = max(cands, key=lambda p: p.get("buildTime", 0))
print(p["version"] + "-" + p["release"])
' "${pkg}" 2>/dev/null || true)
# Skip only when BOTH the package and the catalogue are
# unchanged — a models.toml edit (a new sampling override, a
# corrected quant) must restart cortex even with no new RPM,
# exactly as the neuron drop-in does below.
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ] \
&& [ "${config_changed}" -eq 0 ]; then
echo "${pkg}-${installed} already current; models.toml unchanged — leaving service untouched"
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ]; then
echo "${pkg}-${installed} already current — leaving service untouched"
exit 0
fi
pkg_changed=1
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ]; then
pkg_changed=0
fi
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
echo "installed=${installed} published=${latest:-unknown} — deploying"
if systemctl is-active --quiet cortex.service; then
sudo /usr/bin/systemctl stop cortex.service
fi
# A catalogue-only change still restarts, but must not spend a
# dnf metadata refresh proving the package is already current.
if [ "${pkg_changed}" -eq 1 ]; then
if rpm -q "${pkg}" >/dev/null 2>&1; then
sudo /usr/bin/dnf upgrade --refresh --allowerasing -y cortex
else
sudo /usr/bin/dnf install --refresh --allowerasing -y cortex
fi
if rpm -q "${pkg}" >/dev/null 2>&1; then
sudo /usr/bin/dnf upgrade --refresh --allowerasing -y cortex
else
sudo /usr/bin/dnf install --refresh --allowerasing -y cortex
fi
sudo /usr/bin/systemctl daemon-reload
# enable --now: start the service AND enable it for boot so the
@@ -218,7 +142,7 @@ jobs:
CAPTURE
deploy-neurons:
needs: [config-check, deploy-cortex]
needs: [deploy-cortex]
runs-on: fedora-43
strategy:
# One neuron failing must not cancel the others. Cortex is up
@@ -253,7 +177,6 @@ jobs:
load_timeout: 300
max_prompt_tokens: 16384
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -269,42 +192,6 @@ jobs:
echo "inv=$(ssh gitea_ci@${{ matrix.host }} \
'systemctl show -P InvocationID neuron.service' || true)" >> "$GITHUB_OUTPUT"
# asset/neuron/<short>.toml is this host's neuron.toml. It carries
# the resident-model specs — including the #283 sampling override,
# which is inert until the daemon reloads the model, so a change
# here has to restart the neuron the same way the context-cap
# drop-in does.
#
# Synced before the package step so the restart that step performs
# is the one that picks it up. Digest-compared rather than trusting
# rsync's own change detection, because the restart decision needs
# the answer and a needless neuron restart costs ~5-6 min of 27B
# cold-load (#1).
- name: Sync neuron config (${{ matrix.host }})
id: neuronconf
run: |
short="${{ matrix.host }}"; short="${short%%.*}"
src="asset/neuron/${short}.toml"
if [ ! -f "${src}" ]; then
echo "changed=0" >> "$GITHUB_OUTPUT"
echo "no ${src} in the repo — leaving this host's config alone"
exit 0
fi
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@${{ matrix.host }} \
'sha256sum /etc/neuron/neuron.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "changed=0" >> "$GITHUB_OUTPUT"
echo "${src} unchanged (${local_sum})"
else
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@${{ matrix.host }}:/etc/neuron/neuron.toml
echo "changed=1" >> "$GITHUB_OUTPUT"
echo "${src} updated: ${remote_sum:-<absent>} -> ${local_sum}"
fi
# See deploy-cortex for why gating uses the publish manifest and
# not unprivileged `dnf check-update`.
- name: Deploy helexa-neuron-${{ matrix.flavour }} (skips when already current)
@@ -313,9 +200,6 @@ jobs:
set -eu
pkg=helexa-neuron-${{ matrix.flavour }}
max_prompt_tokens="${{ matrix.max_prompt_tokens }}"
# A neuron.toml change counts as a config change for the
# restart decision below, exactly like the drop-in.
neuron_toml_changed="${{ steps.neuronconf.outputs.changed }}"
# ── Desired per-model systemd drop-in ─────────────────────────
# model.conf carries NEURON_MAX_PROMPT_TOKENS so the context cap
@@ -324,7 +208,7 @@ jobs:
# deploy-managed value wins over any manual local override of the
# same variable. See doc/context-limits.md.
conf=/etc/systemd/system/neuron.service.d/model.conf
config_changed=${neuron_toml_changed:-0}
config_changed=0
if [ -n "${max_prompt_tokens}" ]; then
desired=$(printf '%s\n%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
@@ -358,17 +242,13 @@ jobs:
# Skip only when BOTH the package and the drop-in are unchanged —
# a context-cap change must restart the neuron even with no new RPM.
if [ "${pkg_changed}" -eq 0 ] && [ "${config_changed}" -eq 0 ]; then
echo "${pkg}-${installed} current; NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens:-<unset>} and neuron.toml unchanged — leaving service untouched"
echo "${pkg}-${installed} current; NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens:-<unset>} unchanged — leaving service untouched"
exit 0
fi
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
# Write the drop-in (staged in gitea_ci's dir, installed root-owned).
# Guarded on max_prompt_tokens as well as config_changed: since
# a neuron.toml change also sets config_changed, an entry with
# no max_prompt_tokens would otherwise reach this with `desired`
# unset.
if [ "${config_changed}" -eq 1 ] && [ -n "${max_prompt_tokens}" ]; then
if [ "${config_changed}" -eq 1 ]; then
printf '%s\n' "${desired}" > /var/lib/gitea_ci/model.conf
sudo /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf "${conf}"
rm -f /var/lib/gitea_ci/model.conf
@@ -532,161 +412,6 @@ jobs:
# dependency on cortex or the neurons (the sweep loop is version-aware
# and picks up whatever each neuron reports whenever). So it runs
# alongside the cortex→neurons chain rather than after it.
# The vhost sync gets its own job, with NO `needs`, because it is not
# part of deploying the gateway binary and must not be able to stop
# one.
#
# It was briefly a step inside deploy-cortex, and that broke the fleet
# on its first run (2026-08-27, run 1284): the step hard-fails until an
# operator has run script/infra-setup.sh to install its sudoers grant,
# a failed step fails the job, and `deploy-neurons` has
# `needs: [deploy-cortex]` — so a missing nginx permission silently
# skipped the rollout of all three neuron flavours. The fleet stayed
# two commits behind while the run reported only "deploy-cortex:
# failure".
#
# #287 names this ordering trap. The lesson is narrower than "order the
# steps": a step whose prerequisite is installed by hand does not
# belong in a job that something else depends on, because its failure
# is not a statement about that job. Moving it later within
# deploy-cortex would not have helped — the job still ends red and the
# dependents still skip.
#
# Failing hard here is still right: drift between the repo and the live
# file is a real defect, and this job going red says that and nothing
# else.
deploy-nginx:
runs-on: fedora-43
if: >-
${{
github.event_name == 'workflow_dispatch'
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
gitea_ci@hanzalova.internal 'hostname -f'
# nginx vhost config for the internal site. Version-controlled in
# `asset/nginx/` since the beginning, but until now only ever
# installed by `script/infra-setup.sh` — a one-time, run-by-hand
# script. So the repo copy and the live file could diverge with
# nothing to detect it, which is the #252 shape: two sources of
# truth for one thing and nothing forcing agreement.
#
# Writes to sites-available and (re)points sites-enabled at it as
# a symlink. On this host sites-enabled was a *regular file*, not a
# symlink — two independent copies, of which nginx reads one. An
# operator editing "the config" would reasonably edit
# sites-available, watch `nginx -t` pass, reload, and see nothing
# change. Making the link canonical is part of the sync rather
# than a separate chore, so the layout cannot drift back.
#
# `nginx -t` gates the reload: a config that fails validation is
# left un-reloaded with the previous one still serving.
- name: Sync internal nginx vhost
run: |
src="asset/nginx/helexa.internal.conf"
# The sudoers drop-in that permits this is installed by
# script/infra-setup.sh, which is run by hand, so the live copy
# can lag the repo by however long it has been since anyone ran
# it. Check EVERY grant the step needs, not a representative
# one: the first version of this preflight tested `nginx -t`
# alone, which had existed for ages, so it passed and the job
# then died on the two helexa-ratelimit.conf grants added in the
# same commit as the check. A preflight that names a remedy
# while probing the wrong permission is worse than none — it
# tells you the sudoers are fine seconds before they are not.
#
# Each probe runs the command in a harmless form that still
# matches its sudoers pattern (sha256sum reads; nginx -t and
# `systemctl show` do not mutate). rsync and ln have no such
# form, so they are checked with `sudo -nl`, which reports
# whether a command is permitted without running it.
missing=""
probe() { # probe <description> <command...>
local what="$1"; shift
ssh gitea_ci@hanzalova.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "nginx -t" \
'sudo -n /usr/sbin/nginx -t'
probe "sha256sum /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf'
probe "sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "rsync -> /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/sites-available/helexa.internal.conf'
probe "rsync -> /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "ln -sfn sites-available -> sites-enabled" \
'sudo -nl /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf'
probe "systemctl reload nginx" \
'sudo -nl /usr/bin/systemctl reload nginx'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing nginx sudoers grants on hanzalova:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/cortex-host.conf. Having run" >&2
echo " it before is not enough — grants are added to that file as deploy" >&2
echo " steps grow, and the host keeps whatever revision it last saw." >&2
exit 1
fi
rl="asset/nginx/helexa-ratelimit.conf"
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
remote_rl=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
link_ok=$(ssh gitea_ci@hanzalova.internal \
'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
if [ "${local_sum}" = "${remote_sum}" ] \
&& [ "${local_rl}" = "${remote_rl}" ] \
&& [ "${link_ok}" = "yes" ]; then
echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
exit 0
fi
# Order matters: the vhost references names defined at http
# context by helexa-ratelimit.conf (`limit_req_zone`s and the
# `helexa_noquery` log format). Ship that first, or a vhost
# referencing a not-yet-installed name fails `nginx -t` and the
# deploy stops with the two files out of step.
echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
set -eu
sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
/etc/nginx/sites-enabled/helexa.internal.conf
if ! sudo /usr/sbin/nginx -t; then
echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
exit 1
fi
sudo /usr/bin/systemctl reload nginx
echo "nginx reloaded"
NGINX
# Roll helexa-router + helexa-upstream onto the federation service host
# (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
# two services are independent daemons on one host, so one job handles
@@ -700,10 +425,6 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
# Needed since "Sync router config" reads helexa-router.toml out of
# the tree. This job managed without a checkout while it only ran
# package upgrades over ssh.
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -771,96 +492,6 @@ jobs:
done
DEPLOY
# Router config (#287). Tracked because it holds no secrets, and
# synced here because being untracked is what let its
# `helexa/balanced` alias keep pointing at a retired model until
# beast fell over on 2026-08-27.
#
# After the package step, deliberately: this needs a sudoers grant
# installed by the hand-run script/infra-setup.sh, so on a host that
# has not seen the current grants it fails — and a step that fails
# before the package work would hold the service binaries hostage to
# an unrelated permission. deploy-gallumbits has no dependent jobs,
# so the blast radius stops here either way.
#
# Restarts only on a digest change: the router re-reads its aliases
# at startup, and a needless restart drops in-flight federation
# requests.
- name: Sync router config
run: |
missing=""
probe() {
local what="$1"; shift
ssh gitea_ci@gallumbits.kosherinata.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "sha256sum /etc/helexa-router/helexa-router.toml" \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml'
probe "rsync -> /etc/helexa-router/helexa-router.toml" \
'sudo -nl /usr/bin/rsync -a x /etc/helexa-router/helexa-router.toml'
probe "systemctl restart helexa-router.service" \
'sudo -nl /usr/bin/systemctl restart helexa-router.service'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing sudoers grants on gallumbits:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/gallumbits-host.conf." >&2
exit 1
fi
local_sum=$(sha256sum helexa-router.toml | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "router config unchanged (${local_sum})"
exit 0
fi
# Two questions, not one. The bytes decide whether to SYNC; the
# parsed settings decide whether to RESTART. The first run of
# this step proved why they differ: the only change was a
# comment header, and it bounced the router anyway, dropping
# whatever federation requests were in flight. The router reads
# its aliases at startup, so a comment is not a reason to
# interrupt anyone.
#
# Reading the remote copy into CI is safe here precisely because
# this file holds no secrets — the same property that lets it be
# tracked at all.
#
# The `cat` grant is deliberately NOT in the preflight above: if
# it is missing the read yields nothing, the comparison reports
# "changed", and the step restarts on any byte difference. That
# is the old behaviour, not a broken one, so a host mid-way
# through a sudoers rollout degrades instead of failing.
remote_copy="$(mktemp)"
trap 'rm -f "${remote_copy}"' EXIT
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/cat /etc/helexa-router/helexa-router.toml' \
> "${remote_copy}" 2>/dev/null || true
settings_changed=yes
if python3 script/toml-settings-equal.py "${remote_copy}" helexa-router.toml; then
settings_changed=no
fi
echo "syncing helexa-router.toml: ${remote_sum:-<absent>} -> ${local_sum}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
helexa-router.toml \
gitea_ci@gallumbits.kosherinata.internal:/etc/helexa-router/helexa-router.toml
if [ "${settings_changed}" = yes ]; then
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/systemctl restart helexa-router.service'
echo "helexa-router restarted: settings changed"
else
echo "helexa-router left running: only comments or formatting changed"
fi
- name: Ensure firewalld allows helexa-router + helexa-upstream + helexa-tools + helexa-angels
run: |
ssh gitea_ci@gallumbits.kosherinata.internal '
@@ -903,7 +534,6 @@ jobs:
CAPTURE
deploy-bench:
needs: [config-check]
runs-on: fedora-43
if: >-
${{
@@ -911,7 +541,6 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -927,28 +556,6 @@ jobs:
echo "inv=$(ssh gitea_ci@bob.hanzalova.internal \
'systemctl show -P InvocationID helexa-bench.service' || true)" >> "$GITHUB_OUTPUT"
# Same contract as the gateway and neuron configs: the repo is the
# only writer, so a config edit must reach the host and restart the
# service even when no new RPM landed.
- name: Sync bench config
id: benchconf
run: |
src="asset/helexa-bench/bob.toml"
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@bob.hanzalova.internal \
'sha256sum /etc/helexa-bench/helexa-bench.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "changed=0" >> "$GITHUB_OUTPUT"
echo "${src} unchanged (${local_sum})"
else
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@bob.hanzalova.internal:/etc/helexa-bench/helexa-bench.toml
echo "changed=1" >> "$GITHUB_OUTPUT"
echo "${src} updated: ${remote_sum:-<absent>} -> ${local_sum}"
fi
# See deploy-cortex for why gating uses the publish manifest and
# not unprivileged `dnf check-update`.
- name: Deploy helexa-bench (skips when already current)
@@ -956,8 +563,6 @@ jobs:
ssh gitea_ci@bob.hanzalova.internal 'bash -s' <<'DEPLOY'
set -eu
pkg=helexa-bench
config_changed="${{ steps.benchconf.outputs.changed }}"
config_changed="${config_changed:-0}"
# `rpm -q` prints "package X is not installed" on STDOUT and exits
# 1, so `|| echo not-installed` never fires — it captures the
# message text instead. Test the exit code.
@@ -973,25 +578,18 @@ jobs:
p = max(cands, key=lambda p: p.get("buildTime", 0))
print(p["version"] + "-" + p["release"])
' "${pkg}" 2>/dev/null || true)
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ] \
&& [ "${config_changed}" -eq 0 ]; then
echo "${pkg}-${installed} already current; config unchanged — leaving service untouched"
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ]; then
echo "${pkg}-${installed} already current — leaving service untouched"
exit 0
fi
pkg_changed=1
if [ -n "${latest}" ] && [ "${latest}" = "${installed}" ]; then
pkg_changed=0
fi
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
echo "installed=${installed} published=${latest:-unknown} — deploying"
if systemctl is-active --quiet helexa-bench.service; then
sudo /usr/bin/systemctl stop helexa-bench.service
fi
if [ "${pkg_changed}" -eq 1 ]; then
if rpm -q "${pkg}" >/dev/null 2>&1; then
sudo /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-bench
else
sudo /usr/bin/dnf install --refresh --allowerasing -y helexa-bench
fi
if rpm -q "${pkg}" >/dev/null 2>&1; then
sudo /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-bench
else
sudo /usr/bin/dnf install --refresh --allowerasing -y helexa-bench
fi
sudo /usr/bin/systemctl daemon-reload
# enable --now: start the service AND enable it for boot so the
@@ -1125,12 +723,6 @@ jobs:
cd helexa.ai
npm ci
npm run i18n:check
# Also run here, not just in ci.yml, for the same reason
# i18n:check is: pushes to main skip ci.yml, so this is the only
# gate a merge itself passes through. Cheap, and it is the exact
# case the web CI comment describes — a defect reaching main and
# surfacing as a broken deploy afterwards.
npm test
npm run build
- name: SSH init
+2 -14
View File
@@ -8,23 +8,11 @@ helexa.ai/.env.local
*.swo
.idea/
.vscode/
# Secret-bearing service configs. These carry API keys, a JWT secret, DB
# passwords and SMTP credentials, so they are hand-synced by
# script/infra-setup.sh rather than shipped by CI (#287 Gap 1).
#
# helexa-router.toml is deliberately NOT here: it holds no secrets, only
# a listen address, the product-tier aliases and a mesh endpoint. It was
# ignored by association, and that is precisely how its `helexa/balanced`
# alias came to point at a retired model for long enough to take beast
# down (2026-08-27). A file nothing can diff is a file nothing can catch.
# Operator-owned credential layer, merged after the CI-deployed config
# by cortex / helexa-upstream / helexa-angels. Must never be committed.
secrets.toml
**/secrets.toml
cortex.toml
models.toml
doc/plan/*
/target-cuda/
.claude/
helexa-router.toml
helexa-upstream.toml
helexa-angels.toml
+3 -4
View File
@@ -167,7 +167,7 @@ doc-comment; touch points (the `Job` enum, the dispatch handlers, the
cargo build --release # build all crates
cargo run -p cortex-cli -- serve # run the gateway
cargo test # run all tests
cargo clippy --workspace --all-targets # lint (test code included)
cargo clippy --workspace # lint
```
## CI
@@ -177,7 +177,7 @@ pass before merging:
```sh
cargo fmt --check --all # formatting
cargo clippy --workspace --all-targets -- -D warnings # lint (warnings are errors)
cargo clippy --workspace -- -D warnings # lint (warnings are errors)
cargo test --workspace # tests
```
@@ -191,8 +191,7 @@ Work each change on its own branch; `main` stays releasable.
1. Implement on a feature branch (`fix/<issue>-…`, `feat/<issue>-…`).
2. Run the CI triad locally (`cargo fmt --check --all`,
`cargo clippy --workspace --all-targets -- -D warnings`,
`cargo test --workspace`).
`cargo clippy --workspace -- -D warnings`, `cargo test --workspace`).
Local builds are **CPU-only** — the `#[cfg(feature = "cuda")]` neuron/TP
paths do NOT compile locally. The branch CI's **CUDA type-check** job is
the only thing that validates them, so for any neuron change the push to
-25
View File
@@ -1,10 +1,3 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on the host are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# helexa-bench config for bob.hanzalova.internal.
#
# Synced to /etc/helexa-bench/helexa-bench.toml by script/infra-setup.sh
@@ -22,20 +15,6 @@ iteration_pause_secs = 2
request_timeout_secs = 600
db_path = "/var/lib/helexa-bench/bench.sqlite"
# Identity bench presents to neurons (#288). Without it bench is
# anonymous, and since #262 anonymous callers are served only from
# leftover capacity — capped below max_in_flight and parked at the class
# gate. That moved beast's concurrency:8 from 168.9 tok/s / 0.53s
# ttft_p95 to 104.6 / 11.77s the day #262 landed and held there, so every
# number since measured the yield policy, not serving capacity.
#
# A dedicated pair, not a borrowed one: bench then holds its own
# fair-share allocation (#54) and cannot starve interactive traffic —
# the failure #262 existed to fix.
[bench.principal]
account_id = "helexa-bench"
key_id = "fleet-benchmark"
[scenarios]
prompt_sizes = [128, 4096]
max_tokens = 256
@@ -43,10 +22,6 @@ max_tokens = 256
# carries p95-under-concurrency data before the F3 A/B gate (#94) needs
# it for comparison. Levels mirror the real a0/hermes/opencode fan-out.
concurrency_levels = [2, 4, 8]
# Non-streaming bursts (#285/#288). Kept to the top level: it is where
# serialization shows most starkly and each level costs a full burst per
# sample per build.
concurrency_nonstreaming_levels = [8]
concurrency_prompt_tokens = 512
# Capability probes (#91) — the reasoning/planning axis the speed
+8 -92
View File
@@ -1,16 +1,9 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on the host are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for beast.hanzalova.internal
#
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.8-27B with
# q6k ISQ across both GPUs at activation, matching the validate-neuron
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.6-27B with
# q5k ISQ across both GPUs at activation, matching the validate-neuron
# invocation: `validate-neuron.sh beast.hanzalova.internal
# Qwen/Qwen3.8-27B q6k 2`.
# Qwen/Qwen3.6-27B q5k 2`.
#
# Synced to /etc/neuron/neuron.toml by script/infra-setup.sh. Edits
# take effect after the next deploy workflow run restarts the service
@@ -29,93 +22,16 @@ name = "candle"
# count. NEURON_BATCHING=0 (systemd drop-in) is the kill switch;
# removing this section reverts to batch-1 serialization.
[harness.candle.admission]
# 8. max_in_flight bounds how many requests run concurrently. It does
# NOT partition the KV budget: since #257 (`eca42510`, 2026-08-15) KV is
# a shared pool and each request reserves what its own prompt needs via
# `enter_with_kv(principal, kv_reservation_mb(prompt_len, ...))`. A
# request too big for what is free waits for bytes to come back rather
# than being handed a fixed 1/N slice.
#
# This value was briefly set to 2 (#291) on the stated grounds that "the
# KV budget is divided by it at load, so 8 gives each slot 18,852
# tokens". That was true before #257 and false after it, and #291 was
# filed and actioned after that landed. Nothing in the tree divides the
# budget by the slot count — `kv_budget` is a semaphore of MiB permits
# and the engine's `max_slots` is only the batch width.
#
# The concurrency measurement in #291 stands as data (89.6% of busy time
# at one request in flight, 4.9% utilisation over seven days) but the
# conclusion drawn from it does not, and the traffic it described has
# changed: while it was taken, the router's `helexa/balanced` alias
# pointed web chat at a second 27B, so per-model concurrency was split
# across two models that were evicting each other. With both tiers now
# resolving to this model, chat concurrency lands here.
#
# What actually bounds a busy moment is the pool: 3,689 MiB at
# 32 KiB/token/card is ~118k tokens of KV to share. Eight ordinary chat
# turns fit easily; a few very long agentic sessions will queue on bytes,
# which is the correct behaviour and is what #257 built.
max_in_flight = 8
# Snapshot budget for the cross-request prefix cache (#292).
#
# 3 GiB, not the 1 GiB default. At the measured 32 KiB/token/card, a
# 1 GiB budget cannot hold a snapshot beyond ~32k tokens of K/V alone —
# less once the GatedDeltaNet conv/recurrent state is deep-copied on top,
# which on this arch is 48 of 64 layers in f32. Agentic sessions here run
# well past that: `cached_tokens` was observed pinned at 2,016 for ten
# consecutive turns while the prompt grew to 52,869, so every turn
# re-prefilled ~50k tokens — 37 s each, against ~20 s of actual decode.
# The cache was not missing; the snapshot would not fit.
#
# The snapshot budget is reserved BEFORE the KV budget, not from what is
# left over: measured on this host, kv_budget_mb = 5737 - budget_mb,
# exactly 1:1. So this number is a direct trade against how long a
# single session can get, and both sides have a floor:
#
# budget_mb kv_budget tokens/slot (2 slots) holds a 52.9k snapshot
# 1024 4713 75,400 no ← the #292 symptom
# 3072 2665 42,600 yes, but sessions
# cannot reach 52.9k
# 2048 3689 59,000 yes
#
# 2 GiB is the value that satisfies both: a 52,869-token snapshot is
# ~1,652 MB of K/V plus the GatedDeltaNet state (~1.73 GB total), and
# per-slot KV still covers sessions comfortably past the deepest
# observed. 3 GiB was tried first and starved KV — a session could no
# longer reach the size whose snapshot the cache was being enlarged to
# hold, which is self-defeating.
#
# max_entries stays at the default 8: the byte budget binds first here,
# so the count cap is not what is forcing snapshots to be small.
[harness.candle.prefix_cache]
budget_mb = 2048
[[default_models]]
# Qwen3.8-27B is the resident flagship (#251). It replaced the
# 3.6-27B — architecturally identical, same model_type, layer counts and
# vision tower — so the load spec is unchanged.
#
# The 3.6-27B is no longer catalogued (retired 2026-08-27). While both
# were listed at the same residency class they could displace each other
# in either direction, which is what let a stale `helexa/balanced` alias
# evict this model on every authenticated web-chat turn. See models.toml
# for the incident that ended the arrangement.
# Qwen3.8-27B is the resident flagship (#251). Architecturally identical
# to the 3.6-27B it replaced — same model_type, layer counts and vision
# tower — so the load spec is unchanged. The 3.6-27B stays catalogued at
# the same residency class, so it can still be cold-loaded on demand and
# the two displace each other in either direction.
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
tensor_parallel = 2
devices = [0, 1]
# Sampling override (#283) — MUST match the `[models.sampling]` block on
# this model in models.toml. Two files describe one model; only
# script/check-config-consistency.py forces them to agree, and #252 is
# what happens when nothing does.
#
# The model's own generation_config.json says temperature = 1.0, which
# measured a 20% structural-defect rate on ~2k-token code generation
# against 0/60 at <= 0.6 (Fisher exact one-sided p = 0.0031). Precedence
# is request > operator > model, so a caller naming a temperature still
# gets what it asked for.
[default_models.sampling]
temperature = 0.6
-7
View File
@@ -1,10 +1,3 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on the host are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for benjy.hanzalova.internal
#
# 1x RTX 4090 (24 GB) — largest single-GPU host on the fleet. Pre-warms
-7
View File
@@ -1,10 +1,3 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on the host are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for quadbrat.hanzalova.internal
#
# 1x RTX 3060 (12 GB) — small / quantised tier. Pre-warms Qwen3-1.7B
-21
View File
@@ -24,24 +24,3 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
limit_req_status 429;
# Access-log format for the search locations, which must not record what
# people searched for.
#
# The default `main` format logs `$request` — method, full URI *and
# query string*. For a search proxy that means every query a user types
# is written to /var/log/nginx/access.log next to their IP, for
# authenticated and anonymous visitors alike. SearXNG's own web UI
# defaults to POST partly to avoid exactly this.
#
# `$uri` is the path with the query stripped, so the operational signal
# survives — who called, when, which endpoint, what status, how big, how
# long — and the query itself does not. `$request_method` is kept
# explicit since `$uri` drops it.
#
# Rate-limit debugging and abuse detection both work from this: they
# need counts, statuses and client addresses, never the search terms.
log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
+1 -37
View File
@@ -73,9 +73,6 @@ server {
location = /tools/fetch {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# The `url=` parameter is as revealing as a search query: it is
# the page a user is having read to them. Same treatment.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -86,40 +83,7 @@ server {
# federation service host. GET only; the SPA requests format=json.
location = /tools/web_search {
limit_req zone=helexa_tools burst=10 nodelay;
# GET and POST: SearXNG accepts both, and a client that POSTs
# keeps its query out of any intermediary's request line. Other
# methods have no meaning here.
limit_except GET POST { deny all; }
# A search query is small. Bounding the body stops this becoming
# a way to push arbitrary bytes at the upstream.
client_max_body_size 8k;
# Never log the query — see helexa_noquery in
# conf.d/helexa-ratelimit.conf.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 30s;
}
# The same upstream under SearXNG's own path, for clients that speak
# the SearXNG API rather than our app's.
#
# `/tools/web_search` is our name for it, and a generic SearXNG
# client cannot guess it: they take a base URL and append `/search`,
# because that is what every SearXNG instance serves. Without this
# block such a request falls through to `location /` and is answered
# by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
# client. A 404 would at least be honest; 200-with-HTML sends people
# looking for a bug in their own parser.
#
# Same upstream, same rate-limit zone, same GET-only restriction:
# this is an alias, not a second, laxer door.
location = /search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET POST { deny all; }
client_max_body_size 8k;
access_log /var/log/nginx/access.log helexa_noquery;
limit_except GET { deny all; }
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-12
View File
@@ -13,18 +13,6 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
# deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
# sites-enabled a symlink to it, validate, reload. Scoped to the one
# file and the one service — `nginx -t` is included because a reload
# must never happen on a config nginx has rejected.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service
+3 -25
View File
@@ -4,17 +4,9 @@
# rolls out helexa-router (data plane), helexa-upstream (account/budget
# authority), helexa-tools (grounding fetcher) and helexa-angels (the
# confidential investor portal) package upgrades.
# The SECRET-BEARING configs are not synced by CI — helexa-upstream.toml
# and helexa-angels.toml carry a DB password, a JWT secret and SMTP
# credentials, and ride in script/infra-setup.sh under the operator's own
# sudo instead (#287 Gap 1, still open).
#
# helexa-router.toml is different and IS synced here: it holds no secrets,
# only a listen address, the product-tier aliases and a mesh endpoint. It
# was excluded by association with its neighbours, which is how its
# `helexa/balanced` alias came to point at a model retired from the
# catalogue and took beast down on 2026-08-27. A config CI cannot write is
# a config no check can defend.
# Configs are NOT synced by CI — they carry secrets (DB password, JWT
# secret, operator tokens) and ride in script/infra-setup.sh under the
# operator's own sudo instead.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-router.service
@@ -29,20 +21,6 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload
# deploy-gallumbits' "Sync router config" step. Scoped to the one file:
# the neighbouring service configs stay operator-only until #287 Gap 1
# gives them a render-from-secrets path.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml
# Read-back, so the sync can tell a settings change from a comment
# change and restart only for the former. Safe to grant and safe to read
# into CI because this file holds no secrets — the same property that
# lets it be tracked. If this grant is absent the step degrades to
# restarting on any byte change, which is why it is not in the step's
# fail-hard preflight.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/cat /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl restart helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-upstream
+1 -8
View File
@@ -1,10 +1,4 @@
import type {
Dimensions,
MeasurementRegime,
ReportRow,
RunRow,
SeriesPoint,
} from "./types";
import type { Dimensions, ReportRow, RunRow, SeriesPoint } from "./types";
// Empty default → `fetch('/api/...')` hits the dev proxy (vite.config.ts)
// or the same origin. For a separately-hosted build, set VITE_API_BASE to
@@ -21,7 +15,6 @@ async function getJson<T>(path: string): Promise<T> {
export const getDimensions = () => getJson<Dimensions>("/api/dimensions");
export const getSummary = () => getJson<ReportRow[]>("/api/summary");
export const getRegimes = () => getJson<MeasurementRegime[]>("/api/regimes");
// host is resolved server-side (each model maps to one host today), so the
// public UI selects by model + scenario alone.
+86 -288
View File
@@ -11,8 +11,8 @@ import {
XAxis,
YAxis,
} from "recharts";
import { getDimensions, getRegimes, getSeries } from "../api";
import type { Dimensions, MeasurementRegime, SeriesPoint } from "../types";
import { getDimensions, getSeries } from "../api";
import type { Dimensions, SeriesPoint } from "../types";
import { BASELINE_SOURCE, baselineFor } from "../baseline";
function Picker({
@@ -40,95 +40,11 @@ function Picker({
);
}
type SeriesDef = {
key: string;
name: string;
stroke: string;
dashed?: boolean;
};
/** One titled chart over the shared build timeline.
*
* Every panel draws the same x-axis and the same regime divider, so a
* reader can line a change up across metrics — which is the whole point
* of having more than two of them. */
/** A vertical rule at a build where the metric changed meaning. */
type Rule = { at: string; label: string; detail?: string };
function MetricChart({
title,
hint,
data,
lines,
rules,
unit,
}: {
title: string;
hint?: string;
data: Record<string, unknown>[];
lines: SeriesDef[];
rules?: Rule[];
unit?: string;
}) {
const hasAny = data.some((d) => lines.some((l) => d[l.key] != null));
if (!hasAny) return null;
return (
<>
<h5 className="mt-4">
{title}
{unit ? <span className="text-muted fw-normal"> ({unit})</span> : null}
</h5>
{hint && <p className="text-muted small mb-2">{hint}</p>}
<ResponsiveContainer width="100%" height={260}>
<LineChart data={data} margin={{ top: 8, right: 24, bottom: 8, left: 0 }}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="label" />
<YAxis />
<Tooltip />
<Legend />
{(rules ?? [])
// Only draw a rule whose build is actually on this x-axis;
// recharts would otherwise place it at the origin and imply
// the boundary sits before all the data.
.filter((r) => data.some((d) => d.label === r.at))
.map((r) => (
<ReferenceLine
key={`${r.at}-${r.label}`}
x={r.at}
stroke="#bbb"
strokeDasharray="3 3"
label={{
value: r.label,
position: "top",
fill: "#999",
fontSize: 11,
}}
/>
))}
{lines.map((l) => (
<Line
key={l.key}
type="monotone"
dataKey={l.key}
name={l.name}
stroke={l.stroke}
strokeDasharray={l.dashed ? "5 5" : undefined}
connectNulls
dot={false}
/>
))}
</LineChart>
</ResponsiveContainer>
</>
);
}
export default function Trends() {
const [dims, setDims] = useState<Dimensions | null>(null);
const [model, setModel] = useState("");
const [scenario, setScenario] = useState("");
const [series, setSeries] = useState<SeriesPoint[]>([]);
const [regimes, setRegimes] = useState<MeasurementRegime[]>([]);
const [err, setErr] = useState<string | null>(null);
useEffect(() => {
@@ -139,10 +55,6 @@ export default function Trends() {
if (d.scenarios[0]) setScenario(d.scenarios[0]);
})
.catch((e) => setErr(String(e)));
// A missing/older API just means no rules — not a page failure.
getRegimes()
.then(setRegimes)
.catch(() => setRegimes([]));
}, []);
useEffect(() => {
@@ -156,7 +68,10 @@ export default function Trends() {
// Prepend the pre-helexa-bench baseline (dashed, separate keys) so it
// anchors the timeline without being merged into the live line. Different
// measurement regime — see baseline.ts / doc/benchmarks.md.
const base = useMemo(() => baselineFor(model, scenario), [model, scenario]);
const base = useMemo(
() => baselineFor(model, scenario),
[model, scenario],
);
const data = useMemo(
() => [
...base.map((p) => ({
@@ -170,14 +85,6 @@ export default function Trends() {
ttft: p.ttft_s_median,
decode: p.decode_tps_median,
total: p.total_s_median,
ttftP95: p.ttft_p95_s_median,
queueWait: p.queue_wait_ms_median,
rejected: p.rejected_median,
prefillTps: p.prefill_tps_median,
reasoning: p.reasoning_tokens_median,
cached: p.cached_tokens_median,
completion: p.completion_tokens_median,
tpot: p.tpot_p95_ms_median,
})),
],
[series, base],
@@ -188,55 +95,6 @@ export default function Trends() {
const firstLive = series[0]?.git_sha;
const showDivider = base.length > 0 && series.length > 0;
// The build where the measuring identity changed, derived from the
// data rather than declared — a constant would go stale the moment
// the principal is reconfigured (#288).
const identityShift = useMemo(() => {
for (let i = 1; i < series.length; i++) {
const prev = series[i - 1].principal ?? "anonymous";
const cur = series[i].principal ?? "anonymous";
if (prev !== cur) {
return {
at: series[i].git_sha,
label: `measured as ${cur === "anonymous" ? "anonymous" : "identified"}`,
detail:
"The identity bench measured under changed here. Anonymous " +
"samples are subject to the #262 yield policy and understate " +
"capacity, so points either side are not comparable (#288).",
} as Rule;
}
}
return null;
}, [series]);
/** Declared boundaries touching `metric`, plus the regime-independent
* ones that apply to every panel. */
const rulesFor = (metric: string): Rule[] => {
const out: Rule[] = [];
if (showDivider && firstLive) {
out.push({ at: firstLive, label: "bench.py → helexa-bench" });
}
for (const r of regimes) {
if (r.affects.includes(metric)) {
out.push({ at: r.first_sha, label: r.label, detail: r.detail });
}
}
if (identityShift) out.push(identityShift);
return out;
};
// Anonymous and identified samples are not comparable once #262 is in
// the build: an anonymous caller is capped below max_in_flight and
// parked at the class gate, so it characterises the yield policy
// rather than serving capacity. Say so rather than letting someone
// read a step change as an engine regression (#288).
const identities = useMemo(
() => new Set(series.map((p) => p.principal ?? "anonymous")),
[series],
);
const mixedIdentity = identities.size > 1;
const anyAnonymous = identities.has("anonymous");
if (err) return <Alert variant="danger">{err}</Alert>;
if (!dims) return <Spinner animation="border" />;
@@ -264,25 +122,6 @@ export default function Trends() {
</p>
)}
{mixedIdentity && (
<Alert variant="warning" className="py-2">
<strong>Mixed measurement identity.</strong> Some builds were
sampled anonymously and some under a principal. Since{" "}
<code>#262</code> an anonymous caller is capped below{" "}
<code>max_in_flight</code> and yields to identified traffic, so
those points measure the admission policy rather than serving
capacity. A step change across that boundary is the instrument,
not the engine — see <code>#288</code>.
</Alert>
)}
{!mixedIdentity && anyAnonymous && (
<Alert variant="secondary" className="py-2 small">
Sampled anonymously. Since <code>#262</code> anonymous callers are
served from leftover capacity, so these numbers understate what an
authenticated caller gets (<code>#288</code>).
</Alert>
)}
{data.length === 0 ? (
<Alert variant="info">No data for this selection yet.</Alert>
) : (
@@ -294,128 +133,87 @@ export default function Trends() {
see <code>doc/benchmarks.md</code>.
</p>
)}
<h5 className="mt-3">decode tok/s (higher is better)</h5>
<ResponsiveContainer width="100%" height={280}>
<LineChart data={data} margin={{ top: 8, right: 24, bottom: 8, left: 0 }}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="label" />
<YAxis />
<Tooltip />
<Legend />
{showDivider && firstLive && (
<ReferenceLine
x={firstLive}
stroke="#bbb"
strokeDasharray="3 3"
label={{
value: "bench.py → helexa-bench",
position: "top",
fill: "#999",
fontSize: 11,
}}
/>
)}
<Line
type="monotone"
dataKey="decode"
name="decode tok/s"
stroke="#0d6efd"
connectNulls
/>
{base.length > 0 && (
<Line
type="monotone"
dataKey="baseDecode"
name="baseline (bench.py · gateway)"
stroke="#888"
strokeDasharray="5 5"
connectNulls
/>
)}
</LineChart>
</ResponsiveContainer>
<MetricChart
title="decode tok/s"
unit="higher is better"
data={data}
rules={rulesFor("decode")}
lines={[
{ key: "decode", name: "decode tok/s", stroke: "#0d6efd" },
...(base.length > 0
? [
{
key: "baseDecode",
name: "baseline (bench.py · gateway)",
stroke: "#888",
dashed: true,
},
]
: []),
]}
/>
<MetricChart
title="prefill tok/s"
unit="higher is better"
hint="The other half of serving speed, derived from prefill_tokens / prefill_ms. A prefix-cache hit shortens prefill_ms while the token count stays whole, so a high rate here is itself the cache-hit signal."
data={data}
rules={rulesFor("prefillTps")}
lines={[
{ key: "prefillTps", name: "prefill tok/s", stroke: "#20c997" },
]}
/>
<MetricChart
title="TTFT"
unit="seconds, lower is better"
hint="Median and p95 together on purpose. Under concurrency the median is dominated by whichever streams were admitted immediately; the p95 is the one that moves when a caller is made to wait. Charting only the median is how a 0.53 s → 11.77 s tail went unnoticed for a week."
data={data}
rules={rulesFor("ttft")}
lines={[
{ key: "ttft", name: "TTFT median (s)", stroke: "#dc3545" },
{ key: "ttftP95", name: "TTFT p95 (s)", stroke: "#fd7e14" },
...(base.length > 0
? [
{
key: "baseTtft",
name: "baseline (bench.py · gateway)",
stroke: "#888",
dashed: true,
},
]
: []),
]}
/>
<MetricChart
title="inter-token gap p95"
unit="ms, lower is better"
hint="Stream smoothness. decode tok/s is a mean over the whole window, so a stream that stalls and then catches up is indistinguishable from one that never stalled — this is the number a user feels."
data={data}
rules={rulesFor("tpot")}
lines={[
{ key: "tpot", name: "inter-token p95 (ms)", stroke: "#6f42c1" },
]}
/>
<MetricChart
title="admission"
unit="queue wait ms · requests shed"
hint="Separates “the server is slow” from “you were queued behind someone”. Queue wait is TTFT minus server-measured prefill; rejected counts honest backpressure rather than silent failures."
data={data}
rules={rulesFor("queueWait")}
lines={[
{ key: "queueWait", name: "queue wait (ms)", stroke: "#d63384" },
{ key: "rejected", name: "rejected (count)", stroke: "#adb5bd" },
]}
/>
<MetricChart
title="tokens per sample"
unit="counts"
hint="Cost, not speed. Reasoning tokens are the dominant driver on a reasoning model and move independently of every rate above — a template or sampling change can double what the model thinks before answering while the speed charts stay flat. Cached tokens are why prefill timing varies between otherwise identical samples."
data={data}
rules={rulesFor("completion")}
lines={[
{ key: "completion", name: "completion tokens", stroke: "#0dcaf0" },
{ key: "reasoning", name: "reasoning tokens", stroke: "#ffc107" },
{ key: "cached", name: "cached prompt tokens", stroke: "#198754" },
]}
/>
{(() => {
// Explain every rule actually drawn for this selection. The
// chart label is a name; without the reason, a reader still
// has to go and rediscover what it meant — which is the cost
// this whole feature exists to remove.
const shown = [
...regimes
.filter((r) => series.some((p) => p.git_sha === r.first_sha))
.map((r) => ({ at: r.first_sha, label: r.label, detail: r.detail })),
...(identityShift ? [identityShift] : []),
];
if (shown.length === 0) return null;
return (
<div className="mt-4 pt-3 border-top">
<p className="text-muted small mb-2">
<strong>Dashed rules mark measurement-regime changes</strong> —
builds where a number changed meaning rather than value. A step
across one is the instrument, not the engine.
</p>
<dl className="row small text-muted mb-0">
{shown.map((r) => (
<div key={`${r.at}-${r.label}`} className="mb-2">
<dt className="fw-semibold">
<code>{r.at}</code> — {r.label}
</dt>
<dd className="mb-0">{r.detail}</dd>
</div>
))}
</dl>
</div>
);
})()}
<h5 className="mt-4">TTFT seconds (lower is better)</h5>
<ResponsiveContainer width="100%" height={280}>
<LineChart data={data} margin={{ top: 8, right: 24, bottom: 8, left: 0 }}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="label" />
<YAxis />
<Tooltip />
<Legend />
{showDivider && firstLive && (
<ReferenceLine
x={firstLive}
stroke="#bbb"
strokeDasharray="3 3"
label={{
value: "bench.py → helexa-bench",
position: "top",
fill: "#999",
fontSize: 11,
}}
/>
)}
<Line
type="monotone"
dataKey="ttft"
name="TTFT (s)"
stroke="#dc3545"
connectNulls
/>
{base.length > 0 && (
<Line
type="monotone"
dataKey="baseTtft"
name="baseline (bench.py · gateway)"
stroke="#888"
strokeDasharray="5 5"
connectNulls
/>
)}
</LineChart>
</ResponsiveContainer>
</>
)}
</>
-21
View File
@@ -41,17 +41,6 @@ export interface SeriesPoint {
ttft_s_median: number | null;
decode_tps_median: number | null;
total_s_median: number | null;
ttft_p95_s_median: number | null;
queue_wait_ms_median: number | null;
rejected_median: number | null;
prefill_tps_median: number | null;
reasoning_tokens_median: number | null;
cached_tokens_median: number | null;
completion_tokens_median: number | null;
tpot_p95_ms_median: number | null;
/** Identity the samples were taken under, or null if anonymous.
* Anonymous and identified points are not comparable (#288). */
principal: string | null;
samples: number;
}
@@ -78,13 +67,3 @@ export interface RunRow {
ok: boolean;
error: string | null;
}
/** A build at which a metric changed meaning rather than value (#288).
* Served from the API so the UI and `report` cite one list. */
export interface MeasurementRegime {
first_sha: string;
label: string;
detail: string;
/** Series keys this affects, as Trends.tsx names them. */
affects: string[];
}
-5
View File
@@ -20,11 +20,6 @@ pub struct MessagesRequest {
pub temperature: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub top_p: Option<f64>,
/// Anthropic's Messages API carries `top_k` natively, so it maps
/// straight onto the OpenAI-side field rather than being dropped in
/// translation (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_k: Option<usize>,
#[serde(skip_serializing_if = "Option::is_none")]
pub stream: Option<bool>,
#[serde(flatten)]
+1 -20
View File
@@ -1,7 +1,7 @@
//! Model catalogue — profiles describing how to serve each model.
use crate::discovery::DeviceInfo;
use crate::harness::{ModelCost, ModelLimit, SamplingOverride};
use crate::harness::{ModelCost, ModelLimit};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::Path;
@@ -57,23 +57,6 @@ pub struct ModelProfile {
/// on this being explicit per model rather than implicit.
#[serde(default)]
pub source: Option<String>,
/// Operator sampling override forwarded to neuron on cold load
/// (#283). The counterpart to the same field in a host's
/// `[[default_models]]` — a model that can be both cold-loaded by
/// cortex and resident on a host must carry the same value in both,
/// which `script/check-config-consistency.py` enforces in CI.
///
/// This is the #252 shape of hazard: two config files describing one
/// model, with nothing forcing them to agree.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sampling: Option<SamplingOverride>,
/// The model's own `preserve_thinking` template control, forwarded
/// on cold load so a model this gateway loads behaves the same as
/// one the host was configured to hold resident. Two config files
/// describing one model is how #252 happened; this rides along for
/// the same reason `sampling` does.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
// ── Enrichment (issue #62) ────────────────────────────────
/// Per-model token budget. When present, advertised in `/v1/models`
@@ -257,8 +240,6 @@ mod tests {
pinned_on: vec![],
residency_priority: None,
source: None,
sampling: None,
preserve_thinking: None,
limit: None,
cost: None,
capabilities: vec![],
-141
View File
@@ -143,48 +143,15 @@ pub struct NeuronEndpoint {
impl GatewayConfig {
/// Load configuration from a TOML file, with environment variable overrides.
/// Env vars are prefixed with `CORTEX_` and use `__` as a separator.
///
/// A sibling `secrets.toml`, if present, is merged **after** the main
/// file and therefore wins. It exists so the two have different
/// owners: CI writes the main config, and only the operator writes
/// the secrets — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("CORTEX_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The split exists to make a class of outage impossible rather than
/// merely unlikely. Config that CI cannot write is config no check can
/// defend: `helexa-router.toml` was excluded from git because it sat
/// beside secret-bearing files, and its `helexa/balanced` alias then
/// pointed at a retired model for long enough to take a node down
/// (2026-08-27). The fix is not to hand CI the credentials — it is to
/// stop mixing the two in one file.
///
/// So `cortex.toml` holds structure and is deployed by CI, while
/// `secrets.toml` holds API keys and the upstream bearer, is written
/// only by the operator, and is never read, written or diffed by the
/// pipeline. Because it merges last it also wins, so a value present in
/// both is the operator's.
///
/// A missing file is not an error — figment treats an absent
/// `Toml::file` as an empty source — so a host that keeps everything in
/// one file, and every existing deployment, keeps working unchanged.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
impl Default for GatewayConfig {
fn default() -> Self {
Self {
@@ -203,111 +170,3 @@ impl Default for GatewayConfig {
}
}
}
#[cfg(test)]
mod secrets_layer_tests {
use super::*;
use std::io::Write;
/// Write `main` (and optionally `secrets.toml`) into a fresh directory
/// under the target dir, and load through the real code path.
fn load_with(dir: &str, main: &str, secrets: Option<&str>) -> GatewayConfig {
// `CARGO_TARGET_TMPDIR` is only set for integration tests, not for
// unit tests inside src/, so derive a unique directory instead.
let base = std::env::temp_dir().join(format!("helexa-cortex-config-{dir}"));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).expect("create test dir");
let cfg = base.join("cortex.toml");
write!(
std::fs::File::create(&cfg).expect("create config"),
"{main}"
)
.expect("write");
if let Some(s) = secrets {
let p = base.join("secrets.toml");
write!(std::fs::File::create(&p).expect("create secrets"), "{s}").expect("write");
}
GatewayConfig::load(&cfg).expect("config loads")
}
const MAIN: &str = r#"
[gateway]
listen = "0.0.0.0:31313"
metrics_listen = "0.0.0.0:31314"
[eviction]
strategy = "lru"
defrag_after_cycles = 50
[entitlements]
require_auth = true
[[neurons]]
name = "beast"
endpoint = "http://beast.invalid:13131"
"#;
/// The whole point: a host with no secrets.toml behaves exactly as
/// before. Every current deployment is in this state, so if this
/// regressed the change would take the fleet down on rollout rather
/// than on adoption.
#[test]
fn an_absent_secrets_file_is_not_an_error() {
let cfg = load_with("secrets_absent", MAIN, None);
assert!(cfg.entitlements.require_auth);
assert!(
cfg.entitlements.keys.is_empty(),
"no keys configured anywhere means no keys"
);
}
/// The array-of-tables case that ruled out env-var overrides:
/// `[[entitlements.keys]]` cannot be expressed as `CORTEX_*` env vars,
/// so the second TOML layer is what carries it.
#[test]
fn secrets_file_supplies_the_entitlement_keys() {
let cfg = load_with(
"secrets_keys",
MAIN,
Some(
r#"
[[entitlements.keys]]
key = "sk-test-aaa"
account_id = "acct-a"
key_id = "a"
[[entitlements.keys]]
key = "sk-test-bbb"
account_id = "acct-b"
key_id = "b"
"#,
),
);
assert_eq!(cfg.entitlements.keys.len(), 2);
assert_eq!(cfg.entitlements.keys[0].key, "sk-test-aaa");
assert_eq!(cfg.entitlements.keys[1].account_id, "acct-b");
assert!(
cfg.entitlements.require_auth,
"structure from the CI-owned file must survive the merge"
);
}
/// Ordering is the safety property. If CI ever ships a placeholder in
/// the main file, the operator's real value must still win — otherwise
/// a deploy silently swaps a live credential for a dummy.
#[test]
fn the_operators_value_wins_over_the_deployed_one() {
let cfg = load_with(
"secrets_precedence",
&format!(
"{MAIN}\n[upstream]\nenabled = true\nurl = \"https://example.invalid\"\nbearer = \"PLACEHOLDER\"\n"
),
Some("[upstream]\nbearer = \"real-token\"\n"),
);
assert_eq!(
cfg.upstream.bearer, "real-token",
"secrets.toml merges last, so it wins"
);
assert_eq!(
cfg.upstream.url, "https://example.invalid",
"fields the operator does not override keep the deployed value"
);
}
}
-26
View File
@@ -109,23 +109,6 @@ pub struct ModelLoad {
/// (#54/#137). `#[serde(default)]` for back-compat.
#[serde(default)]
pub rejected_per_principal: u64,
/// Anonymous requests currently holding a seat (#262). Read against
/// `in_flight` it says how much of this model's load is unattributable;
/// against `anon_max_in_flight`, how close that traffic is to the
/// ceiling that keeps it from starving identified callers.
#[serde(default)]
pub anon_in_flight: usize,
/// Seats anonymous traffic may hold at once (#262); `0` = refused
/// outright. `#[serde(default)]` (→ 0) for pre-#262 neurons, which
/// applied no ceiling at all — cortex skips the gauge rather than
/// publish a 0 that reads as the opposite of the truth.
#[serde(default)]
pub anon_max_in_flight: usize,
/// Cumulative anonymous requests refused for want of leftover capacity
/// (#262). Rising here with a flat `rejected_timeout` means the
/// reservation is working, not that the model is overloaded.
#[serde(default)]
pub rejected_anon_yield: u64,
/// Cumulative requests that waited for KV budget and never got it
/// (#257) — the model is saturated by long-context sequences rather
/// than by request count. `#[serde(default)]` for back-compat.
@@ -188,9 +171,6 @@ mod health_load_tests {
kv_available_mb: 0,
rejected_timeout: 0,
rejected_per_principal: 0,
anon_in_flight: 2,
anon_max_in_flight: 7,
rejected_anon_yield: 0,
tok_s_prefill: 0.0,
tok_s_decode: 0.0,
}],
@@ -202,8 +182,6 @@ mod health_load_tests {
assert_eq!(back.models[0].queue_depth, 3);
assert_eq!(back.models[0].max_in_flight, 8);
assert_eq!(back.models[0].max_queue_depth, 8);
assert_eq!(back.models[0].anon_in_flight, 2);
assert_eq!(back.models[0].anon_max_in_flight, 7);
}
#[test]
@@ -215,10 +193,6 @@ mod health_load_tests {
assert_eq!(m.in_flight, 2);
assert_eq!(m.max_in_flight, 0);
assert_eq!(m.max_queue_depth, 0);
// Likewise a pre-#262 neuron, which enforced no anonymous ceiling.
assert_eq!(m.anon_in_flight, 0);
assert_eq!(m.anon_max_in_flight, 0);
assert_eq!(m.rejected_anon_yield, 0);
}
}
+1 -223
View File
@@ -26,115 +26,7 @@ pub struct HarnessHealth {
pub uptime_secs: Option<u64>,
}
/// Operator-set sampling defaults for one model (#283).
///
/// A model's `generation_config.json` is the model authors' statement of
/// intent, and honouring it (#272) is right by default. It is not always
/// right for the workload the operator serves: `Qwen/Qwen3.8-27B`
/// publishes `temperature = 1.0`, which measured a 20% structural-defect
/// rate on ~2k-token code generation against 0/60 at `<= 0.6`
/// (#283, p = 0.0031).
///
/// This is the operator's explicit, visible correction to that default —
/// deliberately *not* a heuristic that inspects the model and guesses.
/// A temperature guesser is a footgun; a number in a config file the
/// operator wrote is not.
///
/// Precedence is **request > operator > model > built-in fallback**: the
/// override replaces what the model published, but a caller that names a
/// value still wins, because an explicit API parameter that is silently
/// ignored is a contract break.
///
/// Every field is optional and overlays independently — setting
/// `temperature` alone leaves the model's `top_p`/`top_k` in force.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct SamplingOverride {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub temperature: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_k: Option<usize>,
/// Multiplier applied to recently-generated tokens before sampling.
/// `1.0` disables it; above that, recently-emitted tokens get less
/// likely.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_penalty: Option<f32>,
/// How many recently-generated tokens the penalty considers.
///
/// The reason this is operator-settable at all: the built-in default
/// is 64, inherited from candle's chat example, and a reasoning
/// model's think block runs to tens of thousands of tokens. Anything
/// restated more than 64 tokens later is invisible to the penalty,
/// which is how a 28,672-token think block can contain the same
/// derivation five times over.
///
/// Raising it is a real trade, not a free win: the penalty is
/// token-level, and source code legitimately repeats tokens
/// constantly (`const`, `function`, brace runs). A wide window at a
/// meaningful penalty distorts exactly the output this fleet is for.
/// Sequence-level repetition wants a sequence-level instrument
/// (frequency/presence penalties, or a DRY-style sampler); this knob
/// exists so the trade can be *measured* per model rather than
/// guessed once and frozen into a constant.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// Sequence-wide one-time penalty for already-seen tokens.
///
/// This, not `repeat_last_n`, is the remedy Qwen prescribe for the
/// endless repetition their models fall into during long reasoning
/// (0..2, 1.5 when severe). It is scored over everything generated
/// so far, so a passage restated 5,000 tokens later is still
/// penalised — the case a trailing window cannot see.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
/// Sequence-wide penalty proportional to a token's existing count.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// Pin the sampler's RNG for every request to this model.
///
/// Exists because parameter search needs a control: comparing two
/// `presence_penalty` values across unseeded runs cannot separate
/// the parameter's effect from sampling variance, and the clients
/// that would otherwise pin it cannot — pi's request builder sends
/// no `seed` at all, so a caller-side seed is not available on the
/// path that matters.
///
/// Note what this does and does not give. It removes sampling noise
/// as a confounder; it does not make two runs comparable
/// token-by-token, because changing a penalty changes the logits
/// and the sequences diverge at the first step where a token choice
/// flips.
///
/// Leaving this set is a product decision, not just a testing one:
/// every identical prompt then yields an identical answer for every
/// caller. That is defensible — predictable and cacheable — but it
/// removes the variation a user may expect from retrying. A request
/// that names its own seed still wins.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub seed: Option<u64>,
}
impl SamplingOverride {
/// True when the operator set nothing — an empty `[sampling]` table
/// must behave exactly as no table at all.
pub fn is_empty(&self) -> bool {
self.temperature.is_none()
&& self.top_p.is_none()
&& self.top_k.is_none()
&& self.repeat_penalty.is_none()
&& self.repeat_last_n.is_none()
&& self.presence_penalty.is_none()
&& self.frequency_penalty.is_none()
&& self.seed.is_none()
}
}
/// Specification for loading a model through a harness.
///
/// Doubles as the `[[default_models]]` entry shape in `neuron.toml`, so
/// a field added here is available to both the load API and the
/// operator's per-host config without a second type to keep in step.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelSpec {
pub model_id: String,
@@ -142,36 +34,6 @@ pub struct ModelSpec {
pub quant: Option<String>,
pub tensor_parallel: Option<u32>,
pub devices: Option<Vec<u32>>,
/// Operator sampling override (#283). Travels with the *model*, not
/// the request: neuron is the only place sampling is resolved, so
/// the override has to reach it whether the load came from cortex's
/// catalogue or from this host's own `[[default_models]]`.
///
/// `#[serde(default)]` so a cortex or neuron predating this field
/// still round-trips the spec.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sampling: Option<SamplingOverride>,
/// Whether prior assistant turns keep their reasoning when the chat
/// template re-renders the conversation.
///
/// This is the model's own `preserve_thinking` template control, not
/// an invention of ours. `Qwen/Qwen3.8-27B` defaults it to *true* —
/// the whole transcript's think blocks are replayed — and `false`
/// keeps reasoning only for turns after the last user query, i.e.
/// the turn in progress.
///
/// `None` leaves the kwarg unset so the template's own default
/// applies, which is the only safe default: the model's authors
/// chose it, and overriding that fleet-wide on a hunch is how you
/// ship a regression you cannot see.
///
/// Exposed as operator config so the premise can be A/B'd on a real
/// workload — full replay is the larger prompt and pushes against
/// the prefix-cache budget and the throughput-derived context
/// ceiling, but whether it *helps* the model is a measurement, not
/// a belief. A request's own `chat_template_kwargs` still wins.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
}
/// Per-model token budget advertised by the catalogue or neuron.
@@ -188,77 +50,8 @@ pub struct ModelLimit {
/// back to `context − output`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub input: Option<usize>,
/// The generation budget a request gets when it names none, and the
/// figure `input = context − output` is derived from.
///
/// A *reserve*, not a ceiling — a request may ask for more and be
/// served it. Named `output` for the shape's history; the number a
/// client should plan against is [`ModelLimit::output_ceiling`].
/// Maximum number of generation tokens.
pub output: usize,
/// The largest output a request may name and be served (#278).
///
/// Distinct from `output`, which is the default and the KV-planning
/// reserve. Publishing the reserve as the ceiling meant a client
/// that trusted the advertisement asked for a fraction of what the
/// model would happily generate — on a reasoning model, often less
/// than its own think block costs.
///
/// `#[serde(default)]` so a neuron predating this field still
/// deserializes; zero means "not advertised", and consumers fall
/// back to `output`.
#[serde(default, skip_serializing_if = "is_zero")]
pub output_ceiling: usize,
}
fn is_zero(v: &usize) -> bool {
*v == 0
}
/// What each reasoning effort level costs, in reasoning tokens (#223).
///
/// OpenAI-shaped clients can only send `minimal|low|medium|high` — the
/// ladder is their entire vocabulary, and they cannot express a token
/// count. So the server picks the numbers, and has to say what they are:
/// a client choosing `low` with no idea whether that buys 2k tokens or
/// 20k is guessing, which is the failure #274 exists to end.
///
/// Ordered rungs rather than a map, so the ladder reads in the order a
/// client would climb it.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ReasoningBudgetRung {
/// The effort level as the caller spells it on the wire.
///
/// Since #290 these are the rungs the **model's own chat template**
/// accepts, discovered by rendering it, rather than a ladder the
/// operator invented. `Qwen/Qwen3.8-27B` accepts exactly `low`,
/// `medium` and `xhigh` and raises on anything else; we previously
/// advertised `minimal`/`low`/`medium`/`high`, two of which the
/// template rejects and which omitted the model's own default.
///
/// Advertise only rungs the template accepts: an invented name is a
/// name a caller can send and the model will reject. Note that a
/// client need not read this at all — pi-ai takes its levels from a
/// static `thinkingLevelMap` in the operator's config — so this is
/// a statement of truth, not a control surface we can rely on.
pub effort: String,
/// Backstop reasoning-token cap for this rung, when the deployment
/// sets one.
///
/// `None` is the normal case and means "named rung, no cap": effort
/// is expressed to the model through its template, and the number of
/// tokens it then spends is the model's business. A cap is a safety
/// net against a runaway think block (#223), not the mechanism by
/// which effort is selected — enforcing effort by truncation is what
/// #290 fixed.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tokens: Option<usize>,
/// Whether the model applies this rung when the caller names none.
///
/// Taken from the template's own `|default(...)`. Without it a
/// caller that says nothing cannot tell what it is getting — on
/// Qwen3.8 that silently means `xhigh`.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub default: bool,
}
/// Operator-set pricing, **USD per 1,000,000 tokens, as JSON numbers**
@@ -333,16 +126,6 @@ pub struct ModelInfo {
/// marker tokens (`<think>` / `<\/think>` or similar).
#[serde(default)]
pub reasoning: bool,
/// The operator's `preserve_thinking` for this loaded model, or
/// `None` when unset and the template's own default applies.
///
/// Advertised so a run can be *stamped* with the value it ran
/// under. This is an A/B knob whose whole purpose is comparison
/// between runs, and a comparison whose arms cannot be told apart
/// afterwards is not a comparison. Recovering it from deploy
/// history is guesswork — a config sync can land mid-session.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
/// Whether this node can actually serve a request for this model
/// right now (#245).
@@ -361,11 +144,6 @@ pub struct ModelInfo {
/// from its own routing table.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub servable: Option<ModelServability>,
/// The reasoning-effort ladder this deployment honours (#223).
/// Absent when the model does not reason, or the harness has no
/// budget to offer.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub reasoning_budget: Vec<ReasoningBudgetRung>,
}
/// Why a loaded model can or cannot be served right now (#245).
+2 -49
View File
@@ -44,12 +44,6 @@ pub struct NodeState {
/// just one whose total topology could. Empty until the first
/// /health poll.
pub device_health: Vec<crate::discovery::DeviceHealth>,
/// The reasoning-effort ladder this neuron honours (#223), copied
/// from its `/models` reply at poll time. Per-node rather than
/// per-model because it is host configuration — every model on a
/// neuron reports the same rungs — and duplicating it onto each
/// model entry would invite the two to disagree.
pub reasoning_budget: Vec<crate::harness::ReasoningBudgetRung>,
}
/// A model registered on a node, with its runtime status.
@@ -193,35 +187,12 @@ pub struct CortexModelEntry {
/// Served max-seq-len in tokens — mirrors `limit.context`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_model_len: Option<usize>,
/// The same window under the two names generic OpenAI clients look
/// for (#278) — a client that probes for one of these rather than
/// `max_model_len` would otherwise find no window at all and fall
/// back to a number its operator had to invent by hand.
///
/// Publishing them is cheap and harmless; do not assume any
/// *specific* client consumes them. An earlier version of this
/// comment credited pi-ai's provider discovery, which was wrong:
/// pi requires `contextWindow` as a static field on its model type
/// and reads none of these keys at runtime (checked against 0.82.1
/// and 0.84.3). The misattribution mattered — it made an operator's
/// hand-set 80k window look like our advertisement failing to land,
/// when in fact nothing was reading it and the number had to be
/// maintained by hand regardless.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub context_window: Option<usize>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub context_length: Option<usize>,
/// Usable input budget — mirrors `limit.input` when present.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_input_tokens: Option<usize>,
/// The largest output a request may name — mirrors
/// `limit.output_ceiling`, falling back to `limit.output` (#278).
/// Maximum generation tokens — mirrors `limit.output`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_output_tokens: Option<usize>,
/// What each effort level buys, in reasoning tokens (#223) — so a
/// client picking `low` knows what it asked for.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub reasoning_budget: Vec<crate::harness::ReasoningBudgetRung>,
}
impl CortexModelEntry {
@@ -233,26 +204,8 @@ impl CortexModelEntry {
/// dropped the limit.
pub fn sync_flat_limit(&mut self) {
self.max_model_len = self.limit.as_ref().map(|l| l.context);
self.context_window = self.limit.as_ref().map(|l| l.context);
self.context_length = self.limit.as_ref().map(|l| l.context);
self.max_input_tokens = self.limit.as_ref().and_then(|l| l.input);
// The ceiling, not the reserve (#278): a client that reads this
// field turns it into the cap it sends on each request, so
// advertising the reserve handed a reasoning model less budget
// than its own think block needs. Falls back to the reserve for
// a neuron too old to publish a ceiling.
//
// Observed with dsh. pi is *not* an example: its `maxTokens` is
// a required static field in models.json and it reads nothing
// from here — an operator has to copy this number across by
// hand, and gets no error if they copy it wrong.
self.max_output_tokens = self.limit.as_ref().map(|l| {
if l.output_ceiling > 0 {
l.output_ceiling
} else {
l.output
}
});
self.max_output_tokens = self.limit.as_ref().map(|l| l.output);
}
}
+4 -45
View File
@@ -18,45 +18,6 @@ pub struct ChatCompletionRequest {
pub temperature: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub top_p: Option<f64>,
/// Truncate sampling to the `k` most likely tokens (#272).
///
/// Not in the OpenAI schema but universal among serving stacks
/// (vLLM, SGLang, llama.cpp, Ollama), and required by models that
/// publish it — Qwen3 specifies `top_k = 20`. A named field rather
/// than `extra`, because a sampling parameter that lands in `extra`
/// is accepted and silently discarded.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_k: Option<usize>,
/// Pin the sampler's RNG for reproducible output (OpenAI core).
///
/// Previously ignored on every text path, which called
/// `unix_subsec_nanos()` regardless — a caller asking for
/// determinism got fresh randomness and a `200`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub seed: Option<u64>,
/// Penalty applied to recently-generated tokens; `1.0` disables.
/// Defaults to 1.1 when absent (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repetition_penalty: Option<f32>,
/// How many recent tokens `repetition_penalty` considers. Defaults
/// to 64 — fine for chat, short for a long reasoning block, which is
/// why it is a knob rather than a constant (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// OpenAI core, `-2.0..=2.0`. Subtracted once from any token that
/// has already appeared, regardless of how often.
///
/// Unlike `repetition_penalty` this is scored over the **whole**
/// generated sequence, not a trailing window — which is what a
/// reasoning model needs, and what Qwen recommend (0..2, 1.5 for
/// severe cases) for the endless-repetition failure their own
/// models exhibit during long thinking.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
/// OpenAI core, `-2.0..=2.0`. Subtracted in proportion to how many
/// times a token has already appeared.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// The deprecated spelling of the output cap. Still what most
/// clients send; see [`ChatCompletionRequest::effective_max_tokens`].
#[serde(skip_serializing_if = "Option::is_none")]
@@ -188,9 +149,8 @@ pub struct Usage {
/// `completion_tokens`, never added into `total_tokens`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub completion_tokens_details: Option<CompletionTokensDetails>,
/// OpenAI-standard breakdown of `prompt_tokens`, carrying
/// `cached_tokens` (#269). Omitted when nothing was reused, so
/// "absent" stays distinguishable from "measured as zero".
/// OpenAI-standard breakdown of `prompt_tokens`. Populated once
/// prompt caching lands (#11); `None` until then.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub prompt_tokens_details: Option<PromptTokensDetails>,
/// helexa extension (non-OpenAI): server-measured prefill/decode
@@ -222,9 +182,8 @@ pub struct CompletionTokensDetails {
/// Sub-counts of `Usage::prompt_tokens`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PromptTokensDetails {
/// Prompt tokens served from neuron's prefix KV cache (#11), so a
/// caller can see the saving rather than assume none (#269). A
/// sub-count of `prompt_tokens`, never added into `total_tokens`.
/// Prompt tokens served from cache (cache-read rate). Populated
/// once prompt caching lands (#11).
pub cached_tokens: u64,
}
+7 -159
View File
@@ -55,26 +55,6 @@ pub struct ResponsesRequest {
pub temperature: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_p: Option<f64>,
/// Truncate sampling to the `k` most likely tokens (#272). Not in
/// the Responses schema, but a sampling parameter that lands in
/// `extra` is accepted and silently discarded, which is the defect
/// this closes.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_k: Option<usize>,
/// Pin the sampler's RNG for reproducible output (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub seed: Option<u64>,
/// Penalty on recently-generated tokens; `1.0` disables (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repetition_penalty: Option<f32>,
/// Window `repetition_penalty` considers (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// OpenAI core; forwarded to the chat path unchanged.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// Chained-conversation identifier. We don't store responses
/// server-side yet; if this is `Some`, the handler returns 400.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -150,22 +130,12 @@ pub enum ResponsesInputItem {
/// string or an array of content parts; the translator renders
/// either form to text rather than losing the tool result.
FunctionCallOutput { call_id: String, output: Value },
/// A reasoning item the model emitted on an earlier turn, replayed
/// by the client so the model can continue its own train of thought.
///
/// Both spellings are modelled because both are in the wild and one
/// of them is ours: OpenAI's o-series carries the text in `content`
/// as `reasoning_text` parts, while neuron emits `summary` with a
/// `summary_text` part. Clients round-trip the completed item
/// verbatim — pi-ai stores our `response.output_item.done` payload
/// and replays it unchanged — so whichever field we emit is the
/// field we get back, and reading only one of them silently loses
/// the turn's reasoning.
/// Reasoning items emitted by o-series models. Accepted but
/// not forwarded to the model — neuron's candle path doesn't
/// surface reasoning separately yet.
Reasoning {
#[serde(default)]
content: Vec<Value>,
#[serde(default)]
summary: Vec<Value>,
},
}
@@ -220,46 +190,11 @@ pub struct ResponsesResponse {
pub status: String,
pub model: String,
pub output: Vec<ResponsesOutputItem>,
/// Why a `status: "incomplete"` response stopped short. Required in
/// practice, not optional decoration: a client cannot tell an
/// honest truncation from a protocol fault without it, and at least
/// one treats the difference as fatal.
///
/// pi-ai's `mapStopReason` maps
/// `incomplete` + `reason == "max_output_tokens"` to
/// `stopReason: "length"`, and `incomplete` with no reason to
/// `stopReason: "error"` ("Response incomplete without a provider
/// reason"). Its agent loop then either continues — failing any
/// truncated tool calls safely and letting the model recover on the
/// next turn — or halts outright. Omitting this field cost a live
/// agentic session its whole run.
///
/// `None` for `completed` and `in_progress`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub incomplete_details: Option<IncompleteDetails>,
/// Populated on completion; `None` while streaming.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage: Option<ResponsesUsage>,
}
/// Why a response is `incomplete`.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct IncompleteDetails {
/// OpenAI spells the output-cap case `"max_output_tokens"`; that is
/// the only reason we produce today. `content_filter` is the other
/// value the upstream API defines.
pub reason: String,
}
impl IncompleteDetails {
/// The response ran into the caller's output budget.
pub fn max_output_tokens() -> Self {
Self {
reason: "max_output_tokens".into(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum ResponsesOutputItem {
@@ -276,15 +211,6 @@ pub enum ResponsesOutputItem {
#[serde(default = "default_item_status")]
status: String,
},
/// A think block, as its own output item ahead of the message —
/// the same shape the streaming projector emits, so a client sees
/// one contract whether or not it streams (#300).
Reasoning {
id: String,
summary: Vec<ResponsesSummaryPart>,
#[serde(default = "default_item_status")]
status: String,
},
/// Reserved for the day tool-call extraction lands. The wire
/// shape mirrors `ResponsesInputItem::FunctionCall`.
FunctionCall {
@@ -297,24 +223,6 @@ pub enum ResponsesOutputItem {
},
}
/// One `summary_text` part of a reasoning output item.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResponsesSummaryPart {
/// Always `"summary_text"`.
#[serde(rename = "type")]
pub part_type: String,
pub text: String,
}
impl ResponsesSummaryPart {
pub fn summary_text(text: impl Into<String>) -> Self {
Self {
part_type: "summary_text".into(),
text: text.into(),
}
}
}
fn default_item_status() -> String {
"completed".into()
}
@@ -341,8 +249,8 @@ pub struct ResponsesUsage {
/// sub-count of `output_tokens`, never added into `total_tokens`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub output_tokens_details: Option<OutputTokensDetails>,
/// OpenAI-standard breakdown of `input_tokens`, carrying
/// `cached_tokens` (#269). Omitted when nothing was reused.
/// OpenAI-standard breakdown of `input_tokens`. Populated once
/// prompt caching lands (#11); `None` until then.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub input_tokens_details: Option<InputTokensDetails>,
}
@@ -357,8 +265,8 @@ pub struct OutputTokensDetails {
/// Sub-counts of `ResponsesUsage::input_tokens`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct InputTokensDetails {
/// Input tokens served from neuron's prefix KV cache (#11), so a
/// caller can see the saving rather than assume none (#269).
/// Input tokens served from cache (cache-read rate). Populated
/// once prompt caching lands (#11).
pub cached_tokens: u64,
}
@@ -381,37 +289,9 @@ pub mod events {
pub const OUTPUT_TEXT_DONE: &str = "response.output_text.done";
pub const CONTENT_PART_DONE: &str = "response.content_part.done";
pub const OUTPUT_ITEM_DONE: &str = "response.output_item.done";
/// Reasoning-item event family. A reasoning model's think block is
/// its own output item, emitted ahead of the message item, with a
/// summary part carrying the text. Streaming these is what keeps the
/// connection observably alive while a model thinks — clients time
/// out on silence, and SSE comment keep-alives don't count because
/// idle timers reset on parsed events, not comments.
pub const REASONING_SUMMARY_PART_ADDED: &str = "response.reasoning_summary_part.added";
pub const REASONING_SUMMARY_TEXT_DELTA: &str = "response.reasoning_summary_text.delta";
pub const REASONING_SUMMARY_TEXT_DONE: &str = "response.reasoning_summary_text.done";
pub const REASONING_SUMMARY_PART_DONE: &str = "response.reasoning_summary_part.done";
pub const FUNCTION_CALL_ARGUMENTS_DELTA: &str = "response.function_call_arguments.delta";
pub const FUNCTION_CALL_ARGUMENTS_DONE: &str = "response.function_call_arguments.done";
pub const COMPLETED: &str = "response.completed";
/// Terminal frame for a response that stopped short. Carries the
/// same `response` payload as [`COMPLETED`], with
/// `status: "incomplete"` and `incomplete_details.reason`.
///
/// A client keyed on the event name — rather than on
/// `response.status` — sees no terminal event at all if a truncated
/// response is announced as `response.completed`, and waits out its
/// idle timeout.
pub const INCOMPLETE: &str = "response.incomplete";
/// Terminal frame for a response that did not finish because the
/// server failed — a poisoned device, an OOM, a producer that died
/// mid-stream. Carries `status: "failed"` and an `error` object.
///
/// The alternative is worse than it looks: defaulting a missing
/// finish to a clean stop reports a crash as a complete answer, and
/// the caller believes a truncated or empty reply was the model's
/// considered response.
pub const FAILED: &str = "response.failed";
}
#[cfg(test)]
@@ -616,7 +496,6 @@ mod tests {
}],
status: "completed".into(),
}],
incomplete_details: None,
usage: Some(ResponsesUsage {
input_tokens: 5,
output_tokens: 3,
@@ -629,36 +508,5 @@ mod tests {
let parsed: ResponsesResponse = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.id, "resp_1");
assert_eq!(parsed.output.len(), 1);
assert!(
!json.contains("incomplete_details"),
"a completed response must not carry an incomplete reason"
);
}
/// The field pi-ai's `mapStopReason` keys on. Absent, it maps
/// `incomplete` to `stopReason: "error"` and its agent loop halts;
/// present with `max_output_tokens`, it maps to `"length"` and the
/// loop continues.
#[test]
fn an_incomplete_response_serialises_its_reason() {
let r = ResponsesResponse {
id: "resp_2".into(),
object: "response".into(),
created_at: 1700,
status: "incomplete".into(),
model: "m".into(),
output: vec![],
incomplete_details: Some(IncompleteDetails::max_output_tokens()),
usage: None,
};
let v: serde_json::Value = serde_json::to_value(&r).unwrap();
assert_eq!(v["status"], "incomplete");
assert_eq!(v["incomplete_details"]["reason"], "max_output_tokens");
let parsed: ResponsesResponse = serde_json::from_value(v).unwrap();
assert_eq!(
parsed.incomplete_details,
Some(IncompleteDetails::max_output_tokens())
);
}
}
+7 -233
View File
@@ -94,28 +94,6 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
if let Some(tc) = tool_choice {
obj.insert("tool_choice".into(), tc);
}
// Anthropic's extended-thinking budget is the one quantitative
// reasoning control any of our wire formats offers (#223).
// Normalised onto `reasoning.max_tokens`, the shape the harness
// resolves, so all three inlets converge on one field instead of
// each generation loop learning three spellings.
let budget = obj
.get("thinking")
.filter(|t| t.get("type").and_then(Value::as_str) != Some("disabled"))
.and_then(|t| t.get("budget_tokens"))
.and_then(Value::as_u64)
.filter(|n| *n > 0);
if let Some(budget) = budget {
let reasoning = obj
.entry("reasoning")
.or_insert_with(|| Value::Object(Default::default()));
if !reasoning.is_object() {
*reasoning = Value::Object(Default::default());
}
if let Some(r) = reasoning.as_object_mut() {
r.entry("max_tokens").or_insert_with(|| Value::from(budget));
}
}
}
ChatCompletionRequest {
@@ -123,15 +101,6 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
messages,
temperature: req.temperature,
top_p: req.top_p,
// Anthropic models top_k too, so it survives the hop (#272).
top_k: req.top_k,
// Anthropic's Messages API has no seed or penalty knobs; absent
// means the model's own generation_config decides.
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
max_tokens: Some(req.max_tokens),
// Anthropic has one spelling; it maps onto the legacy field.
max_completion_tokens: None,
@@ -415,26 +384,17 @@ fn anthropic_content_to_text(content: AnthropicContent) -> String {
pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
let choice = resp.choices.into_iter().next();
let (content_text, reasoning_text, stop_reason) = match choice {
let (content_text, stop_reason) = match choice {
Some(c) => {
let text = match c.message.content {
MessageContent::Text(t) => t,
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
MessageContent::Null => String::new(),
};
// The model's think block, which the harness returns on
// `message.reasoning_content` (#300).
let reasoning = c
.message
.extra
.get("reasoning_content")
.and_then(|v| v.as_str())
.map(str::to_string)
.filter(|t| !t.is_empty());
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
(text, reasoning, stop)
(text, stop)
}
None => (String::new(), None, None),
None => (String::new(), None),
};
let usage = resp.usage.unwrap_or(Usage {
@@ -450,32 +410,10 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
id: resp.id,
response_type: "message".into(),
role: "assistant".into(),
// A `thinking` block ahead of the text, which is Anthropic's
// own ordering. This path accepted `thinking.budget_tokens` on
// the way in (see `anthropic_to_openai`) and returned nothing on
// the way out: the caller spent the tokens, could not see them,
// and could not replay the model's reasoning next turn (#300).
//
// Emitted without a `signature`. Anthropic signs its thinking
// blocks and a client that round-trips one may expect the field;
// we cannot mint a valid signature and inventing one would be
// worse than omitting it. If a real client rejects an unsigned
// block, the fix is to gate this behind a flag — not to go back
// to silently deleting the model's work.
content: {
let mut blocks = Vec::with_capacity(2);
if let Some(thinking) = reasoning_text {
blocks.push(ContentBlock {
block_type: "thinking".into(),
data: json!({ "thinking": thinking }),
});
}
blocks.push(ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
});
blocks
},
content: vec![ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
}],
model: resp.model,
stop_reason,
usage: AnthropicUsage {
@@ -532,7 +470,6 @@ pub struct AnthropicStreamTranslator {
enum OpenBlock {
Text,
ToolUse,
Thinking,
}
impl AnthropicStreamTranslator {
@@ -572,32 +509,6 @@ impl AnthropicStreamTranslator {
}
for choice in &chunk.choices {
// The model's think block. Dropped entirely until #300: this
// translator read `content` and `tool_calls` and nothing
// else, so an Anthropic client asked for extended thinking
// (`thinking.budget_tokens`, which `anthropic_to_openai`
// accepts), was billed for the tokens, and received a stream
// with no thinking in it — every turn, silently. Fixing only
// the non-streaming translation would have missed this,
// because the clients that ask for thinking stream.
if let Some(text) = choice
.delta
.get("reasoning_content")
.and_then(Value::as_str)
&& !text.is_empty()
{
self.ensure_thinking_block(&mut out);
let index = self.open_block.as_ref().map(|(i, _)| *i).unwrap_or(0);
out.push((
"content_block_delta".to_string(),
json!({
"type": "content_block_delta",
"index": index,
"delta": { "type": "thinking_delta", "thinking": text }
}),
));
}
if let Some(text) = choice.delta.get("content").and_then(Value::as_str)
&& !text.is_empty()
{
@@ -711,32 +622,6 @@ impl AnthropicStreamTranslator {
out
}
/// Open a `thinking` block, closing whatever preceded it.
///
/// Emitted without a `signature`. Anthropic signs its thinking
/// blocks; we cannot mint a valid one, and inventing a value would
/// be worse than omitting it. If a client rejects an unsigned block
/// the fix is a flag — not a return to deleting the model's work.
fn ensure_thinking_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Thinking)) => {}
_ => {
self.close_open_block(out);
let index = self.next_index;
self.next_index += 1;
self.open_block = Some((index, OpenBlock::Thinking));
out.push((
"content_block_start".to_string(),
json!({
"type": "content_block_start",
"index": index,
"content_block": { "type": "thinking", "thinking": "" }
}),
));
}
}
}
fn ensure_text_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Text)) => {}
@@ -1146,115 +1031,4 @@ mod request_tests {
let tools = openai.extra.get("tools").and_then(Value::as_array).unwrap();
assert_eq!(tools[0]["function"]["name"], "x");
}
/// The Anthropic path must return the model's thinking (#300).
///
/// It accepted `thinking.budget_tokens` on the way in and returned
/// a bare text block on the way out: the caller spent the tokens,
/// never saw them, and could not replay the model's reasoning on
/// the next turn. Taking a parameter and silently discarding what
/// it buys is the defect, not the missing field.
#[test]
fn anthropic_response_carries_a_thinking_block() {
let mut extra = serde_json::Map::new();
extra.insert("reasoning_content".into(), json!("weighing the options"));
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("the answer".into()),
extra: Value::Object(extra),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 2, "expected thinking + text");
assert_eq!(a.content[0].block_type, "thinking", "thinking comes first");
assert_eq!(a.content[0].data["thinking"], "weighing the options");
assert_eq!(a.content[1].block_type, "text");
assert_eq!(a.content[1].data["text"], "the answer");
}
/// No thinking means the unchanged single text block.
#[test]
fn anthropic_response_without_thinking_is_unchanged() {
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("plain".into()),
extra: Value::Object(Default::default()),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 1);
assert_eq!(a.content[0].block_type, "text");
}
/// Streaming Anthropic must carry thinking too (#300).
///
/// Fixing only the non-streaming translation would have missed this
/// entirely — and missed it for the clients that matter, since the
/// ones asking for extended thinking stream.
#[test]
fn anthropic_stream_emits_thinking_deltas() {
use crate::openai::{ChatCompletionChunk, ChunkChoice};
let mut t = AnthropicStreamTranslator::new();
let chunk = |delta: Value| ChatCompletionChunk {
id: "c".into(),
object: "chat.completion.chunk".into(),
created: 0,
model: "m".into(),
choices: vec![ChunkChoice {
index: 0,
delta,
finish_reason: None,
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let ev = t.on_chunk(&chunk(json!({ "reasoning_content": "weighing it" })));
let start = ev
.iter()
.find(|(n, _)| n == "content_block_start")
.expect("a block must open");
assert_eq!(start.1["content_block"]["type"], "thinking");
let delta = ev
.iter()
.find(|(n, _)| n == "content_block_delta")
.expect("a delta must follow");
assert_eq!(delta.1["delta"]["type"], "thinking_delta");
assert_eq!(delta.1["delta"]["thinking"], "weighing it");
// Visible text opens a separate block, after thinking closes.
let ev = t.on_chunk(&chunk(json!({ "content": "the answer" })));
let names: Vec<&str> = ev.iter().map(|(n, _)| n.as_str()).collect();
assert!(
names.contains(&"content_block_stop"),
"the thinking block must close first: {names:?}"
);
let start = ev.iter().find(|(n, _)| n == "content_block_start").unwrap();
assert_eq!(start.1["content_block"]["type"], "text");
}
}
-23
View File
@@ -798,10 +798,6 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
// Catalogue-seeded: no neuron has reported rungs yet.
reasoning_budget: Vec::new(),
},
);
}
@@ -843,13 +839,6 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
// so a client never overflows the most-constrained
// deployment that might serve it.
e.limit = tightest_limit(e.limit.take(), entry.limit.clone());
// The reasoning ladder this neuron honours (#223).
// First non-empty wins: a catalogue-seeded entry
// starts with none, and neurons serving the same
// model are configured alike.
if e.reasoning_budget.is_empty() {
e.reasoning_budget = node.reasoning_budget.clone();
}
})
.or_insert_with(|| CortexModelEntry {
id: model_id.clone(),
@@ -869,10 +858,6 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
// What this neuron says each effort level buys (#223).
reasoning_budget: node.reasoning_budget.clone(),
});
}
}
@@ -934,10 +919,6 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
// What this neuron says each effort level buys (#223).
reasoning_budget: node.reasoning_budget.clone(),
});
}
}
@@ -975,10 +956,6 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
// Catalogue-seeded: no neuron has reported rungs yet.
reasoning_budget: Vec::new(),
},
);
}
+1 -10
View File
@@ -141,15 +141,6 @@ pub fn usage_sink(
/// returns a guard the caller settles with actual usage; on refusal returns
/// the #63 envelope (`rate_limit_exceeded` + `Retry-After` for a resetting
/// window, `insufficient_quota` for a hard balance — never `402`).
// The `Err` variant is the #63 envelope itself, which is a data-carrying
// struct (status, type, code, message, param, retry hint, diagnostics) and
// is passed by value everywhere it is built, matched and rendered. Boxing it
// at this one boundary would push `Box` conversions through three handlers
// and both HTTP adapters to save an allocation on a path taken once per
// rejected request. clippy 1.98 lowered what it considers large enough to
// flag; the shape is deliberate. Same call as the existing allows on the
// figment-returning config loaders.
#[allow(clippy::result_large_err)]
pub async fn reserve_or_reject(
provider: Arc<dyn EntitlementProvider>,
principal: &Principal,
@@ -288,7 +279,7 @@ mod tests {
// Requested output dominates; prompt estimate is small for a tiny body.
let body = br#"{"model":"m","max_tokens":1000}"#;
let est = reservation_estimate(body, Some(8192));
assert!((1000..1100).contains(&est), "est was {est}");
assert!(est >= 1000 && est < 1100, "est was {est}");
}
#[test]
+5 -120
View File
@@ -4,7 +4,7 @@
//! in Prometheus text format.
use anyhow::Result;
use metrics_exporter_prometheus::{Matcher, PrometheusBuilder};
use metrics_exporter_prometheus::PrometheusBuilder;
use std::net::SocketAddr;
/// Install the Prometheus metrics recorder and return a handle.
@@ -12,7 +12,8 @@ use std::net::SocketAddr;
pub fn install(listen: &str) -> Result<()> {
let addr: SocketAddr = listen.parse()?;
with_buckets(PrometheusBuilder::new().with_http_listener(addr))?
PrometheusBuilder::new()
.with_http_listener(addr)
.install()
.map_err(|e| anyhow::anyhow!("failed to install Prometheus exporter: {e}"))?;
@@ -24,77 +25,13 @@ pub fn install(listen: &str) -> Result<()> {
/// Install a recorder for testing (no HTTP listener). Returns a handle
/// that can render the current metrics as Prometheus text.
pub fn install_test_recorder() -> Result<metrics_exporter_prometheus::PrometheusHandle> {
let handle = with_buckets(PrometheusBuilder::new())?
let handle = PrometheusBuilder::new()
.install_recorder()
.map_err(|e| anyhow::anyhow!("failed to install test recorder: {e}"))?;
describe_metrics();
Ok(handle)
}
/// Give every histogram explicit buckets, so it is exported as a
/// Prometheus **histogram** rather than a summary.
///
/// Without this, `metrics-exporter-prometheus` renders every
/// `histogram!` as a summary: `{quantile="0.95"}` series and no
/// `_bucket` series at all. Three things break as a result, and the
/// third is what made it visible.
///
/// 1. `histogram_quantile()` has nothing to read, so any dashboard
/// panel written the idiomatic way returns *No data* forever. The
/// fleet dashboard's TTFT panel did exactly that while cortex was
/// serving thousands of requests.
/// 2. Summary quantiles are computed per process over a rolling
/// window, so they **cannot be aggregated**. Averaging p95 across
/// two gateways is not p95 of anything.
/// 3. They decay: with no samples in the window every quantile reads
/// `0`, which is indistinguishable from "genuinely instant" on a
/// graph.
///
/// Buckets are per-metric because the quantities differ by orders of
/// magnitude — a request lasting minutes and a TTFT of milliseconds
/// have no useful shared scale. Ranges are chosen from observed fleet
/// behaviour rather than round numbers: decode runs for minutes on a
/// long turn, prefill is seconds on a long prompt, and decode
/// throughput sits in the tens of tokens/sec.
fn with_buckets(builder: PrometheusBuilder) -> Result<PrometheusBuilder> {
let seconds_short = &[
0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 30.0, 60.0, 120.0,
];
let seconds_long = &[
0.1, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0, 600.0, 1200.0,
];
let tokens_per_second = &[
1.0, 2.5, 5.0, 10.0, 15.0, 20.0, 30.0, 40.0, 60.0, 80.0, 120.0, 200.0,
];
builder
// Whole-request latency: a long agentic turn legitimately runs
// for many minutes, so the tail has to reach there.
.set_buckets_for_metric(
Matcher::Full("cortex_request_duration_seconds".into()),
seconds_long,
)?
// Prefill: sub-second for a short prompt, seconds for a long
// one. Anything past a minute is pathological and belongs in
// the overflow bucket.
.set_buckets_for_metric(
Matcher::Full("cortex_time_to_first_token_seconds".into()),
seconds_short,
)?
// Image generation is inherently slower than a text turn and
// scales with resolution and step count.
.set_buckets_for_metric(
Matcher::Full("cortex_images_generation_seconds".into()),
seconds_long,
)?
// Not a duration — decode throughput, tens of tokens/sec on
// this fleet.
.set_buckets_for_metric(
Matcher::Full("cortex_tokens_per_second".into()),
tokens_per_second,
)
.map_err(|e| anyhow::anyhow!("failed to configure histogram buckets: {e}"))
}
fn describe_metrics() {
metrics::describe_histogram!(
"cortex_request_duration_seconds",
@@ -121,10 +58,6 @@ fn describe_metrics() {
"cortex_request_errors_total",
"Total number of failed proxy requests"
);
metrics::describe_counter!(
"cortex_cached_prompt_tokens_total",
"Prompt tokens served from neuron's prefix KV cache; over cortex_prompt_tokens_total this is the cache-hit rate (#269)"
);
metrics::describe_counter!("cortex_evictions_total", "Total number of model evictions");
metrics::describe_counter!(
"cortex_cold_starts_total",
@@ -178,15 +111,7 @@ fn describe_metrics() {
);
metrics::describe_counter!(
"cortex_model_rejections_total",
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal / anon_yield — the load-shedding signal (#137, #262)"
);
metrics::describe_gauge!(
"cortex_model_anon_in_flight",
"Anonymous (unattributable) requests holding a seat on a neuron:model (#262)"
);
metrics::describe_gauge!(
"cortex_model_anon_max_in_flight",
"Seats anonymous traffic may hold at once, so it cannot starve identified callers (#262)"
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal — the load-shedding signal (#137)"
);
metrics::describe_gauge!(
"cortex_model_tok_s_decode",
@@ -197,43 +122,3 @@ fn describe_metrics() {
"Live prefill throughput per neuron:model, tokens/sec EMA (#137)"
);
}
#[cfg(test)]
mod bucket_tests {
/// Histograms must export as Prometheus **histograms**, with
/// `_bucket` series — not as summaries.
///
/// This asserts the exported *shape*, not the recording call,
/// because the recording was never the problem: cortex measured
/// TTFT correctly for months while the fleet dashboard's panel sat
/// on "No data", since `histogram_quantile()` reads `_bucket`
/// series and a summary has none. A metric that is collected but
/// cannot be queried is indistinguishable from one that was never
/// collected — from the graph, and from the operator's chair.
#[test]
fn histograms_export_buckets_not_summary_quantiles() {
let handle = match super::install_test_recorder() {
Ok(h) => h,
// Another test in this binary owns the global recorder;
// it installs the same buckets, so skipping is honest.
Err(_) => return,
};
metrics::histogram!("cortex_time_to_first_token_seconds", "node" => "n", "model" => "m")
.record(0.42);
let rendered = handle.render();
assert!(
rendered.contains("cortex_time_to_first_token_seconds_bucket"),
"no _bucket series — histogram_quantile() cannot read this:\n{rendered}"
);
assert!(
rendered.contains("# TYPE cortex_time_to_first_token_seconds histogram"),
"exported as the wrong metric type:\n{rendered}"
);
assert!(
!rendered
.contains(r#"cortex_time_to_first_token_seconds{node="n",model="m",quantile="#),
"still exporting summary quantiles, which cannot be aggregated across gateways"
);
}
}
-26
View File
@@ -130,17 +130,6 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
Ok(resp) if resp.status().is_success() => {
match resp.json::<Vec<ModelInfo>>().await {
Ok(models) => {
// The reasoning-effort ladder is host configuration:
// every model on a neuron reports the same rungs, so
// it is held once per node rather than copied onto
// each model entry where the two could drift (#223).
// Taken from the first model that offers one — a
// non-reasoning model advertises none.
node.reasoning_budget = models
.iter()
.map(|m| m.reasoning_budget.clone())
.find(|rungs| !rungs.is_empty())
.unwrap_or_default();
let mut seen = std::collections::HashSet::new();
for upstream in &models {
seen.insert(upstream.id.clone());
@@ -391,21 +380,6 @@ fn export_health_metrics(node: &str, h: &HealthResponse) {
counter!("cortex_model_rejections_total",
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "per_principal")
.absolute(m.rejected_per_principal);
counter!("cortex_model_rejections_total",
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "anon_yield")
.absolute(m.rejected_anon_yield);
// How much of this model's load is unattributable (#262), and the
// ceiling that keeps it from starving identified callers. 0 for a
// pre-#262 neuron, which had no ceiling — skip rather than publish
// a 0 that reads as "anonymous is refused" when it means the
// opposite.
gauge!("cortex_model_anon_in_flight", "node" => node.to_string(), "model" => m.id.clone())
.set(m.anon_in_flight as f64);
if m.anon_max_in_flight > 0 {
gauge!("cortex_model_anon_max_in_flight",
"node" => node.to_string(), "model" => m.id.clone())
.set(m.anon_max_in_flight as f64);
}
}
for d in &h.devices {
let device = d.index.to_string();
-12
View File
@@ -171,9 +171,6 @@ impl IntoResponse for ProxyError {
// over the decode window (first→last chunk); falls back to the
// full request duration for single-chunk (non-streaming) bodies
// cortex_prompt_tokens_total / cortex_completion_tokens_total (counters)
// cortex_cached_prompt_tokens_total (counter) — prompt tokens served
// from neuron's prefix cache; over prompt_tokens_total this is the
// fleet's cache-hit rate (#269)
/// Cap on the retained body tail. The usage object rides on the final
/// chunk, so a generous tail is plenty; the cap bounds memory on huge
@@ -246,15 +243,6 @@ impl ChunkObserver for CortexMetrics {
if let Some(prompt) = prompt {
metrics::counter!("cortex_prompt_tokens_total", &self.labels).increment(prompt);
}
// Prefix-cache reuse (#269), read from the same usage object.
// Against `cortex_prompt_tokens_total` this gives fleet-wide
// cache-hit rate as a ratio, which was previously only
// obtainable by grepping `reused=` out of a neuron journal —
// per-host, unaggregated, and gone on log rotation.
if let Some(cached) = last_count_for(self.tail.as_str(), "cached_tokens") {
metrics::counter!("cortex_cached_prompt_tokens_total", &self.labels)
.increment(cached);
}
if let Some(completion) = completion.filter(|c| *c > 0) {
metrics::counter!("cortex_completion_tokens_total", &self.labels)
.increment(completion);
-7
View File
@@ -535,11 +535,6 @@ async fn profile_to_spec(
quant: profile.quant.clone(),
tensor_parallel,
devices: Some(devices),
// The catalogue's operator override rides along with the load
// (#283), so a cold-loaded model samples the same way as one
// this host was configured to hold resident.
sampling: profile.sampling.clone(),
preserve_thinking: profile.preserve_thinking,
}
}
@@ -647,8 +642,6 @@ mod tests {
pinned_on: vec![],
residency_priority: None,
source: source.map(String::from),
sampling: None,
preserve_thinking: None,
limit: None,
cost: None,
capabilities: vec![],
+1 -25
View File
@@ -66,9 +66,6 @@ impl CortexState {
endpoint: nc.endpoint.clone(),
healthy: false,
models: HashMap::new(),
// Filled from the neuron's /models reply at the
// first poll (#223).
reasoning_budget: Vec::new(),
lifecycle_cycles: 0,
last_poll: None,
discovery: None,
@@ -101,29 +98,8 @@ impl CortexState {
neuron_configs: config.neurons.clone(),
eviction: config.eviction.clone(),
catalogue,
// READ timeout, not a total one. reqwest's `.timeout()` is a
// total deadline — "from when the request starts connecting
// until the response body has finished" — which for a streamed
// completion is a hard cap on how long a model may generate,
// not a liveness check. At 300s and ~13 tok/s that capped every
// stream at roughly 3.9k tokens: cortex severed the connection
// mid-answer, the client reported the stream as terminated, and
// neuron carried on generating into a socket nobody was reading.
// A long agentic turn could therefore never complete through the
// gateway however healthy the fleet was, and the failure looked
// like a model fault because the model was blameless.
//
// `.read_timeout()` resets on every successful read, so it times
// out a stalled upstream while letting a producing one run as
// long as it needs. The same 300s is kept deliberately: it is
// the headroom a cold model load needs before its first byte,
// which is what the original total timeout was reaching for.
//
// This pairs with neuron's 1s SSE keep-alive: a healthy stream
// puts bytes on the wire every second, so idle here means idle,
// and the deadline finally measures what it was meant to.
http_client: reqwest::Client::builder()
.read_timeout(std::time::Duration::from_secs(300))
.timeout(std::time::Duration::from_secs(300))
.build()
.expect("failed to build HTTP client"),
entitlements,
@@ -1,9 +1,9 @@
//! Integration tests for budget enforcement (#52) — the A0 seatbelt.
//!
//! A reservation over the key's hard cap is refused *before* neuron is hit,
//! with the #63 code matching the cap-window semantics: `rate_limit_exceeded`
//! plus `Retry-After` for a resetting window, `insufficient_quota` for a hard
//! balance. Spend never exceeds the cap. No 402, ever.
//! with the #63 code matching the cap-window semantics (rate_limit_exceeded
//! + Retry-After for a resetting window, insufficient_quota for a hard
//! balance). Spend never exceeds the cap. No 402, ever.
use axum::Json;
use axum::extract::Path;
@@ -55,9 +55,6 @@ async fn seed_loaded(fleet: &CortexState, node: &str, in_flight: usize, queue_de
kv_available_mb: 0,
rejected_timeout: 0,
rejected_per_principal: 0,
anon_in_flight: 0,
anon_max_in_flight: 7,
rejected_anon_yield: 0,
tok_s_prefill: 0.0,
tok_s_decode: 0.0,
},
+2 -72
View File
@@ -69,21 +69,6 @@ capabilities = ["text"]
let mut nodes = fleet.nodes.write().await;
let node = nodes.get_mut("mock-node").expect("node exists");
node.healthy = true;
// The rungs the neuron reported at poll time. Since #290 these
// come from the model's own chat template rather than operator
// configuration, so the fixture uses the shape a real reasoning
// model ships: Qwen3.8-27B accepts exactly low/medium/xhigh and
// defaults to xhigh.
node.reasoning_budget = ["low", "medium", "xhigh"]
.into_iter()
.map(|effort| cortex_core::harness::ReasoningBudgetRung {
default: effort == "xhigh",
effort: effort.into(),
// No cap: effort is expressed to the model through its
// template, not enforced by truncating its thinking.
tokens: None,
})
.collect();
node.models.insert(
"test-model".into(),
ModelEntry {
@@ -100,9 +85,6 @@ capabilities = ["text"]
context: 49152,
input: Some(40960),
output: 8192,
// The ceiling a request may name, distinct from the
// 8192-token reserve above (#278).
output_ceiling: 32768,
}),
servable: None,
},
@@ -165,53 +147,7 @@ capabilities = ["text"]
// vLLM-convention probes (Hermes Agent) auto-detect the window.
assert_eq!(entry["max_model_len"], 49152);
assert_eq!(entry["max_input_tokens"], 40960);
// The ceiling, not the reserve (#278). A client that reads this
// field turns it into the cap it sends on each request; advertising
// the 8192 reserve told a reasoning model's harness to stop below
// the cost of a single think block. Observed with dsh — pi takes
// `maxTokens` from static config and reads nothing here.
assert_eq!(entry["max_output_tokens"], 32768);
// The rungs the model itself accepts (#290). A caller can only send
// a rung name, so a name we invent is one the template will reject.
// Publishing the true set is correct regardless of who reads it —
// pi-ai, for one, takes its levels from static config instead.
let rungs = entry["reasoning_budget"]
.as_array()
.expect("reasoning_budget advertised for a reasoning model");
let names: Vec<&str> = rungs
.iter()
.map(|r| r["effort"].as_str().expect("effort"))
.collect();
assert_eq!(names, ["low", "medium", "xhigh"]);
// Which rung applies when the caller names none. Without this a
// silent caller cannot tell what it is getting — on Qwen3.8 that is
// xhigh, the most expensive rung on the ladder.
let default_rung: Vec<&str> = rungs
.iter()
.filter(|r| r["default"].as_bool().unwrap_or(false))
.map(|r| r["effort"].as_str().expect("effort"))
.collect();
assert_eq!(
default_rung,
["xhigh"],
"the model's own default must be visible"
);
// A named rung carries no token count: effort reaches the model
// through its template, and enforcing it by truncation is the defect
// #290 fixed.
assert!(
rungs.iter().all(|r| r.get("tokens").is_none()),
"a named rung must not masquerade as a token budget"
);
// The window under the two names a generic client may read, for one
// that probes these rather than `max_model_len`. Publish all three;
// do not assume a particular client consumes any of them. pi, in
// particular, reads none — `contextWindow` is a required static
// field on its model type (checked against 0.82.1 and 0.84.3), so
// its window is hand-maintained whatever we advertise.
assert_eq!(entry["context_window"], 49152);
assert_eq!(entry["context_length"], 49152);
assert_eq!(entry["max_output_tokens"], 8192);
// No limit → flat fields omitted entirely, never 0 or a guess.
let unknown = body["data"]
@@ -221,13 +157,7 @@ capabilities = ["text"]
.find(|m| m["id"] == "no-limit-model")
.expect("no-limit-model present in /v1/models");
assert!(unknown.get("limit").is_none());
for key in [
"max_model_len",
"max_input_tokens",
"max_output_tokens",
"context_window",
"context_length",
] {
for key in ["max_model_len", "max_input_tokens", "max_output_tokens"] {
assert!(
unknown.get(key).is_none(),
"{key} must be omitted when the window is unknown"
@@ -96,7 +96,6 @@ async fn spawn_gateway(neuron: &str, context: usize) -> String {
context,
input: None,
output: 16,
output_ceiling: 0,
}),
servable: None,
},
+5 -5
View File
@@ -80,7 +80,7 @@ The fastest path: env-var single-endpoint config.
```sh
export HELEXA_ACP_BASE_URL=http://hanzalova.internal:31313/v1
export HELEXA_ACP_MODEL=Qwen/Qwen3.8-27B
export HELEXA_ACP_MODEL=Qwen/Qwen3.6-27B
helexa-acp # speaks ACP over stdin/stdout; not interactive
```
@@ -117,7 +117,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.8-27B"
default_model = "Qwen/Qwen3.6-27B"
max_tokens = 8192
context_window = 32768
@@ -138,7 +138,7 @@ default_model = "claude-opus-4"
Restart Zed. The model dropdown lists every model from every
configured endpoint with the `endpoint:model` selector
(`helexa:Qwen/Qwen3.8-27B`, `openrouter:anthropic/claude-opus-4`,
(`helexa:Qwen/Qwen3.6-27B`, `openrouter:anthropic/claude-opus-4`,
…). Switch mid-session; the next prompt routes to the new endpoint.
When only one endpoint is configured the prefix is dropped (model
@@ -166,7 +166,7 @@ Copy-pasteable blocks. Mix and match.
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.8-27B"
default_model = "Qwen/Qwen3.6-27B"
max_tokens = 8192
context_window = 32768
```
@@ -241,7 +241,7 @@ vLLM.
name = "lan"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.8-27B"
default_model = "Qwen/Qwen3.6-27B"
[[endpoints]]
name = "cloud"
+1 -29
View File
@@ -214,39 +214,11 @@ fn default_notify_to() -> String {
}
impl AngelsConfig {
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> anyhow::Result<Self> {
let path = path.as_ref();
let cfg: Self = Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Toml::file(path.as_ref()))
.merge(Env::prefixed("ANGELS_").split("__"))
.extract()?;
Ok(cfg)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
-9
View File
@@ -34,7 +34,6 @@ pub fn api_routes(state: ApiState) -> Router {
Router::new()
.route("/api/health", get(health))
.route("/api/dimensions", get(dimensions))
.route("/api/regimes", get(regimes))
.route("/api/summary", get(summary))
.route("/api/scaling", get(scaling))
.route("/api/concurrency", get(concurrency))
@@ -66,14 +65,6 @@ async fn health(State(s): State<ApiState>) -> Result<Json<serde_json::Value>, Ap
Ok(Json(json!({ "status": "ok", "run_count": count })))
}
/// Declared measurement-regime boundaries (#288).
///
/// Static — no store access — but served from the API rather than
/// hardcoded in the frontend so `report` and the UI cite one list.
async fn regimes() -> Json<&'static [crate::regime::MeasurementRegime]> {
Json(crate::regime::REGIMES)
}
async fn dimensions(State(s): State<ApiState>) -> Result<Json<crate::store::Dimensions>, ApiError> {
let store = s.lock().await;
store.dimensions().map(Json).map_err(err500)
+4 -55
View File
@@ -6,7 +6,6 @@ use crate::config::{TargetConfig, TargetKind};
use anyhow::{Context, Result, anyhow};
use cortex_core::build_info::BuildInfo;
use cortex_core::discovery::{DiscoveryResponse, HealthResponse};
use cortex_core::entitlements::{HEADER_ACCOUNT_ID, HEADER_KEY_ID};
use cortex_core::harness::{ModelInfo, ModelSpec};
use cortex_core::openai::ModelsResponse;
use std::time::Duration;
@@ -20,49 +19,10 @@ pub struct TargetClient {
impl TargetClient {
pub fn new(request_timeout: Duration) -> Result<Self> {
Self::with_principal(request_timeout, None)
}
/// Build a client that stamps `principal`'s identity headers on every
/// request (#288).
///
/// Set on the client rather than at each call site so no scenario can
/// be added later that silently measures as anonymous — which is the
/// failure this fixes, and it was invisible for a week.
///
/// `None` keeps the historical anonymous behaviour, which is correct
/// for `openai` targets (the headers mean nothing to a foreign
/// engine) and for a fresh install with no principal configured.
pub fn with_principal(
request_timeout: Duration,
principal: Option<&crate::config::PrincipalSettings>,
) -> Result<Self> {
let mut builder = reqwest::Client::builder().timeout(request_timeout);
if let Some(p) = principal {
let mut headers = reqwest::header::HeaderMap::new();
headers.insert(
HEADER_ACCOUNT_ID,
reqwest::header::HeaderValue::from_str(&p.account_id)
.context("principal.account_id is not a valid header value")?,
);
headers.insert(
HEADER_KEY_ID,
reqwest::header::HeaderValue::from_str(&p.key_id)
.context("principal.key_id is not a valid header value")?,
);
tracing::info!(
account_id = %p.account_id,
key_id = %p.key_id,
"bench authenticates: measurements reflect identified-caller capacity (#288)"
);
builder = builder.default_headers(headers);
} else {
tracing::warn!(
"bench has no [bench.principal]: requests are anonymous, so since #262 \
they measure the anonymous-yield policy rather than serving capacity (#288)"
);
}
let http = builder.build().context("building HTTP client")?;
let http = reqwest::Client::builder()
.timeout(request_timeout)
.build()
.context("building HTTP client")?;
Ok(TargetClient { http })
}
@@ -209,15 +169,6 @@ impl TargetClient {
quant: None,
tensor_parallel: (info.devices.len() > 1).then_some(info.devices.len() as u32),
devices: Some(info.devices.clone()),
// `GET /models` does not report the operator sampling
// override (#283), so a spec reconstructed here cannot carry
// it — same limitation as `quant` directly above. Benign
// while this is only used to re-load a model the host
// already has a `[[default_models]]` entry for, since that
// entry supplies the override; it would matter if this were
// ever used to load a model the host has no config for.
sampling: None,
preserve_thinking: None,
})
}
@@ -282,11 +233,9 @@ impl TargetClient {
cost: None,
tool_call: false,
reasoning: false,
preserve_thinking: None,
// A generic OpenAI endpoint says nothing about
// serviceability, and absent means "no opinion".
servable: None,
reasoning_budget: Vec::new(),
})
.collect())
}
-94
View File
@@ -67,44 +67,6 @@ pub struct BenchSettings {
/// SQLite system-of-record path.
#[serde(default = "default_db_path")]
pub db_path: String,
/// Identity bench presents to a neuron (#288).
///
/// Without this bench is *anonymous*, and since #262 anonymous
/// callers are served only from capacity left over once identified
/// traffic is satisfied — they are capped below `max_in_flight` and
/// park at the class gate ahead of every other gate. Measured on
/// beast, that moved `concurrency:8` from 168.9 tok/s / 0.53 s
/// ttft_p95 to 104.6 / 11.77 s the day #262 landed, and held there.
///
/// So an unauthenticated bench does not measure serving capacity; it
/// measures the yield policy. Real traffic arrives through cortex
/// with a principal and is subject to none of it.
///
/// Deliberately not an admission *exemption*: bench should be
/// subject to the same rules as any caller, or it measures something
/// no user experiences — the same class of error in the other
/// direction. Give it a principal and let it queue like everyone
/// else.
#[serde(default)]
pub principal: Option<PrincipalSettings>,
}
/// The account/key pair bench stamps on inference requests (#288).
///
/// These are the headers cortex asserts after a bearer resolves (#49).
/// Bench talks to neurons **directly** over the WireGuard mesh, so it
/// stamps them itself — the same trust model the link already relies on,
/// since a neuron accepts them from cortex on exactly that basis.
///
/// Give bench its *own* pair rather than borrowing a real caller's: it
/// then holds its own fair-share allocation (#54) and cannot starve
/// interactive traffic, which is the failure #262 was fixing.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PrincipalSettings {
/// Stamped as `x-helexa-account-id`.
pub account_id: String,
/// Stamped as `x-helexa-key-id`.
pub key_id: String,
}
impl Default for BenchSettings {
@@ -115,7 +77,6 @@ impl Default for BenchSettings {
iteration_pause_secs: default_iter_pause(),
request_timeout_secs: default_timeout(),
db_path: default_db_path(),
principal: None,
}
}
}
@@ -149,18 +110,6 @@ pub struct ScenarioConfig {
/// deliberately, e.g. `concurrency_levels = [2, 4, 8]`.
#[serde(default)]
pub concurrency_levels: Vec<u32>,
/// Concurrency levels to run **non-streaming** (#285). Separate from
/// `concurrency_levels` because the two shapes are not
/// interchangeable: streaming multiplexes through the batch engine,
/// non-streaming did not until #285 and serialized completely — a
/// measured 1.00x aggregate throughput at every level against 3.98x
/// streamed.
///
/// Defaults to the top level only. The signal is starkest there and
/// each extra level costs a full burst per sample per build; an
/// operator who wants the whole curve can list it.
#[serde(default = "default_concurrency_nonstreaming_levels")]
pub concurrency_nonstreaming_levels: Vec<u32>,
/// Square image sizes (px per side) — one `image:<px>` scenario per
/// entry (#203), run only against models advertising the `image`
/// capability. Defaults to `[1024]`.
@@ -199,7 +148,6 @@ impl Default for ScenarioConfig {
prompt_sizes: default_prompt_sizes(),
max_tokens: default_max_tokens(),
concurrency_levels: Vec::new(),
concurrency_nonstreaming_levels: Vec::new(),
concurrency_prompt_tokens: default_concurrency_prompt_tokens(),
image_sizes: default_image_sizes(),
capability_probes: Vec::new(),
@@ -267,10 +215,6 @@ fn default_timeout() -> u64 {
fn default_db_path() -> String {
"/var/lib/helexa-bench/bench.sqlite".to_string()
}
fn default_concurrency_nonstreaming_levels() -> Vec<u32> {
vec![8]
}
fn default_api_enabled() -> bool {
true
}
@@ -299,44 +243,6 @@ fn default_capability_max_tokens() -> u64 {
// figment's, not ours, so suppress the lint here.
#[allow(clippy::result_large_err)]
mod tests {
/// #288: a config with no `[bench.principal]` must still parse — the
/// anonymous behaviour is the historical one and correct for
/// `openai` targets — but a configured one must round-trip, because
/// a silently-dropped principal reverts the measurement to the
/// anonymous-yield path without changing a visible number.
#[test]
fn principal_is_optional_and_round_trips() {
let parse = |t: &str| -> BenchConfig {
figment::Figment::new()
.merge(figment::providers::Toml::string(t))
.extract()
.expect("parse")
};
let none = parse("[bench]\nsamples_per_version = 3\n");
assert!(none.bench.principal.is_none());
let some = parse(
"[bench]\nsamples_per_version = 3\n\n\
[bench.principal]\naccount_id = \"acct\"\nkey_id = \"k1\"\n",
);
let p = some.bench.principal.expect("principal");
assert_eq!((p.account_id.as_str(), p.key_id.as_str()), ("acct", "k1"));
}
/// The non-streaming burst defaults on (#285). If it defaulted off,
/// the path that serialized completely would stay unmeasured on every
/// fleet that had not opted in — which is how it went unnoticed.
#[test]
fn nonstreaming_concurrency_defaults_to_the_top_level() {
let cfg: BenchConfig = figment::Figment::new()
.merge(figment::providers::Toml::string(
"[scenarios]\nprompt_sizes = [128]\n",
))
.extract()
.expect("parse");
assert_eq!(cfg.scenarios.concurrency_nonstreaming_levels, vec![8]);
}
use super::*;
use figment::Jail;
-1
View File
@@ -7,7 +7,6 @@
pub mod api;
pub mod client;
pub mod config;
pub mod regime;
pub mod report;
pub mod scenario;
pub mod store;
-148
View File
@@ -1,148 +0,0 @@
//! Measurement-regime boundaries — builds where a number changed
//! meaning rather than value.
//!
//! A trend line silently spans them, and every reader re-derives the
//! same thing: that a cliff was the instrument, not the engine. Three
//! have already cost real investigation time:
//!
//! - beast's `concurrency:8` fell 168.9 → 104.6 tok/s with `ttft_p95`
//! rising 0.53 → 11.77 s, and it read as a 38% throughput regression
//! until the one commit in the window turned out to be an admission
//! *policy* change that bench, being anonymous, was subject to (#288).
//! - quadbrat's `capability:debug-reason` TTFT fell 88.8 → 0.1 s, which
//! read as a latency win and was a redefinition of when the clock
//! stops.
//! - the same redefinition put decode rates at ~250 tok/s on a 3060
//! beforehand, by dividing reasoning-inclusive token counts by a
//! visible-content-only window.
//!
//! So the boundaries are declared, not remembered. Declared here rather
//! than in the frontend because `report` renders the same series into
//! `benchmarks.md` and needs the same caveats.
//!
//! ## What belongs here
//!
//! Only changes to **what a metric means**. A change that made the
//! server genuinely faster, slower, or differently behaved does *not*
//! belong — that is the signal these lines exist to keep readable. #280
//! (fetch the model's own chat template) moved quadbrat's
//! `completion_tokens` from 4096 to 1053 by ending a truncation loop;
//! that is a real behavioural change and stays unmarked.
//!
//! The identified/anonymous split is deliberately absent too: it is
//! recorded per-row in `runs.principal`, so the UI derives it from the
//! data instead of trusting a constant to stay true.
use serde::Serialize;
/// A build at which one or more metrics changed meaning.
#[derive(Debug, Clone, Serialize)]
pub struct MeasurementRegime {
/// Short SHA of the first build measured under the new semantics.
/// Matched against `SeriesPoint::git_sha`, so a boundary whose build
/// never appears in a selection simply does not draw.
pub first_sha: &'static str,
/// Terse label for the chart rule.
pub label: &'static str,
/// What changed, and why the step is not a performance result.
pub detail: &'static str,
/// Series keys the change affects, as the API/UI name them. A
/// boundary only draws on panels carrying one of these — marking
/// every panel would train people to ignore the rules.
pub affects: &'static [&'static str],
}
/// The declared boundaries, oldest first.
pub const REGIMES: &[MeasurementRegime] = &[
MeasurementRegime {
first_sha: "92dddc0",
label: "#262 anonymous callers yield",
detail: "Anonymous traffic became servable only from capacity left over \
once identified traffic is satisfied — capped below max_in_flight \
and parked at the class gate. An anonymous bench therefore \
characterises the yield policy rather than serving capacity. \
Steps here are admission policy, not the engine (#288).",
affects: &["decode", "ttft", "ttftP95", "queueWait", "rejected"],
},
MeasurementRegime {
first_sha: "0fe7aa3",
label: "#117 reasoning counts as liveness",
detail: "TTFT previously waited for the first *visible content* chunk, so \
on a thinking model it measured the whole reasoning span (88.8 s \
on quadbrat). It now stops at the first delta of any kind. The \
same change fixed decode rates computed by dividing \
reasoning-inclusive token counts by a content-only window, which \
had produced ~250 tok/s on a 3060.",
affects: &["decode", "ttft"],
},
];
/// Boundaries affecting `metric`, in declaration order.
pub fn regimes_for(metric: &str) -> Vec<&'static MeasurementRegime> {
REGIMES
.iter()
.filter(|r| r.affects.contains(&metric))
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
/// A boundary that names no metric would draw nowhere and mislead by
/// omission; one naming an unknown key would draw nowhere and look
/// fine. Both are typos this catches.
#[test]
fn every_regime_affects_a_known_series_key() {
// The keys `Trends.tsx` maps `SeriesPoint` onto.
const KNOWN: &[&str] = &[
"decode",
"ttft",
"ttftP95",
"queueWait",
"rejected",
"prefillTps",
"reasoning",
"cached",
"completion",
"tpot",
];
for r in REGIMES {
assert!(
!r.affects.is_empty(),
"{} affects nothing, so it would never draw",
r.first_sha
);
for key in r.affects {
assert!(
KNOWN.contains(key),
"{} names unknown series key {key:?}",
r.first_sha
);
}
}
}
#[test]
fn regimes_for_selects_by_metric() {
assert!(regimes_for("ttft").iter().any(|r| r.first_sha == "0fe7aa3"));
assert!(
regimes_for("completion").is_empty(),
"#280 changed behaviour, not meaning — it must not be marked"
);
}
/// Short SHAs are matched against `SeriesPoint::git_sha`, which the
/// store records at the width `/version` reports.
#[test]
fn declared_shas_are_short_form() {
for r in REGIMES {
assert_eq!(
r.first_sha.len(),
7,
"{} is not a 7-char short sha; it would never match a series point",
r.first_sha
);
}
}
}
+16 -403
View File
@@ -9,7 +9,6 @@
use crate::config::ScenarioConfig;
use anyhow::{Context, Result, anyhow};
use async_trait::async_trait;
use cortex_core::entitlements::HEADER_KEY_ID;
use cortex_core::harness::ModelInfo;
use cortex_core::openai::ChatCompletionChunk;
use eventsource_stream::Eventsource;
@@ -47,17 +46,6 @@ pub struct RunCtx<'a> {
pub model_id: String,
pub max_tokens: u64,
pub timeout: Duration,
/// Base `x-helexa-key-id` bench presents (#288), when configured.
///
/// The client already sends this as a default header, so every
/// scenario is identified without having to remember. A concurrency
/// burst overrides it per stream — see
/// [`ConcurrencyScenario`] — because fair-share (#54) bounds one
/// principal to `max_per_principal` in-flight-or-queued requests,
/// which defaults to 2. Eight streams under one identity would be
/// capped at two and the rest rejected, measuring the fair-share cap
/// instead of the server.
pub principal_key_id: Option<String>,
}
/// Operator-felt metrics for a single measured request.
@@ -84,34 +72,6 @@ pub struct ScenarioMetrics {
pub decode_ms: Option<u64>,
/// Tokens submitted to prefill — the denominator for prefill tok/s.
pub prefill_tokens: Option<u64>,
/// Tokens spent inside the reasoning span, from
/// `usage.completion_tokens_details.reasoning_tokens` (#223). A
/// sub-count of `completion_tokens`.
///
/// Worth a trend of its own: on a reasoning model this is the
/// dominant cost driver and it moves independently of speed. A
/// template change, a sampling change (#283) or a reasoning-budget
/// change can double what the model thinks before answering while
/// every tok/s number stays flat — the bill doubles and no chart
/// moves.
pub reasoning_tokens: Option<u64>,
/// p95 inter-token arrival gap in milliseconds — the tail of the
/// stream's smoothness, client-observed.
///
/// `decode_tps` is a mean over the whole decode window, so a stream
/// that stalls for a second and then catches up is indistinguishable
/// from one that never stalled. This is the number a user actually
/// feels, and the one a batching stall or a mid-stream rebatch shows
/// up in. `None` for non-streaming, which has no inter-token gaps by
/// construction, and for streams too short to have a tail.
pub tpot_p95_ms: Option<f64>,
/// Prompt tokens served from neuron's prefix KV cache (#269), from
/// `usage.prompt_tokens_details.cached_tokens`.
///
/// The reason prefill timing varies between otherwise identical
/// samples. #269 exists because the saving was invisible and every
/// client reported a 0% hit rate; bench then threw the number away.
pub cached_tokens: Option<u64>,
// ── Concurrency / agentic-load fields (#89) ──────────────────────────
// Set only by the concurrency scenario, which fans out N simultaneous
// streams to characterize the real a0/hermes/opencode workload that
@@ -243,9 +203,6 @@ impl Scenario for ImageLatencyScenario {
.as_u64()
.map(|d| d + timing["decode_ms"].as_u64().unwrap_or(0)),
prefill_tokens: None,
reasoning_tokens: None,
cached_tokens: None,
tpot_p95_ms: None,
concurrency: None,
ttft_p95_s: None,
queue_wait_ms_median: None,
@@ -276,18 +233,6 @@ pub fn build_scenarios(cfg: &ScenarioConfig) -> Vec<Box<dyn Scenario>> {
id: format!("concurrency:{n}"),
concurrency: n,
approx_prompt_tokens: cfg.concurrency_prompt_tokens,
streaming: true,
}) as Box<dyn Scenario>);
}
// Non-streaming bursts (#285). A separate cell id so the two shapes
// never average together — they measured 1.00x against 3.98x before
// #285, and a combined number would have hidden both.
for &n in &cfg.concurrency_nonstreaming_levels {
scenarios.push(Box::new(ConcurrencyScenario {
id: format!("concurrency:{n}:nostream"),
concurrency: n,
approx_prompt_tokens: cfg.concurrency_prompt_tokens,
streaming: false,
}) as Box<dyn Scenario>);
}
for &side in &cfg.image_sizes {
@@ -312,7 +257,7 @@ pub fn build_scenarios(cfg: &ScenarioConfig) -> Vec<Box<dyn Scenario>> {
pub async fn cold_probe(ctx: &RunCtx<'_>) -> Result<ScenarioMetrics> {
let prompt = build_prompt(128);
let payload = chat_payload(ctx, &prompt);
tokio::time::timeout(ctx.timeout, stream_and_measure(ctx, &payload, None))
tokio::time::timeout(ctx.timeout, stream_and_measure(ctx, &payload))
.await
.map_err(|_| anyhow!("cold probe timed out after {:?}", ctx.timeout))?
}
@@ -350,7 +295,7 @@ impl Scenario for ChatLatencyScenario {
async fn run(&self, ctx: &RunCtx) -> Result<ScenarioMetrics> {
let prompt = build_prompt(self.approx_prompt_tokens);
let payload = chat_payload(ctx, &prompt);
let fut = stream_and_measure(ctx, &payload, None);
let fut = stream_and_measure(ctx, &payload);
tokio::time::timeout(ctx.timeout, fut)
.await
.map_err(|_| anyhow!("request timed out after {:?}", ctx.timeout))?
@@ -368,16 +313,6 @@ pub struct ConcurrencyScenario {
id: String,
concurrency: u32,
approx_prompt_tokens: u32,
/// Whether the burst streams (#285).
///
/// Both shapes matter and they are not interchangeable. Streaming
/// multiplexes through the batch engine; non-streaming did not until
/// #285, and serialized completely — 1.00x aggregate throughput at
/// every concurrency level against 3.98x streamed. A bench that only
/// streams cannot see that, which is why it went unmeasured: the
/// neuron reports `in_flight: 8`, admission accepts, `/health` looks
/// healthy, and throughput is single-stream.
streaming: bool,
}
#[async_trait]
@@ -392,26 +327,13 @@ impl Scenario for ConcurrencyScenario {
async fn run(&self, ctx: &RunCtx) -> Result<ScenarioMetrics> {
let prompt = build_prompt(self.approx_prompt_tokens);
let payload = if self.streaming {
chat_payload(ctx, &prompt)
} else {
nonstreaming_payload(ctx, &prompt)
};
let payload = chat_payload(ctx, &prompt);
// Fire all requests at once; each is independently timed and capped
// by the per-request timeout so one hung request can't stall the
// burst.
// Fire all streams at once; each is independently timed and capped by
// the per-request timeout so one hung stream can't stall the burst.
let burst_start = Instant::now();
let streaming = self.streaming;
let payload = &payload;
let futs = (0..self.concurrency).map(move |i| async move {
// Each stream presents its own identity — see
// `with_stream_identity`.
if streaming {
tokio::time::timeout(ctx.timeout, stream_and_measure(ctx, payload, Some(i))).await
} else {
tokio::time::timeout(ctx.timeout, request_and_measure(ctx, payload, Some(i))).await
}
let futs = (0..self.concurrency).map(|_| async {
tokio::time::timeout(ctx.timeout, stream_and_measure(ctx, &payload)).await
});
let results = futures::future::join_all(futs).await;
let burst_window = burst_start.elapsed().as_secs_f64();
@@ -429,7 +351,7 @@ impl Scenario for ConcurrencyScenario {
}
if streams.is_empty() {
return Err(anyhow!(
"all {} concurrent requests failed ({rejected} shed by admission)",
"all {} concurrent streams failed ({rejected} shed by admission)",
self.concurrency
));
}
@@ -458,50 +380,9 @@ impl Scenario for ConcurrencyScenario {
total_s: burst_window,
prompt_tokens: streams.iter().find_map(|m| m.prompt_tokens),
completion_tokens: total_tokens,
// Median server-side phase timing across the burst. These
// were `None`, which left the prefill-tok/s panel blank for
// exactly the cells where queue behaviour matters most, and
// made `queue_wait_ms` (TTFT minus server prefill, #85)
// uncomputable for a burst — the one number that separates
// "the server is slow" from "you were queued".
//
// Median rather than sum: every stream prefills its own copy
// of the same prompt, so the representative per-stream cost
// is the meaningful figure and it composes with `ttft_s`,
// which is also a median.
prefill_ms: median(
&streams
.iter()
.filter_map(|m| m.prefill_ms.map(|v| v as f64))
.collect::<Vec<_>>(),
)
.map(|v| v as u64),
decode_ms: median(
&streams
.iter()
.filter_map(|m| m.decode_ms.map(|v| v as f64))
.collect::<Vec<_>>(),
)
.map(|v| v as u64),
prefill_tokens: median(
&streams
.iter()
.filter_map(|m| m.prefill_tokens.map(|v| v as f64))
.collect::<Vec<_>>(),
)
.map(|v| v as u64),
// Summed across the burst, like `completion_tokens`: the cell
// describes the whole burst, so a per-stream figure here
// would not compose with it.
reasoning_tokens: sum_opt(&streams, |m| m.reasoning_tokens),
cached_tokens: sum_opt(&streams, |m| m.cached_tokens),
// The worst stream in the burst, not a median of medians:
// under load the question is how bad it got for somebody,
// and averaging tails is how a stall stays invisible.
tpot_p95_ms: streams
.iter()
.filter_map(|m| m.tpot_p95_ms)
.max_by(|a, b| a.total_cmp(b)),
prefill_ms: None,
decode_ms: None,
prefill_tokens: None,
concurrency: Some(self.concurrency),
ttft_p95_s: percentile(&ttfts, 95.0),
queue_wait_ms_median: median(&queue_waits),
@@ -512,146 +393,6 @@ impl Scenario for ConcurrencyScenario {
}
}
/// Sum an optional per-stream count across a burst, `None` when no
/// stream reported it — so "nobody measured it" stays distinguishable
/// from "measured as zero", the same distinction #269 exists to
/// preserve.
fn sum_opt(
streams: &[ScenarioMetrics],
f: impl Fn(&ScenarioMetrics) -> Option<u64>,
) -> Option<u64> {
let vals: Vec<u64> = streams.iter().filter_map(f).collect();
(!vals.is_empty()).then(|| vals.iter().sum())
}
/// Per-stream identity for a concurrency burst (#288/#54).
///
/// Fair-share bounds *one principal* to `max_per_principal`
/// in-flight-or-queued requests, defaulting to 2. A burst of 8 under a
/// single identity would therefore have 6 rejected with `PrincipalCap`,
/// and the scenario would measure the fair-share cap rather than the
/// server.
///
/// Giving each stream its own key is also the more faithful model: a
/// concurrency-8 burst is meant to stand in for eight simultaneous
/// *callers*, not one caller issuing eight requests. Fair-share is still
/// exercised — each identity is subject to it — just not self-inflicted.
fn with_stream_identity(
rb: reqwest::RequestBuilder,
ctx: &RunCtx<'_>,
stream: Option<u32>,
) -> reqwest::RequestBuilder {
match (ctx.principal_key_id.as_deref(), stream) {
(Some(base), Some(i)) => rb.header(HEADER_KEY_ID, format!("{base}-{i}")),
// No principal configured, or a single-request scenario: the
// client's default headers already carry the base identity.
_ => rb,
}
}
/// The non-streaming counterpart to [`chat_payload`] (#285).
///
/// `stream_options` is deliberately absent: usage is part of the body on
/// this shape, and sending the streaming-only option to a strict server
/// is a needless compatibility risk.
fn nonstreaming_payload(ctx: &RunCtx, prompt: &str) -> serde_json::Value {
json!({
"model": ctx.model_id,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": ctx.max_tokens,
"temperature": 0,
"stream": false,
})
}
/// Time one non-streaming chat completion (#285).
///
/// There is no first-chunk moment on this shape — the whole body arrives
/// at once — so `ttft_s` is the full request wall-clock rather than a
/// separate measurement. That is the honest reading: for a non-streaming
/// caller, time-to-first-token *is* time-to-everything, and it is exactly
/// what makes the serialization this scenario exists to detect so
/// expensive.
///
/// `decode_tps` is per-request; the burst aggregate is computed by the
/// caller over the whole window, the same way the streaming path does it.
async fn request_and_measure(
ctx: &RunCtx<'_>,
payload: &serde_json::Value,
stream: Option<u32>,
) -> Result<ScenarioMetrics> {
let start = Instant::now();
let resp = with_stream_identity(ctx.client.post(&ctx.chat_url), ctx, stream)
.json(payload)
.send()
.await
.context("sending non-streaming chat request")?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(anyhow!("upstream returned {status}: {}", body.trim()));
}
let v: serde_json::Value = resp
.json()
.await
.context("non-streaming chat response was not JSON")?;
let total_s = start.elapsed().as_secs_f64();
let usage = v.get("usage");
let completion_tokens = usage
.and_then(|u| u.get("completion_tokens"))
.and_then(|t| t.as_u64())
.unwrap_or(0);
let prompt_tokens = usage
.and_then(|u| u.get("prompt_tokens"))
.and_then(|t| t.as_u64());
let reasoning_tokens = usage
.and_then(|u| u.get("completion_tokens_details"))
.and_then(|d| d.get("reasoning_tokens"))
.and_then(|t| t.as_u64());
let cached_tokens = usage
.and_then(|u| u.get("prompt_tokens_details"))
.and_then(|d| d.get("cached_tokens"))
.and_then(|t| t.as_u64());
let timing = usage.and_then(|u| u.get("helexa_timing"));
let field = |name: &str| timing.and_then(|t| t.get(name)).and_then(|x| x.as_u64());
let (prefill_ms, decode_ms, prefill_tokens) = (
field("prefill_ms"),
field("decode_ms"),
field("prefill_tokens"),
);
// Prefer the server's own decode window when it reports one (#85);
// it excludes prefill, so it is comparable with the streaming path's
// decode-window rate rather than being diluted by it.
let decode_tps = match decode_ms {
Some(ms) if ms > 200 => Some(completion_tokens as f64 / (ms as f64 / 1000.0)),
_ if total_s > 0.2 => Some(completion_tokens as f64 / total_s),
_ => None,
};
Ok(ScenarioMetrics {
ttft_s: total_s,
decode_tps,
total_s,
prompt_tokens,
completion_tokens,
prefill_ms,
decode_ms,
prefill_tokens,
reasoning_tokens,
cached_tokens,
// Non-streaming has no inter-token gaps: the body arrives whole.
tpot_p95_ms: None,
concurrency: None,
ttft_p95_s: None,
queue_wait_ms_median: None,
rejected: None,
artifact: None,
image_units: None,
})
}
/// Quality probe (#91): runs a fixed prompt and stores the full generated
/// text as an artifact for later scoring (manual now, LLM-judge later). The
/// point is to compare reasoning/planning quality across models — the axis
@@ -684,7 +425,7 @@ impl Scenario for CapabilityScenario {
"stream": true,
"stream_options": {"include_usage": true},
});
let fut = stream_and_measure_inner(ctx, &payload, true, None);
let fut = stream_and_measure_inner(ctx, &payload, true);
tokio::time::timeout(ctx.timeout, fut)
.await
.map_err(|_| anyhow!("capability probe timed out after {:?}", ctx.timeout))?
@@ -727,9 +468,8 @@ fn percentile(values: &[f64], p: f64) -> Option<f64> {
async fn stream_and_measure(
ctx: &RunCtx<'_>,
payload: &serde_json::Value,
stream: Option<u32>,
) -> Result<ScenarioMetrics> {
stream_and_measure_inner(ctx, payload, false, stream).await
stream_and_measure_inner(ctx, payload, false).await
}
/// As [`stream_and_measure`] but accumulates the full visible text when
@@ -739,10 +479,11 @@ async fn stream_and_measure_inner(
ctx: &RunCtx<'_>,
payload: &serde_json::Value,
capture_text: bool,
stream_ix: Option<u32>,
) -> Result<ScenarioMetrics> {
let start = Instant::now();
let resp = with_stream_identity(ctx.client.post(&ctx.chat_url), ctx, stream_ix)
let resp = ctx
.client
.post(&ctx.chat_url)
.json(payload)
.send()
.await
@@ -762,10 +503,6 @@ async fn stream_and_measure_inner(
let mut prefill_ms: Option<u64> = None;
let mut decode_ms: Option<u64> = None;
let mut prefill_tokens: Option<u64> = None;
let mut reasoning_tokens: Option<u64> = None;
let mut cached_tokens: Option<u64> = None;
// Inter-token arrival gaps, for the p95 tail.
let mut gaps_ms: Vec<f64> = Vec::new();
let mut captured = String::new();
while let Some(event) = stream.next().await {
@@ -803,11 +540,6 @@ async fn stream_and_measure_inner(
if content.is_some() || reasoning.is_some() {
if first.is_none() {
first = Some(now);
} else if let Some(prev) = last {
// Gaps only *between* generated deltas: the wait for
// the first one is TTFT and belongs to prefill, not
// to stream smoothness.
gaps_ms.push(now.duration_since(prev).as_secs_f64() * 1000.0);
}
last = Some(now);
chunk_count += 1;
@@ -819,14 +551,6 @@ async fn stream_and_measure_inner(
if let Some(usage) = chunk.usage {
prompt_tokens = Some(usage.prompt_tokens);
completion_tokens = Some(usage.completion_tokens);
reasoning_tokens = usage
.completion_tokens_details
.as_ref()
.map(|d| d.reasoning_tokens);
cached_tokens = usage
.prompt_tokens_details
.as_ref()
.map(|d| d.cached_tokens);
if let Some(t) = usage.helexa_timing {
prefill_ms = Some(t.prefill_ms);
decode_ms = Some(t.decode_ms);
@@ -865,9 +589,6 @@ async fn stream_and_measure_inner(
prefill_ms,
decode_ms,
prefill_tokens,
reasoning_tokens,
cached_tokens,
tpot_p95_ms: percentile(&gaps_ms, 95.0),
// Concurrency fields unset on the single-request path; the
// concurrency scenario builds its own aggregate (#89).
concurrency: None,
@@ -883,113 +604,6 @@ async fn stream_and_measure_inner(
mod tests {
use super::*;
/// A burst must report server-side phase timing, or `queue_wait_ms`
/// (TTFT minus server prefill, #85) cannot be computed for the very
/// cells it was designed for, and the prefill-tok/s panel stays
/// blank under concurrency.
#[test]
fn a_burst_reports_median_phase_timing_not_none() {
let s = |prefill_ms, decode_ms, prefill_tokens| ScenarioMetrics {
ttft_s: 0.1,
decode_tps: Some(10.0),
total_s: 1.0,
prompt_tokens: Some(100),
completion_tokens: 10,
prefill_ms,
decode_ms,
prefill_tokens,
reasoning_tokens: None,
cached_tokens: None,
tpot_p95_ms: Some(5.0),
concurrency: None,
ttft_p95_s: None,
queue_wait_ms_median: None,
rejected: None,
artifact: None,
image_units: None,
};
let streams = [
s(Some(10), Some(100), Some(512)),
s(Some(20), Some(200), Some(512)),
s(Some(30), Some(300), Some(512)),
];
let med = |f: fn(&ScenarioMetrics) -> Option<u64>| {
median(
&streams
.iter()
.filter_map(|m| f(m).map(|v| v as f64))
.collect::<Vec<_>>(),
)
.map(|v| v as u64)
};
assert_eq!(med(|m| m.prefill_ms), Some(20));
assert_eq!(med(|m| m.decode_ms), Some(200));
// Every stream prefills the same prompt, so the median is the
// prompt size rather than a sum across the burst.
assert_eq!(med(|m| m.prefill_tokens), Some(512));
}
/// #288/#54: a burst must present one identity *per stream*.
///
/// Fair-share bounds a single principal to `max_per_principal`
/// in-flight-or-queued requests, which defaults to 2. Eight streams
/// under one key would have six rejected with `PrincipalCap`, and
/// `concurrency:8` would silently measure the fair-share cap instead
/// of the server — a worse failure than the anonymity it replaced,
/// because it looks like backpressure.
#[test]
fn a_burst_presents_one_identity_per_stream() {
let client = reqwest::Client::new();
let ctx = RunCtx {
client: &client,
chat_url: "http://x/v1/chat/completions".into(),
model_id: "m".into(),
max_tokens: 16,
timeout: Duration::from_secs(1),
principal_key_id: Some("fleet-benchmark".into()),
};
let key_of = |stream: Option<u32>| -> Option<String> {
with_stream_identity(client.post(&ctx.chat_url), &ctx, stream)
.build()
.expect("build")
.headers()
.get(HEADER_KEY_ID)
.map(|v| v.to_str().expect("utf8").to_string())
};
let keys: Vec<Option<String>> = (0..3).map(|i| key_of(Some(i))).collect();
assert_eq!(
keys,
vec![
Some("fleet-benchmark-0".to_string()),
Some("fleet-benchmark-1".to_string()),
Some("fleet-benchmark-2".to_string()),
]
);
// Distinct, or the cap bites anyway.
let uniq: std::collections::HashSet<_> = keys.iter().collect();
assert_eq!(uniq.len(), 3);
}
/// A single-request scenario adds no override: the client's default
/// headers already carry the base identity, and a per-request copy
/// would be a second place for it to drift.
#[test]
fn a_single_request_scenario_keeps_the_client_identity() {
let client = reqwest::Client::new();
let ctx = RunCtx {
client: &client,
chat_url: "http://x/v1/chat/completions".into(),
model_id: "m".into(),
max_tokens: 16,
timeout: Duration::from_secs(1),
principal_key_id: Some("fleet-benchmark".into()),
};
let req = with_stream_identity(client.post(&ctx.chat_url), &ctx, None)
.build()
.expect("build");
assert!(req.headers().get(HEADER_KEY_ID).is_none());
}
#[test]
fn prompt_grows_with_token_target() {
let small = build_prompt(128);
@@ -1032,7 +646,6 @@ mod tests {
prompt_sizes: vec![128],
max_tokens: 64,
concurrency_levels: vec![2, 8],
concurrency_nonstreaming_levels: vec![8],
concurrency_prompt_tokens: 512,
capability_probes: vec![CapabilityProbe {
name: "plan".into(),
+3 -107
View File
@@ -14,22 +14,6 @@ use std::path::Path;
#[derive(Debug, Clone)]
pub struct RunRecord {
pub ts: String, // RFC3339
/// Identity this sample was taken under (#288), `account/key`, or
/// `None` for an anonymous run. Anonymous samples measure the
/// admission yield policy rather than serving capacity once #262 is
/// in the build, so this is the column that says whether two rows
/// are comparable at all.
pub principal: Option<String>,
/// Tokens spent reasoning (#223) — the dominant cost driver on a
/// reasoning model, and one that moves independently of every tok/s
/// number on the dashboard.
pub reasoning_tokens: Option<u64>,
/// Prompt tokens served from the prefix cache (#269) — the reason
/// prefill timing varies between otherwise identical samples.
pub cached_tokens: Option<u64>,
/// p95 inter-token arrival gap (ms) — stream smoothness, which a
/// mean tok/s cannot express.
pub tpot_p95_ms: Option<f64>,
// target
pub target_name: String,
pub target_kind: String,
@@ -220,18 +204,6 @@ impl Store {
("artifact", "TEXT"),
("quality_score", "REAL"),
("scorer", "TEXT"),
// #288: which identity the sample was taken under. Rows
// predating this are NULL, which is honest — they were
// anonymous, and since #262 that measures a different
// thing. Without the column the 2026-08-18 discontinuity
// is folklore rather than data.
("principal", "TEXT"),
// #223 / #269: both are on the wire and were being discarded.
("reasoning_tokens", "INTEGER"),
("cached_tokens", "INTEGER"),
// #285: inter-token tail. `decode_tps` is a mean and hides
// a stall that later catches up.
("tpot_p95_ms", "REAL"),
],
)?;
Ok(())
@@ -292,7 +264,6 @@ impl Store {
swap_unload_ms, swap_load_ms,
artifact, quality_score, scorer,
image_units,
principal, reasoning_tokens, cached_tokens, tpot_p95_ms,
ok, error
) VALUES (
?1, ?2, ?3, ?4,
@@ -309,8 +280,7 @@ impl Store {
?42, ?43,
?44, ?45, ?46,
?47,
?48, ?49, ?50, ?51,
?52, ?53
?48, ?49
)",
params![
r.ts,
@@ -360,10 +330,6 @@ impl Store {
r.quality_score,
r.scorer,
r.image_units,
r.principal,
r.reasoning_tokens,
r.cached_tokens,
r.tpot_p95_ms,
r.ok as i64,
r.error,
],
@@ -737,15 +703,7 @@ impl Store {
}
};
let mut stmt = self.conn.prepare(
"SELECT git_sha, build_timestamp, package_version, ttft_s, decode_tps, total_s,
ttft_p95_s, queue_wait_ms, rejected,
-- Prefill rate derived here rather than stored: the
-- inputs are already recorded and a stored rate
-- would be a third thing to keep consistent.
CASE WHEN prefill_ms > 0 AND prefill_tokens IS NOT NULL
THEN (prefill_tokens * 1000.0) / prefill_ms END AS prefill_tps,
reasoning_tokens, cached_tokens, completion_tokens,
tpot_p95_ms, principal, ts
"SELECT git_sha, build_timestamp, package_version, ttft_s, decode_tps, total_s, ts
FROM runs
WHERE ok=1 AND target_name=?1 AND model_id=?2 AND scenario_id=?3
ORDER BY id",
@@ -759,16 +717,7 @@ impl Store {
ttft_s: r.get(3)?,
decode_tps: r.get(4)?,
total_s: r.get(5)?,
ttft_p95_s: r.get(6)?,
queue_wait_ms: r.get(7)?,
rejected: r.get(8)?,
prefill_tps: r.get(9)?,
reasoning_tokens: r.get(10)?,
cached_tokens: r.get(11)?,
completion_tokens: r.get(12)?,
tpot_p95_ms: r.get(13)?,
principal: r.get(14)?,
ts: r.get(15)?,
ts: r.get(6)?,
})
})?
.collect::<rusqlite::Result<_>>()?;
@@ -889,33 +838,6 @@ pub struct SeriesPoint {
pub ttft_s_median: Option<f64>,
pub decode_tps_median: Option<f64>,
pub total_s_median: Option<f64>,
// ── Added because the two above cannot show what changed ──────
/// Tail TTFT. The median above stayed flat at ~0.3 s across the
/// build where p95 went 0.53 s → 11.77 s (#288); charting only the
/// median is why that went unnoticed for a week.
pub ttft_p95_s_median: Option<f64>,
/// Admission queue wait — separates "the server is slow" from
/// "you were queued behind someone".
pub queue_wait_ms_median: Option<f64>,
/// Requests shed by admission during the burst.
pub rejected_median: Option<f64>,
/// Prefill throughput, derived from `prefill_tokens / prefill_ms`.
/// The other half of serving speed; only decode was ever charted.
pub prefill_tps_median: Option<f64>,
/// Tokens spent reasoning (#223) — the dominant cost driver on a
/// reasoning model, and independent of every speed metric here.
pub reasoning_tokens_median: Option<f64>,
/// Prompt tokens served from the prefix cache (#269).
pub cached_tokens_median: Option<f64>,
/// Total generated tokens — catches a model that starts truncating
/// or rambling, which no rate metric shows.
pub completion_tokens_median: Option<f64>,
/// p95 inter-token gap (ms) — stream smoothness.
pub tpot_p95_ms_median: Option<f64>,
/// Identity the samples were taken under (#288), or `None` if
/// anonymous. Anonymous and identified points are **not
/// comparable** once #262 is in the build.
pub principal: Option<String>,
pub samples: usize,
}
@@ -926,15 +848,6 @@ struct SeriesRaw {
ttft_s: Option<f64>,
decode_tps: Option<f64>,
total_s: Option<f64>,
ttft_p95_s: Option<f64>,
queue_wait_ms: Option<f64>,
rejected: Option<f64>,
prefill_tps: Option<f64>,
reasoning_tokens: Option<f64>,
cached_tokens: Option<f64>,
completion_tokens: Option<f64>,
tpot_p95_ms: Option<f64>,
principal: Option<String>,
ts: String,
}
@@ -967,19 +880,6 @@ fn aggregate_series(raws: Vec<SeriesRaw>) -> Vec<SeriesPoint> {
ttft_s_median: median(rows.iter().filter_map(|r| r.ttft_s)),
decode_tps_median: median(rows.iter().filter_map(|r| r.decode_tps)),
total_s_median: median(rows.iter().filter_map(|r| r.total_s)),
ttft_p95_s_median: median(rows.iter().filter_map(|r| r.ttft_p95_s)),
queue_wait_ms_median: median(rows.iter().filter_map(|r| r.queue_wait_ms)),
rejected_median: median(rows.iter().filter_map(|r| r.rejected)),
prefill_tps_median: median(rows.iter().filter_map(|r| r.prefill_tps)),
reasoning_tokens_median: median(rows.iter().filter_map(|r| r.reasoning_tokens)),
cached_tokens_median: median(rows.iter().filter_map(|r| r.cached_tokens)),
completion_tokens_median: median(rows.iter().filter_map(|r| r.completion_tokens)),
tpot_p95_ms_median: median(rows.iter().filter_map(|r| r.tpot_p95_ms)),
// Any row's principal: a build's samples are all taken
// under one identity, and a mixture would mean the
// config changed mid-build, which the chart should show
// as a discontinuity rather than average away.
principal: rows.iter().find_map(|r| r.principal.clone()),
samples: rows.len(),
};
(sort_key, point)
@@ -1381,10 +1281,6 @@ mod tests {
fn rec(target: &str, sha: &str, model: &str, scenario: &str, ok: bool) -> RunRecord {
RunRecord {
ts: "2026-06-13T00:00:00Z".into(),
principal: None,
reasoning_tokens: None,
cached_tokens: None,
tpot_p95_ms: None,
target_name: target.into(),
target_kind: "neuron".into(),
endpoint: "http://x:13131".into(),
+1 -18
View File
@@ -78,10 +78,7 @@ pub struct Sweeper {
impl Sweeper {
pub fn new(cfg: BenchConfig) -> Result<Self> {
let client = TargetClient::with_principal(
cfg.bench.request_timeout(),
cfg.bench.principal.as_ref(),
)?;
let client = TargetClient::new(cfg.bench.request_timeout())?;
let store = Store::open(&cfg.bench.db_path)?;
Ok(Sweeper { cfg, client, store })
}
@@ -172,7 +169,6 @@ impl Sweeper {
model_id: model.id.clone(),
max_tokens: self.cfg.scenarios.max_tokens,
timeout: self.cfg.bench.request_timeout(),
principal_key_id: self.cfg.bench.principal.as_ref().map(|p| p.key_id.clone()),
};
let cold = crate::scenario::cold_probe(&ctx).await;
let swap = SwapTiming { unload_ms, load_ms };
@@ -256,7 +252,6 @@ impl Sweeper {
model_id: model.id.clone(),
max_tokens: self.cfg.scenarios.max_tokens,
timeout: self.cfg.bench.request_timeout(),
principal_key_id: self.cfg.bench.principal.as_ref().map(|p| p.key_id.clone()),
};
// One unmeasured warmup when the cell is empty (matches
@@ -351,15 +346,6 @@ impl Sweeper {
RunRecord {
ts: chrono::Utc::now().to_rfc3339(),
// #288: stamp the identity this sample was taken under, so a
// later reader can tell an anonymous row from an identified
// one instead of inferring it from the date.
principal: self
.cfg
.bench
.principal
.as_ref()
.map(|p| format!("{}/{}", p.account_id, p.key_id)),
target_name: target.name.clone(),
target_kind: kind_str(target.kind).to_string(),
endpoint: target.endpoint.clone(),
@@ -397,9 +383,6 @@ impl Sweeper {
prefill_ms: m.and_then(|m| m.prefill_ms),
decode_ms: m.and_then(|m| m.decode_ms),
prefill_tokens: m.and_then(|m| m.prefill_tokens),
reasoning_tokens: m.and_then(|x| x.reasoning_tokens),
cached_tokens: m.and_then(|x| x.cached_tokens),
tpot_p95_ms: m.and_then(|x| x.tpot_p95_ms),
vram_used_mb: health.map(|h| h.vram_used_mb),
gpu_util_pct: health.map(|h| h.gpu_util_pct),
gpu_temp_c: health.map(|h| h.gpu_temp_c),
-4
View File
@@ -15,10 +15,6 @@ fn rec(
ok: bool,
) -> RunRecord {
RunRecord {
principal: None,
reasoning_tokens: None,
cached_tokens: None,
tpot_p95_ms: None,
ts: "2026-06-13T00:00:00Z".into(),
target_name: host.into(),
target_kind: "neuron".into(),
@@ -80,7 +80,6 @@ async fn spawn_mock(sha: &str) -> (String, Arc<Mutex<String>>) {
fn config_for(endpoint: String, db_path: String) -> BenchConfig {
BenchConfig {
bench: BenchSettings {
principal: None,
sweep_interval_secs: 1,
samples_per_version: 2,
iteration_pause_secs: 0,
@@ -92,7 +91,6 @@ fn config_for(endpoint: String, db_path: String) -> BenchConfig {
prompt_sizes: vec![128], // single scenario keeps assertions simple
max_tokens: 16,
concurrency_levels: Vec::new(),
concurrency_nonstreaming_levels: vec![],
concurrency_prompt_tokens: 512,
capability_probes: Vec::new(),
},
+4 -18
View File
@@ -80,19 +80,10 @@ fn router_entry(cortex: &str, e: &CortexModelEntry) -> CortexModelEntry {
cost: e.cost.clone(),
tool_call: e.tool_call,
reasoning: e.reasoning,
// Every flat ecosystem field is derived from `limit` by the
// final sync pass in aggregate_models, so they are cleared here
// rather than copied. Spread from `e` keeps a field added to the
// entry later from being silently dropped at this hop.
..CortexModelEntry {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
reasoning_budget: Vec::new(),
..e.clone()
}
// Derived from `limit` by the final sync pass in aggregate_models.
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
}
}
@@ -173,9 +164,6 @@ mod tests {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
reasoning_budget: Vec::new(),
}
}
@@ -236,7 +224,6 @@ mod tests {
context: 32_768,
input: None,
output: 4096,
output_ceiling: 0,
});
a.cost = Some(ModelCost {
input: 0.50,
@@ -249,7 +236,6 @@ mod tests {
context: 16_384, // tighter
input: None,
output: 4096,
output_ceiling: 0,
});
b.cost = Some(ModelCost {
input: 0.20, // cheaper
-3
View File
@@ -42,9 +42,6 @@ fn model_entry(loaded: bool, feasible: bool) -> CortexModelEntry {
max_model_len: None,
max_input_tokens: None,
max_output_tokens: None,
context_window: None,
context_length: None,
reasoning_budget: Vec::new(),
}
}
-5
View File
@@ -58,11 +58,6 @@ mod tests {
assert!(cfg.max_redirects > 0);
}
// The closure's error type is `figment::Error`, which is 208 bytes and
// not ours to shrink; `Jail::expect_with` dictates the signature. The
// usual remedy (box the Err) is not available through a third-party
// callback contract, and this is a test closure run once.
#[allow(clippy::result_large_err)]
#[test]
fn loads_toml_with_env_override() {
figment::Jail::expect_with(|jail| {
-29
View File
@@ -287,44 +287,15 @@ fn default_from_addr() -> String {
impl UpstreamConfig {
/// Load from a TOML file with `UPSTREAM_`-prefixed env overrides
/// (`__` nesting separator).
///
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("UPSTREAM_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
#[cfg(test)]
mod tests {
use super::*;
+16 -4
View File
@@ -53,6 +53,18 @@ fn unique_email() -> String {
format!("u-{}@test.local", uuid::Uuid::new_v4())
}
/// A fixture value that is unique per run.
///
/// Rows keyed on a hash of these (`email_tokens.token_hash`,
/// `api_keys.key_hash`) are uniquely indexed, so a hard-coded literal
/// passes against a virgin database and then fails with `23505` on every
/// run afterwards — the tests only looked deterministic because CI always
/// gets a fresh one. Anything inserted under a unique constraint gets a
/// fresh value here.
fn unique_secret(prefix: &str) -> String {
format!("{prefix}-{}", uuid::Uuid::new_v4())
}
async fn post(url: String, body: Value, bearer: Option<&str>) -> reqwest::Response {
let c = reqwest::Client::new();
let mut req = c.post(url).json(&body);
@@ -85,13 +97,13 @@ async fn verify_endpoint_consumes_token_once() {
.await
.unwrap()
.get("id");
let raw = "verify-raw-token-xyz";
let raw = unique_secret("verify-raw-token");
pool.execute(
sqlx::query(
"INSERT INTO email_tokens (token_hash, user_id, kind, expires_at) \
VALUES ($1, $2, 'verify', now() + interval '1 hour')",
)
.bind(sha256(raw))
.bind(sha256(&raw))
.bind(user_id),
)
.await
@@ -273,13 +285,13 @@ async fn fingerprint_abuse_silently_deactivates_all_no_clue() {
.await
.unwrap()
.get("id");
let raw = "sk-helexa-deactivated-probe";
let raw = unique_secret("sk-helexa-deactivated-probe");
pool.execute(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix) VALUES ($1, $2, 'sk-helexa-')",
)
.bind(acct)
.bind(sha256(raw)),
.bind(sha256(&raw)),
)
.await
.unwrap();
+12 -119
View File
@@ -2,7 +2,6 @@
use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::candle::WireSurface;
use crate::harness::candle::{CandleHarness, InferenceError};
use crate::harness::preflight::PreflightError;
use crate::health::HealthCache;
@@ -22,7 +21,7 @@ use futures::stream::{self, StreamExt};
use serde_json::{Value, json};
use std::convert::Infallible;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::sync::RwLock;
use tokio_stream::wrappers::ReceiverStream;
@@ -273,35 +272,6 @@ fn default_enable_thinking(req: &mut ChatCompletionRequest, include_thinking: bo
/// client-supplied copy and asserts the authoritative value, so over the
/// trusted WireGuard link these are safe to key fair-share on. `None` for an
/// unauthenticated/direct request — exempt from the per-principal cap.
/// SSE keep-alive for the inference streams.
///
/// A streamed completion emits its `role` delta immediately and then says
/// **nothing at all until prefill finishes** — there is no token to send
/// yet. On a long prompt that silence is seconds long: a 7.4k-token prompt
/// on a TP-2 27B prefills in about six.
///
/// `KeepAlive::default()` is axum's 15s, which is longer than almost every
/// prefill we serve, so in practice it never fired and the gap went out on
/// the wire as dead air. Clients read dead air as a dropped stream: the
/// DeepSeek Harness gives up after roughly a second and reports
/// `terminated`, then retries — and because the abandoned request keeps its
/// admission permit and the TP pool lock until it finishes, each retry
/// queues behind the last one and waits *longer*, so the session never
/// recovers on its own. Every attempt failed while the model was healthy
/// and, by then, three sequences deep in work nobody was listening to.
///
/// One second is chosen to sit under the shortest client idle timeout we
/// have met rather than to be tuned to any particular one. The cost is a
/// 3-byte comment frame per idle second, which SSE parsers discard by
/// spec; the benefit is that a prefill of any length looks alive.
///
/// This does not make a slow prefill fast — see the `chunked prefill`
/// path for that — it stops a slow prefill being indistinguishable from a
/// dead connection.
fn prefill_keep_alive() -> KeepAlive {
KeepAlive::new().interval(Duration::from_secs(1))
}
fn principal_key(headers: &axum::http::HeaderMap) -> Option<String> {
let account = headers.get(HEADER_ACCOUNT_ID)?.to_str().ok()?;
let key = headers.get(HEADER_KEY_ID)?.to_str().ok()?;
@@ -363,7 +333,7 @@ async fn chat_completions(
if req.stream.unwrap_or(false) {
match candle
.chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
.chat_completion_stream_with(req, chat_config, principal)
.await
{
Ok(rx) => {
@@ -376,16 +346,13 @@ async fn chat_completions(
let done_stream =
stream::once(async { Ok::<_, Infallible>(Event::default().data("[DONE]")) });
Sse::new(body_stream.chain(done_stream))
.keep_alive(prefill_keep_alive())
.keep_alive(KeepAlive::default())
.into_response()
}
Err(e) => inference_error_response(e),
}
} else {
match candle
.chat_completion(req, principal, WireSurface::Chat)
.await
{
match candle.chat_completion(req, principal).await {
Ok(resp) => Json(resp).into_response(),
Err(e) => inference_error_response(e),
}
@@ -438,13 +405,7 @@ async fn responses(
if stream_requested {
match candle
.responses_stream(
chat_req,
response_id,
message_item_id,
principal,
WireSurface::Responses,
)
.responses_stream(chat_req, response_id, message_item_id, principal)
.await
{
Ok(rx) => {
@@ -458,7 +419,7 @@ async fn responses(
Ok::<_, Infallible>(Event::default().event(frame.event_name).data(body))
});
Sse::new(body_stream)
.keep_alive(prefill_keep_alive())
.keep_alive(KeepAlive::default())
.into_response()
}
Err(e) => inference_error_response(e),
@@ -468,10 +429,7 @@ async fn responses(
// and translate the result. We don't currently re-tokenise
// to compute usage; the harness returns it via the chat
// response and we pass it through.
match candle
.chat_completion(chat_req, principal, WireSurface::Responses)
.await
{
match candle.chat_completion(chat_req, principal).await {
Ok(chat_resp) => {
// Extract the assistant text (chat completions
// always emits one choice on the candle path).
@@ -499,9 +457,9 @@ async fn responses(
input_tokens: u.prompt_tokens,
output_tokens: u.completion_tokens,
total_tokens: u.prompt_tokens + u.completion_tokens,
// Carry the reasoning sub-count through from the
// chat usage: the collector counts the `<think>`
// span and returns its text separately.
// Carry the reasoning sub-count through from the chat
// usage — the non-streaming path now splits off the
// `<think>` span and counts it (see `split_off_reasoning`).
output_tokens_details: u.completion_tokens_details.as_ref().map(|d| {
OutputTokensDetails {
reasoning_tokens: d.reasoning_tokens,
@@ -515,18 +473,8 @@ async fn responses(
model_id,
message_item_id: mint_message_item_id(),
};
// The model's think block, which the chat collector now
// returns on `message.reasoning_content` (#300). Without
// this the caller is billed for `reasoning_tokens` it
// never sees, and cannot replay the model's own
// reasoning on the next turn.
let reasoning = chat_resp
.choices
.first()
.and_then(|c| c.message.extra.get("reasoning_content"))
.and_then(|v| v.as_str())
.map(str::to_string);
let resp = openai_responses::build_response(&meta, text, finish, usage, reasoning);
let _ = chat_resp; // make the borrow-checker happy if `text` consumed it
let resp = openai_responses::build_response(&meta, text, finish, usage);
Json(resp).into_response()
}
Err(e) => inference_error_response(e),
@@ -781,18 +729,6 @@ fn inference_error_response(err: InferenceError) -> axum::response::Response {
"too many concurrent requests for this key; retry shortly",
)
.with_retry_after(retry_after_secs),
// Draining (#256). 503 + Retry-After, like any transient capacity
// condition: the node is coming back, and a well-behaved client
// should retry rather than surface a failure. Deliberately not
// `rate_limit_exceeded` — nothing about this is the caller's
// fault or their rate.
InferenceError::ShuttingDown => OpenAiError::new(
503,
"api_error",
"shutting_down",
"neuron is shutting down; retry shortly",
)
.with_retry_after(15),
InferenceError::Other(e) => OpenAiError::without_code(500, "api_error", format!("{e:#}")),
};
// Every failed request says why, once, here (#63). Before this the
@@ -1109,46 +1045,3 @@ where
}
}
}
#[cfg(test)]
mod keep_alive_tests {
use super::*;
use futures::StreamExt;
/// A stream that emits its opening frame and then goes quiet — the
/// shape of every streamed completion while it prefills — must keep
/// putting bytes on the wire.
///
/// The deadline here is the point: with axum's 15s default this test
/// times out, which is exactly the production failure. A 7.4k-token
/// prompt prefills in ~6s, so the default never fired once, the
/// connection sat silent for the whole prefill, and clients dropped it
/// as dead — the DeepSeek Harness after ~1s, reporting `terminated`.
///
/// Asserted against observable output rather than the configured
/// interval because `KeepAlive` exposes no getter, and because what
/// broke was the behaviour, not the number.
#[tokio::test]
async fn a_silent_stream_still_sends_bytes_while_prefilling() {
let silent = stream::pending::<Result<Event, Infallible>>();
let response = Sse::new(silent)
.keep_alive(prefill_keep_alive())
.into_response();
let mut body = response.into_body().into_data_stream();
let frame = tokio::time::timeout(Duration::from_secs(3), body.next())
.await
.expect("a silent stream must emit a keep-alive well inside any client idle timeout")
.expect("stream ended instead of keeping alive")
.expect("keep-alive frame");
// SSE comment frames start with ':' and are discarded by parsers
// per spec, so this costs the client nothing but proves liveness.
assert_eq!(
frame.first(),
Some(&b':'),
"expected an SSE comment frame, got {:?}",
String::from_utf8_lossy(&frame)
);
}
}
+1 -57
View File
@@ -86,16 +86,6 @@ pub struct CandleHarnessConfig {
#[serde(default)]
pub context_limit: ContextLimitConfig,
/// What each OpenAI effort level costs, in reasoning tokens (#223).
///
/// OpenAI-shaped clients cannot send a token count — their whole
/// vocabulary is the `minimal|low|medium|high` ladder — so the
/// numbers live here and are advertised on `/models`. Operator
/// configurable because the right values depend on the model and the
/// hardware.
#[serde(default)]
pub reasoning_budget: crate::harness::reasoning_budget::ReasoningBudgetLadder,
/// Admission control (#53): bounds the per-model wait queue so a busy
/// model returns a fast, retryable `429`/`503` instead of stalling new
/// requests until their client times out.
@@ -190,36 +180,9 @@ pub struct AdmissionConfig {
/// for any single principal (resolved from the `x-helexa-*` headers
/// cortex stamps), so one client can't monopolize the queue while others
/// wait. Over-cap → `429 rate_limit_exceeded` + `Retry-After`. `0`
/// disables the cap.
///
/// Anonymous requests carry no principal and so cannot be capped by it.
/// They are bounded instead by `anon_max_in_flight` / `anon_max_pending`
/// below (#262).
/// disables the cap; anonymous requests are always exempt.
#[serde(default = "default_admission_max_per_principal")]
pub max_per_principal: usize,
/// Ceiling on **anonymous** requests holding an in-flight seat (#262).
///
/// Anonymous traffic is served from capacity left over once identified
/// traffic is satisfied: it never takes a seat while an identified
/// request is waiting, and it may never hold every seat at once — which
/// is what this bounds, because a request already running cannot be
/// preempted. Reserving one seat is what caps how long a newly arrived
/// identified request waits: one request's duration, not all of them.
///
/// Unset (the default) derives `max_in_flight - 1`, floored at 1 so a
/// single-seat model still serves anonymous callers. `0` refuses
/// anonymous traffic outright.
#[serde(default)]
pub anon_max_in_flight: Option<usize>,
/// Ceiling on anonymous requests in flight **plus** queued (#262).
///
/// The queue needs the same reservation as the seats: without it an
/// anonymous flood fills the queue and identified callers are refused at
/// the door, before priority can help them.
///
/// Unset derives `max_in_flight + max_queue_depth - 1`, floored at 1.
#[serde(default)]
pub anon_max_pending: Option<usize>,
/// Maximum seconds a request waits for enough **KV budget** to free
/// before it is refused (#257). Deliberately far longer than
/// `max_wait_secs`: that one waits for a slot to turn over, this one
@@ -237,8 +200,6 @@ impl Default for AdmissionConfig {
max_queue_depth: default_admission_max_queue_depth(),
max_wait_secs: default_admission_max_wait_secs(),
max_per_principal: default_admission_max_per_principal(),
anon_max_in_flight: None,
anon_max_pending: None,
kv_max_wait_secs: default_admission_kv_max_wait_secs(),
}
}
@@ -343,24 +304,8 @@ pub struct ContextLimitConfig {
/// Generation reserve (tokens) left below the context wall:
/// `input = context − output_reserve_tokens`. Defaults to neuron's
/// default `max_tokens`.
///
/// This is the budget a request gets when it names none, and the
/// figure the input/output split is derived from. It is **not** the
/// most a request may ask for — see `max_output_tokens` (#278).
#[serde(default = "default_output_reserve_tokens")]
pub output_reserve_tokens: usize,
/// The largest output a request may name, advertised as
/// `max_output_tokens` on `/v1/models` (#278).
///
/// `None` derives it as `min(max_position_embeddings,
/// DEFAULT_OUTPUT_CEILING)`. Separate from the reserve because the
/// two answer different questions and were previously the same
/// number: publishing the reserve told every client that read the
/// advertisement to cap itself at the default, which on a reasoning
/// model can be less than one think block.
#[serde(default)]
pub max_output_tokens: Option<usize>,
}
impl Default for ContextLimitConfig {
@@ -372,7 +317,6 @@ impl Default for ContextLimitConfig {
activation_headroom_mb: default_activation_headroom_mb(),
min_free_floor_mb: default_context_min_free_floor_mb(),
output_reserve_tokens: default_output_reserve_tokens(),
max_output_tokens: None,
}
}
}
File diff suppressed because it is too large Load Diff
@@ -380,23 +380,7 @@ impl GatedDeltaNet {
self.head_v_dim,
)?;
// Stash the updated recurrent state for the next call.
//
// The delta rule ran in f32 and `state_init` upcasts back to f32
// on entry, so storing at the model dtype is a round trip that
// buys only memory (#284). Decode takes it once per token.
//
// Storing f32 keeps `new_state` as-is, which on the CUDA decode
// path is the same buffer the kernel updated in place — so the
// stored state aliases it, where the `to_dtype` branch always
// allocated afresh. That is sound: the pre-update value is dead
// by this point, and prefix snapshots deep-copy
// (`snapshot_state`) precisely so they never share storage with
// a buffer a later forward will overwrite.
self.state.recurrent_state = Some(if gdn_state_f32() {
new_state
} else {
new_state.to_dtype(dtype)?
});
self.state.recurrent_state = Some(new_state.to_dtype(dtype)?);
// core_attn_out: (B, H, L, D_v) → (B, L, H, D_v) → (B*L*H, D_v).
let core_attn_out = core_attn_out.transpose(1, 2)?.contiguous()?; // (B, L, H, D_v)
@@ -480,33 +464,6 @@ fn chunked_prefill_enabled() -> bool {
})
}
/// `NEURON_GDN_STATE_F32=1` keeps the recurrent state in f32 between
/// forwards instead of storing it at the model's activation dtype (#284).
///
/// The delta rule computes in f32 throughout, so storing the result at
/// BF16 and upcasting it again next call is a pure round-trip loss.
/// Measured on the chunked path: bit-exact carrying f32 at every chunk
/// size, against 1.3e-3 relative error after 16 prefill chunks and
/// **1.0e-2 after 4096 decode steps** — decode round-trips once per
/// token, across 48 of this arch's 64 layers.
///
/// Off by default on purpose. Every #283 measurement was taken against
/// the bf16 baseline, and flipping this silently would move the ground
/// underneath the sampling A/B and the decode-rate work (#282, #285)
/// at the same time. Turn it on deliberately, measure, then decide the
/// default.
///
/// Costs ~2× the recurrent slab: +37.7 MB per GPU per sequence on the
/// deployed 27B under TP=2, so +302 MB/GPU at `max_in_flight = 8`.
fn gdn_state_f32() -> bool {
static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*ENABLED.get_or_init(|| {
std::env::var("NEURON_GDN_STATE_F32")
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false)
})
}
/// Chunk-parallel gated delta rule — a faithful port of the HF
/// reference `torch_chunk_gated_delta_rule` (chunk_size = 64) in
/// `transformers/models/qwen3_5/modeling_qwen3_5.py`, minus the steps
@@ -1140,94 +1097,6 @@ mod tests {
assert!(d_state < 2e-4, "final state diverged: {d_state}");
}
/// #284: the state carried *between* forwards must not lose
/// precision. Every other parity test here is single-call and
/// all-f32, so none of them crosses the boundary where
/// `GatedDeltaNet::forward` stores the state at the model's
/// activation dtype — which is why the defect lived.
///
/// Two properties, one test:
///
/// 1. carrying f32, an N-chunk run equals a single call **exactly**,
/// so the chunk-boundary math itself is sound (this kills the
/// "chunked prefill corrupts state" hypothesis, #283 H1);
/// 2. carrying the model dtype loses a measurable, chunk-count-
/// dependent amount — the defect the `NEURON_GDN_STATE_F32` flag
/// exists to switch off.
///
/// Asserting (2) is deliberate: if a future change makes the bf16
/// carry lossless this test fails loudly and the flag can go, rather
/// than the flag quietly outliving its reason.
#[test]
fn state_carried_across_chunks_is_exact_in_f32_and_lossy_in_bf16() {
let (b, h, dk, dv) = (1, 2, 32, 32);
let dev = Device::Cpu;
let l = 1024;
// Long-memory decay: g near zero keeps the state alive across
// every boundary, which is both the realistic regime for
// long-context retrieval and the one that accumulates most.
let scale = 1.0 / (dk as f64).sqrt();
let q = Tensor::randn(0f32, 1.0, (b, h, l, dk), &dev).unwrap();
let q = (l2norm(&q, 1e-6).unwrap() * scale).unwrap();
let k = Tensor::randn(0f32, 1.0, (b, h, l, dk), &dev).unwrap();
let k = l2norm(&k, 1e-6).unwrap();
let v = (Tensor::randn(0f32, 1.0, (b, h, l, dv), &dev).unwrap() * 0.5).unwrap();
let g = (Tensor::rand(0f32, 1f32, (b, h, l), &dev).unwrap() * -0.02).unwrap();
let beta = Tensor::rand(0.05f32, 0.95f32, (b, h, l), &dev).unwrap();
let zeros = || Tensor::zeros((b, h, dk, dv), DType::F32, &dev).unwrap();
// Ground truth: the whole sequence in one call, f32 throughout.
let (out_ref, _) = run_chunk_gated_delta_rule(&q, &k, &v, &g, &beta, zeros()).unwrap();
// Replay it in `chunk`-sized pieces, carrying the state the way
// `forward` does.
let replay = |chunk: usize, carry: Option<DType>| {
let mut state = zeros();
let mut outs = Vec::new();
let mut pos = 0;
while pos < l {
let n = chunk.min(l - pos);
let take = |t: &Tensor| t.narrow(2, pos, n).unwrap().contiguous().unwrap();
let (o, s) = run_chunk_gated_delta_rule(
&take(&q),
&take(&k),
&take(&v),
&take(&g),
&take(&beta),
state,
)
.unwrap();
state = match carry {
Some(dt) => s.to_dtype(dt).unwrap().to_dtype(DType::F32).unwrap(),
None => s,
};
outs.push(o);
pos += n;
}
Tensor::cat(&outs, 2).unwrap()
};
for chunk in [128usize, 256, 512] {
let exact = max_abs_diff(&out_ref, &replay(chunk, None));
assert_eq!(
exact, 0.0,
"chunk={chunk}: an f32 state carry must reproduce the \
single-call result bit for bit, got {exact:e}"
);
}
// And the dtype round trip is what costs us. Compared on the
// final chunk, where the accumulated state has done the most work.
let tail = |t: &Tensor| t.narrow(2, l - 128, 128).unwrap().contiguous().unwrap();
let lossy = max_abs_diff(&tail(&out_ref), &tail(&replay(128, Some(DType::BF16))));
assert!(
lossy > 1e-5,
"storing the state at BF16 between chunks is expected to lose \
precision (#284); if this now passes losslessly the flag can \
be retired, got {lossy:e}"
);
}
/// Adversarially correlated inputs: near-identical keys with
/// beta ≈ 1 and negligible decay make the UT-transform matrix T
/// maximally coherent — raw powers of T grow combinatorially
@@ -562,9 +562,9 @@ mod tests {
.forward_batch_decode(&input, &positions, mask.as_ref())
.expect("batched step");
assert_eq!(h.dims()[0], 3);
for (row, exp) in expected.iter().enumerate() {
for row in 0..3 {
let got: Vec<f32> = h.i((row, 0, ..)).unwrap().to_vec1().unwrap();
let diff = max_abs_diff(&exp[t], &got);
let diff = max_abs_diff(&expected[row][t], &got);
assert!(diff < 1e-4, "row {row} step {t} diverged: {diff}");
}
}
@@ -610,9 +610,9 @@ mod tests {
let h = model
.forward_batch_decode(&input, &[3, 3], None)
.expect("step");
for (row, exp) in expected.iter().enumerate() {
for row in 0..2 {
let got: Vec<f32> = h.i((row, 0, ..)).unwrap().to_vec1().unwrap();
let diff = max_abs_diff(exp, &got);
let diff = max_abs_diff(&expected[row], &got);
assert!(diff < 1e-4, "row {row} diverged: {diff}");
}
}
@@ -674,9 +674,9 @@ mod tests {
let h = model
.forward_batch_decode(&input, &positions, mask.as_ref())
.expect("batched step");
for (row, exp) in expected.iter().enumerate() {
for row in 0..2 {
let got: Vec<f32> = h.i((row, 0, ..)).unwrap().to_vec1().unwrap();
let diff = max_abs_diff(&exp[t], &got);
let diff = max_abs_diff(&expected[row][t], &got);
assert!(diff < 1e-4, "batched row {row} step {t}: {diff}");
}
}
File diff suppressed because it is too large Load Diff
+1 -223
View File
@@ -233,24 +233,7 @@ pub fn render_chat_template(
MessageContent::Null => Value::Null,
};
let mut obj = serde_json::Map::new();
// `developer` is OpenAI's successor to `system` and is what
// a client sends once a model is declared reasoning-capable
// (pi-ai: `model.reasoning && supportsDeveloperRole`). Most
// chat templates — every one on this fleet — only know the
// older spelling and `raise_exception` on anything else, so
// an unmodified client got a 422 for a system prompt it sent
// correctly.
//
// Rewritten only when the template does not handle the role
// itself: a template that knows `developer` may treat it as
// distinct from `system`, and substituting there would be us
// deciding something the model's authors already decided.
let role = if m.role == "developer" && !template.contains("developer") {
"system"
} else {
m.role.as_str()
};
obj.insert("role".into(), Value::String(role.to_string()));
obj.insert("role".into(), Value::String(m.role.clone()));
obj.insert("content".into(), content_value);
// Forward extras (e.g. tool_calls on assistant turns,
// tool_call_id on tool result turns). HF templates that
@@ -438,139 +421,6 @@ mod tests {
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
}
/// A file read back through a tool result must reach the model
/// byte-for-byte.
///
/// The reported symptom: the model writes a file correctly, reads it
/// back, and sees errors that are not in the file — consistently,
/// after every write. Anything that mangles `<`, `&`, `"` or
/// newlines on this path would produce exactly that, and would make
/// the model burn its budget reconciling a file against a corrupted
/// view of itself.
#[test]
fn a_file_read_back_through_a_tool_result_is_byte_exact() {
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
// The characters most likely to be mangled, in the shapes a real
// web project contains them.
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n</head>\n<body>\n <script>\n if (a && b || c < d) { g.fillStyle = '#b8905a'; }\n const s = `rgba(${r},${g},0.5)`;\n\n // a comment & an ampersand\n </script>\n</body>\n</html>\n";
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{"id":"call_0","type":"function",
"function":{"name":"read","arguments":"{\"path\":\"/tmp/index.html\"}"}}]),
);
let messages = vec![
user_msg("read the file back"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text(file.to_string()),
extra: Value::Object(tool_extras),
},
user_msg("is it correct?"),
];
let out =
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
for probe in [
"<!DOCTYPE html>",
"<html lang=\"en\">",
"if (a && b || c < d)",
"g.fillStyle = '#b8905a';",
"`rgba(${r},${g},0.5)`",
"// a comment & an ampersand",
] {
assert!(
out.contains(probe),
"the model would not see {probe:?} verbatim — rendered prompt mangles it"
);
}
// And nothing HTML-escaped anywhere.
for bad in ["&lt;", "&gt;", "&amp;", "&quot;", "&#x27;", "&#39;"] {
assert!(
!out.contains(bad),
"found {bad:?} in the rendered prompt: content is being HTML-escaped"
);
}
}
/// Blank lines inside a replayed tool-call argument must survive
/// rendering intact.
///
/// A model that writes a source file and then re-reads it from its
/// own replayed tool call is comparing what it remembers writing
/// against what the prompt now shows. Any whitespace the pipeline
/// eats makes those disagree, and the model has no way to tell a
/// rendering artifact from a mistake of its own — which is a
/// plausible route into the announce-then-stop stall (#296).
///
/// The JSON string→object normalisation above is the step that
/// could plausibly lose them, since it round-trips the arguments
/// through a parser.
#[test]
fn a_replayed_tool_call_keeps_the_blank_lines_in_its_arguments() {
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
// Blank lines between sections, exactly as a written source file
// carries them.
let file_body = "'use strict';\n\n// ---- constants ----\nconst A = 1;\n\n\nfunction f() {\n return A;\n}\n";
let args = json!({ "path": "/tmp/game.js", "content": file_body });
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{
"id": "call_0", "type": "function",
"function": {
"name": "write",
// The OpenAI wire form: arguments as a JSON *string*.
"arguments": serde_json::to_string(&args).unwrap()
}
}]),
);
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let messages = vec![
user_msg("write the file"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text("Successfully wrote 84 bytes".into()),
extra: Value::Object(tool_extras),
},
user_msg("now check it parses"),
];
let out =
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
assert!(
out.contains(file_body),
"the file body must appear verbatim in the rendered prompt"
);
let blanks_in = file_body.lines().filter(|l| l.is_empty()).count();
assert_eq!(blanks_in, 3, "fixture sanity");
// And specifically the double newline the model would notice.
assert!(
out.contains("'use strict';\n\n// ---- constants ----"),
"blank line after the prologue was eaten"
);
assert!(
out.contains("const A = 1;\n\n\nfunction f()"),
"consecutive blank lines were collapsed"
);
}
/// The second turn of an OpenAI-native agentic loop: the client
/// replays its own assistant turn, which carries `tool_calls` and
/// `"content": null`. That must render as the tool-call block —
@@ -648,78 +498,6 @@ mod tests {
/// Minimal Qwen3-style template — enough surface to confirm
/// our renderer threads role + content correctly without
/// loading a real model's tokenizer_config.json.
/// A reasoning-capable model makes OpenAI clients send the system
/// prompt as `developer` — and every chat template on this fleet
/// raises on an unknown role, so the caller got a 422 for a prompt
/// it had sent correctly. Observed live: pi with `reasoning: true`,
/// `template_render_failed: Unexpected message role`.
#[test]
fn the_developer_role_reaches_a_template_that_only_knows_system() {
const SYSTEM_ONLY: &str = "{%- for message in messages -%}\
{%- if message.role == 'system' -%}SYS:{{ message.content }}\
{%- elif message.role == 'user' -%}USR:{{ message.content }}\
{%- else -%}{{ raise_exception('Unexpected message role.') }}\
{%- endif -%}\
{%- endfor -%}";
let messages = vec![
ChatMessage {
role: "developer".into(),
content: MessageContent::Text("be terse".into()),
extra: Value::Object(Default::default()),
},
user_msg("hi"),
];
let out = render_chat_template(SYSTEM_ONLY, &messages, &Value::Null, &Value::Null)
.expect("a developer role must not blow up a system-only template");
assert!(out.contains("SYS:be terse"), "got {out:?}");
}
/// A template that models `developer` itself keeps it: the role may
/// mean something distinct there, and substituting would override a
/// decision the model's authors already made.
#[test]
fn a_template_that_knows_developer_keeps_it() {
const KNOWS_DEVELOPER: &str = "{%- for message in messages -%}\
{%- if message.role == 'developer' -%}DEV:{{ message.content }}\
{%- else -%}{{ message.role }}:{{ message.content }}\
{%- endif -%}\
{%- endfor -%}";
let messages = vec![ChatMessage {
role: "developer".into(),
content: MessageContent::Text("be terse".into()),
extra: Value::Object(Default::default()),
}];
let out = render_chat_template(KNOWS_DEVELOPER, &messages, &Value::Null, &Value::Null)
.expect("render");
assert!(out.contains("DEV:be terse"), "got {out:?}");
}
/// The fetcher and the reader must agree on the filenames, or the
/// reader looks for something nobody downloaded and falls back to a
/// built-in prompt format without anything failing (#280 — observed
/// live: every model neuron downloaded itself was served with the
/// fallback because no sidecar was ever fetched).
#[test]
fn every_template_source_this_reads_is_one_the_loader_fetches() {
for name in crate::harness::candle::CandleHarness::SIDECAR_FILES {
assert!(
!name.is_empty(),
"sidecar names are the contract between fetch and read"
);
}
// The three the reader consults, in its own priority order.
for source in [
"chat_template.jinja",
"chat_template.json",
"tokenizer_config.json",
] {
assert!(
crate::harness::candle::CandleHarness::SIDECAR_FILES.contains(&source),
"{source} is read by load_chat_template_alongside but never fetched"
);
}
}
const QWEN3_LIKE: &str = "{%- for message in messages -%}\
<|im_start|>{{ message.role }}\n{{ message.content }}<|im_end|>\n\
{%- endfor -%}\
+1 -190
View File
@@ -36,28 +36,6 @@ use crate::config::ContextLimitConfig;
/// few turns.
const RATE_EMA_ALPHA: f64 = 0.3;
/// Smallest prompt whose prefill time says anything about throughput.
///
/// A prefill has a fixed per-request cost — tokenize, launch, sync —
/// that dominates a short prompt completely. The deploy's own smoke
/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
/// host that measures ~1,400. Folding that in does not make the
/// estimate slightly worse; it replaces it.
///
/// That mattered because the fallback it displaces is the whole
/// cold-start guard: with no sample at all, `derive_limit` uses
/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
/// the host advertised **17,408** — a sixth of what it can serve —
/// until real traffic happened to warm it (#295).
///
/// 512 excludes the smoke probe and the short auxiliary calls agent
/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
/// so fixed overhead is roughly a tenth of it — informative rather than
/// dominant.
const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
/// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
/// `bits`. No-op for degenerate inputs so a probe request or a clock blip
/// can't poison the average.
@@ -121,15 +99,7 @@ impl ThroughputEma {
/// Fold one prefill measurement (`prompt_tokens` processed in
/// `elapsed`) into the prefill EMA.
///
/// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
/// measure fixed per-request cost rather than throughput, and the
/// value they displace — the bootstrap estimate — is better than
/// they are.
pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
return;
}
fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
}
@@ -264,23 +234,10 @@ fn round_down(tokens: usize, granularity: usize) -> usize {
const CONTEXT_GRANULARITY: usize = 1024;
/// Default ceiling on what a request may name as its output budget.
///
/// Not derived from anything — a deliberate figure, and configurable
/// per host (`[context_limit] max_output_tokens`). The evidence behind
/// it: a reasoning model asked to build a small application front-loads
/// one long think block before it acts, measured at 27,504 tokens for a
/// frontier model on a task Qwen3.8-27B was cut off attempting at
/// 24,433 (#271, #278). A ceiling below that turns a capable model into
/// one that never reaches its first tool call, so the default sits
/// above the observed cost of that class of turn.
const DEFAULT_OUTPUT_CEILING: usize = 32_768;
/// Derive `limit{context,input,output}` for a loaded model.
///
/// ```text
/// output = output_reserve_tokens (default + KV planning)
/// output_ceiling = min(max_output_tokens, max_position_embeddings)
/// output = output_reserve_tokens
/// vram_ceiling = (free_tightest − activation_headroom − min_free_floor) / kv_bytes_per_token_per_card
/// throughput_ceiling = target_prefill_latency_secs × prefill_tok_per_sec
/// context = min(max_position_embeddings, vram_ceiling, throughput_ceiling) [clamped by `hard_ceiling` if set]
@@ -361,163 +318,17 @@ pub fn derive_limit(
);
let input = context.saturating_sub(output);
// The ceiling a request may name, distinct from the reserve above
// (#278). Bounded by the model's own position limit rather than by
// the live context, because the advertisement is read once and
// cached by clients: a ceiling that shrank with fleet VRAM would be
// stale in whichever direction hurt — either rejecting what we said
// we would serve, or leaving budget unclaimed. A request that will
// not actually fit is refused at request time, with the numbers,
// instead of being under-promised here.
let output_ceiling = round_down(
cfg.max_output_tokens
.unwrap_or(DEFAULT_OUTPUT_CEILING)
.min(profile.max_position_embeddings),
CONTEXT_GRANULARITY,
)
.max(output);
ModelLimit {
context,
input: Some(input),
output,
output_ceiling,
}
}
#[cfg(test)]
mod tests {
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
/// ~1,400. Before this guard that one sample displaced the
/// bootstrap estimate and the host advertised 17,408 instead of
/// ~96,000 — until real traffic happened along (#295).
#[test]
fn a_smoke_probe_does_not_poison_the_prefill_rate() {
let ema = ThroughputEma::new();
ema.record(19, Duration::from_millis(130));
assert_eq!(
ema.get(),
None,
"a 19-token prefill must leave the EMA unset so the bootstrap value stands"
);
}
/// The short auxiliary calls agent harnesses make are the same
/// shape as the probe and equally uninformative — observed at 143
/// and 469 tokens against prompts three orders of magnitude larger.
#[test]
fn short_auxiliary_prompts_are_ignored_too() {
let ema = ThroughputEma::new();
for tokens in [143usize, 469] {
ema.record(tokens, Duration::from_millis(200));
}
assert_eq!(ema.get(), None);
}
/// A real prompt still records, and still tracks a shift.
#[test]
fn representative_prefills_still_measure() {
let ema = ThroughputEma::new();
ema.record(1400, Duration::from_secs(1));
let first = ema.get().expect("a 1400-token sample must record");
assert!((first - 1400.0).abs() < 1.0, "got {first}");
// A slower host is tracked, not ignored.
for _ in 0..20 {
ema.record(700, Duration::from_secs(1));
}
let after = ema.get().unwrap();
assert!(after < first && after > 600.0, "got {after}");
}
/// Exactly at the threshold counts — the boundary is inclusive so
/// the constant means what it says.
#[test]
fn the_threshold_itself_is_a_valid_sample() {
let ema = ThroughputEma::new();
ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
assert!(ema.get().is_some());
}
use super::*;
/// The ceiling a client is told to plan against must not be the
/// reserve. Advertising the reserve is what told a reasoning model's
/// harness to cap itself below the cost of one think block (#278).
#[test]
fn the_advertised_ceiling_is_not_the_reserve() {
let cfg = ContextLimitConfig::default();
let profile = ContextProfile {
max_position_embeddings: 131_072,
kv_bytes_per_token_per_card: 65_536,
world_size: 2,
};
let limit = derive_limit(&profile, 0, 0.0, None, &cfg);
assert_eq!(limit.output, cfg.output_reserve_tokens, "reserve unchanged");
assert_eq!(limit.output_ceiling, DEFAULT_OUTPUT_CEILING);
assert!(limit.output_ceiling > limit.output);
}
/// A model whose own position limit is below the default ceiling
/// cannot generate past it, so the advertisement must not claim it.
#[test]
fn the_ceiling_never_exceeds_the_models_own_position_limit() {
let cfg = ContextLimitConfig::default();
let profile = ContextProfile {
max_position_embeddings: 16_384,
kv_bytes_per_token_per_card: 65_536,
world_size: 1,
};
let limit = derive_limit(&profile, 0, 0.0, None, &cfg);
assert_eq!(limit.output_ceiling, 16_384);
}
/// The operator's number wins, which is what makes the default a
/// starting point rather than a guess baked into the binary.
#[test]
fn a_configured_ceiling_overrides_the_default() {
let cfg = ContextLimitConfig {
max_output_tokens: Some(65_536),
..ContextLimitConfig::default()
};
let profile = ContextProfile {
max_position_embeddings: 131_072,
kv_bytes_per_token_per_card: 65_536,
world_size: 2,
};
assert_eq!(
derive_limit(&profile, 0, 0.0, None, &cfg).output_ceiling,
65_536
);
}
/// The ceiling is deliberately *not* a function of live VRAM: it is
/// read once and cached by clients, so it has to mean the same thing
/// tomorrow. The live context still moves — that is what a request
/// is judged against — but the contract does not.
#[test]
fn the_ceiling_holds_still_while_the_context_moves() {
let cfg = ContextLimitConfig::default();
let profile = ContextProfile {
max_position_embeddings: 131_072,
kv_bytes_per_token_per_card: 65_536,
world_size: 2,
};
let roomy = derive_limit(&profile, 60_000, 5_000.0, None, &cfg);
// Tight enough that the VRAM term binds — the same shape as the
// fleet advertising 109568 one hour and 131072 the next.
let tight = derive_limit(&profile, 6_000, 5_000.0, None, &cfg);
assert!(
tight.context < roomy.context,
"the live window should shrink under VRAM pressure"
);
assert_eq!(
tight.output_ceiling, roomy.output_ceiling,
"the advertised ceiling must not move with fleet load"
);
}
/// beast Qwen3.6-27B: 16 full-attn layers, 4 kv heads, head_dim 256,
/// f16 (2 B), TP=2 → 64 KiB/token total, 32 KiB/token/card.
fn beast_profile() -> ContextProfile {
+40 -209
View File
@@ -297,23 +297,14 @@ async fn op_drop_snap(cfg: &EngineConfig, session: &mut ActiveSession, id: u64)
/// Prefill one joining request at B=1 through the backend's existing
/// chunked-prefill + prefix-cache paths. Returns the prefill logits.
/// Prefill, returning the logits **and** how many leading tokens the
/// prefix cache let us skip (#269).
///
/// The reuse count is computed here and was previously only logged, so
/// no caller could report it — every client saw a 0% cache hit rate on
/// a cache that was working. Returning it is what lets `usage` carry
/// `cached_tokens`.
async fn op_prefill(
cfg: &EngineConfig,
session: &mut ActiveSession,
prompt_tokens: &[u32],
) -> Result<(Vec<f32>, usize)> {
) -> Result<Vec<f32>> {
let prompt_len = prompt_tokens.len();
let prefill_start = std::time::Instant::now();
// Each arm yields its logits plus how many leading tokens the
// prefix cache served, so neither can be forgotten.
let (logits, cached_tokens) = match (&cfg.backend, session) {
let logits = match (&cfg.backend, session) {
(
BackendConfig::Single {
worker,
@@ -349,7 +340,7 @@ async fn op_prefill(
None => chunked_prefill_via_worker(worker, *handle, prompt_tokens, reused).await?,
};
prefill_rate.record(prompt_len, prefill_start.elapsed());
(logits, reused)
logits
}
#[cfg(feature = "cuda")]
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => {
@@ -393,12 +384,12 @@ async fn op_prefill(
}
};
tp.prefill_rate.record(prompt_len, prefill_start.elapsed());
(logits, reused)
logits
}
#[cfg(feature = "cuda")]
_ => anyhow::bail!("engine backend/session mismatch"),
};
Ok((logits, cached_tokens))
Ok(logits)
}
/// One queued request. Admission has already been passed — the permit
@@ -406,14 +397,9 @@ async fn op_prefill(
pub struct EngineRequest {
pub prompt_tokens: Vec<u32>,
pub max_new: usize,
/// Sampling resolved once from request → model `generation_config`
/// → fallback (#272), rather than loose scalars each caller could
/// default differently.
pub sampling: crate::harness::sampling::SamplingParams,
/// Reasoning-token budget for this request (#223). `None` is
/// unbounded — the default, and the behaviour before it existed.
/// Per-request because slots in one batch can differ.
pub reasoning_budget: Option<usize>,
pub temperature: f64,
pub top_p: Option<f64>,
pub seed: u64,
pub eos_id: Option<u32>,
pub tool_schemas: ToolSchemas,
pub tx: mpsc::Sender<InferenceEvent>,
@@ -453,8 +439,6 @@ enum RouterMsg {
reason: FinishReason,
prompt_tokens: u32,
completion_tokens: u32,
/// Leading prompt tokens the prefix cache let us skip (#269).
cached_tokens: u32,
timing: FinishTiming,
},
}
@@ -471,12 +455,6 @@ struct Slot {
max_new: usize,
eos_id: Option<u32>,
lp: LogitsProcessor,
/// The slot's resolved sampling (#272). Held per-slot because the
/// repetition penalty and its window are per-request knobs, and
/// slots in one batch can carry different ones.
sampling: crate::harness::sampling::SamplingParams,
/// Bounds this slot's think block (#223); `None` = unbounded.
governor: Option<crate::harness::reasoning_budget::ReasoningGovernor>,
router: mpsc::Sender<RouterMsg>,
/// Set by the router when the consumer hangs up; the engine stops
/// feeding the slot and compacts it out.
@@ -484,9 +462,6 @@ struct Slot {
finished: Option<FinishReason>,
prefill_ms: u32,
prefill_tokens: u32,
/// Prompt tokens served from the prefix cache (#269), reported to
/// the caller as `cached_tokens` so a working cache is visible.
cached_tokens: u32,
decode_start: std::time::Instant,
_admit: AdmissionPermit,
}
@@ -584,13 +559,7 @@ async fn run_engine(cfg: EngineConfig, mut rx: mpsc::Receiver<EngineRequest>) {
break;
}
};
let nt = match sample_with_penalty(
&logits,
&slot.generated,
&mut slot.lp,
&slot.sampling,
slot.governor.as_mut(),
) {
let nt = match sample_with_penalty(&logits, &slot.generated, &mut slot.lp) {
Ok(t) => t,
Err(e) => {
let health = logits_health_slice(&logits_vec);
@@ -658,7 +627,6 @@ async fn finish_slot(slot: &mut Slot, reason: FinishReason, rate: &PrefillRateEm
reason,
prompt_tokens: slot.prompt_len as u32,
completion_tokens: slot.generated.len() as u32,
cached_tokens: slot.cached_tokens,
timing: FinishTiming {
prefill_ms: slot.prefill_ms,
decode_ms: slot.decode_start.elapsed().as_millis() as u32,
@@ -791,11 +759,14 @@ async fn prefill_join(
session: &mut ActiveSession,
req: EngineRequest,
) -> Result<Option<(Slot, u64)>> {
use candle_transformers::generation::Sampling;
let EngineRequest {
prompt_tokens,
max_new,
sampling,
reasoning_budget,
temperature,
top_p,
seed,
eos_id,
tool_schemas,
tx,
@@ -803,37 +774,37 @@ async fn prefill_join(
span,
} = req;
let mut lp = LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling());
// Built before prefill's first sample so a prompt that opens the
// think block is governed from its very first token (#223).
let mut governor = crate::harness::reasoning_budget::ReasoningGovernor::new(
reasoning_budget,
cfg.reasoning_tokens
.as_ref()
.map(|p| (p.open_id, p.close_id)),
prompt_opens_reasoning(&prompt_tokens, cfg.reasoning_tokens.as_ref()),
);
let mut lp = {
let sampling = if temperature <= 0.0 {
Sampling::ArgMax
} else {
match top_p {
Some(p) => Sampling::TopP { p, temperature },
None => Sampling::All { temperature },
}
};
LogitsProcessor::from_sampling(seed, sampling)
};
let prompt_len = prompt_tokens.len();
let prefill_start = std::time::Instant::now();
let (logits_vec, cached_tokens) = op_prefill(cfg, session, &prompt_tokens).await?;
let logits_vec = op_prefill(cfg, session, &prompt_tokens).await?;
let prefill_elapsed = prefill_start.elapsed();
// First token from the prefill logits.
let generated: Vec<u32> = Vec::new();
let logits = Tensor::new(logits_vec.as_slice(), &Device::Cpu)?;
let first =
match sample_with_penalty(&logits, &generated, &mut lp, &sampling, governor.as_mut()) {
Ok(t) => t,
Err(e) => {
let health = logits_health_slice(&logits_vec);
tracing::warn!(
?health,
"batch engine: prefill sample failed; logits unhealthy"
);
return Err(e);
}
};
let first = match sample_with_penalty(&logits, &generated, &mut lp) {
Ok(t) => t,
Err(e) => {
let health = logits_health_slice(&logits_vec);
tracing::warn!(
?health,
"batch engine: prefill sample failed; logits unhealthy"
);
return Err(e);
}
};
// Router task for this slot.
let hangup = Arc::new(AtomicBool::new(false));
@@ -856,14 +827,11 @@ async fn prefill_join(
let mut slot = Slot {
prefix_len: prompt_len,
prompt_len,
cached_tokens: cached_tokens as u32,
generated,
next_token: first,
max_new,
eos_id,
lp,
sampling,
governor,
router: router_tx,
hangup,
finished: None,
@@ -922,9 +890,6 @@ async fn run_router(
let mut tool_call_idx: usize = 0;
let mut emitted_tool_call = false;
let mut consumer_alive = true;
// Whether a Finish was seen, so the fall-through below can tell a
// clean end from a dropped slot.
let mut finished = false;
while let Some(msg) = rx.recv().await {
match msg {
@@ -1013,43 +978,17 @@ async fn run_router(
mut reason,
prompt_tokens,
completion_tokens,
cached_tokens,
timing,
} => {
if emitted_tool_call && reason == FinishReason::Stop {
reason = FinishReason::ToolCalls;
}
// Close the journal trail for this turn (#268). The
// non-streaming path has always logged this; streamed
// turns logged nothing at all, so whether one finished,
// hit its output cap, or was abandoned mid-flight was
// unanswerable afterwards — and an absence of lines was
// repeatedly misread as an absence of work.
//
// `run_router` is instrumented with the request's span,
// so req_id and model come along without being passed.
tracing::info!(
prompt_tokens,
completion_tokens,
reasoning_tokens = reasoning_token_count,
cached_tokens,
finish_reason = ?reason,
prefill_ms = timing.prefill_ms,
decode_ms = timing.decode_ms,
// The turn ran to completion with nobody reading it:
// it held its admission permit and the pool lock the
// whole time. Invisible until now.
client_gone = !consumer_alive,
"chat_completion (stream): done"
);
finished = true;
let _ = tx
.send(InferenceEvent::Finish {
reason,
prompt_tokens,
completion_tokens,
reasoning_tokens: reasoning_token_count,
cached_tokens,
timing: Some(timing),
})
.await;
@@ -1057,17 +996,6 @@ async fn run_router(
}
}
}
// Fell out of the loop without a Finish: the engine dropped this
// slot's channel (fatal worker error, shutdown). The client's stream
// ends either way, but silently — so say so rather than leaving the
// turn's last trace to be a prefill line (#268).
if !finished {
tracing::warn!(
reasoning_tokens = reasoning_token_count,
client_gone = !consumer_alive,
"chat_completion (stream): ended without a finish event — engine dropped the slot"
);
}
}
/// `tracing::Instrument` without importing the trait at every use
@@ -1085,92 +1013,6 @@ mod tests {
use crate::config::AdmissionConfig;
use crate::harness::admission::AdmissionController;
/// A streamed turn must close its own journal trail (#268).
///
/// Streaming used to end silently: no token counts, no
/// `finish_reason`, no duration. That made a finished turn, a turn
/// that hit its output cap, and a turn abandoned by a disconnected
/// client indistinguishable after the fact — and during the #267
/// investigation an absence of log lines was twice mistaken for an
/// absence of work on a daemon that was healthy and decoding.
///
/// Asserts the emitted line rather than the struct, because a
/// `Finish` event that reaches the client but leaves no server-side
/// record is exactly the state this fixes.
#[tokio::test]
async fn a_streamed_turn_logs_how_it_ended() {
use std::sync::{Arc, Mutex};
use tracing_subscriber::prelude::*;
#[derive(Clone, Default)]
struct Capture(Arc<Mutex<Vec<u8>>>);
impl std::io::Write for Capture {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.0
.lock()
.expect("capture poisoned")
.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for Capture {
type Writer = Self;
fn make_writer(&'a self) -> Self::Writer {
self.clone()
}
}
let capture = Capture::default();
let subscriber = tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer().with_writer(capture.clone()));
let _guard = tracing::subscriber::set_default(subscriber);
let (router_tx, router_rx) = mpsc::channel::<RouterMsg>(8);
let (tx, mut rx) = mpsc::channel::<InferenceEvent>(8);
let handle = tokio::spawn(run_router(
tiny_tokenizer(8),
None,
None,
ToolSchemas::default(),
false,
tx,
Arc::new(AtomicBool::new(false)),
router_rx,
));
router_tx
.send(RouterMsg::Finish {
reason: FinishReason::Length,
prompt_tokens: 7413,
completion_tokens: 24576,
cached_tokens: 2068,
timing: FinishTiming {
prefill_ms: 6000,
decode_ms: 900_000,
prefill_tokens: 7413,
},
})
.await
.expect("router accepts finish");
drop(router_tx);
handle.await.expect("router task joins");
while rx.recv().await.is_some() {}
let logged = String::from_utf8(capture.0.lock().expect("capture poisoned").clone())
.expect("utf8 log output");
assert!(
logged.contains("chat_completion (stream): done"),
"streamed turn left no completion line:\n{logged}"
);
// The counts are the point — "it ended" without them cannot
// distinguish a complete answer from an exhausted budget.
assert!(logged.contains("24576"), "no completion_tokens:\n{logged}");
assert!(logged.contains("Length"), "no finish_reason:\n{logged}");
}
/// A WordLevel tokenizer whose vocab covers the whole fixture
/// vocab (`w0`..`w511`), so every decoded token maps to a unique
/// word and the emitted text uniquely encodes the token sequence.
@@ -1204,20 +1046,9 @@ mod tests {
.submit(EngineRequest {
prompt_tokens: prompt,
max_new,
// Greedy, deterministic: the fixture asserts exact
// token sequences.
sampling: crate::harness::sampling::SamplingParams::resolve(
&crate::harness::sampling::RequestedSampling {
temperature: Some(0.0),
seed: Some(0),
..Default::default()
},
&crate::harness::sampling::ModelGenerationDefaults::default(),
0,
),
// Unbounded: this fixture asserts exact token sequences,
// so nothing may substitute a close marker (#223).
reasoning_budget: None,
temperature: 0.0, // greedy — deterministic
top_p: None,
seed: 0,
eos_id: None,
tool_schemas: ToolSchemas::new(),
tx,
-3
View File
@@ -12,9 +12,6 @@ pub mod image;
pub mod prefix_cache;
pub mod preflight;
pub mod preprocess;
pub mod reasoning_budget;
pub mod reasoning_effort;
pub mod sampling;
pub mod speculative;
pub mod tp;
-2
View File
@@ -401,8 +401,6 @@ mod tests {
quant: quant.map(String::from),
tensor_parallel: tp,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
@@ -1,305 +0,0 @@
//! Bounding how long a model may think before it must answer (#223).
//!
//! A reasoning model with no stopping criterion will fill whatever
//! output budget it is given. Measured on Qwen3.8-27B: given 16,384
//! tokens it used 16,384; given 32,768 it used 32,768; both times it was
//! still opening new topics when the cap bit, and both times the caller
//! received no answer at all.
//!
//! The fix is not to stop the model thinking — the run that did produce
//! files designed for 27,157 tokens first, and that design was good. It
//! is to bound the thinking so the rest of the output budget survives
//! for the answer.
//!
//! Two halves live here:
//!
//! - [`ReasoningBudget`] — how many reasoning tokens this request may
//! spend, resolved from whichever control the caller's wire format
//! gave them.
//! - [`ReasoningGovernor`] — the per-request state that enforces it, by
//! substituting the model's own `</think>` token when the budget runs
//! out so generation transitions to answering instead of being cut off.
use serde::{Deserialize, Serialize};
/// Per-model mapping from an OpenAI effort level to a token budget.
///
/// OpenAI-shaped clients cannot send a token count — pi-ai's vocabulary
/// is the ladder `off | minimal | low | medium | high | xhigh | max` and
/// nothing else — so the numbers have to live on the server. Operator
/// configurable, because the right values depend on the model and the
/// hardware; a derived default would be a guess baked into the binary.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReasoningBudgetLadder {
pub minimal: usize,
pub low: usize,
pub medium: usize,
pub high: usize,
/// `xhigh` and `max` are only offered by a client when the model
/// declares them; absent here means "not offered", and a request
/// naming one falls back to `high` rather than being rejected.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub xhigh: Option<usize>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max: Option<usize>,
/// Tokens of the caller's **own** output budget held back for the
/// answer, when the caller names no explicit `reasoning.max_tokens`.
///
/// This is what an OpenAI-shaped client gets instead of the rungs
/// above, and it is deliberately not derived from effort. Effort is
/// a prompt instruction the model is trained to follow (#290);
/// making it *also* set a guillotine is what told Qwen3.8-27B to
/// reason at `xhigh` and then cut it off at `medium`'s 12,288.
///
/// A reserve has no such double meaning. The model thinks as long
/// as it likes within the budget the caller declared, minus enough
/// to answer — so the caller always gets an answer, and the size of
/// the think block stays the model's decision rather than ours.
///
/// Set to `0` to disable, restoring fully unbounded reasoning.
#[serde(default = "default_answer_reserve")]
pub answer_reserve_tokens: usize,
}
/// Enough for a substantive answer with tool calls, not so much that it
/// meaningfully shortens a think block: at the 32,768 budget pi sends,
/// this leaves ~28.7 k for reasoning — above the ~27.2 k a full agentic
/// design turn was measured to use.
fn default_answer_reserve() -> usize {
4_096
}
impl Default for ReasoningBudgetLadder {
/// Rungs sized against observed behaviour rather than round numbers:
/// a long agentic design turn on this class of model runs to ~27k
/// tokens, so `high` sits above that (bounded but rarely binding),
/// `medium` at roughly half (enough to plan a whole implementation),
/// and `low` at the scale of a focused single-file change.
fn default() -> Self {
Self {
minimal: 1_024,
low: 4_096,
medium: 12_288,
high: 32_768,
xhigh: None,
max: None,
answer_reserve_tokens: default_answer_reserve(),
}
}
}
impl ReasoningBudgetLadder {
/// Resolve an effort level to a token budget.
///
/// `none`/`off` are handled before this — they disable thinking
/// outright via the chat template — so they have no budget here.
/// An unrecognised level resolves to `None` (unbounded) rather than
/// to a guess: silently substituting a budget nobody asked for is
/// the failure mode this whole area has been full of.
pub fn for_effort(&self, effort: &str) -> Option<usize> {
match effort.trim().to_ascii_lowercase().as_str() {
"minimal" => Some(self.minimal),
"low" => Some(self.low),
"medium" => Some(self.medium),
"high" => Some(self.high),
"xhigh" => Some(self.xhigh.unwrap_or(self.high)),
"max" => Some(self.max.or(self.xhigh).unwrap_or(self.high)),
_ => None,
}
}
/// The mapping as a client should see it on `/v1/models` — the rungs
/// this deployment actually offers, so a caller picking `low` knows
/// what it bought instead of guessing (#274).
pub fn advertised(&self) -> Vec<(String, usize)> {
let mut out = vec![
("minimal".to_string(), self.minimal),
("low".to_string(), self.low),
("medium".to_string(), self.medium),
("high".to_string(), self.high),
];
if let Some(v) = self.xhigh {
out.push(("xhigh".to_string(), v));
}
if let Some(v) = self.max {
out.push(("max".to_string(), v));
}
out
}
}
/// Enforces a resolved budget over one request's token stream.
///
/// Fed every sampled token; returns the token that should actually be
/// emitted. While the model is inside a think block and over budget, it
/// returns the close marker instead, exactly once — after which the
/// model is answering and the governor is inert.
#[derive(Debug, Clone)]
pub struct ReasoningGovernor {
limit: usize,
open_id: u32,
close_id: u32,
inside: bool,
used: usize,
forced: bool,
}
impl ReasoningGovernor {
/// `None` when the request named no budget, or the model has no
/// reasoning markers to enforce one with — in both cases generation
/// proceeds exactly as it did before this existed.
pub fn new(
limit: Option<usize>,
markers: Option<(u32, u32)>,
prompt_opened_reasoning: bool,
) -> Option<Self> {
let limit = limit?;
let (open_id, close_id) = markers?;
Some(Self {
limit,
open_id,
close_id,
inside: prompt_opened_reasoning,
used: 0,
forced: false,
})
}
/// Whether the forced close has already been spent.
pub fn forced(&self) -> bool {
self.forced
}
/// Reasoning tokens counted so far.
pub fn used(&self) -> usize {
self.used
}
/// Govern one sampled token, returning the token to emit.
pub fn govern(&mut self, sampled: u32) -> u32 {
if sampled == self.open_id {
self.inside = true;
return sampled;
}
if sampled == self.close_id {
self.inside = false;
return sampled;
}
if !self.inside {
return sampled;
}
if self.used >= self.limit {
// Budget spent: hand the model its own close marker so it
// transitions to answering. Cutting the stream here instead
// would reproduce the very failure this exists to prevent.
self.inside = false;
self.forced = true;
return self.close_id;
}
self.used += 1;
sampled
}
}
#[cfg(test)]
mod tests {
use super::*;
const OPEN: u32 = 100;
const CLOSE: u32 = 101;
fn governor(limit: usize, opened: bool) -> ReasoningGovernor {
ReasoningGovernor::new(Some(limit), Some((OPEN, CLOSE)), opened).expect("governor")
}
#[test]
fn a_request_with_no_budget_is_ungoverned() {
assert!(ReasoningGovernor::new(None, Some((OPEN, CLOSE)), true).is_none());
}
/// A model with no detected markers cannot be governed — better
/// unbounded than truncated at an arbitrary token.
#[test]
fn a_model_without_markers_is_ungoverned() {
assert!(ReasoningGovernor::new(Some(10), None, true).is_none());
}
/// The case that motivated this: the chat template left the think
/// block open, so the model is reasoning from the first token.
#[test]
fn a_prompt_opened_block_is_bounded_and_force_closed() {
let mut g = governor(3, true);
assert_eq!(g.govern(7), 7);
assert_eq!(g.govern(8), 8);
assert_eq!(g.govern(9), 9);
assert_eq!(g.govern(10), CLOSE, "the fourth token must close the block");
assert!(g.forced());
assert_eq!(g.used(), 3);
// Answering now — the governor is inert.
assert_eq!(g.govern(11), 11);
assert_eq!(g.govern(12), 12);
}
/// Content before the block opens is never counted against the
/// reasoning budget.
#[test]
fn only_tokens_inside_the_block_are_counted() {
let mut g = governor(2, false);
assert_eq!(g.govern(5), 5);
assert_eq!(g.govern(6), 6);
assert_eq!(g.used(), 0);
assert_eq!(g.govern(OPEN), OPEN);
assert_eq!(g.govern(7), 7);
assert_eq!(g.govern(8), 8);
assert_eq!(g.govern(9), CLOSE);
assert_eq!(g.used(), 2);
}
/// A model that finishes thinking on its own is untouched — the
/// budget is a ceiling, not a target.
#[test]
fn a_model_that_stops_on_its_own_is_not_forced() {
let mut g = governor(100, true);
for t in 20..30 {
assert_eq!(g.govern(t), t);
}
assert_eq!(g.govern(CLOSE), CLOSE);
assert!(!g.forced());
for t in 40..50 {
assert_eq!(g.govern(t), t);
}
assert!(!g.forced(), "content after the block must not be governed");
}
#[test]
fn the_ladder_resolves_the_levels_clients_can_send() {
let l = ReasoningBudgetLadder::default();
assert_eq!(l.for_effort("minimal"), Some(1_024));
assert_eq!(l.for_effort("LOW"), Some(4_096));
assert_eq!(l.for_effort(" medium "), Some(12_288));
assert_eq!(l.for_effort("high"), Some(32_768));
// Not offered by default → falls back to high rather than
// rejecting a level the client was told it could send.
assert_eq!(l.for_effort("xhigh"), Some(32_768));
assert_eq!(l.for_effort("max"), Some(32_768));
// Unknown → unbounded, never a substituted guess.
assert_eq!(l.for_effort("enthusiastic"), None);
assert_eq!(l.for_effort("none"), None);
}
#[test]
fn the_advertised_ladder_lists_only_what_is_offered() {
let l = ReasoningBudgetLadder::default();
let names: Vec<_> = l.advertised().into_iter().map(|(k, _)| k).collect();
assert_eq!(names, ["minimal", "low", "medium", "high"]);
let l2 = ReasoningBudgetLadder {
xhigh: Some(49_152),
..ReasoningBudgetLadder::default()
};
assert!(
l2.advertised()
.iter()
.any(|(k, v)| k == "xhigh" && *v == 49_152)
);
}
}
@@ -1,302 +0,0 @@
//! Which reasoning-effort rungs a model actually supports, discovered
//! from the model itself (#290).
//!
//! ## Why this is discovered rather than configured
//!
//! `Qwen/Qwen3.8-27B` accepts exactly three efforts and hard-fails on
//! anything else:
//!
//! ```jinja
//! {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
//! {%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
//! {{- raise_exception('Unexpected reasoning effort ...') }}
//! ```
//!
//! We previously advertised an invented ladder — `minimal`, `low`,
//! `medium`, `high` — of which two values the template rejects outright
//! and which omits `xhigh`, the model's own default and strongest rung.
//!
//! A caution about *why* that mattered, because the obvious story is
//! wrong. It is tempting to say clients read our advertisement and were
//! misled by it. pi-ai does not: its `getSupportedThinkingLevels` reads
//! `model.thinkingLevelMap`, a static field in the operator's
//! `models.json`, and treats `xhigh`/`max` as unavailable unless that
//! map names them explicitly. So a pi user saw exactly
//! `minimal, low, medium, high` — the invented ladder, to the value —
//! for a reason that had nothing to do with what we published.
//!
//! The coincidence is the trap: our wrong advertisement and pi's
//! default produced the same symptom, which made the advertisement look
//! causal. Publishing rungs the model actually accepts is still correct
//! — it is the truth, other clients may read it, and it is what makes
//! `apply_effort_kwarg` right — but do not assume a given client
//! consumes it. For pi the operator must also set `thinkingLevelMap`,
//! by hand, and nothing validates that they match.
//!
//! A hardcoded per-model table would drift the moment a model ships a
//! new template, which is exactly the class of defect #280 fixed by
//! fetching the template instead of assuming it. So the rungs come from
//! the artifact.
//!
//! ## How the probe works
//!
//! `raise_exception` is already wired to a render error, so the template
//! answers the question directly: render it once per candidate effort
//! and see which ones survive.
//!
//! Two ambiguities the probe has to resolve, or it would report
//! confident nonsense:
//!
//! 1. **A template that ignores `reasoning_effort` entirely** renders
//! successfully for every candidate. Supporting "all six" would be a
//! lie. So a candidate only counts when the rendered prompt actually
//! *differs* across efforts — if every render is identical the model
//! has no effort control and we advertise none.
//! 2. **The default** is whichever supported candidate renders
//! identically to a render with no effort supplied. Qwen3.8 defaults
//! to `xhigh`; without detecting that, a caller who says nothing gets
//! the strongest rung while we believe it got nothing.
use anyhow::Result;
use cortex_core::openai::{ChatMessage, MessageContent};
use serde_json::{Value, json};
/// Effort names in ascending order of thinking, as clients spell them.
///
/// Deliberately a superset of what any one model supports: this is the
/// *candidate* set the probe tries, and the conventional vocabulary
/// OpenAI-compatible clients use (pi-ai's `EXTENDED_THINKING_LEVELS`
/// minus `off`, which is not an effort but a separate switch handled by
/// `enable_thinking`).
pub const CANDIDATE_EFFORTS: &[&str] = &["minimal", "low", "medium", "high", "xhigh", "max"];
/// What one model's template accepts.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct SupportedEfforts {
/// Accepted efforts, ascending, as the template spells them.
pub levels: Vec<String>,
/// The effort applied when the caller names none — the template's
/// own `|default(...)`. `None` when it cannot be determined.
pub default: Option<String>,
}
impl SupportedEfforts {
pub fn is_empty(&self) -> bool {
self.levels.is_empty()
}
pub fn supports(&self, effort: &str) -> bool {
self.levels.iter().any(|l| l == effort)
}
/// Map a caller's effort onto the nearest rung this model has.
///
/// Exact match wins. Otherwise walk **down** from the requested
/// level first, then up — a caller asking for more thinking than the
/// model offers should get the most it has, and a caller asking for
/// less should not be silently upgraded into a slower, costlier
/// response than they asked for.
///
/// Descending-first is the opposite of pi-ai's `clampThinkingLevel`,
/// which searches upward first. That is deliberate: pi is choosing
/// among levels it believes exist, whereas we are the ones who know
/// what exists, and over-delivering thinking is the failure that
/// costs the caller money and latency.
///
/// `None` when the model has no rungs at all.
pub fn nearest(&self, requested: &str) -> Option<&str> {
if self.levels.is_empty() {
return None;
}
if let Some(exact) = self.levels.iter().find(|l| *l == requested) {
return Some(exact.as_str());
}
let idx = CANDIDATE_EFFORTS.iter().position(|c| *c == requested)?;
for c in CANDIDATE_EFFORTS[..idx].iter().rev() {
if let Some(hit) = self.levels.iter().find(|l| l.as_str() == *c) {
return Some(hit.as_str());
}
}
for c in &CANDIDATE_EFFORTS[idx + 1..] {
if let Some(hit) = self.levels.iter().find(|l| l.as_str() == *c) {
return Some(hit.as_str());
}
}
None
}
}
/// Render a probe prompt at `effort`, or `None` if the template rejects
/// it.
fn probe_render(template: &str, effort: Option<&str>) -> Option<String> {
let messages = [ChatMessage {
role: "user".into(),
content: MessageContent::Text("probe".into()),
extra: Value::Null,
}];
let kwargs = match effort {
Some(e) => json!({ "reasoning_effort": e }),
None => Value::Null,
};
super::chat_template::render_chat_template(template, &messages, &Value::Null, &kwargs).ok()
}
/// Discover which efforts `template` accepts (#290).
///
/// Errors are not propagated: a template that cannot render even the
/// baseline probe yields no rungs, and the caller behaves as it did
/// before this existed.
pub fn probe(template: &str, model_id: &str) -> Result<SupportedEfforts> {
let Some(baseline) = probe_render(template, None) else {
tracing::debug!(
model = %model_id,
"reasoning effort: template does not render a baseline probe; advertising no rungs"
);
return Ok(SupportedEfforts::default());
};
let mut rendered: Vec<(&str, String)> = Vec::new();
for candidate in CANDIDATE_EFFORTS {
if let Some(out) = probe_render(template, Some(candidate)) {
rendered.push((candidate, out));
}
}
// A template that never reads `reasoning_effort` renders the same
// bytes whatever we pass. Advertising every candidate in that case
// would be the invented-ladder mistake again, in a new place.
let effort_aware = rendered.windows(2).any(|w| w[0].1 != w[1].1)
|| rendered.iter().any(|(_, out)| *out != baseline);
if !effort_aware {
tracing::debug!(
model = %model_id,
"reasoning effort: template ignores reasoning_effort; advertising no rungs"
);
return Ok(SupportedEfforts::default());
}
let default = rendered
.iter()
.find(|(_, out)| *out == baseline)
.map(|(name, _)| (*name).to_string());
let levels: Vec<String> = rendered
.iter()
.map(|(name, _)| (*name).to_string())
.collect();
tracing::info!(
model = %model_id,
levels = %levels.join(","),
default = ?default,
"reasoning effort: rungs discovered from the model's chat template (#290)"
);
Ok(SupportedEfforts { levels, default })
}
#[cfg(test)]
mod tests {
use super::*;
/// The probe against the **real** deployed Qwen3.8-27B template,
/// not a hand-written approximation of it. A synthetic template
/// proves the probe logic; only the shipped artifact proves the
/// answer, and the artifact is what #290 is about.
#[test]
fn the_real_qwen38_template_yields_low_medium_xhigh_defaulting_to_xhigh() {
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/data/qwen3.8-27b.chat_template.jinja"
);
let Ok(template) = std::fs::read_to_string(path) else {
// The fixture is checked in; if it is ever removed this test
// should fail loudly rather than silently pass.
panic!("missing fixture {path}");
};
let s = probe(&template, "Qwen/Qwen3.8-27B").expect("probe");
assert_eq!(
s.levels,
vec!["low", "medium", "xhigh"],
"the shipped template accepts exactly these"
);
assert_eq!(s.default.as_deref(), Some("xhigh"));
// The two rungs we used to advertise are rejected outright.
assert!(!s.supports("minimal"));
assert!(!s.supports("high"));
}
/// The shape Qwen3.8-27B actually ships: three rungs, `xhigh`
/// default, `raise_exception` on anything else.
const QWEN38: &str = r#"
{%- set resolved = reasoning_effort|default('xhigh') %}
{%- if resolved not in ('xhigh', 'medium', 'low') %}
{{- raise_exception('Unexpected reasoning effort ' ~ resolved) }}
{%- endif %}
{%- if resolved == 'xhigh' %}EFFORT-XHIGH{% elif resolved == 'low' %}EFFORT-LOW{% else %}EFFORT-MEDIUM{% endif %}
{%- for m in messages %}<|im_start|>{{ m.role }}
{{ m.content }}<|im_end|>
{%- endfor %}
"#;
/// A template with no effort control at all.
const PLAIN: &str = r#"
{%- for m in messages %}<|im_start|>{{ m.role }}
{{ m.content }}<|im_end|>
{%- endfor %}
"#;
#[test]
fn discovers_exactly_the_rungs_the_template_accepts() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.levels, vec!["low", "medium", "xhigh"]);
assert!(!s.supports("minimal"), "template raises on minimal");
assert!(!s.supports("high"), "template raises on high");
}
/// Without this, a caller who names no effort silently receives the
/// strongest rung while we believe it received nothing — which is
/// how the xhigh instruction ended up on every request in #290.
#[test]
fn detects_the_templates_own_default() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.default.as_deref(), Some("xhigh"));
}
/// A template that ignores the kwarg must advertise nothing rather
/// than appearing to support all six.
#[test]
fn a_template_without_effort_control_advertises_no_rungs() {
let s = probe(PLAIN, "test").expect("probe");
assert!(s.is_empty(), "got {:?}", s.levels);
assert_eq!(s.nearest("medium"), None);
}
#[test]
fn an_exact_match_is_returned_unchanged() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.nearest("medium"), Some("medium"));
assert_eq!(s.nearest("xhigh"), Some("xhigh"));
}
/// `high` does not exist on this model. Descending first means it
/// resolves to `medium`, not `xhigh`: a caller is never given more
/// thinking (slower, dearer) than it asked for.
#[test]
fn an_unsupported_level_resolves_downward_not_upward() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.nearest("high"), Some("medium"));
}
/// `minimal` is below every rung the model has, so there is nothing
/// to descend to and the nearest is the weakest available.
#[test]
fn a_level_below_every_rung_resolves_up_to_the_weakest() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.nearest("minimal"), Some("low"));
}
#[test]
fn an_unknown_level_name_resolves_to_nothing_rather_than_a_guess() {
let s = probe(QWEN38, "test").expect("probe");
assert_eq!(s.nearest("turbo"), None);
}
}
-687
View File
@@ -1,687 +0,0 @@
//! Sampling parameter resolution (#272, #283).
//!
//! Four sources, in priority order:
//!
//! 1. what the **request** asked for,
//! 2. what the **operator** set for this model (#283),
//! 3. what the **model** published in `generation_config.json`,
//! 4. a built-in fallback.
//!
//! Tier (2) exists because honouring the model's own file is right by
//! default and still wrong sometimes: `Qwen/Qwen3.8-27B` publishes
//! `temperature = 1.0`, which measured a 20% structural-defect rate on
//! ~2k-token code generation against 0/60 at `<= 0.6` (#283,
//! p = 0.0031). The operator's correction is explicit and lives in a
//! config file — never a heuristic that inspects the model and guesses,
//! which would be a footgun.
//!
//! The operator sits *below* the request on purpose. An explicit API
//! parameter that is silently ignored is a contract break; the observed
//! failure came from callers that named nothing, so a default-tier
//! override is sufficient to fix it.
//!
//! Mechanically, tier (2) is folded into tier (3) once at load time by
//! [`ModelGenerationDefaults::with_operator_override`], so the
//! per-request resolve path stays a two-tier lookup and every call site
//! gets the override for free — there is no site that can forget it.
//!
//! Before this, (2) did not exist and (3) was a hardcoded
//! `temperature = 0.7` with `top_p = None` — and `None` selects
//! `Sampling::All`, meaning no truncation at all. So a caller that
//! omitted sampling fields (the common case: the DeepSeek Harness sends
//! only `temperature`) got untruncated sampling across the full ~250k
//! vocabulary on a model whose authors published `top_k = 20,
//! top_p = 0.95, temperature = 1.0`.
//!
//! Picking "no truncation" for an absent field is not a neutral default,
//! it is the widest possible one. The neutral default is what the model
//! shipped with.
//!
//! Resolved **once** per request and threaded whole. The eight
//! `LogitsProcessor` construction sites this replaces could each have
//! disagreed about a default — the same shape of bug as #252, where two
//! config files described one model and drifted.
use serde::{Deserialize, Serialize};
use std::path::Path;
/// Fallback temperature when neither the request nor the model says.
/// Retained from the previous hardcoded value so a model without a
/// `generation_config.json` behaves exactly as it did before.
const FALLBACK_TEMPERATURE: f64 = 0.7;
/// Repetition penalty applied to recently-generated tokens before
/// sampling. 1.0 disables it; >1.0 makes recently-emitted tokens less
/// likely. mistral.rs and llama.cpp default to 1.1, which is enough to
/// stop small quantized models from degenerating into "Wait, no, no..."
/// loops without distorting normal output.
pub const DEFAULT_REPEAT_PENALTY: f32 = 1.1;
/// Number of recently-generated tokens fed into the repetition penalty.
/// Matches the candle quantized-qwen3 example default.
///
/// Defensible for chat and poorly suited to a 24k-token reasoning block,
/// where a derivation repeated 500 tokens later is well outside the
/// window — which is why it is now a per-request knob rather than a
/// constant.
pub const DEFAULT_REPEAT_LAST_N: usize = 64;
/// What a model published in its `generation_config.json`.
///
/// Every field optional: the file is a HuggingFace convention, not a
/// schema, and models populate different subsets of it.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ModelGenerationDefaults {
pub temperature: Option<f64>,
pub top_p: Option<f64>,
pub top_k: Option<usize>,
/// Not published in `generation_config.json` — this struct is the
/// resolved per-model default layer, and the operator override
/// (#283) lands here so the repetition knobs sit in the same
/// precedence chain as the rest.
pub repeat_penalty: Option<f32>,
pub repeat_last_n: Option<usize>,
pub presence_penalty: Option<f32>,
pub frequency_penalty: Option<f32>,
pub seed: Option<u64>,
}
impl ModelGenerationDefaults {
/// Read `generation_config.json` from a model snapshot directory.
///
/// Absent or unparseable is not an error — we fall back exactly as
/// before — but it is logged, because silently guessing the model's
/// sampling is how the untruncated default went unnoticed.
pub fn load_from_dir(dir: &Path) -> Self {
let path = dir.join("generation_config.json");
let text = match std::fs::read_to_string(&path) {
Ok(t) => t,
Err(e) => {
tracing::debug!(
path = %path.display(),
error = %e,
"sampling: no generation_config.json; using built-in fallbacks"
);
return Self::default();
}
};
match serde_json::from_str::<Self>(&text) {
Ok(d) => {
tracing::info!(
path = %path.display(),
temperature = ?d.temperature,
top_p = ?d.top_p,
top_k = ?d.top_k,
"sampling: model generation defaults loaded"
);
d
}
Err(e) => {
tracing::warn!(
path = %path.display(),
error = %e,
"sampling: generation_config.json did not parse; using built-in fallbacks"
);
Self::default()
}
}
}
/// Overlay the operator's per-model override (#283) onto what the
/// model published, field by field.
///
/// Applied **once at load**, not per request, so the override cannot
/// be forgotten at one of the resolve sites — the same reasoning
/// that made #272 resolve sampling once rather than at eight
/// `LogitsProcessor` construction sites.
///
/// Each field overlays independently: setting only `temperature`
/// leaves the model's `top_p`/`top_k` in force, because an operator
/// correcting one number should not silently blank the other two
/// back to "no truncation".
///
/// Logged at `info` whenever it changes something — an operator
/// override that is in force but invisible is how a fleet ends up
/// with two configs that disagree and nobody noticing (#252).
pub fn with_operator_override(
mut self,
model_id: &str,
operator: Option<&cortex_core::harness::SamplingOverride>,
) -> Self {
let Some(op) = operator else {
return self;
};
if op.is_empty() {
return self;
}
let before = self.clone();
if let Some(t) = op.temperature {
self.temperature = Some(t);
}
if let Some(p) = op.top_p {
self.top_p = Some(p);
}
if let Some(k) = op.top_k {
self.top_k = Some(k);
}
if let Some(r) = op.repeat_penalty {
self.repeat_penalty = Some(r);
}
if let Some(n) = op.repeat_last_n {
self.repeat_last_n = Some(n);
}
if let Some(v) = op.presence_penalty {
self.presence_penalty = Some(v);
}
if let Some(v) = op.frequency_penalty {
self.frequency_penalty = Some(v);
}
if let Some(v) = op.seed {
self.seed = Some(v);
}
tracing::info!(
model = %model_id,
model_temperature = ?before.temperature,
model_top_p = ?before.top_p,
model_top_k = ?before.top_k,
temperature = ?self.temperature,
top_p = ?self.top_p,
top_k = ?self.top_k,
repeat_penalty = ?self.repeat_penalty,
repeat_last_n = ?self.repeat_last_n,
presence_penalty = ?self.presence_penalty,
frequency_penalty = ?self.frequency_penalty,
seed = ?self.seed,
"sampling: operator override applied over the model's published defaults (#283)"
);
self
}
}
/// What the caller asked for. All optional — absent means "no opinion",
/// which defers to the model's published value.
#[derive(Debug, Clone, Default)]
pub struct RequestedSampling {
pub temperature: Option<f64>,
pub top_p: Option<f64>,
pub top_k: Option<usize>,
pub seed: Option<u64>,
pub repeat_penalty: Option<f32>,
pub repeat_last_n: Option<usize>,
pub presence_penalty: Option<f32>,
pub frequency_penalty: Option<f32>,
}
/// Fully-resolved sampling for one request.
#[derive(Debug, Clone, PartialEq)]
pub struct SamplingParams {
pub temperature: f64,
pub top_p: Option<f64>,
pub top_k: Option<usize>,
pub seed: u64,
pub repeat_penalty: f32,
pub repeat_last_n: usize,
/// Sequence-wide penalties, OpenAI semantics. `0.0` disables.
pub presence_penalty: f32,
pub frequency_penalty: f32,
}
impl SamplingParams {
/// Resolve request over model defaults over fallback.
///
/// `random_seed` is the per-request seed used when the caller did
/// not pin one; it is passed in rather than generated here so this
/// stays a pure function and the "same seed reproduces" property is
/// testable without a clock.
pub fn resolve(
requested: &RequestedSampling,
model: &ModelGenerationDefaults,
random_seed: u64,
) -> Self {
Self {
temperature: requested
.temperature
.or(model.temperature)
.unwrap_or(FALLBACK_TEMPERATURE),
// No `unwrap_or` here on purpose: `None` still means "do not
// truncate", but it is now only reachable when neither the
// caller nor the model expressed a preference, rather than
// whenever the caller stayed quiet.
top_p: requested.top_p.or(model.top_p),
top_k: requested.top_k.or(model.top_k),
seed: requested.seed.or(model.seed).unwrap_or(random_seed),
repeat_penalty: requested
.repeat_penalty
.or(model.repeat_penalty)
.unwrap_or(DEFAULT_REPEAT_PENALTY),
repeat_last_n: requested
.repeat_last_n
.or(model.repeat_last_n)
.unwrap_or(DEFAULT_REPEAT_LAST_N),
presence_penalty: requested
.presence_penalty
.or(model.presence_penalty)
.unwrap_or(0.0),
frequency_penalty: requested
.frequency_penalty
.or(model.frequency_penalty)
.unwrap_or(0.0),
}
}
/// The candle sampling strategy these parameters select.
///
/// `temperature <= 0` is greedy regardless of the rest — a caller
/// asking for determinism that way should not have it silently
/// softened by a `top_k` the model happened to publish.
pub fn to_sampling(&self) -> candle_transformers::generation::Sampling {
use candle_transformers::generation::Sampling;
if self.temperature <= 0.0 {
return Sampling::ArgMax;
}
let temperature = self.temperature;
match (self.top_k, self.top_p) {
(Some(k), Some(p)) => Sampling::TopKThenTopP { k, p, temperature },
(Some(k), None) => Sampling::TopK { k, temperature },
(None, Some(p)) => Sampling::TopP { p, temperature },
(None, None) => Sampling::All { temperature },
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use candle_transformers::generation::Sampling;
use cortex_core::harness::SamplingOverride;
fn qwen3_defaults() -> ModelGenerationDefaults {
ModelGenerationDefaults {
temperature: Some(1.0),
top_p: Some(0.95),
top_k: Some(20),
..Default::default()
}
}
/// The case that motivated #271: a caller that names no sampling
/// parameters must get the model's published ones, not untruncated
/// sampling at a hardcoded 0.7.
#[test]
fn a_silent_request_inherits_the_models_published_sampling() {
let p = SamplingParams::resolve(&RequestedSampling::default(), &qwen3_defaults(), 42);
assert_eq!(p.temperature, 1.0);
assert_eq!(p.top_p, Some(0.95));
assert_eq!(p.top_k, Some(20));
assert_eq!(
p.to_sampling(),
Sampling::TopKThenTopP {
k: 20,
p: 0.95,
temperature: 1.0
},
"the combination Qwen3 asks for must be expressible"
);
}
/// The caller still wins. A harness that deliberately sets a low
/// temperature must not have it overridden by the model's file.
#[test]
fn the_request_overrides_the_model() {
let requested = RequestedSampling {
temperature: Some(0.2),
top_k: Some(5),
..Default::default()
};
let p = SamplingParams::resolve(&requested, &qwen3_defaults(), 42);
assert_eq!(p.temperature, 0.2);
assert_eq!(p.top_k, Some(5));
// Untouched fields still fall through to the model.
assert_eq!(p.top_p, Some(0.95));
}
// ── Operator override (#283) ──────────────────────────────
fn op(temperature: Option<f64>, top_p: Option<f64>, top_k: Option<usize>) -> SamplingOverride {
SamplingOverride {
temperature,
top_p,
top_k,
..Default::default()
}
}
fn with_op(o: SamplingOverride) -> ModelGenerationDefaults {
qwen3_defaults().with_operator_override("Qwen/Qwen3.8-27B", Some(&o))
}
/// The #283 case: the model publishes `temperature = 1.0`, which
/// measured a 20% structural-defect rate on code generation. The
/// operator sets 0.6 and a silent caller must get 0.6.
#[test]
fn the_operator_override_replaces_what_the_model_published() {
let p = SamplingParams::resolve(
&RequestedSampling::default(),
&with_op(op(Some(0.6), None, None)),
42,
);
assert_eq!(p.temperature, 0.6);
}
/// Precedence is request > operator > model. An explicit API
/// parameter that is silently ignored is a contract break, so a
/// caller naming a temperature still wins over the operator.
#[test]
fn an_explicit_request_still_beats_the_operator() {
let requested = RequestedSampling {
temperature: Some(0.9),
..Default::default()
};
let p = SamplingParams::resolve(&requested, &with_op(op(Some(0.6), None, None)), 42);
assert_eq!(p.temperature, 0.9);
}
/// Overriding one field must not blank the others back to "no
/// truncation" — that was the #271 failure mode, reintroduced by a
/// wholesale replace instead of a field-by-field overlay.
#[test]
fn overriding_temperature_alone_leaves_top_p_and_top_k_intact() {
let d = with_op(op(Some(0.6), None, None));
assert_eq!(d.temperature, Some(0.6));
assert_eq!(d.top_p, Some(0.95), "the model's top_p must survive");
assert_eq!(d.top_k, Some(20), "the model's top_k must survive");
}
/// Each field overlays independently.
#[test]
fn every_field_can_be_overridden_independently() {
let d = with_op(op(None, Some(0.8), Some(40)));
assert_eq!(d.temperature, Some(1.0), "untouched, so the model's");
assert_eq!(d.top_p, Some(0.8));
assert_eq!(d.top_k, Some(40));
}
/// No override, and an override that sets nothing, must both be
/// indistinguishable from the pre-#283 behaviour.
#[test]
fn absent_or_empty_override_changes_nothing() {
let base = qwen3_defaults();
let none = qwen3_defaults().with_operator_override("m", None);
let empty =
qwen3_defaults().with_operator_override("m", Some(&SamplingOverride::default()));
for d in [none, empty] {
assert_eq!(d.temperature, base.temperature);
assert_eq!(d.top_p, base.top_p);
assert_eq!(d.top_k, base.top_k);
}
}
/// An operator override on a model that ships no
/// `generation_config.json` must still take effect, rather than
/// falling through to the built-in 0.7.
#[test]
fn the_override_applies_even_without_a_generation_config() {
let d = ModelGenerationDefaults::default()
.with_operator_override("m", Some(&op(Some(0.6), None, None)));
let p = SamplingParams::resolve(&RequestedSampling::default(), &d, 42);
assert_eq!(p.temperature, 0.6);
assert_ne!(p.temperature, FALLBACK_TEMPERATURE);
}
/// `temperature = 0` from an operator must select greedy decoding,
/// not be treated as "unset" by an `unwrap_or`-shaped bug.
#[test]
fn an_operator_temperature_of_zero_is_greedy_not_unset() {
let p = SamplingParams::resolve(
&RequestedSampling::default(),
&with_op(op(Some(0.0), None, None)),
42,
);
assert_eq!(p.temperature, 0.0);
assert_eq!(p.to_sampling(), Sampling::ArgMax);
}
/// A model with no `generation_config.json` must behave exactly as
/// it did before this landed, so nothing regresses on the models
/// that do not ship one.
/// An operator can pin the seed, because the client that needs it
/// pinned cannot send one: pi's request builder emits no `seed`, so
/// without this a parameter search runs unseeded and sampling
/// variance is indistinguishable from the effect being measured.
#[test]
fn an_operator_can_pin_the_seed() {
let op = cortex_core::harness::SamplingOverride {
seed: Some(424_242),
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
// The random seed argument must be ignored once an operator pinned one.
let p = SamplingParams::resolve(&RequestedSampling::default(), &defaults, 999);
assert_eq!(p.seed, 424_242);
}
/// Precedence holds: a caller that names its own seed keeps it.
#[test]
fn a_requested_seed_outranks_the_operator() {
let op = cortex_core::harness::SamplingOverride {
seed: Some(424_242),
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let req = RequestedSampling {
seed: Some(7),
..Default::default()
};
assert_eq!(SamplingParams::resolve(&req, &defaults, 999).seed, 7);
}
/// With neither, the per-request random seed still applies — an
/// unset override must not accidentally freeze the fleet's RNG.
#[test]
fn no_pinned_seed_leaves_generation_random() {
let d = ModelGenerationDefaults::default().with_operator_override("m", None);
assert_eq!(
SamplingParams::resolve(&RequestedSampling::default(), &d, 999).seed,
999
);
}
/// The gap this closes: `repeat_last_n` defaults to 64 — candle's
/// chat example value — while a reasoning model's think block runs
/// to tens of thousands of tokens. Anything restated more than 64
/// tokens later is invisible to the penalty, and before this there
/// was no operator route to the knob at all: `SamplingOverride`
/// carried only temperature/top_p/top_k, and OpenAI-shaped clients
/// do not send repetition parameters. The 64 was effectively
/// hardcoded in production.
#[test]
fn an_operator_can_widen_the_repetition_window() {
let op = cortex_core::harness::SamplingOverride {
repeat_penalty: Some(1.05),
repeat_last_n: Some(2048),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let p = SamplingParams::resolve(&RequestedSampling::default(), &defaults, 7);
assert_eq!(p.repeat_last_n, 2048);
assert!((p.repeat_penalty - 1.05).abs() < f32::EPSILON);
}
/// Precedence is request > operator > built-in, same as every other
/// sampling field — a caller that named a window keeps it.
#[test]
fn a_requested_repetition_window_outranks_the_operator() {
let op = cortex_core::harness::SamplingOverride {
repeat_last_n: Some(2048),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let req = RequestedSampling {
repeat_last_n: Some(128),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
assert_eq!(
SamplingParams::resolve(&req, &defaults, 7).repeat_last_n,
128
);
}
/// An operator who sets nothing must not change the constants.
#[test]
fn no_operator_setting_leaves_the_repetition_defaults_alone() {
let d = ModelGenerationDefaults::default().with_operator_override("m", None);
let p = SamplingParams::resolve(&RequestedSampling::default(), &d, 7);
assert_eq!(p.repeat_last_n, DEFAULT_REPEAT_LAST_N);
assert!((p.repeat_penalty - DEFAULT_REPEAT_PENALTY).abs() < f32::EPSILON);
}
#[test]
fn without_model_defaults_the_old_behaviour_is_preserved() {
let p = SamplingParams::resolve(
&RequestedSampling::default(),
&ModelGenerationDefaults::default(),
42,
);
assert_eq!(p.temperature, FALLBACK_TEMPERATURE);
assert_eq!(p.top_p, None);
assert_eq!(p.top_k, None);
assert_eq!(p.to_sampling(), Sampling::All { temperature: 0.7 });
}
/// A pinned seed is honoured; an absent one takes the per-request
/// random value. Reproducibility is the whole point — the text paths
/// previously called `unix_subsec_nanos()` unconditionally, so
/// `seed` was accepted and discarded.
#[test]
fn seed_is_pinned_when_given_and_random_otherwise() {
let pinned = SamplingParams::resolve(
&RequestedSampling {
seed: Some(7),
..Default::default()
},
&ModelGenerationDefaults::default(),
42,
);
assert_eq!(pinned.seed, 7);
let unpinned = SamplingParams::resolve(
&RequestedSampling::default(),
&ModelGenerationDefaults::default(),
42,
);
assert_eq!(unpinned.seed, 42);
}
/// Greedy beats every truncation setting: a caller asking for
/// `temperature = 0` wants determinism, not a softened version of it.
#[test]
fn zero_temperature_is_greedy_even_with_model_top_k() {
let requested = RequestedSampling {
temperature: Some(0.0),
..Default::default()
};
let p = SamplingParams::resolve(&requested, &qwen3_defaults(), 42);
assert_eq!(p.to_sampling(), Sampling::ArgMax);
}
/// Each truncation knob alone selects the matching strategy.
#[test]
fn single_knobs_select_their_own_strategy() {
let only_k = SamplingParams::resolve(
&RequestedSampling {
top_k: Some(20),
..Default::default()
},
&ModelGenerationDefaults::default(),
1,
);
assert_eq!(
only_k.to_sampling(),
Sampling::TopK {
k: 20,
temperature: 0.7
}
);
let only_p = SamplingParams::resolve(
&RequestedSampling {
top_p: Some(0.9),
..Default::default()
},
&ModelGenerationDefaults::default(),
1,
);
assert_eq!(
only_p.to_sampling(),
Sampling::TopP {
p: 0.9,
temperature: 0.7
}
);
}
/// Repetition penalty and its window are now per-request, defaulting
/// to the values that were previously constants.
#[test]
fn repetition_knobs_default_to_the_former_constants() {
let d = SamplingParams::resolve(
&RequestedSampling::default(),
&ModelGenerationDefaults::default(),
1,
);
assert_eq!(d.repeat_penalty, DEFAULT_REPEAT_PENALTY);
assert_eq!(d.repeat_last_n, DEFAULT_REPEAT_LAST_N);
let tuned = SamplingParams::resolve(
&RequestedSampling {
repeat_penalty: Some(1.05),
repeat_last_n: Some(512),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
},
&ModelGenerationDefaults::default(),
1,
);
assert_eq!(tuned.repeat_penalty, 1.05);
assert_eq!(tuned.repeat_last_n, 512);
}
#[test]
fn absent_generation_config_is_not_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let d = ModelGenerationDefaults::load_from_dir(dir.path());
assert!(d.temperature.is_none() && d.top_p.is_none() && d.top_k.is_none());
}
#[test]
fn a_real_generation_config_parses() {
let dir = tempfile::tempdir().expect("tempdir");
// Verbatim from Qwen/Qwen3.8-27B, extra fields included — the
// reader must tolerate the ones it does not model.
std::fs::write(
dir.path().join("generation_config.json"),
r#"{"bos_token_id":248044,"do_sample":true,"eos_token_id":[248046,248044],
"pad_token_id":248044,"temperature":1.0,"top_k":20,"top_p":0.95}"#,
)
.expect("write");
let d = ModelGenerationDefaults::load_from_dir(dir.path());
assert_eq!(d.temperature, Some(1.0));
assert_eq!(d.top_p, Some(0.95));
assert_eq!(d.top_k, Some(20));
}
#[test]
fn malformed_generation_config_falls_back_rather_than_failing() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("generation_config.json"), "{not json").expect("write");
let d = ModelGenerationDefaults::load_from_dir(dir.path());
assert!(d.temperature.is_none());
}
}
+16 -214
View File
@@ -30,10 +30,6 @@ pub mod worker;
use anyhow::{Context, Result};
use std::path::{Path, PathBuf};
use std::process::Stdio;
#[cfg(any(feature = "cuda", test))]
use std::sync::Arc;
#[cfg(any(feature = "cuda", test))]
use std::sync::atomic::{AtomicBool, Ordering};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Lines};
use tokio::process::{Child, ChildStdin, ChildStdout, Command};
@@ -323,87 +319,6 @@ pub struct WorkerPool {
leader_comm: Option<nccl_state::SendComm>,
}
/// How long the watchdog waits for `ncclCommAbort` to return before
/// concluding the process is unrecoverable (#265). Generous relative to a
/// healthy abort (milliseconds) and short relative to a human noticing.
#[cfg(any(feature = "cuda", test))]
fn tp_abort_deadline() -> std::time::Duration {
let secs = std::env::var("NEURON_TP_ABORT_DEADLINE_S")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|s| *s > 0)
.unwrap_or(60);
std::time::Duration::from_secs(secs)
}
/// Wait for the abort thread to signal completion. `true` if it returned
/// in time (recovery proceeds normally), `false` if the deadline elapsed
/// (the process is unrecoverable).
///
/// Split out from the deadman thread so the decision is testable without
/// a GPU: the branch that ends the process is the one most worth
/// exercising, and the one hardest to reach by accident.
#[cfg(any(feature = "cuda", test))]
fn wait_for_abort(done: &AtomicBool, deadline: std::time::Duration) -> bool {
let start = std::time::Instant::now();
while start.elapsed() < deadline {
if done.load(Ordering::Acquire) {
return true;
}
std::thread::sleep(std::time::Duration::from_millis(25));
}
// One last look: the abort may have landed inside the final sleep.
done.load(Ordering::Acquire)
}
/// Exit non-zero so systemd's `Restart=on-failure` restarts us (#265).
///
/// Called only when the process cannot recover on its own: the NCCL
/// collective is wedged and the abort that would unblock it has itself
/// hung. The daemon then serves nothing — on beast 2026-08-17 it sat in
/// that state for 70 minutes with GPUs idle, weights resident, 27
/// connections queued on the listener, and `systemctl` reporting it
/// active. A process that knows it is unrecoverable should not wait for a
/// human.
///
/// **Everything diagnostic is logged first.** A restart destroys the
/// state that explains the hang, so trading a diagnosable outage for an
/// undiagnosable self-healing one would be a poor bargain.
///
/// `process::exit` deliberately skips destructors and the graceful-drain
/// path. That is normally the wrong instinct, but draining a wedged
/// process cannot work — and `TimeoutStopSec=120s` plus
/// `TimeoutStopFailureMode=abort` would add two more minutes of outage on
/// top of the wedge (#256).
#[cfg(feature = "cuda")]
fn exit_unrecoverable(model_id: &str, reason: &str) -> ! {
// Read straight from the driver rather than any cached value: the
// cache is refreshed by the very machinery that is wedged.
let vram = std::process::Command::new("nvidia-smi")
.args([
"--query-gpu=index,memory.used,memory.free,utilization.gpu",
"--format=csv,noheader",
])
.output()
.ok()
.map(|o| String::from_utf8_lossy(&o.stdout).replace('\n', " | "))
.unwrap_or_else(|| "unavailable".into());
tracing::error!(
model = %model_id,
reason = %reason,
vram = %vram,
pid = std::process::id(),
"tp watchdog: UNRECOVERABLE — exiting non-zero so systemd restarts this daemon (#265). \
The GPU state above is from immediately before the exit; the wedged collective and its \
request id are in the preceding watchdog lines."
);
// Give the logging layer a moment to flush before the process dies —
// an exit that eats its own explanation is the failure mode this
// whole change exists to avoid.
std::thread::sleep(std::time::Duration::from_millis(500));
std::process::exit(70) // EX_SOFTWARE
}
/// Per-step deadline for a TP forward (#17 Stage 2). A healthy decode
/// step or chunked prefill completes in well under a second; a wedged
/// NCCL collective never returns. Generous default so no legitimate step
@@ -434,66 +349,23 @@ impl WorkerPool {
"tp watchdog: leader forward exceeded deadline — NCCL collective wedged; \
aborting comm to unblock the leader thread for auto-recovery"
);
let Some(comm) = self.leader_comm.as_ref().map(|c| c.0.clone()) else {
tracing::error!(
match &self.leader_comm {
Some(c) => match c.0.abort() {
Ok(()) => tracing::error!(
model = %model_id,
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
failing the step so the model auto-recovers (unload+reload)"
),
Err(e) => tracing::error!(
model = %model_id, error = ?e,
"tp watchdog: ncclCommAbort failed — recovery may stall until a process restart"
),
},
None => tracing::error!(
model = %model_id,
"tp watchdog: no cached leader comm handle — cannot abort; exiting so \
systemd restarts us (#265)"
);
exit_unrecoverable(model_id, "no leader comm handle to abort");
};
// `ncclCommAbort` can itself block forever — observed on beast
// 2026-08-17, where neither the success nor the failure branch below
// was ever reached and the daemon sat wedged for 70 minutes with 27
// connections queued on its listener (#265). So the abort runs on its
// own thread and the deadline below does NOT depend on it returning:
// an abort that hangs must not take the supervisor with it.
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
let owned_model = model_id.to_string();
std::thread::Builder::new()
.name("tp-watchdog-abort".into())
.spawn(move || {
match comm.abort() {
Ok(()) => tracing::error!(
model = %owned_model,
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
failing the step so the model auto-recovers (unload+reload)"
),
Err(e) => tracing::error!(
model = %owned_model, error = ?e,
"tp watchdog: ncclCommAbort failed — the collective is still wedged"
),
}
// Returning at all is the signal, success or failure: a
// returned abort leaves the process responsive, so
// auto-recovery can run and we must not exit under it.
signal.store(true, Ordering::Release);
})
.map(|_| ())
.unwrap_or_else(|e| {
// Cannot even spawn a thread — nothing here can recover.
tracing::error!(model = %model_id, error = %e, "tp watchdog: abort thread spawn failed");
exit_unrecoverable(model_id, "could not spawn the abort thread");
});
// Deadman. Disarmed only by the abort returning.
let deadline = tp_abort_deadline();
let owned_model = model_id.to_string();
if let Err(e) = std::thread::Builder::new()
.name("tp-watchdog-deadman".into())
.spawn(move || {
if wait_for_abort(&done, deadline) {
return; // abort came back; the normal path owns recovery
}
exit_unrecoverable(
&owned_model,
"ncclCommAbort did not return within the deadline",
);
})
{
tracing::error!(model = %model_id, error = %e, "tp watchdog: deadman spawn failed");
"tp watchdog: no cached leader comm handle — cannot abort; recovery will rely \
on a process restart"
),
}
}
@@ -1570,73 +1442,3 @@ impl WorkerPool {
&self.exe
}
}
#[cfg(test)]
mod watchdog_tests {
use super::*;
use std::time::Duration;
/// An abort that returns disarms the deadman — the normal path then
/// owns recovery and the process must NOT exit.
#[test]
fn abort_returning_disarms_the_deadman() {
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
signal.store(true, Ordering::Release);
});
assert!(
wait_for_abort(&done, Duration::from_secs(5)),
"a returning abort must disarm the deadman"
);
}
/// The #265 case: `ncclCommAbort` never returns. The deadline must
/// elapse and report unrecoverable, which is what triggers the exit.
#[test]
fn abort_that_never_returns_trips_the_deadline() {
let done = Arc::new(AtomicBool::new(false)); // never set
let start = std::time::Instant::now();
assert!(
!wait_for_abort(&done, Duration::from_millis(200)),
"an abort that never returns must trip the deadline"
);
assert!(
start.elapsed() >= Duration::from_millis(200),
"must actually wait the deadline rather than failing fast"
);
}
/// An abort landing in the last instant still counts as returned —
/// exiting on a race would restart a process that had just recovered.
#[test]
fn abort_landing_at_the_deadline_still_counts() {
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(180));
signal.store(true, Ordering::Release);
});
assert!(
wait_for_abort(&done, Duration::from_millis(200)),
"an abort landing at the deadline must not be treated as a hang"
);
}
/// The deadline is operator-tunable but never zero — a zero deadline
/// would exit the moment any watchdog fired.
#[test]
fn deadline_defaults_sanely_and_rejects_zero() {
assert_eq!(tp_abort_deadline(), Duration::from_secs(60));
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "0") };
assert_eq!(
tp_abort_deadline(),
Duration::from_secs(60),
"zero must fall back, not exit instantly on every watchdog"
);
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "15") };
assert_eq!(tp_abort_deadline(), Duration::from_secs(15));
unsafe { std::env::remove_var("NEURON_TP_ABORT_DEADLINE_S") };
}
}
+1 -1
View File
@@ -2445,7 +2445,7 @@ mod tests {
}
#[cfg(not(feature = "cuda"))]
fn vb_over(path: &std::path::Path) -> ShardedVarBuilder<'_> {
fn vb_over(path: &std::path::Path) -> ShardedVarBuilder {
// SAFETY: mmap of a file the test just wrote; nothing mutates it.
unsafe {
candle_nn::var_builder::ShardedSafeTensors::var_builder(
+1 -4
View File
@@ -186,9 +186,6 @@ async fn daemon(args: Args) -> Result<()> {
let registry = HarnessRegistry::from_configs(&cfg.harnesses, &bind_url, &cfg.harness);
discovery_result.harnesses = registry.names();
let candle = registry.candle();
// A second handle for the shutdown future: it must close admission
// before the drain begins, and `candle` is moved into NeuronState.
let shutdown_candle = candle.clone();
let health_cache = Arc::new(health::HealthCache::new());
health_cache
@@ -249,7 +246,7 @@ async fn daemon(args: Args) -> Result<()> {
}
axum::serve(listener, app)
.with_graceful_shutdown(startup::shutdown_signal(shutdown_candle))
.with_graceful_shutdown(startup::shutdown_signal())
.await?;
// Deactivation: serve has returned (graceful shutdown signal
+1 -20
View File
@@ -9,7 +9,6 @@ use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::preflight::PreflightError;
use cortex_core::harness::ModelSpec;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::signal;
@@ -166,18 +165,7 @@ fn preflight_kind(err: &PreflightError) -> &'static str {
/// Wired into `axum::serve(...).with_graceful_shutdown(shutdown_signal())`
/// so the HTTP listener stops accepting new connections, lets in-flight
/// requests drain, and then yields control back to main for cleanup.
///
/// **Before returning, it closes admission and wakes every waiter (#256).**
/// Order matters: axum's graceful shutdown waits for *all* in-flight
/// requests, and a request queued behind the TP pool lock is in-flight as
/// far as axum is concerned. Those waits are unbounded — held by whichever
/// request is mid-forward — so without this the drain cannot finish and
/// systemd SIGKILLs at `TimeoutStopSec`. Observed 2026-08-15: SIGTERM at
/// 13:03:14, one queued request logging "still waiting on pool lock" every
/// two seconds, `Failed with result 'timeout'` at 13:05:24. The same node
/// drained in about a second when idle, which is what identified the
/// queued waiter as the cause.
pub async fn shutdown_signal(candle: Option<Arc<crate::harness::candle::CandleHarness>>) {
pub async fn shutdown_signal() {
let ctrl_c = async {
signal::ctrl_c().await.ok();
};
@@ -191,13 +179,6 @@ pub async fn shutdown_signal(candle: Option<Arc<crate::harness::candle::CandleHa
_ = ctrl_c => tracing::info!("received SIGINT, shutting down"),
_ = terminate => tracing::info!("received SIGTERM, shutting down"),
}
// Refuse new admissions and abandon the unbounded waits, so the drain
// that starts the moment this future resolves has a bounded amount of
// work left to wait for.
crate::harness::candle::begin_shutdown();
if let Some(candle) = candle {
candle.close_all_admission().await;
}
}
/// Unload every model currently registered. Called from `main.rs` after
-7
View File
@@ -84,13 +84,6 @@ pub enum InferenceEvent {
/// `output_tokens_details.reasoning_tokens` (responses).
/// Zero for non-reasoning models.
reasoning_tokens: u32,
/// Leading prompt tokens served from the prefix cache (#269) — a
/// sub-count of `prompt_tokens`, never added to `total_tokens`.
/// Surfaced as `prompt_tokens_details.cached_tokens` (chat) /
/// `input_tokens_details.cached_tokens` (responses). The cache
/// has worked since #11; without this the saving was invisible
/// and every client reported a 0% hit rate.
cached_tokens: u32,
/// Server-measured prefill/decode timing for the request, or
/// `None` on paths that don't measure it (CPU fallback that
/// doesn't instrument, tests). Streaming projectors surface
+21 -581
View File
@@ -27,8 +27,7 @@
//! propagate without us writing any logic.
use cortex_core::openai::{
ChatCompletionChoice, ChatCompletionChunk, ChatCompletionResponse, ChatMessage, ChunkChoice,
CompletionTokensDetails, HelexaTiming, MessageContent, PromptTokensDetails, Usage,
ChatCompletionChunk, ChunkChoice, CompletionTokensDetails, HelexaTiming, Usage,
};
use serde_json::json;
use tokio::sync::mpsc;
@@ -140,35 +139,11 @@ pub fn project_chat_stream_with(
}
InferenceEvent::ReasoningDelta(text) => {
if !config.include_thinking {
// Default path. This used to `continue` —
// dropping the token and emitting nothing at
// all — to keep model-internal scratchpad out
// of naïve clients' UIs. That goal was right;
// the method silenced the stream.
//
// A reasoning model can think for minutes on a
// hard prompt, and for that whole span not one
// SSE event went out. Clients time that out: the
// DeepSeek Harness reports `pi-ai stream idle
// timeout after 300000ms` and retries, so a task
// that reasons for over five minutes could never
// complete no matter how healthy the model was.
// SSE comment keep-alives do not help — idle
// timers reset on events, not comments.
//
// Emitting `reasoning_content` instead keeps the
// original guarantee (it is not `content`, so
// unaware clients still show clean output) while
// the stream stays visibly alive, and hands
// reasoning-aware clients the field they are
// already looking for.
if text.is_empty() {
continue;
}
let chunk = reasoning_chunk(&id, created, &model_id, &text);
if tx.send(chunk).await.is_err() {
return;
}
// Default path — reasoning has no slot in
// chat completions, so it's dropped. Naïve
// clients (Zed commit-message generator,
// any vanilla OpenAI client) get clean
// output.
continue;
}
let Some(markers) = config.reasoning_markers.as_ref() else {
@@ -220,7 +195,6 @@ pub fn project_chat_stream_with(
prompt_tokens,
completion_tokens,
reasoning_tokens,
cached_tokens,
timing,
} => {
// The finish_reason chunk, then an OpenAI-style
@@ -235,12 +209,9 @@ pub fn project_chat_stream_with(
&id,
created,
&model_id,
UsageCounts {
prompt_tokens,
completion_tokens,
reasoning_tokens,
cached_tokens,
},
prompt_tokens,
completion_tokens,
reasoning_tokens,
timing,
),
]
@@ -275,35 +246,6 @@ fn role_chunk(id: &str, created: u64, model_id: &str) -> ChatCompletionChunk {
}
}
/// A reasoning delta, in the field reasoning-aware clients read.
///
/// `reasoning_content` is not in the OpenAI spec, but it is the
/// de-facto slot: DeepSeek's API emits it, and so do vLLM and SGLang
/// for reasoning models. Our own consumers already expect it —
/// `helexa-acp`'s OpenAI provider reads `choice.delta.reasoning_content`,
/// and `helexa-bench` times a stream's first reasoning token from it.
///
/// Crucially it is NOT `content`, so a client that has never heard of
/// reasoning ignores the unknown field and sees clean output — the
/// property the old drop-it-entirely default was protecting, kept
/// without the silence that came with it.
fn reasoning_chunk(id: &str, created: u64, model_id: &str, text: &str) -> ChatCompletionChunk {
ChatCompletionChunk {
id: id.into(),
object: "chat.completion.chunk".into(),
created,
model: model_id.into(),
choices: vec![ChunkChoice {
index: 0,
delta: json!({ "reasoning_content": text }),
finish_reason: None,
extra: serde_json::Value::Object(Default::default()),
}],
usage: None,
extra: serde_json::Value::Object(Default::default()),
}
}
fn content_chunk(id: &str, created: u64, model_id: &str, text: &str) -> ChatCompletionChunk {
ChatCompletionChunk {
id: id.into(),
@@ -390,32 +332,15 @@ fn final_chunk(
/// ignore the empty-choices chunk; clients that do (opencode, and
/// cortex's Anthropic translator) get the token counts they need to
/// track context.
/// The token counts a finished turn reports. Grouped rather than passed
/// loose: adding `cached_tokens` (#269) put the list past the point
/// where four adjacent bare `u32`s are safe to read at a call site.
struct UsageCounts {
prompt_tokens: u32,
completion_tokens: u32,
/// Sub-count of `completion_tokens` — tokens spent reasoning.
reasoning_tokens: u32,
/// Sub-count of `prompt_tokens` — tokens served from the prefix
/// cache. Neither sub-count is added into `total_tokens`.
cached_tokens: u32,
}
fn usage_chunk(
id: &str,
created: u64,
model_id: &str,
counts: UsageCounts,
prompt_tokens: u32,
completion_tokens: u32,
reasoning_tokens: u32,
timing: Option<FinishTiming>,
) -> ChatCompletionChunk {
let UsageCounts {
prompt_tokens,
completion_tokens,
reasoning_tokens,
cached_tokens,
} = counts;
ChatCompletionChunk {
id: id.into(),
object: "chat.completion.chunk".into(),
@@ -431,12 +356,7 @@ fn usage_chunk(
completion_tokens_details: (reasoning_tokens > 0).then_some(CompletionTokensDetails {
reasoning_tokens: reasoning_tokens as u64,
}),
// Prefix-cache reuse (#269). Omitted at zero so a client
// sees unchanged JSON when nothing was cached, and so
// "absent" stays distinguishable from "measured as none".
prompt_tokens_details: (cached_tokens > 0).then_some(PromptTokensDetails {
cached_tokens: cached_tokens as u64,
}),
prompt_tokens_details: None,
// helexa extension (#85): server-measured prefill/decode
// timing for the bench harness. Omitted on paths that don't
// measure it so standard clients see unchanged JSON.
@@ -450,379 +370,10 @@ fn usage_chunk(
}
}
/// Drain an [`InferenceEvent`] stream into a single non-streaming
/// [`ChatCompletionResponse`] (#285).
///
/// Non-streaming used to be a *separate inference path* that took the
/// per-model `inference_lock` for the whole request. That made it
/// invisible to the batch engine (#98), so concurrent non-streaming
/// callers serialized completely — measured on beast at 1.00x aggregate
/// throughput across 1/2/4/8 concurrency, against 3.98x for the same
/// work streamed. `max_in_flight` bought those requests admission and
/// nothing else.
///
/// Expressing non-streaming as "run the stream, buffer the result"
/// collapses the two paths onto one producer, so batching, prefix-cache
/// reuse, tool-call extraction, the reasoning governor and server-side
/// timing all apply identically to both. It also removes a second place
/// where the same wire object could be assembled differently — the
/// shape of defect #252 and #272 both were.
///
/// Reasoning is kept out of `content` — it is not the assistant's
/// reply — but it is **returned** on `message.reasoning_content`, the
/// same field the streaming path emits as
/// `choice.delta.reasoning_content`.
///
/// It used to be discarded here (#300). That was inherited from the
/// path this replaced and was wrong on both counts: the tokens are
/// metered in `usage.completion_tokens_details.reasoning_tokens`, so a
/// caller paid for text it could never see; and a client cannot replay
/// what it never received, so on the next turn the model saw its own
/// prior turn as a bare assertion with the reasoning that produced it
/// deleted.
///
/// `finish_reason` becomes `"tool_calls"` whenever any tool call was
/// parsed.
///
/// Two fields that the old path hardcoded to `None` now populate,
/// because the events carry them and the struct already had the slots:
/// `prompt_tokens_details.cached_tokens` and `helexa_timing`.
///
/// A stream that ends without a `Finish` is an error, not an empty
/// reply: that is how a dropped engine slot or a fatal worker fault
/// presents, and returning a 200 with partial text would hide it.
pub async fn collect_chat_completion(
mut rx: mpsc::Receiver<InferenceEvent>,
id: String,
created: u64,
model_id: String,
) -> Result<ChatCompletionResponse, String> {
let mut content = String::new();
let mut reasoning = String::new();
let mut tool_calls: Vec<serde_json::Value> = Vec::new();
let mut finish: Option<(FinishReason, u32, u32, u32, u32, Option<FinishTiming>)> = None;
while let Some(event) = rx.recv().await {
match event {
InferenceEvent::Start => {}
InferenceEvent::TextDelta(t) => content.push_str(&t),
// Kept out of `content`, returned on `reasoning_content`.
InferenceEvent::ReasoningDelta(t) => reasoning.push_str(&t),
InferenceEvent::ToolCall {
index,
id: call_id,
name,
arguments,
} => tool_calls.push(json!({
"index": index,
"id": call_id,
"type": "function",
"function": { "name": name, "arguments": arguments },
})),
InferenceEvent::Finish {
reason,
prompt_tokens,
completion_tokens,
reasoning_tokens,
cached_tokens,
timing,
} => {
finish = Some((
reason,
prompt_tokens,
completion_tokens,
reasoning_tokens,
cached_tokens,
timing,
));
break;
}
}
}
let Some((reason, prompt_tokens, completion_tokens, reasoning_tokens, cached_tokens, timing)) =
finish
else {
return Err(
"inference stream ended without a finish event (engine slot dropped?)".to_string(),
);
};
// A parsed tool call outranks whatever the sampler stopped on, so a
// model that emits a call and then hits EOS still reports
// `tool_calls` and the client dispatches instead of rendering.
let finish_reason = if tool_calls.is_empty() {
reason.as_openai_str().to_string()
} else {
"tool_calls".to_string()
};
let mut message_extra = serde_json::Map::new();
if !tool_calls.is_empty() {
message_extra.insert("tool_calls".into(), json!(tool_calls));
}
// Absent rather than empty when the model did not think: a
// non-reasoning model must keep the unchanged message shape.
if !reasoning.is_empty() {
message_extra.insert("reasoning_content".into(), json!(reasoning));
}
let message_extra = serde_json::Value::Object(message_extra);
Ok(ChatCompletionResponse {
id,
object: "chat.completion".into(),
created,
model: model_id,
choices: vec![ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text(content),
extra: message_extra,
},
finish_reason: Some(finish_reason),
extra: serde_json::Value::Object(Default::default()),
}],
usage: Some(Usage {
prompt_tokens: prompt_tokens as u64,
completion_tokens: completion_tokens as u64,
total_tokens: (prompt_tokens + completion_tokens) as u64,
completion_tokens_details: (reasoning_tokens > 0).then_some(CompletionTokensDetails {
reasoning_tokens: reasoning_tokens as u64,
}),
prompt_tokens_details: (cached_tokens > 0).then_some(PromptTokensDetails {
cached_tokens: cached_tokens as u64,
}),
helexa_timing: timing.map(|t| HelexaTiming {
prefill_ms: t.prefill_ms as u64,
decode_ms: t.decode_ms as u64,
prefill_tokens: t.prefill_tokens as u64,
}),
}),
extra: serde_json::Value::Object(Default::default()),
})
}
#[cfg(test)]
mod tests {
/// The model's think block must come back on the non-streaming
/// path too (#300).
///
/// It used to be dropped here while the streaming path emitted it
/// as `choice.delta.reasoning_content`. The tokens were metered
/// either way, so a non-streaming caller paid for text it could
/// never see — and, worse, could not replay the model's reasoning
/// on the next turn, leaving the model to face its own prior turn
/// as a bare assertion.
#[tokio::test]
async fn non_streaming_returns_the_models_reasoning() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::ReasoningDelta(
"weighing the options".into(),
))
.await
.unwrap();
tx.send(InferenceEvent::TextDelta("the answer".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 5,
completion_tokens: 7,
reasoning_tokens: 3,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
let msg = &r.choices[0].message;
assert_eq!(
msg.extra["reasoning_content"], "weighing the options",
"reasoning must be returned, not discarded"
);
// And it must NOT be folded into the visible reply.
assert!(
matches!(&msg.content, MessageContent::Text(t) if t == "the answer"),
"reasoning leaked into content"
);
}
/// A non-reasoning model must keep the unchanged message shape —
/// absent, not an empty string.
#[tokio::test]
async fn no_reasoning_means_no_reasoning_field() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::TextDelta("plain".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 1,
completion_tokens: 1,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
assert!(
r.choices[0]
.message
.extra
.get("reasoning_content")
.is_none()
);
}
use super::*;
// ── Non-streaming collector (#285) ────────────────────────────
async fn collect_from(events: Vec<InferenceEvent>) -> Result<ChatCompletionResponse, String> {
let (tx, rx) = mpsc::channel(64);
for e in events {
tx.send(e).await.expect("send");
}
drop(tx);
collect_chat_completion(rx, "chatcmpl-test".into(), 1_700_000_000, "m".into()).await
}
fn finish(reason: FinishReason) -> InferenceEvent {
InferenceEvent::Finish {
reason,
prompt_tokens: 10,
completion_tokens: 4,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
}
}
#[tokio::test]
async fn collector_concatenates_text_deltas() {
let r = collect_from(vec![
InferenceEvent::Start,
InferenceEvent::TextDelta("hel".into()),
InferenceEvent::TextDelta("lo".into()),
finish(FinishReason::Stop),
])
.await
.expect("collected");
let c = &r.choices[0];
assert!(matches!(&c.message.content, MessageContent::Text(t) if t == "hello"));
assert_eq!(c.finish_reason.as_deref(), Some("stop"));
assert_eq!(r.object, "chat.completion");
}
/// Reasoning is counted but never rendered — the behaviour of the
/// path this replaced. A regression here would leak scratchpad into
/// `content` for every non-streaming caller.
#[tokio::test]
async fn collector_drops_reasoning_text_but_keeps_the_count() {
let (tx, rx) = mpsc::channel(64);
tx.send(InferenceEvent::ReasoningDelta("secret".into()))
.await
.expect("send");
tx.send(InferenceEvent::TextDelta("answer".into()))
.await
.expect("send");
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 10,
completion_tokens: 9,
reasoning_tokens: 5,
cached_tokens: 0,
timing: None,
})
.await
.expect("send");
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 1, "m".into())
.await
.expect("collected");
let c = &r.choices[0];
assert!(matches!(&c.message.content, MessageContent::Text(t) if t == "answer"));
let u = r.usage.expect("usage");
assert_eq!(
u.completion_tokens_details
.expect("details")
.reasoning_tokens,
5
);
}
/// A parsed tool call outranks the sampler's stop reason, matching
/// the streaming projector.
#[tokio::test]
async fn collector_promotes_finish_reason_to_tool_calls() {
let r = collect_from(vec![
InferenceEvent::ToolCall {
index: 0,
id: "call_1".into(),
name: "read_file".into(),
arguments: r#"{"path":"a.txt"}"#.into(),
},
finish(FinishReason::Stop),
])
.await
.expect("collected");
let c = &r.choices[0];
assert_eq!(c.finish_reason.as_deref(), Some("tool_calls"));
let calls = c.message.extra.get("tool_calls").expect("tool_calls");
assert_eq!(calls[0]["function"]["name"], "read_file");
assert_eq!(calls[0]["type"], "function");
assert_eq!(calls[0]["id"], "call_1");
}
/// cached_tokens and helexa_timing were hardcoded to `None` on the
/// old non-streaming path; the events carried them all along.
#[tokio::test]
async fn collector_surfaces_cached_tokens_and_timing() {
let (tx, rx) = mpsc::channel(8);
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 100,
completion_tokens: 5,
reasoning_tokens: 0,
cached_tokens: 64,
timing: Some(FinishTiming {
prefill_ms: 12,
decode_ms: 34,
prefill_tokens: 100,
}),
})
.await
.expect("send");
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 1, "m".into())
.await
.expect("collected");
let u = r.usage.expect("usage");
assert_eq!(u.prompt_tokens_details.expect("details").cached_tokens, 64);
let t = u.helexa_timing.expect("timing");
assert_eq!((t.prefill_ms, t.decode_ms, t.prefill_tokens), (12, 34, 100));
}
/// A dropped engine slot ends the channel with no Finish. Returning
/// a 200 with partial text would hide a fatal worker fault.
#[tokio::test]
async fn collector_errors_when_the_stream_ends_without_finish() {
let err = collect_from(vec![InferenceEvent::TextDelta("partial".into())])
.await
.expect_err("must not succeed");
assert!(err.contains("without a finish"), "unexpected: {err}");
}
/// Drain the projection's output into a Vec for assertion.
async fn collect(mut rx: mpsc::Receiver<ChatCompletionChunk>) -> Vec<ChatCompletionChunk> {
let mut out = Vec::new();
@@ -854,7 +405,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -893,7 +443,6 @@ mod tests {
prompt_tokens: 128,
completion_tokens: 64,
reasoning_tokens: 0,
cached_tokens: 0,
timing: Some(FinishTiming {
prefill_ms: 200,
decode_ms: 1500,
@@ -918,73 +467,6 @@ mod tests {
assert_eq!(timing.prefill_tokens, 128);
}
/// A cache hit must reach the caller (#269).
///
/// Prefix caching has worked since #11 — neuron logs `reused=2068`
/// on prefills — but nothing carried the number onto the wire, so
/// every client reported a 0% hit rate on a cache that was saving
/// real prefill. dsh showed `Cache hit 0%` against a 42.1K-token
/// context, and an agentic harness that believes nothing is cached
/// may restructure the very prefix that was being reused.
#[tokio::test]
async fn cached_tokens_reach_the_usage_chunk() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream(rx, "id".into(), 1, "m".into());
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 7413,
completion_tokens: 12,
reasoning_tokens: 0,
cached_tokens: 2068,
timing: None,
})
.await
.unwrap();
drop(tx);
let out = collect(out_rx).await;
let usage = out
.iter()
.find_map(|c| c.usage.as_ref())
.expect("usage chunk present");
let details = usage
.prompt_tokens_details
.as_ref()
.expect("prompt_tokens_details populated when the cache was hit");
assert_eq!(details.cached_tokens, 2068);
// A sub-count of prompt_tokens, never added into the total —
// otherwise a cache hit would inflate what a caller is billed.
assert_eq!(usage.prompt_tokens, 7413);
assert_eq!(usage.total_tokens, 7425);
}
/// No reuse means no `prompt_tokens_details` at all, so a client
/// sees byte-identical JSON to before this landed — and "absent"
/// stays distinguishable from "measured as zero".
#[tokio::test]
async fn no_cache_hit_omits_the_details_object() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream(rx, "id".into(), 1, "m".into());
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 10,
completion_tokens: 1,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let usage = collect(out_rx)
.await
.iter()
.find_map(|c| c.usage.clone())
.expect("usage chunk present");
assert!(usage.prompt_tokens_details.is_none());
}
#[tokio::test]
async fn empty_text_delta_is_dropped() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
@@ -1006,7 +488,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -1019,24 +500,10 @@ mod tests {
}
#[tokio::test]
/// Reasoning is streamed as `reasoning_content`, never as `content`.
///
/// Both halves matter. Keeping it out of `content` is what stops
/// scratchpad material appearing in the UI of a client that has
/// never heard of reasoning — the guarantee the previous
/// drop-everything default provided.
///
/// But emitting *something* is what keeps the connection alive. A
/// model can think for minutes; while it did, this projection sent
/// no events at all, and clients closed the stream as dead. The
/// DeepSeek Harness reports `pi-ai stream idle timeout after
/// 300000ms`, so any task whose reasoning outlasts the client's
/// idle timeout could never complete. SSE comment keep-alives do
/// not help, because idle timers reset on events, not comments.
async fn reasoning_delta_is_streamed_as_reasoning_content() {
async fn reasoning_delta_is_dropped_in_chat_projection() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream(rx, "id".into(), 1, "m".into());
tx.send(InferenceEvent::ReasoningDelta("pondering".into()))
tx.send(InferenceEvent::ReasoningDelta("<think>".into()))
.await
.unwrap();
tx.send(InferenceEvent::TextDelta("real".into()))
@@ -1044,27 +511,8 @@ mod tests {
.unwrap();
drop(tx);
let out = collect(out_rx).await;
assert_eq!(out.len(), 2, "reasoning must produce a chunk, not silence");
assert_eq!(out[0].choices[0].delta["reasoning_content"], "pondering");
assert!(
out[0].choices[0].delta.get("content").is_none(),
"reasoning must never arrive as content"
);
assert_eq!(out[1].choices[0].delta["content"], "real");
}
/// An empty reasoning delta is still nothing to say — it must not
/// manufacture a chunk, matching the text path.
#[tokio::test]
async fn empty_reasoning_delta_is_dropped() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream(rx, "id".into(), 1, "m".into());
tx.send(InferenceEvent::ReasoningDelta(String::new()))
.await
.unwrap();
drop(tx);
let out = collect(out_rx).await;
assert!(out.is_empty(), "empty reasoning deltas produce no chunks");
assert_eq!(out.len(), 1);
assert_eq!(out[0].choices[0].delta["content"], "real");
}
fn pair() -> ReasoningTokenPair {
@@ -1103,7 +551,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -1156,7 +603,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -1203,7 +649,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -1222,12 +667,10 @@ mod tests {
}
#[tokio::test]
/// With the flag off, reasoning stays out of `content` whether or
/// not markers are configured — the flag is the gate, not the
/// marker presence. It now travels as `reasoning_content` instead
/// of being discarded, so this asserts the containment guarantee
/// only: nothing model-internal reaches a naive client's `content`.
async fn include_thinking_off_keeps_reasoning_out_of_content() {
async fn include_thinking_off_drops_reasoning_even_with_markers() {
// Default behaviour even when markers happen to be
// configured. The flag is the gate, not the marker
// presence.
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream_with(
rx,
@@ -1250,7 +693,6 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
@@ -1279,7 +721,6 @@ mod tests {
prompt_tokens: 42,
completion_tokens: 5,
reasoning_tokens: 2,
cached_tokens: 0,
timing: None,
})
.await
@@ -1314,7 +755,6 @@ mod tests {
prompt_tokens: 10,
completion_tokens: 7,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
File diff suppressed because it is too large Load Diff
-8
View File
@@ -29,8 +29,6 @@ async fn test_load_default_models_skips_unknown_harness() {
quant: None,
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
},
ModelSpec {
model_id: "model-b".into(),
@@ -38,8 +36,6 @@ async fn test_load_default_models_skips_unknown_harness() {
quant: None,
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
},
];
@@ -99,8 +95,6 @@ async fn test_load_default_models_skipped_on_driver_mismatch() {
quant: Some("q6k".into()),
tensor_parallel: Some(2),
devices: None,
sampling: None,
preserve_thinking: None,
}];
let activation = ActivationTracker::new(&specs);
let reason = "host NVIDIA driver/library mismatch (userspace NVML 580.159 vs loaded \
@@ -202,8 +196,6 @@ fn qwen_spec() -> ModelSpec {
quant: None,
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
-2
View File
@@ -53,8 +53,6 @@ async fn test_candle_qwen3_load_unload_lifecycle() {
quant: Some(quant),
tensor_parallel: None,
devices: Some(vec![0]),
sampling: None,
preserve_thinking: None,
};
registry
@@ -1,170 +0,0 @@
{%- set image_count = namespace(value=0) %}
{%- set video_count = namespace(value=0) %}
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
{%- if content is string %}
{{- content }}
{%- elif content is iterable and content is not mapping %}
{%- for item in content %}
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
{%- if is_system_content %}
{{- raise_exception('System message cannot contain images.') }}
{%- endif %}
{%- if do_vision_count %}
{%- set image_count.value = image_count.value + 1 %}
{%- endif %}
{%- if add_vision_id %}
{{- 'Picture ' ~ image_count.value ~ ': ' }}
{%- endif %}
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
{%- elif 'video' in item or item.type == 'video' %}
{%- if is_system_content %}
{{- raise_exception('System message cannot contain videos.') }}
{%- endif %}
{%- if do_vision_count %}
{%- set video_count.value = video_count.value + 1 %}
{%- endif %}
{%- if add_vision_id %}
{{- 'Video ' ~ video_count.value ~ ': ' }}
{%- endif %}
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
{%- elif 'text' in item %}
{{- item.text }}
{%- else %}
{{- raise_exception('Unexpected item type in content.') }}
{%- endif %}
{%- endfor %}
{%- elif content is none or content is undefined %}
{{- '' }}
{%- else %}
{{- raise_exception('Unexpected content type.') }}
{%- endif %}
{%- endmacro %}
{%- if not messages %}
{{- raise_exception('No messages provided.') }}
{%- endif %}
{%- set reasoning_instructions = '' %}
{%- if enable_thinking is undefined or enable_thinking is true %}
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
{%- endif %}
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif resolved_reasoning_effort == 'low' %}
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
{%- endif %}
{%- endif %}
{%- if tools and tools is iterable and tools is not mapping %}
{{- '<|im_start|>system\n' }}
{%- if reasoning_instructions %}
{{- reasoning_instructions + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>" }}
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
{%- if messages[0].role == 'system' %}
{%- set content = render_content(messages[0].content, false, true)|trim %}
{%- if content %}
{{- '\n\n' + content }}
{%- endif %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- else %}
{%- if messages[0].role == 'system' %}
{%- set content = render_content(messages[0].content, false, true)|trim %}
{%- if content %}
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + content + '<|im_end|>\n' }}
{%- elif reasoning_instructions %}
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
{%- endif %}
{%- elif reasoning_instructions %}
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" %}
{%- set content = render_content(message.content, false)|trim %}
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if ns.multi_step_tool %}
{{- raise_exception('No user query found in messages.') }}
{%- endif %}
{%- for message in messages %}
{%- set content = render_content(message.content, true)|trim %}
{%- if message.role == "system" %}
{%- if not loop.first %}
{{- raise_exception('System message must be at the beginning.') }}
{%- endif %}
{%- elif message.role == "user" %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- endif %}
{%- set reasoning_content = reasoning_content|trim %}
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{%- if loop.first %}
{%- if content|trim %}
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
{%- else %}
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
{%- endif %}
{%- else %}
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
{%- endif %}
{%- if tool_call.arguments is defined and tool_call.arguments != '' %}
{%- for args_name, args_value in tool_call.arguments|items %}
{{- '<parameter=' + args_name + '>\n' }}
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
{{- args_value }}
{{- '\n</parameter>\n' }}
{%- endfor %}
{%- endif %}
{{- '</function>\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.previtem and loop.previtem.role != "tool" %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content }}
{{- '\n</tool_response>' }}
{%- if not loop.last and loop.nextitem.role != "tool" %}
{{- '<|im_end|>\n' }}
{%- elif loop.last %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- else %}
{{- raise_exception('Unexpected message role.') }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- else %}
{{- '<think>\n' }}
{%- endif %}
{%- endif %}
+1 -3
View File
@@ -60,9 +60,7 @@ fn read_f32(path: &Path) -> Vec<f32> {
let bytes = std::fs::read(path).unwrap_or_else(|e| panic!("read {path:?}: {e}"));
assert!(bytes.len().is_multiple_of(4), "truncated f32 file {path:?}");
bytes
.as_chunks::<4>()
.0
.iter()
.chunks_exact(4)
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]]))
.collect()
}
-2
View File
@@ -106,8 +106,6 @@ fn spec(model_id: &str, tp: Option<u32>, quant: Option<&str>) -> ModelSpec {
quant: quant.map(String::from),
tensor_parallel: tp,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
+1 -3
View File
@@ -37,9 +37,7 @@ fn fixture_dir() -> PathBuf {
fn read_f32(path: &Path) -> Vec<f32> {
let bytes = std::fs::read(path).unwrap_or_else(|e| panic!("read {path:?}: {e}"));
bytes
.as_chunks::<4>()
.0
.iter()
.chunks_exact(4)
.map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
.collect()
}
-21
View File
@@ -2,22 +2,6 @@
Description=Neuron — per-node GPU discovery and harness daemon for cortex
After=network-online.target
Wants=network-online.target
# Start-limit bound for the self-restart path (#265). When a TP forward
# wedges and the NCCL abort that would unblock it also hangs, neuron
# exits 70 so Restart=on-failure recovers it in seconds instead of the
# 70-minute outage that needed a human on 2026-08-17.
#
# That only helps if the wedge is transient. A deterministic one — a bad
# build, a sick card, a model that wedges on load — would otherwise
# restart forever, and a flapping node is WORSE than a down one: cortex
# marks it healthy between restarts and keeps routing to it, so every
# client sees intermittent failures instead of one node cleanly out.
#
# Four failures in ten minutes stops the unit and leaves it `failed`,
# where `systemctl status` and an alert can see it. Visibly broken beats
# invisibly cycling.
StartLimitIntervalSec=600
StartLimitBurst=4
[Service]
Type=simple
@@ -40,11 +24,6 @@ TimeoutStartSec=1800s
# On stop, neuron drains in-flight requests then unloads every model
# to release CUDA contexts cleanly. Allow generous time for big-model
# unloads; systemd will SIGKILL after this bound.
#
# Note this bound does NOT apply to the self-restart path (#265): a
# wedged process exits directly rather than draining, precisely because
# draining cannot work when the collective is stuck — and waiting this
# out would add two minutes to an outage that is already total.
TimeoutStopSec=120s
KillSignal=SIGTERM
+1 -1
View File
@@ -35,7 +35,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.8-27B"
default_model = "Qwen/Qwen3.6-27B"
max_tokens = 8192
# Compaction kicks in when the rolling history grows past this token
# budget. Set to your model's context window. Disable by removing
+1 -1
View File
@@ -23,7 +23,7 @@ listen = "0.0.0.0:8088"
#
# [aliases]
# "helexa/small" = "Qwen/Qwen3-8B"
# "helexa/balanced" = "Qwen/Qwen3.8-27B"
# "helexa/balanced" = "Qwen/Qwen3.6-27B"
# -- Downstream cortexes -------------------------------------------------
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
-37
View File
@@ -1,37 +0,0 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on gallumbits are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# helexa-router — federation data plane on gallumbits.
#
# Tracked despite living beside the secret-bearing service configs,
# because it holds no secrets: a listen address, the product-tier
# aliases, and a mesh endpoint. Being untracked cost an outage on
# 2026-08-27 — `helexa/balanced` still resolved to Qwen/Qwen3.6-27B
# after that model had been retired from the catalogue, so every
# authenticated web-chat turn evicted beast's resident flagship and
# cold-loaded a model nobody had asked for. Nothing could diff the two
# files, so nothing did.
#
# `script/check-alias-targets.py` now fails CI if any alias below names
# a model that models.toml does not define.
[router]
listen = "0.0.0.0:8088"
poll_interval_secs = 10
region = "eu-central"
# Product tiers → real model ids (#166). The SPA sends the tier names.
[aliases]
"helexa/small" = "Qwen/Qwen3-8B"
"helexa/balanced" = "Qwen/Qwen3.8-27B"
# hanzalova operator cortex, plaintext over the WireGuard mesh (the mesh
# is the trust boundary; no TLS pin needed for http://).
[[cortexes]]
name = "hanzalova"
endpoint = "http://hanzalova.internal:31313"
region = "eu-central"
@@ -19,7 +19,7 @@ gateway.
```
┌──────────────┐
clients ────▶│ cortex │ :31313 API, :31314 metrics
clients ─────▶│ cortex │ :31313 API, :31314 metrics
└──┬────┬────┬─┘
│ │ │
┌────▼┐ ┌─▼──┐ ┌▼────┐
@@ -44,70 +44,6 @@ change.
VRAM fragmentation has had time to accumulate. It is a signal to restart
that neuron, not an automatic action.
## secrets.toml
Credentials do not belong in the file CI deploys. Put them in a
`secrets.toml` beside the main config and the service will merge it
last, so its values win:
```
/etc/cortex/cortex.toml structure — deployed by CI, safe to diff
/etc/cortex/secrets.toml credentials — yours, root-only, never in git
```
```toml
# /etc/cortex/secrets.toml (root:cortex 0640)
[[entitlements.keys]]
key = "sk-…"
account_id = "acct-a"
key_id = "a"
[upstream]
bearer = "…"
```
`helexa-upstream` and `helexa-angels` read a `secrets.toml` beside their
own configs the same way — the DB URL, JWT secret and SMTP URL belong
there.
**An absent `secrets.toml` is not an error.** A single-file deployment
keeps working exactly as before, so this is opt-in per host and can be
adopted one service at a time.
### Why bother
Because a config CI cannot write is a config no check can defend, and
the cost of that is not hypothetical. `helexa-router.toml` was kept out
of git for sitting beside secret-bearing files. Nothing could diff it
against the catalogue, so its `helexa/balanced` alias went on pointing
at a model that had been retired — and since both models were
deliberately given equal residency priority so they could displace each
other, every authenticated chat turn evicted the resident flagship to
cold-load the retired one. Four of those overlapped and the node ended
up holding 32 GB of GPU memory with no model loaded and nothing
serving.
Splitting secrets from structure is what lets the structure be
version-controlled, reviewed and checked. The ordering matters too: the
operator's file merges last, so a deploy can never replace a live
credential with a placeholder.
### Moving existing secrets
One-time, per host, and reversible — the service reads both files, so
you can move a value and restart before moving the next:
1. Create `secrets.toml` beside the config, owned `root:<service>` mode
`0640` (`0600` where the service runs as root).
2. Move the credential blocks across. Leave everything else in place.
3. Restart the service and confirm it still authenticates — for cortex,
a request with a real key should be billed rather than treated as
anonymous.
4. Delete the moved lines from the main file only once step 3 passes.
Until step 4 the value exists in both files, which is safe: the
operator's copy wins.
## models.toml
The catalogue says what each model needs and where it may run:
@@ -142,7 +78,7 @@ Clients should not hardcode model ids. Aliases let them ask for a role:
[aliases]
"helexa/small" = "Qwen/Qwen3-1.7B"
"helexa/balanced" = "Qwen/Qwen3-8B"
"helexa/large" = "Qwen/Qwen3.8-27B"
"helexa/large" = "Qwen/Qwen3.6-27B"
"helexa/image" = "Tongyi-MAI/Z-Image-Turbo"
```
@@ -58,32 +58,6 @@ Two things that bite:
- **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
very different experience from the same model on NVMe.
### Where weights come from
By default a bare `org/name` id is fetched from HuggingFace. Operators
running a mirror, or serving models from more than one place, can
declare sources explicitly:
```toml
[harness.candle]
default_source = "huggingface"
[harness.candle.sources.huggingface]
endpoint = "https://huggingface.co"
auth_env = "HF_TOKEN" # env var name, never the token itself
cache_dir = "/archive/llm" # optional; per-source cache
```
`auth_env` names an environment variable rather than holding a token, so
credentials stay out of the config file — worth keeping to even when the
file is not obviously shared, because config is exactly what gets copied
between hosts.
Give a source its own `cache_dir` when two sources can serve the same
`org/name`. The cache tree is keyed on the model id alone, so sharing a
directory between sources that disagree about what `org/name` means will
collide on disk.
## Default models
Models listed as defaults load when the daemon activates, so the host
@@ -92,7 +66,7 @@ load cost:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.8-27B"
model_id = "Qwen/Qwen3.6-27B"
harness = "candle"
quant = "q6k"
tensor_parallel = 2
@@ -117,17 +91,10 @@ Each loaded model has bounded admission control:
[harness.candle.admission]
max_in_flight = 8 # concurrent requests actually running
max_queue_depth = 8 # waiting before rejection
max_wait_secs = 30 # how long a queued request waits for a slot
kv_max_wait_secs = 300 # how long it waits for enough KV budget
max_wait_secs = 30 # how long a queued request waits
max_per_principal = 2 # fair share, so one caller cannot monopolise
```
`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
The first waits for a slot to turn over; the second waits for a whole
long-context sequence to finish, which on a 27B can be minutes. A
request that needs KV budget nobody has yet is not a request to reject
in thirty seconds.
Over-capacity requests are rejected promptly with `429`/`503` and a
`Retry-After` rather than being allowed to queue invisibly. A fast
rejection a client can retry is far better than a request that hangs
@@ -135,82 +102,18 @@ until something times out.
`max_in_flight` above 1 enables batched decode on architectures that
support it, so concurrent requests share a decode step instead of
serialising.
**`max_in_flight` does not partition the KV budget.** KV is a shared
pool, and each request reserves what its own prompt actually needs when
it is admitted — which is why `kv_max_wait_secs` above exists at all. A
request larger than what is currently free waits for bytes to come back
rather than being handed a fixed 1/N slice, and a single long session
can legitimately hold most of the pool.
So the slot count and the context window are separate questions. What
bounds a busy moment is the pool: divide `kv_budget_mb` by the model's
KV cost per token (`GET /health` reports both) to see how many tokens
are available to share. Many ordinary chat turns fit at once; a few very
long agentic sessions will queue on bytes.
Size `max_in_flight` against the concurrency you actually want to serve
concurrently, not against context. Anything above it queues on admission
with a `Retry-After` rather than being refused, so setting it too low
converts capacity you have into latency your callers feel. Setting it
very high mainly costs batch-step efficiency, since the engine pads to
the widest active sequence.
Measure before assuming you need a high number — but measure the traffic
you will have, not the traffic you had. On one production host, seven
days of spans showed 89.6% of busy time at a single request in flight;
that host's chat tier was at the time being routed to a *different*
model, and the figure stopped describing it the moment that was
corrected.
serialising. Raise it on big cards, leave it at 1 on small ones — the
KV cache for several concurrent long contexts is itself substantial.
`max_per_principal` is what stops one busy client consuming the whole
model.
### Anonymous callers
Fair share is keyed on the account and key cortex resolves from the
bearer token. A caller that sends no credential — or one that does not
resolve, which `require_auth = false` deliberately tolerates — has no
key to be shared out, so `max_per_principal` cannot bind it.
Anonymous traffic is therefore served from the capacity left over once
identified traffic is satisfied. It never takes a seat while an
authenticated request is waiting for one, whoever arrived first, and it
cannot hold every seat at once:
```toml
anon_max_in_flight = 7 # unset: max_in_flight - 1, floored at 1
anon_max_pending = 15 # unset: max_in_flight + max_queue_depth - 1
```
The defaults hold back one seat and one queue place, which is what
bounds how long an authenticated request can be made to wait: a request
already running cannot be preempted, so without a reserved seat an
anonymous burst arriving during an idle moment locks the model for as
long as all of it takes to finish. On a single-seat model the floor
keeps anonymous callers served — priority alone decides who wins there.
Set `anon_max_in_flight = 0` to refuse anonymous traffic outright, or
`require_auth = true` on the gateway to make attribution universal.
Anonymous callers still contend with each other, and under sustained
authenticated load they are refused rather than served slowly — the
refusal is a retryable `503`, so a client that backs off gets served
when the load passes.
Watch it with `cortex_model_anon_in_flight` against
`cortex_model_in_flight` (how much of a model's load is unattributable)
and `cortex_model_rejections_total{reason="anon_yield"}`. That last
counter rising while `reason="wait_timeout"` stays flat means the
reservation is doing its job, not that the model is overloaded.
## VRAM guards
neuron refuses work it cannot finish rather than crashing partway:
- a **static floor** of free VRAM below which prefill is refused
(`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
`[harness.candle.context_limit]`; default 1500)
(`NEURON_MIN_FREE_VRAM_MB`, default 1500)
- a **length-aware check** that estimates the KV cache the request will
actually need and refuses if it will not fit
@@ -219,207 +122,15 @@ can pass the static floor and still be impossible, because KV grows with
prompt *and* generation length. See [context
limits](/docs/operating/context-limits) for how the numbers relate.
`activation_headroom_mb` reserves VRAM per card for prefill activations
on top of the weights and the KV cache, before the context ceiling is
computed. Raise it if prefill OOMs on a host whose steady-state
occupancy looks fine — activations are transient and do not show up in
the resident figure.
A host that cannot currently serve a model reports that on `/health`,
and cortex stops routing to it — an unservable replica is not a routing
target.
## Image models
```toml
[harness.candle.image]
te_device = "cpu" # where the text encoder runs
te_resident = true # keep it loaded between generations
max_dim = 2048 # resolution ceiling per side
```
The text encoder is the placement decision. On CPU the ~8 GB encoder
never touches the GPU — prompt encoding is one short forward and the
features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
all. Move it to `cuda` only on a card with room to spare.
`te_resident` trades memory for latency: resident costs ~16 GB of system
RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
request. CPU defaults to resident, CUDA to rebuilding per request, which
are the right defaults for where each is scarce.
`max_dim` rejects oversized requests **before admission**, so an
impossible resolution costs nothing. That matters more than it sounds:
an OOM on the device poisons the worker context, so refusing early is
the difference between one rejected request and a model that needs
restarting.
## Prompt caps
`NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
rejected before any GPU work, so a runaway client costs nothing.
## Sampling
Every model publishes its own sampling defaults in
`generation_config.json`, and a caller can override them per request.
Between those two sits an operator override, for the cases where the
model's published defaults are wrong for how you serve it:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.8-27B"
[default_models.sampling]
temperature = 0.6
presence_penalty = 1.0
```
Precedence is **request > operator > model > built-in**. Anything you
leave unset changes nothing, so an empty table behaves exactly as no
table at all.
Set the same values in the gateway's `models.toml` under
`[models.sampling]`. A model that is cold-loaded on demand gets its
sampling from the catalogue, not from this file, so the two must agree
or the same model samples differently depending on how it arrived —
`script/check-config-consistency.py` fails the deploy if they diverge.
### Which knob for which symptom
| symptom | knob |
|---|---|
| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
| The think block restates the same passage over and over | `presence_penalty` |
| Common tokens over-represented across a long answer | `frequency_penalty` |
| Short immediate loops — "no, no, no…" | `repeat_penalty` |
| You need two runs to be comparable | `seed` |
**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
measured a 20% structural-defect rate on long agentic turns against 0/60
at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
Qwen recommend for their thinking variants, so this is not fighting the
model.
**`presence_penalty`** and **`frequency_penalty`** are scored over the
*whole* generated sequence. They are the ones that matter for reasoning
models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
remedy for the endless repetition their models fall into during long
thinking. Both default to `0.0`, which is off.
**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
`64`) are the older, *windowed* form: the penalty only sees the last
`repeat_last_n` tokens. That is fine for chat and close to useless for a
think block — a 20,000-token deliberation restating a passage 5,000
tokens later is entirely outside a 64-token window. Measured across
recorded sessions, **86% of think blocks over 60k characters contained
verbatim repeated sentences**, one of them repeating 76% of its own
content.
Widening the window is the obvious move and the wrong one: the penalty
is token-level, and source code legitimately repeats tokens constantly
(`const`, `function`, brace runs), so a wide window at a meaningful
penalty degrades exactly the output you care about. Reach for
`presence_penalty` first — what recurs is passages, not tokens.
**`seed`** pins the sampler's RNG for every request to this model. It
exists because parameter search needs a control and the clients that
would otherwise pin it often cannot. Leaving it set is a product
decision, not just a testing one: every identical prompt then returns an
identical answer to every caller, which is predictable and cacheable but
removes the variation a user may expect from retrying.
## Reasoning
```toml
[default_models.sampling]
# nothing here — reasoning settings live in their own places, below
```
**`preserve_thinking`** controls whether prior turns keep their think
blocks when the conversation is re-rendered. It is the model's own chat
template control, not ours:
```toml
[[default_models]]
preserve_thinking = false # keep reasoning only for the turn in progress
```
Unset is the default and means the template decides. On Qwen3.8-27B that
is `true` — every prior turn's reasoning is replayed. Full replay is
what makes prompt growth track *total* completion rather than visible
output: one measured session produced 28,672 reasoning tokens on turn 1
and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
prefill on every turn thereafter.
Setting it to `false` is a legitimate experiment, not a recommendation.
Whether narrowing replay *helps* the model is unmeasured. Set it in both
`models.toml` and this file, like sampling, and the value is reported on
`/v1/models` and logged at load so a session can be attributed to the
arm it ran under.
**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
part of the caller's output budget for the answer:
```toml
[harness.candle.reasoning_budget]
answer_reserve_tokens = 4096
```
A reasoning model with no stopping criterion fills whatever budget it is
given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
given 32,768 it used 32,768 — and both times returned no answer at all.
The reserve bounds the think block at `max_tokens - answer_reserve` so
the rest survives for the reply.
Deliberately *not* derived from reasoning effort. Effort is a prompt
instruction the model is trained to follow; making it also set a
guillotine is how a model came to be told to reason at `xhigh` and then
cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
reasoning.
The trade to know: when the reserve fires, the model is interrupted
mid-deliberation and its answer phase tends to state an *intention*
rather than a conclusion. That is better than returning nothing, but it
is not free.
## The prefix cache
Snapshots of previously-served prefixes, so a follow-up turn re-uses the
KV cache instead of re-prefilling the whole conversation:
```toml
[harness.candle.prefix_cache]
enabled = true
budget_mb = 2048 # snapshot bytes per loaded model
max_entries = 8 # live snapshots per model, regardless of budget
```
**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
That is the symptom: the prompt keeps growing, the cached prefix does
not, and every turn re-prefills the difference. Observed on a 1 GiB
default — `cached_tokens` pinned at 2,016 for ten consecutive turns
while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
at roughly 37 s each, against ~20 s of actual decode.
The number is not free, and the direction of the trade is the opposite
of what it looks like. **The snapshot budget is reserved before the KV
budget, not from what is left over.** On a 2×32 GB host serving a 27B,
measured 1:1 — the token column is the pool all concurrent requests
share, not a per-slot allowance:
| `budget_mb` | KV budget | tokens in the pool | holds a 53k snapshot |
|---|---|---|---|
| 1024 | 4713 MB | 150,800 | no |
| 2048 | 3689 MB | 118,000 | yes |
| 3072 | 2665 MB | 85,300 | yes, but a session cannot reach 53k |
So raising it too far starves the KV cache and shortens how long a
single session can run — the model can no longer reach the context whose
snapshot the larger cache was meant to hold. Size it against the deepest
session you actually serve, and check `kv_budget_mb` in the journal
after a reload.
## Checking on it
```sh
@@ -24,7 +24,7 @@ confined anywhere.
```toml
[[models]]
id = "Qwen/Qwen3.8-27B"
id = "Qwen/Qwen3.6-27B"
harness = "candle"
min_devices = 2
min_device_vram_mb = 24000
@@ -32,43 +32,6 @@ daemon now does automatically.
| `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
| `limit.output` | same | generation reserve opencode leaves below the wall |
### Tuning the derived limit
Since #67 the neuron computes `limit` itself (see below), and three
`[harness.candle.context_limit]` settings shape that computation:
```toml
[harness.candle.context_limit]
target_prefill_latency_secs = 120 # longest acceptable prefill per turn
bootstrap_prefill_tok_per_sec = 800 # cold-start estimate until measured
output_reserve_tokens = 8192 # what a request gets when it names no cap
max_output_tokens = 32768 # the largest output a request may name
```
**`target_prefill_latency_secs`** is the one that usually binds. The
throughput ceiling is `target_prefill_latency_secs × measured
prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
advertises about 168k — above most hard ceilings, in which case it stops
mattering. On a slower host, or one whose recent traffic has been small
prompts, it is what caps the window.
**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
its own rate, and stops the advertised window collapsing on a
freshly-restarted host. Prefill samples below 512 tokens are ignored
when computing that rate: a short prompt is dominated by fixed
per-request cost and says nothing about throughput — the deploy's own
19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
So a host that has served only small requests keeps the bootstrap
estimate rather than adopting a misleading one, and no warm-up is
needed after a restart.
**`output_reserve_tokens`** is what a request that names no cap
receives. **`max_output_tokens`** is the ceiling advertised to clients
as `max_output_tokens`, and it is the *ceiling*, not the reserve —
publishing the reserve tells every client that reads the advertisement
to cap itself there, which on a reasoning model can be less than a
single think block.
## How they must relate
For a single model on a single neuron, all of these must hold:
@@ -97,9 +60,9 @@ Notes:
lands, treat the VRAM-safe ceiling in rule 5 as a hard limit you set
`NEURON_MAX_PROMPT_TOKENS` below, not something the daemon enforces.
## VRAM cost of context (Qwen3.8-27B on beast)
## VRAM cost of context (Qwen3.6-27B on beast)
Qwen3.8-27B is a hybrid linear-attention model: of its 64 layers only
Qwen3.6-27B is a hybrid linear-attention model: of its 64 layers only
every 4th is full-attention (`full_attention_interval = 4` → **16**
full-attn layers); the rest are `linear_attention` with constant-size
recurrent state. KV cache grows **only** on the 16 full-attn layers
@@ -168,7 +131,7 @@ value wins over a hand override of the same variable. Use `local.conf`
only for genuinely host-local, transient experiments — and remember a
later deploy will re-assert `model.conf`.
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.8-27B".limit`:
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.6-27B".limit`:
```json
{ "context": 131072, "input": 122880, "output": 8192 }
+1 -451
View File
@@ -36,8 +36,7 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0",
"vitest": "^3.2.7"
"vite": "^7.2.0"
}
},
"node_modules/@babel/code-frame": {
@@ -1815,17 +1814,6 @@
"@swc/counter": "^0.1.3"
}
},
"node_modules/@types/chai": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz",
"integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/deep-eql": "*",
"assertion-error": "^2.0.1"
}
},
"node_modules/@types/debug": {
"version": "4.1.13",
"resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.13.tgz",
@@ -1835,13 +1823,6 @@
"@types/ms": "*"
}
},
"node_modules/@types/deep-eql": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz",
"integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
@@ -2262,121 +2243,6 @@
"vite": "^4 || ^5 || ^6 || ^7 || ^8"
}
},
"node_modules/@vitest/expect": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz",
"integrity": "sha512-E8eBXaKibuvH2pSZErOjdVb5vF4PbKYcrnluBTYxEk1l/VhhwZg1kZQsdtjq+CsF5CFydf2Rdkz7jDHKSisi3w==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/mocker": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.7.tgz",
"integrity": "sha512-Trr0hYO9CM3Wj6ksWHRhK9IZpIY6wTMO5u/MqXurMxT57sWBaOPEtP3Oq60ihZuh5JsiagKfz95OcxdEP6dBrA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "3.2.7",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.17"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"msw": "^2.4.9",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"peerDependenciesMeta": {
"msw": {
"optional": true
},
"vite": {
"optional": true
}
}
},
"node_modules/@vitest/pretty-format": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.7.tgz",
"integrity": "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/runner": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.7.tgz",
"integrity": "sha512-sB9y4ovltoQP+WaUPwmSxO9WIg9Ig694Di5PalVPsYHklAdE027mehpWF2SQSVq+k6sFgaivbTjTJwZLSHbedA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "3.2.7",
"pathe": "^2.0.3",
"strip-literal": "^3.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/snapshot": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.7.tgz",
"integrity": "sha512-7C+MwShwtBSI5Buwoyg3s/iY1eHL9PKAf+O1wVh/TdnjXUtkoL/9YQtre90i4MtNXM6edP1wJ2zOBpfCyhIS7g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"magic-string": "^0.30.17",
"pathe": "^2.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/spy": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.7.tgz",
"integrity": "sha512-Q2eQGI6d2L/hBtZ0qNuKcAGid68XK6cv1xsoaIma6PaJhHPoqcEJhYpXZ/5myCMqkNgtP6UKuBhbc0nHKnrkuQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyspy": "^4.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/utils": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.7.tgz",
"integrity": "sha512-x6BDOd7dyo3PFLY3I9/HJ25X/6OurhGXk2/B9gOZNPF7XDVjeBK4k01lQE5uvDpbuheErh91qYuE1E2OEjK3Rw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"loupe": "^3.1.4",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/acorn": {
"version": "8.17.0",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.17.0.tgz",
@@ -2452,16 +2318,6 @@
"node": ">=10"
}
},
"node_modules/assertion-error": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz",
"integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
}
},
"node_modules/bail": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/bail/-/bail-2.0.2.tgz",
@@ -2556,16 +2412,6 @@
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/cac": {
"version": "6.7.14",
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
"integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -2607,23 +2453,6 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/chai": {
"version": "5.3.3",
"resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
"integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==",
"dev": true,
"license": "MIT",
"dependencies": {
"assertion-error": "^2.0.1",
"check-error": "^2.1.1",
"deep-eql": "^5.0.1",
"loupe": "^3.1.0",
"pathval": "^2.0.0"
},
"engines": {
"node": ">=18"
}
},
"node_modules/chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
@@ -2681,16 +2510,6 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/check-error": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz",
"integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 16"
}
},
"node_modules/classnames": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz",
@@ -2814,16 +2633,6 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/deep-eql": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz",
"integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/deep-is": {
"version": "0.1.4",
"resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz",
@@ -2886,13 +2695,6 @@
"dev": true,
"license": "ISC"
},
"node_modules/es-module-lexer": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
"integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==",
"dev": true,
"license": "MIT"
},
"node_modules/esbuild": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
@@ -3142,16 +2944,6 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/estree-walker": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz",
"integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/estree": "^1.0.0"
}
},
"node_modules/esutils": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz",
@@ -3162,16 +2954,6 @@
"node": ">=0.10.0"
}
},
"node_modules/expect-type": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz",
"integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -3757,13 +3539,6 @@
"loose-envify": "cli.js"
}
},
"node_modules/loupe": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz",
"integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==",
"dev": true,
"license": "MIT"
},
"node_modules/lowlight": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/lowlight/-/lowlight-3.3.0.tgz",
@@ -3789,16 +3564,6 @@
"yallist": "^3.0.2"
}
},
"node_modules/magic-string": {
"version": "0.30.21",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
"integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/sourcemap-codec": "^1.5.5"
}
},
"node_modules/markdown-table": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/markdown-table/-/markdown-table-3.0.4.tgz",
@@ -4826,23 +4591,6 @@
"node": ">=8"
}
},
"node_modules/pathe": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz",
"integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==",
"dev": true,
"license": "MIT"
},
"node_modules/pathval": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz",
"integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 14.16"
}
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -5348,13 +5096,6 @@
"node": ">=8"
}
},
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
"integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==",
"dev": true,
"license": "ISC"
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
@@ -5375,20 +5116,6 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/stackback": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
"integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==",
"dev": true,
"license": "MIT"
},
"node_modules/std-env": {
"version": "3.10.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz",
"integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==",
"dev": true,
"license": "MIT"
},
"node_modules/stringify-entities": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz",
@@ -5416,26 +5143,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/strip-literal": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz",
"integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"js-tokens": "^9.0.1"
},
"funding": {
"url": "https://github.com/sponsors/antfu"
}
},
"node_modules/strip-literal/node_modules/js-tokens": {
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz",
"integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==",
"dev": true,
"license": "MIT"
},
"node_modules/style-to-js": {
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.21.tgz",
@@ -5467,20 +5174,6 @@
"node": ">=8"
}
},
"node_modules/tinybench": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
"integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz",
"integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyglobby": {
"version": "0.2.17",
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz",
@@ -5498,36 +5191,6 @@
"url": "https://github.com/sponsors/SuperchupuDev"
}
},
"node_modules/tinypool": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz",
"integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==",
"dev": true,
"license": "MIT",
"engines": {
"node": "^18.0.0 || >=20.0.0"
}
},
"node_modules/tinyrainbow": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz",
"integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/tinyspy": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz",
"integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/trim-lines": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz",
@@ -5894,102 +5557,6 @@
}
}
},
"node_modules/vite-node": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz",
"integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==",
"dev": true,
"license": "MIT",
"dependencies": {
"cac": "^6.7.14",
"debug": "^4.4.1",
"es-module-lexer": "^1.7.0",
"pathe": "^2.0.3",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"bin": {
"vite-node": "vite-node.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/vitest": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.7.tgz",
"integrity": "sha512-KrxIJ62Fd89gfysR4WotlgZABiz2dqFPgqGzX7s+CwsqLFomRH7777ZcrOD6+WVAh7khPQP41A+BKbpcJFrdEg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/expect": "3.2.7",
"@vitest/mocker": "3.2.7",
"@vitest/pretty-format": "^3.2.7",
"@vitest/runner": "3.2.7",
"@vitest/snapshot": "3.2.7",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"debug": "^4.4.1",
"expect-type": "^1.2.1",
"magic-string": "^0.30.17",
"pathe": "^2.0.3",
"picomatch": "^4.0.2",
"std-env": "^3.9.0",
"tinybench": "^2.9.0",
"tinyexec": "^0.3.2",
"tinyglobby": "^0.2.14",
"tinypool": "^1.1.1",
"tinyrainbow": "^2.0.0",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0",
"vite-node": "3.2.4",
"why-is-node-running": "^2.3.0"
},
"bin": {
"vitest": "vitest.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"@edge-runtime/vm": "*",
"@types/debug": "^4.1.12",
"@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
"@vitest/browser": "3.2.7",
"@vitest/ui": "3.2.7",
"happy-dom": "*",
"jsdom": "*"
},
"peerDependenciesMeta": {
"@edge-runtime/vm": {
"optional": true
},
"@types/debug": {
"optional": true
},
"@types/node": {
"optional": true
},
"@vitest/browser": {
"optional": true
},
"@vitest/ui": {
"optional": true
},
"happy-dom": {
"optional": true
},
"jsdom": {
"optional": true
}
}
},
"node_modules/void-elements": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/void-elements/-/void-elements-3.1.0.tgz",
@@ -6024,23 +5591,6 @@
"node": ">= 8"
}
},
"node_modules/why-is-node-running": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz",
"integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==",
"dev": true,
"license": "MIT",
"dependencies": {
"siginfo": "^2.0.0",
"stackback": "0.0.2"
},
"bin": {
"why-is-node-running": "cli.js"
},
"engines": {
"node": ">=8"
}
},
"node_modules/word-wrap": {
"version": "1.2.5",
"resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.5.tgz",
+1 -4
View File
@@ -9,8 +9,6 @@
"preview": "vite preview",
"lint": "eslint .",
"typecheck": "tsc -b",
"test": "vitest run",
"test:watch": "vitest",
"i18n:check": "node ./scripts/check-i18n-keys.mjs",
"i18n:meta": "node ./scripts/check-i18n-metadata.mjs",
"i18n:lang-labels": "node ./scripts/check-i18n-lang-labels.mjs",
@@ -45,7 +43,6 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0",
"vitest": "^3.2.7"
"vite": "^7.2.0"
}
}
-105
View File
@@ -2063,108 +2063,3 @@ html[dir="rtl"] .hx-landing-composer textarea {
display: none;
}
}
/* Assistant working — reasoning and tool calls (#TurnBlocks).
*
* Deliberately quieter than the answer: muted text, no bubble of its
* own, hairline separators. The whole point is that it can be present
* without competing, so it reads as an annotation on the reply rather
* than as more reply. */
.hx-blocks {
display: flex;
flex-direction: column;
gap: 0.25rem;
margin-bottom: 0.5rem;
}
.hx-block {
border-left: 2px solid var(--color-border-subtle);
padding-left: 0.5rem;
}
.hx-block-open {
border-left-color: var(--color-accent-soft);
}
.hx-block-head {
display: flex;
align-items: center;
gap: 0.35rem;
width: 100%;
padding: 0.15rem 0;
background: none;
border: 0;
color: var(--color-text-muted);
font-size: 0.78rem;
text-align: start;
cursor: pointer;
}
.hx-block-head:hover {
color: var(--color-text);
}
.hx-block-caret {
flex: none;
transition: transform 0.12s ease;
}
.hx-block-caret-open {
transform: rotate(90deg);
}
.hx-block-label {
flex: none;
font-weight: 600;
}
.hx-block-running {
flex: none;
opacity: 0.7;
font-style: italic;
}
/* The peek is the first thing to give up space — it is a hint, and a
* wrapped one would cost the header its single-line shape. */
.hx-block-peek {
flex: 1 1 auto;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
opacity: 0.6;
}
.hx-block-body {
/* Capped and scrollable: a long think block would otherwise push the
* answer off-screen, which defeats the point of showing it. */
max-height: 16rem;
overflow-y: auto;
margin: 0.15rem 0 0.35rem;
padding-inline-end: 0.4rem;
color: var(--color-text-muted);
font-size: 0.82rem;
line-height: 1.45;
white-space: pre-wrap;
overflow-wrap: anywhere;
}
/* Reasoning toggle in the composer. Sits with the send button rather
* than in a settings panel: it changes what the next message does, so it
* belongs where the message is written. */
.hx-thinking-toggle {
display: inline-flex;
align-items: center;
gap: 0.35rem;
margin: 0;
padding: 0 0.5rem;
color: var(--color-text-muted);
font-size: 0.78rem;
white-space: nowrap;
cursor: pointer;
user-select: none;
}
.hx-thinking-toggle:hover {
color: var(--color-text);
}
.hx-thinking-toggle input {
margin: 0;
cursor: pointer;
}
/* Narrow screens: the composer is already tight, and the label is the
* expendable half — the checkbox still reads as a control. */
@media (max-width: 30rem) {
.hx-thinking-toggle span {
display: none;
}
}
-129
View File
@@ -1,129 +0,0 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { FaChevronRight, FaBrain, FaWrench } from "react-icons/fa6";
import type { TurnBlock } from "../data/db";
import { isOpen } from "../lib/turnBlockState";
/**
* The assistant's working — reasoning and tool calls — shown above its
* answer.
*
* Two things are in tension. All of this should be *available*, because a
* model that reasons for two minutes and then answers has done work worth
* inspecting, and a tool call the user cannot see is a claim they cannot
* check. But none of it is what they asked for, so it must not bury the
* answer.
*
* The resolution is that visibility follows activity:
*
* * While a block is the live one — still streaming, no answer text yet
* — it is open. That span is otherwise dead air; a spinner would say
* "working", this says what it is working *on*.
* * The moment answer text starts, every block collapses to a one-line
* header. The thing the user asked for takes the space.
* * Completed turns load collapsed, so scrollback stays readable.
*
* A block the user opens or closes by hand stays that way. Explicit
* intent outranks the automatic rule, including for the live block —
* someone who closes a noisy reasoning stream should not have it reopen
* itself on the next token.
*/
export function TurnBlocks({
blocks,
streaming,
answerStarted,
}: {
blocks: TurnBlock[];
/** This turn is still being generated. */
streaming: boolean;
/** Answer text has begun arriving, so the working is now history. */
answerStarted: boolean;
}) {
const { t } = useTranslation();
// Only blocks the user has touched appear here; everything else follows
// the automatic rule. Keyed by index, which is stable because blocks
// are only ever appended within a turn.
const [overrides, setOverrides] = useState<Record<number, boolean>>({});
if (blocks.length === 0) return null;
return (
<div className="hx-blocks">
{blocks.map((b, i) => {
const open = isOpen(i, blocks.length, streaming, answerStarted, overrides[i]);
const body = b.kind === "reasoning" ? b.text : blockToolBody(b);
return (
<div key={i} className={`hx-block${open ? " hx-block-open" : ""}`}>
<button
type="button"
className="hx-block-head"
aria-expanded={open}
onClick={() => setOverrides((o) => ({ ...o, [i]: !open }))}
>
<FaChevronRight
className={`hx-block-caret${open ? " hx-block-caret-open" : ""}`}
size={10}
aria-hidden="true"
/>
{b.kind === "reasoning" ? (
<FaBrain size={11} aria-hidden="true" />
) : (
<FaWrench size={11} aria-hidden="true" />
)}
<span className="hx-block-label">
{b.kind === "reasoning"
? t("chat:blockReasoning")
: t("chat:blockTool", { name: b.name })}
</span>
{/* A pending tool call is the one case where the header
* alone should say something is happening: its body is
* empty until the result lands, so an open block would
* otherwise look like a stall. */}
{b.kind === "tool" && b.result === undefined && streaming && (
<span className="hx-block-running">{t("chat:blockRunning")}</span>
)}
{!open && (
<span className="hx-block-peek">{peek(body)}</span>
)}
</button>
{open && <div className="hx-block-body">{body}</div>}
</div>
);
})}
</div>
);
}
/** Arguments, then the result once it arrives. */
function blockToolBody(b: Extract<TurnBlock, { kind: "tool" }>): string {
const args = prettyArgs(b.args);
return b.result === undefined ? args : `${args}\n\n${b.result}`;
}
/**
* Tool arguments as sent, pretty-printed when they are JSON.
*
* Models emit malformed arguments often enough that this must never
* throw — a UI that blanks because the model produced bad JSON hides the
* very thing worth seeing.
*/
function prettyArgs(raw: string): string {
try {
return JSON.stringify(JSON.parse(raw), null, 2);
} catch {
return raw;
}
}
/**
* The tail of a collapsed block, so a closed row still carries a hint of
* its content.
*
* The tail rather than the head: while a block streams, the end is where
* the model currently is. A head-anchored preview freezes on the first
* few words and stops telling you anything.
*/
function peek(text: string): string {
const flat = text.replace(/\s+/g, " ").trim();
return flat.length <= 80 ? flat : `…${flat.slice(-79)}`;
}
-27
View File
@@ -39,23 +39,6 @@ export interface MessageSource {
url: string;
}
/**
* Work the assistant did on the way to an answer: a stretch of reasoning,
* or one tool call and its result.
*
* Kept as an ordered list rather than two separate fields because the
* order is the story — a turn can reason, call a tool, reason about what
* came back, and call another. Flattening that into "the reasoning" and
* "the tool calls" would lose which thought preceded which call.
*
* `text` and `result` grow while streaming, so a block is also its own
* progress indicator: a reasoning block with text and no following block
* is what the model is doing *right now*.
*/
export type TurnBlock =
| { kind: "reasoning"; text: string }
| { kind: "tool"; name: string; args: string; result?: string };
export interface Message {
id: string;
conversationId: string;
@@ -67,16 +50,6 @@ export interface Message {
promptTokens?: number;
completionTokens?: number;
sources?: MessageSource[];
/**
* Reasoning and tool calls, in the order they happened. Absent on user
* turns and on assistant turns that answered directly.
*
* Not indexed, so no schema version bump: Dexie stores the whole
* object and only the declared indexes constrain it. Messages written
* before this field existed simply have none, which renders as an
* answer with no working shown — correct, since none was captured.
*/
blocks?: TurnBlock[];
}
/**
+1 -14
View File
@@ -10,7 +10,6 @@ import {
type Message,
type MessageRole,
type Project,
type TurnBlock,
} from "./db";
function uuid(): string {
@@ -139,22 +138,10 @@ export async function setMessageContent(id: string, content: string): Promise<vo
await db.messages.update(id, { content });
}
/**
* Write the whole block list as an absolute snapshot.
*
* Same discipline as `setMessageContent`: the caller owns the array and
* writes it entire, because a read-modify-write append races itself when
* deltas arrive faster than the IndexedDB round-trip — which is how the
* content path dropped tokens before it was written this way.
*/
export async function setMessageBlocks(id: string, blocks: TurnBlock[]): Promise<void> {
await db.messages.update(id, { blocks });
}
export async function finalizeMessage(
id: string,
patch: Partial<
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources" | "blocks">
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources">
>,
): Promise<void> {
await db.messages.update(id, patch);
+1 -6
View File
@@ -29,10 +29,5 @@
"reading": "{{host}} እየተነበበ ነው…",
"copyCode": "ኮድ ቅዳ",
"copiedCode": "ተቀድቷል",
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።",
"blockReasoning": "በማሰብ ላይ",
"blockTool": "{{name}}",
"blockRunning": "በማስኬድ ላይ…",
"showThinking": "ማመዛዘን",
"showThinkingHint": "ሞዴሉ ከመመለሱ በፊት እንዲያስብ ጠይቅ፣ እና ማመዛዘኑን አሳይ"
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።"
}
+1 -6
View File
@@ -29,10 +29,5 @@
"reading": "تجري قراءة {{host}}…",
"copyCode": "نسخ الكود",
"copiedCode": "تم النسخ",
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني.",
"blockReasoning": "جارٍ التفكير",
"blockTool": "{{name}}",
"blockRunning": "قيد التشغيل…",
"showThinking": "التفكير",
"showThinkingHint": "اطلب من النموذج التفكير قبل الإجابة، واعرض تفكيره"
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني."
}
+1 -6
View File
@@ -29,10 +29,5 @@
"reading": "четене на {{host}}…",
"copyCode": "Копиране на кода",
"copiedCode": "Копирано",
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота.",
"blockReasoning": "Разсъждаване",
"blockTool": "{{name}}",
"blockRunning": "изпълнява се…",
"showThinking": "Разсъждение",
"showThinkingHint": "Моделът да мисли преди да отговори и да показва разсъжденията си"
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота."
}
+1 -6
View File
@@ -29,10 +29,5 @@
"reading": "s'està llegint {{host}}…",
"copyCode": "Copia el codi",
"copiedCode": "Copiat",
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta.",
"blockReasoning": "Pensant",
"blockTool": "{{name}}",
"blockRunning": "en execució…",
"showThinking": "Raonament",
"showThinkingHint": "Demana al model que pensi abans de respondre i mostra'n el raonament"
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta."
}
+1 -6
View File
@@ -29,10 +29,5 @@
"reading": "čtení {{host}}…",
"copyCode": "Kopírovat kód",
"copiedCode": "Zkopírováno",
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl.",
"blockReasoning": "Přemýšlení",
"blockTool": "{{name}}",
"blockRunning": "probíhá…",
"showThinking": "Uvažování",
"showThinkingHint": "Ať model před odpovědí přemýšlí a ukáže své uvažování"
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl."
}

Some files were not shown because too many files have changed in this diff Show More