Compare commits

...
Author SHA1 Message Date
grenadeandClaude Opus 5 3241680bfd fix(admission): serve anonymous callers from leftover capacity
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m5s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m27s
CI / Test (push) Successful in 9m7s
CI / CUDA type-check (push) Successful in 19m18s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Fair-share (#54) bounds what one identified caller may hold. A caller
with no principal cannot be bounded by it — cortex stamps the
`x-helexa-*` headers only after a bearer resolves, and both a missing
and an unrecognised key proceed unstamped. So the callers we know least
about were the only ones nothing could cap.

Observed 2026-08-16: a credential-less batch client at concurrency 4
held every seat on an 8-seat model while an interactive session waited
out `max_wait` and was abandoned. `rejected_per_principal: 0` recorded
that the cap never fired, because it was never eligible to. KV was not
the constraint — seats were.

Anonymous traffic is now served from the capacity left over once
identified traffic is satisfied, enforced at three points:

- Priority. An anonymous request never takes a seat while an identified
  one is waiting, whoever arrived first. It parks on a class gate ahead
  of the KV and seat gates, so it yields holding nothing.
- A seat ceiling (`anon_max_in_flight`, default `max_in_flight - 1`).
  Priority alone is not enough: a running request cannot be preempted,
  so without a reserved seat an anonymous burst arriving during an idle
  moment still locks the model for as long as all of it takes.
- A queue ceiling (`anon_max_pending`), or an anonymous flood refuses
  identified callers at the door before priority can favour them.

The class gate makes anonymous requests opportunistic rather than
queued: they take a seat only when one is free at that instant. Queueing
would defeat the rule, since the seat semaphore is FIFO and knows
nothing of classes — an anonymous waiter in line first would be handed
the next seat ahead of an identified caller arriving later. Parking
re-tests priority on every wakeup instead of once on the way in, and
keeps a yielding request from holding KV budget an identified caller
needs, which is what would otherwise deadlock against the KV gate.

Refusals stay diagnostic: `AnonYield` only when capacity existed and the
reservation withheld it, ordinary `Timeout` when the model was simply
full, and #257's `KvTimeout` — with its far longer patience — when a
long-context request is waiting for budget rather than a seat.

Both ceilings floor at 1, so a single-seat model still serves anonymous
callers; `anon_max_in_flight = 0` refuses them outright. Surfaced on
/health as `anon_in_flight` / `anon_max_in_flight` and exported by
cortex as `cortex_model_anon_in_flight` plus a `reason="anon_yield"`
rejection counter: rising there while `wait_timeout` stays flat means
the reservation is working, not that the model is overloaded.

Closes #262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGhUi88ANnSfvk86DtYc9g
2026-08-18 13:26:08 +03:00
grenade 4878924b9b ci: stop docs commits from re-running the whole Rust gate
CI / Classify changes (push) Successful in 11s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m58s
CI / Test (push) Successful in 8m6s
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m30s
build-prerelease / Build helexa-router binary (push) Successful in 1m50s
build-prerelease / Build helexa-bench binary (push) Successful in 2m3s
build-prerelease / Build neuron-ampere (push) Successful in 2m24s
build-prerelease / Build neuron-ada (push) Successful in 2m24s
build-prerelease / Build cortex binary (push) Successful in 2m29s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m45s
build-prerelease / Build helexa-angels binary (push) Successful in 3m4s
build-prerelease / Test (push) Successful in 10m58s
build-prerelease / Build neuron-blackwell (push) Successful in 14m38s
build-prerelease / Package helexa-router RPM (push) Successful in 25s
build-prerelease / Package helexa-bench RPM (push) Successful in 25s
build-prerelease / Package helexa-upstream RPM (push) Successful in 23s
build-prerelease / Package cortex RPM (push) Successful in 29s
build-prerelease / Package helexa-angels RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m16s
build-prerelease / Build helexa-tools binary (push) Successful in 1m29s
build-prerelease / Package helexa-tools RPM (push) Successful in 21s
CI / CUDA type-check (push) Successful in 27m38s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m17s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 31s
A one-character whitespace fix to a doc ran fmt, clippy and the full
test suite. The cause was not missing detection — detection exists and
correctly decided every `build_*` was false. It was the base it asks
against.

`CHECK_RUST` was decided by diffing against **cortex's last published
sha**. Cortex is only rebuilt when cortex paths change, so after a run of
neuron-only work its base sits days behind, and the diff from it sweeps
in every unrelated Rust commit since. Measured on the offending push:
diffing from cortex's base (fec5b33) matched 8 Rust files; diffing the
push's own range matched none. Those 8 had already been linted on the
branches that introduced them.

So when nothing is being built, ask the honest question — did THIS PUSH
touch Rust — using the push's own range. The invariant that matters is
untouched: if any `build_*` is true, `CHECK_RUST` is forced true, so
nothing reaches the fleet unlinted. The changed branch only runs when
there is no artefact at all. `decide` still returns true for an unusable
base, so a new branch (all-zero before-sha) or a force-push keeps the
full gate.

ci.yml, which handles branches and PRs, gets the same treatment from the
other direction:

- documentation joins the SPA as inert for Rust — no crate
  `include_str!`s a markdown file, no spec packages `doc/`, no build
  script reads one. Markdown is matched only at the repo root and under
  `doc/`; a `.md` inside `crates/` still falls through, since
  `include_str!` could make one load-bearing without this file knowing.
- the web job is now gated too. It ran on every push, including Rust-only
  ones that cannot affect the SPA — three minutes of `npm ci`, lint,
  typecheck and build to prove a Rust change did not break TypeScript.

The classifier stays one-directional: it narrows only when *every*
changed path is known-inert, and a tag, an unknown base, an empty diff or
a mixed changeset still takes the full gate. A new top-level directory is
covered the day it appears.

Verified by replaying twelve changesets through the classifier: the
reported docs commit (rust=false, web=true — it is SPA content), root
markdown and `doc/` (both false), mixed Rust+docs, markdown under
`crates/`, workflow edits, spec files, `Cargo.lock` and an unknown
directory all correctly take the full gate.
2026-08-18 11:54:26 +03:00
grenade 4661557ed7 doc: whitespace adjustment
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m38s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Test (push) Successful in 8m52s
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-18 11:45:49 +03:00
grenade 01d034df08 Merge branch 'fix/neuron-bounded-drain'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m15s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m31s
build-prerelease / Build neuron-ampere (push) Successful in 3m2s
build-prerelease / Test (push) Successful in 6m57s
build-prerelease / Build neuron-blackwell (push) Successful in 14m33s
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m4s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-08-17 22:45:20 +03:00
grenade 3d28e016e4 fix(neuron): bound the shutdown drain so restarts stop ending in SIGKILL
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m4s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 6m57s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Every restart of a busy node cost two minutes and then a hard kill.
2026-08-15: SIGTERM at 13:03:14, one queued request logging "still
waiting on pool lock" every two seconds, `Failed with result 'timeout'`
at 13:05:24. The same node drained in about a second when idle — which is
what identified the queued waiter as the cause rather than the model
unload.

axum's graceful shutdown waits for all in-flight requests, and a request
queued behind the TP pool lock is in-flight as far as axum can tell. That
wait is unbounded: it is held by whichever request is mid-forward, so the
drain cannot finish, and TimeoutStopSec expires.

Two waits had to become abandonable, because closing one alone leaves the
other stuck:

- `acquire_pool_lock` now selects on a shutdown notify and returns
  `InferenceError::ShuttingDown` instead of waiting for a lock nobody is
  going to release in time. It also refuses up front when already
  draining — a request that has not started has nothing to preserve.
- `AdmissionController::close()` closes both semaphores, so queued
  waiters wake immediately. `enter_with_kv` already treated a closed
  semaphore as a rejection ("Semaphore is never closed; treat a
  closed/elapsed wait the same"), so this needed no new rejection path —
  the code was already written for a case that could not happen yet.

`shutdown_signal` does both before returning, so the drain that starts
when it resolves has a bounded amount of work left. Requests already
running are untouched: they hold their permits and either finish or die
with the process.

Clients get `503 shutting_down` + `Retry-After: 15`. Deliberately not
`rate_limit_exceeded` — nothing about this is the caller's fault or their
rate, and a client that distinguishes them can retry a draining node
differently from an overloaded one.

Tests cover a slot waiter, a KV-budget waiter, and a later arrival, and
`close_wakes_queued_waiters` was verified to fail when `close()` forgets
the slots semaphore — the shape of the original bug.

Refs #256.
2026-08-17 22:23:06 +03:00
grenade fdfdb19c64 Merge branch 'fix/single-flight-model-load'
build-prerelease / Resolve version stamps + change detection (push) Successful in 12s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m17s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m35s
build-prerelease / Build neuron-ampere (push) Successful in 3m11s
build-prerelease / Test (push) Successful in 7m0s
build-prerelease / Build neuron-blackwell (push) Successful in 14m52s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-17 18:55:01 +03:00
grenade 9853c5b08a fix(candle): single-flight model loads so concurrent callers cannot race
CI / Classify changes (push) Successful in 12s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m27s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m29s
CI / Test (push) Successful in 7m6s
CI / CUDA type-check (push) Successful in 19m42s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`load_model` opened with a check-then-act:

    {
        let models = self.models.read().await;
        if models.contains_key(&spec.model_id) { bail!("already loaded") }
    }                                   // ← lock released, load happens after

Concurrent callers for one id all took the read lock, all saw the id
absent, all released, and all proceeded. On beast 2026-08-17 that gave
FIVE `--worker --rank 1` processes loading the same shard onto one card:
each logged the same device's free VRAM marching down together until
every attempt failed, and because the failures left traffic unserved the
next request started another. It does not converge — the node stayed
down ~40 minutes and needed a human.

Now each model id has a gate, taken before the membership check and held
for the whole load, with the membership check repeated under it. The
second caller waits and then sees the model present, returning the same
"already loaded" error cortex already treats as success on its cold-load
path — so a coalesced caller gets the outcome it wanted and no client
changes.

The gate is per model id, not global. A single load lock would serialise
every cold-load on the node behind whichever model got there first, which
on a multi-model node trades this bug for a worse one. Tested both ways:
same id contends, different ids do not — the second test was verified to
fail when the gate is made global.

Gates are never reclaimed. The key space is the catalogue, so it is
bounded and tiny, and freeing them safely would mean proving no task is
waiting on one — cost without benefit.

This becomes more load-bearing with #265: a self-restarting neuron lands
a fresh process into whatever traffic queued during the outage, which is
exactly the stampede that produced the five workers.

Refs #255.
2026-08-17 18:34:00 +03:00
grenade 1e3d3f76b7 Merge branch 'fix/neuron-exit-on-wedge'
build-prerelease / Resolve version stamps + change detection (push) Successful in 12s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m29s
build-prerelease / Build neuron-ada (push) Successful in 3m5s
build-prerelease / Test (push) Successful in 6m34s
build-prerelease / Build neuron-blackwell (push) Successful in 14m47s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-17 18:12:45 +03:00
grenade 8b0ed91380 test(tp): run the deadman tests instead of only compiling them
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m46s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 6m40s
CI / CUDA type-check (push) Successful in 19m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The watchdog tests were gated on `feature = "cuda"`, so CI compiled them
in the type-check job and never ran them anywhere — tests for the branch
that terminates the process, which had never once executed.

`wait_for_abort` and `tp_abort_deadline` contain no CUDA: an atomic flag,
a clock, and an env var. Gating them `any(feature = "cuda", test)` lets
the ordinary CPU suite exercise them while keeping them out of non-test
CPU builds, where they would be dead code.

All four now run in the normal Test job, and the #265 case — an abort
that never returns — was verified to fail when `wait_for_abort` is made
to always report success.
2026-08-17 17:50:50 +03:00
grenade c9f7383d91 fix(tp): exit non-zero when a wedged collective cannot be aborted
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m46s
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m53s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for 70 minutes on 2026-08-17 and recovered only
because a human noticed. The unit already had Restart=on-failure and
RestartSec=5; the process simply never exited, so systemd saw a healthy
daemon — correctly, by its own rules — while the listener held 27
unaccepted connections, both GPUs sat at 0%, and clients got 503s.

The watchdog already diagnosed the remedy in its own log text ("recovery
may stall until a process restart") and then waited for someone to apply
it.

Exiting on that Err branch would not have helped. There is no
`ncclCommAbort succeeded` line and no `... failed` line in the incident:
the abort call itself blocked and never returned, taking its thread with
it. Control never reached either branch. So:

- the abort runs on its own named thread, where hanging costs one thread
  rather than the daemon;
- an independent deadman waits NEURON_TP_ABORT_DEADLINE_S (default 60)
  for it to return, and does NOT depend on it returning;
- if the deadline elapses, log the forensics and exit 70 (EX_SOFTWARE).

`wait_for_abort` is split out so the decision is testable without a GPU —
the branch that ends the process is both the most important to get right
and the hardest to reach by accident. Tests cover the abort returning,
never returning, and landing in the last instant (exiting on that race
would restart a process that had just recovered).

Forensics are logged BEFORE the exit, including a live nvidia-smi read
rather than the cached VRAM value, since the cache is refreshed by the
machinery that is wedged. A self-healing outage nobody can diagnose is a
poor trade for one that at least stayed put.

The unit gains StartLimitIntervalSec=600 / StartLimitBurst=4. Self-
restarting a deterministic wedge would otherwise flap, and a flapping
node is worse than a down one: cortex marks it healthy between restarts
and keeps routing to it, so every client sees intermittent failures
instead of one node cleanly out. Four failures in ten minutes leaves the
unit `failed` and visible.

process::exit skips destructors and the drain path deliberately —
normally the wrong instinct, but draining cannot work when the collective
is stuck, and TimeoutStopSec=120s would add two minutes to an outage that
is already total (#256).

Refs #265.
2026-08-17 17:23:25 +03:00
grenade 18683d904d Merge branch 'feat/upstream-key-auth-allocation'
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m37s
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Successful in 2m19s
build-prerelease / Test (push) Successful in 12m21s
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 15s
2026-08-17 15:51:34 +03:00
grenade 46fe3d525c feat(upstream): an API key can read and refill its own allocation
CI / Classify changes (push) Successful in 12s
CI / Format (push) Successful in 26s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m4s
CI / Clippy (push) Successful in 3m18s
CI / Test (push) Successful in 12m50s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Everything under /web/v1 required a session token from email+password.
That included reading the balance and requesting a top-up — the two
things a long-running service actually needs. So a service watching its
own allocation had to store its owner's account password: the credential
that can mint keys, archive them and read the account. To watch a number
it was already entitled to spend against, it had to hold the one secret
that can take over the account.

The router now splits by credential class rather than by "protected":

- session-only — /account, /keys, /keys/{id}/archive, /keys/{id}/limit
- session OR API key — /allocation, /redeem, /topup/request

Key management stays session-only deliberately, and that is the line
worth defending. If an inference key could mint keys, one leaked
credential would become a foothold that SURVIVES revoking the original —
a spending problem turning into an eviction problem.

`/allocation` is a new narrow projection rather than widened auth on
`/account`. `/account` also returns `angel_access`, and whatever field is
added to it next would otherwise reach every inference key by default. A
separate route means widening the account view can never widen what a key
can see.

`require_account` dispatches on the `sk-helexa-` prefix rather than
attempting a JWT decode and falling back, so a valid key never logs a
decode failure. Archived keys and deactivated accounts need no extra
check: `ledger::resolve_key` already requires both to be `active`.

The test asserts the refusals, not just the grants — mint, list, archive
and /account are each 401 for a key — and it was verified to fail when
key management is moved onto the key-authenticated router, so it is
testing the property rather than restating the implementation. It also
covers an archived key authenticating nothing and both bearer shapes
being refused when bogus.

Refs #264.
2026-08-17 14:54:52 +03:00
grenade fec5b33195 Merge branch 'fix/failure-observability'
build-prerelease / Resolve version stamps + change detection (push) Successful in 12s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m31s
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 1m58s
build-prerelease / Test (push) Successful in 10m39s
build-prerelease / Build neuron-ada (push) Successful in 2m10s
build-prerelease / Build neuron-ampere (push) Successful in 3m2s
build-prerelease / Build neuron-blackwell (push) Successful in 14m45s
build-prerelease / Package cortex RPM (push) Successful in 25s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m2s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 27s
2026-08-15 23:10:28 +03:00
grenade 5ce4d8481e fix(observability): failures say why, in the journal and on the wire
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m43s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 6m54s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Three incidents today were each diagnosable only by reading source and
reproducing arithmetic by hand, because every failure path logged
nothing useful and returned nothing parseable:

- a 422 that killed a harness session (#253) — axum's `Json` extractor
  rejects before any handler runs, with a bare string body and no log;
- a 503 that discarded 30 minutes of agentic work (#257) — the envelope
  reached the client, but neither neuron nor cortex recorded the reason;
- a device fault that poisoned a model — cause present in the error but
  formatted away before anyone saw it.

Four fixes, in rough order of what they cost us.

**Error chains survive (56 sites).** `anyhow!("ctx: {e}")` keeps only the
outermost context, so wrapping an error discarded everything beneath it.
`{e:#}` keeps the chain.

This is not just cosmetic: `is_device_fault` is *default-true* — it
poisons unless it sees a known non-device marker like "shape mismatch",
and those markers live in the inner error. A flattened chain therefore
turns a benign pre-kernel failure into a device fault, poisoning the
model and forcing an unload+reload for nothing. Pinned by a test that
shows the same error classified both ways depending on formatting.

It also means today's `AssembleKvBatch` fault was never established as a
device fault at all — it was classified as one by default, with its
cause invisible.

**`ApiJson` replaces `Json` on every body-taking route.** Same #63
envelope as every other error, plus a WARN naming endpoint and detail.
422 for a shape we don't accept, 400 for malformed JSON, 415 for a bad
content-type — distinguished because they need different fixes from the
caller.

**Every rejection logs once, at the boundary.** `inference_error_response`
now emits status, type, code, message and the `extra` numbers that make
it actionable (free_mb/required_mb, prompt_len/max, retry_after), so the
line stands alone instead of pointing at source to go and read.

**cortex logs upstream error bodies.** Previously only `status=503`, on
the grounds that the body was a stream. Streaming is what a *success*
body needs; an error body is a few hundred bytes and streaming it only
guarantees nobody sees the reason. Non-2xx is now buffered (64 KiB cap),
logged truncated, and handed on unchanged. Over the cap it says the body
was dropped rather than passing an empty one off as the upstream's answer.

Refs #253, #257.
2026-08-15 19:29:05 +03:00
grenade c62cfc6a50 fix(servability): a model with KV budget is servable, however saturated
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m34s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 9m10s
build-prerelease / Build neuron-ada (push) Successful in 2m14s
build-prerelease / Build neuron-ampere (push) Successful in 3m5s
build-prerelease / Build neuron-blackwell (push) Successful in 14m33s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 23s
Observed live within an hour of #257 deploying, on the session it was
meant to protect:

    router:  skipping loaded-but-unservable location node=beast
    evictor: evicting model node=beast model=Qwen/Qwen3.8-27B
    evictor: failed to evict — status=404
    router:  cold-swap eviction failed
    router:  cold-loading via /models/load
    router:  cold-load saw 'already loaded' — treating as success
    proxy:   proxying request node=beast

cortex tried to EVICT the flagship model out from under a live session,
twice, and only got away with it because the eviction failed. A
successful one would have unloaded the model mid-conversation.

The probe (#245) asks its question against live free VRAM. That was a
fair proxy for "can this serve" while an over-VRAM request was rejected
outright — the node never stayed saturated for long. With the KV gate
(#257) it does: free VRAM sitting near zero while long sequences are
resident is admission *working*, and a request that does not fit queues
rather than failing. The probe read that as a dead location.

So when a KV budget is published, defer to it: capacity is admission's
job now, and a model whose budget can still admit work is servable. The
live-VRAM question is kept for the case the gate cannot answer — no
budget published (CPU load, or an arch with no context profile) — which
is also the case #245 was written for: VRAM eaten by something *outside*
neuron, which no reservation accounting would ever see.

The decision is extracted into `servability_from` so it can be tested
without a loaded model on a GPU, and the regression is pinned.

This is the drift the `check_vram` doc-comment warns about — "two copies
of 'can this serve' will always drift" — and I introduced it by leaving
the probe alone while changing what saturation means. The lesson is that
the probe's *input* changed even though its code did not.

Refs #257, #245.
2026-08-15 18:37:51 +03:00
grenade adebbd429a Merge branch 'fix/vram-aware-admission'
build-prerelease / Resolve version stamps + change detection (push) Successful in 12s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m41s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m40s
build-prerelease / Build helexa-bench binary (push) Successful in 1m54s
build-prerelease / Build cortex binary (push) Successful in 2m15s
build-prerelease / Build neuron-ada (push) Successful in 2m23s
build-prerelease / Build neuron-ampere (push) Successful in 2m26s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m33s
build-prerelease / Test (push) Successful in 7m7s
build-prerelease / Build neuron-blackwell (push) Successful in 15m8s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Package helexa-router RPM (push) Successful in 31s
build-prerelease / Package cortex RPM (push) Successful in 32s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m15s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 25s
2026-08-15 16:46:53 +03:00
grenade 8f7edd7d00 fix(admission): borrow the context-limit config in the TP non-streaming path
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m55s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m43s
CI / Test (push) Successful in 9m21s
CI / CUDA type-check (push) Successful in 19m22s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
A by-value binding where the other three call sites hold a reference, so
it only failed under `--features cuda` — the one configuration that
cannot be compiled locally. Caught by the branch CUDA type-check, which
is the point of that job.

Refs #257.
2026-08-15 16:24:29 +03:00
grenade 4fd23c5322 fix(admission): reserve the prefix cache's VRAM out of the KV budget
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m2s
CI / Format (push) Successful in 13s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 6m53s
CI / CUDA type-check (push) Failing after 19m9s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The budget is measured at load, while the prefix cache (#11) is still
empty — but its snapshots live on the model's device and grow into that
same free VRAM. Handing it out to sequences would over-admit by up to
`prefix_cache.budget_mb`: 1 GiB by default, ~18% of beast's budget, and
precisely the kind of over-commitment #257 exists to stop.

This has a real cost, so state it plainly: on beast a 67,881-token turn
reserves 2377 MiB against a budget that falls from 5738 to 4714, so the
card now admits ONE such turn at a time and queues the second. Live, two
did coexist — because the cache held ~433 MiB of its 1024 MiB
entitlement at that moment, not all of it. Reserving the entitlement
rather than the current usage is the conservative direction on purpose:
over-admitting re-creates the OOM, while over-reserving only queues a
request that is still served. Teaching the cache to evict snapshots to
admit a sequence, instead of the two silently competing for the same
VRAM, is the natural follow-up.

Also re-points #65's acceptance test at where its guard now lives. That
test proved a cap derived from *polled* free VRAM could not admit a
prompt that no longer fit. The protection is now the KV gate, and the
staleness it guarded against is structurally gone — the budget is fixed
at load rather than re-read from a poll, so it cannot disagree with
itself between poll and request. The test now proves the same prompt is
still refused (fast, as KvUnservable) on a tight card and still admitted
on an ample one.

Refs #257.
2026-08-15 16:01:24 +03:00
grenade eca4251054 fix(admission): reserve KV budget at admission instead of rejecting
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m42s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m39s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Admission bounded request *count*; capacity is bounded by KV *bytes*. On
long-context sessions those disagree badly — eight concurrent 68k
sequences need ~19 GB of KV that does not exist — so the real limit was
enforced by a VRAM backstop that ran BEFORE admission and rejected
outright. A 30-minute, 26-step agentic session died on two 503s 1.2 s
apart while the admission queue sat completely empty (queue_depth 0 of 8,
every rejected_* counter zero). There was capacity to wait; the code
refused instead.

Reordering alone would not have helped: admission grants on
in_flight (2) < max_in_flight (8) and would have passed the request
straight through to the same failure.

So make the budget itself a gate. `AdmissionController` gains a KV
semaphore whose permits are MiB, filled once at load — measured while
nothing is in flight, the only moment a free-VRAM reading is not already
net of some running sequence's reservation. Each request reserves the KV
its sequence will hold (prompt + output reserve) and returns it on drop,
so a request that does not fit *yet* queues until one that does.

The order between the two gates is load-bearing: KV first, then the
in-flight slot. Taking the slot first deadlocks — every slot could be
held by a request waiting for KV while the KV is held by requests waiting
for a slot, and nothing would run to release either.

A reservation larger than the entire budget can never be satisfied, so it
fails fast as 413 `prompt_too_long_for_vram` rather than hanging until
the deadline. That is a client-side length error, not backpressure: an
idle node would refuse it too.

`kv_max_wait_secs` defaults to 240 — far longer than `max_wait_secs`,
because this gate waits for a whole long-context sequence to finish
rather than for a slot to turn over, and a short deadline would only
convert a fast rejection into a slow one.

The pre-admission backstop is reduced to the catastrophic floor; keeping
the length-aware form there would reinstate the same rejection one call
earlier. The serviceability probe (#245) keeps it, asking a deliberately
different question — "could a minimal request start right now" — and the
shared floor keeps one definition of out-of-memory.

/health gains kv_budget_mb, kv_available_mb, rejected_kv_timeout and
rejected_kv_unservable, so saturation-by-context is visible as something
other than silence.

The single-GPU non-streaming path admits before the prompt is built and
so has no length to price yet; it is left unpriced and commented. Long-
context agentic traffic is streaming, which is where the budget has to
hold.

Pinned by the session's own numbers: 9286 MiB free at load → 5738 MiB
budget; a 67,881-token turn reserves 2377 MiB; two fit, three do not.

Refs #257.
2026-08-15 15:52:37 +03:00
grenade 9e1a0fa231 Merge branch 'fix/max-completion-tokens'
build-prerelease / Resolve version stamps + change detection (push) Successful in 21s
build-prerelease / Build helexa-bench binary (push) Successful in 1m50s
build-prerelease / Build cortex binary (push) Successful in 2m25s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m28s
build-prerelease / Build neuron-ampere (push) Successful in 2m51s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m8s
build-prerelease / Test (push) Successful in 9m31s
build-prerelease / Build neuron-blackwell (push) Successful in 15m33s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Successful in 25s
build-prerelease / Package helexa-upstream RPM (push) Successful in 23s
build-prerelease / Package cortex RPM (push) Successful in 24s
build-prerelease / Build helexa-router binary (push) Successful in 1m20s
build-prerelease / Package helexa-router RPM (push) Successful in 23s
build-prerelease / Build neuron-ada (push) Successful in 2m14s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m3s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 37s
2026-08-15 12:36:55 +03:00
grenade af7a1e578b Merge branch 'fix/openai-null-assistant-content'
build-prerelease / Build neuron-blackwell (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 12s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m42s
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Build cortex binary (push) Successful in 2m33s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m58s
build-prerelease / Build neuron-ampere (push) Successful in 3m5s
build-prerelease / Test (push) Successful in 7m25s
build-prerelease / Package helexa-upstream RPM (push) Successful in 20s
build-prerelease / Package cortex RPM (push) Successful in 24s
build-prerelease / Build helexa-router binary (push) Has been cancelled
build-prerelease / Build helexa-tools binary (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package helexa-router RPM (push) Has been cancelled
build-prerelease / Package helexa-tools RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-15 12:24:36 +03:00
grenade e4a45f00ca fix(wire): honour max_completion_tokens as the output cap
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m5s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 10m1s
CI / CUDA type-check (push) Successful in 23m2s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
OpenAI deprecated `max_tokens` in favour of `max_completion_tokens`, and
current SDKs and clients send only the new field. neuron read only
`max_tokens`, so the new spelling fell into the flattened `extra` and
every generation loop missed it — a caller asking for 3 tokens got 149,
generating to the server default instead of the requested cap.

cortex already meters against `max_completion_tokens` in preference to
`max_tokens` (`metering::requested_max_output`), so the gateway was
reserving budget for a ceiling the node did not enforce.

Make it a named field, resolve both spellings through
`effective_max_tokens()` — current field wins, so what we bill and what
we generate agree — and collapse them onto `max_tokens` once at
neuron's chat entry point, which covers all four generation loops and
the batch engine without touching them.

Sending both spellings stays valid rather than becoming a duplicate-
field error, which a serde alias would have made it.
2026-08-15 12:06:55 +03:00
grenade 3c1dececeb fix(wire): accept "content": null on assistant tool-call turns
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m6s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 7m15s
CI / CUDA type-check (push) Successful in 22m4s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
An OpenAI-native agentic client replays its own assistant turn on the
follow-up request that carries the tool result. OpenAI emits that turn
with `"content": null` — the payload was only `tool_calls` — so the
replay carries null too. `ChatMessage::content` was a non-optional
untagged enum of `String | [parts]`, so axum's `Json` extractor
rejected the whole request with 422 before neuron's handler ran:

    messages[1].content: data did not match any variant of
    untagged enum MessageContent

The failure lands on turn two, after the first tool call has already
succeeded, and the 422 body is axum's raw deserialization message
rather than an error envelope — so the client sees its agentic loop
die with nothing it can parse.

Model the third wire shape explicitly: `MessageContent::Null`, matching
`null` or an absent `content` key (`#[serde(default)]`), serializing
back to `null`. HF chat templates already model it the same way
(`content is none` renders as empty), so it maps straight through to
the prompt and the assistant's `tool_calls` render as they should.

The Anthropic path was unaffected — its translation synthesises an
empty-string content for tool-use turns — which is why Claude Code
worked and OpenAI-native tool callers did not.
2026-08-15 11:59:18 +03:00
27 changed files with 3073 additions and 170 deletions
+27 -1
View File
@@ -105,6 +105,11 @@ jobs:
echo "commit_timestamp=${COMMIT_TIMESTAMP}" >> "$GITHUB_OUTPUT"
- id: changes
env:
# The push's own range. Empty on workflow_dispatch and all
# zeroes for a new branch; `decide` treats both as unusable and
# falls back to the full gate.
PUSH_BEFORE: ${{ github.event.before }}
run: |
set -ux
# Default: build everything. Detection only ever narrows
@@ -189,9 +194,30 @@ jobs:
BUILD_TOOLS=$(decide "$TOOLS_BASE" "$TOOLS_RE")
BUILD_ANGELS=$(decide "$ANGELS_BASE" "$ANGELS_RE")
if [ "$BUILD_CORTEX" = "true" ] || [ "$BUILD_NEURON" = "true" ] || [ "$BUILD_BENCH" = "true" ] || [ "$BUILD_UPSTREAM" = "true" ] || [ "$BUILD_ROUTER" = "true" ] || [ "$BUILD_TOOLS" = "true" ] || [ "$BUILD_ANGELS" = "true" ]; then
# Anything we are about to build gets linted and tested.
# This is the invariant that matters: no artefact reaches
# the fleet unchecked, however the branch below behaves.
CHECK_RUST=true
else
CHECK_RUST=$(decide "$CORTEX_BASE" "$RUST_RE")
# Nothing is being built, so nothing will be published.
# The only question left is whether THIS PUSH touched
# Rust — and the honest base for that is the push's own
# range, not a package's publication lag.
#
# This used to ask "has anything Rust changed since
# cortex was last published". Cortex is only rebuilt when
# cortex paths change, so after a run of neuron-only work
# its base sits days behind and the diff from it sweeps in
# every unrelated Rust commit since. A one-character
# whitespace fix to a doc then re-ran fmt, clippy and the
# full test suite over a delta that had already been
# linted on the branch that introduced it (2026-08-18,
# base fec5b33).
#
# `decide` returns true for an unusable base, so a new
# branch (all-zero before-sha) or a force-push still gets
# the full gate.
CHECK_RUST=$(decide "$PUSH_BEFORE" "$RUST_RE")
fi
fi
fi
+46 -9
View File
@@ -39,24 +39,34 @@ env:
CARGO_TARGET_DIR: target-${{ github.job }}
jobs:
# Two independent questions: does this change need the Rust gate, and
# does it need the web gate.
#
# The SPA under helexa.ai/ is built and shipped by its own deploy job and
# is not read by any crate, build script or spec file — so a change
# confined to it cannot alter a Rust result. Waiting up to 70 minutes for
# a CUDA type-check to prove that is pure latency, and the wait is what
# tempts people to merge front-end work unchecked.
# tempts people to merge front-end work unchecked. Documentation is inert
# the same way, and more obviously: no crate `include_str!`s a markdown
# file, no spec packages `doc/`, no build script reads one.
#
# The classification is deliberately one-directional: it only ever
# narrows the gate when *every* changed path is front-end, and every
# Conversely a Rust-only change cannot break the SPA, so it need not
# spend three minutes on `npm ci` + lint + typecheck + build.
#
# The classification stays deliberately one-directional: it only ever
# narrows the gate when *every* changed path is known-inert, and every
# other outcome — a tag, an unknown base, an empty diff, a mixed
# changeset — falls through to the full gate. A new top-level directory
# is therefore covered on the day it appears, without anyone
# remembering to add it here.
# remembering to add it here. Adding an inert pattern is the only way to
# widen what gets skipped, which is the property worth keeping.
changes:
name: Classify changes
timeout-minutes: 10
runs-on: fedora-43
outputs:
rust: ${{ steps.classify.outputs.rust }}
web: ${{ steps.classify.outputs.web }}
steps:
- uses: actions/checkout@v4
with:
@@ -64,7 +74,12 @@ jobs:
- id: classify
run: |
set -u
full() { echo "→ full gate: $1"; echo "rust=true" >> "$GITHUB_OUTPUT"; exit 0; }
full() {
echo "→ full gate: $1"
echo "rust=true" >> "$GITHUB_OUTPUT"
echo "web=true" >> "$GITHUB_OUTPUT"
exit 0
}
# Tags build the SRPMs from the whole tree, and the srpm jobs
# gate on fmt/clippy/test/cuda-check. Never narrow there.
@@ -86,11 +101,31 @@ jobs:
echo "changed files:"
printf '%s\n' "${files}" | sed 's/^/ /'
other="$(printf '%s\n' "${files}" | grep -v '^helexa\.ai/' || true)"
[ -z "${other}" ] || full "changes outside helexa.ai/"
# Inert for Rust: the SPA, and documentation. Markdown is
# matched only at the repo root and under doc/ — a .md inside
# crates/ is left to fall through, since `include_str!` could
# make one load-bearing without this file knowing.
rust_relevant="$(printf '%s\n' "${files}" \
| grep -v '^helexa\.ai/' \
| grep -v '^doc/' \
| grep -vE '^[^/]+\.md$' \
|| true)"
if [ -n "${rust_relevant}" ]; then
echo "rust=true" >> "$GITHUB_OUTPUT"
echo "→ Rust gate: required"
else
echo "rust=false" >> "$GITHUB_OUTPUT"
echo "→ Rust gate: skipped (front-end and/or docs only)"
fi
echo "→ front-end only: skipping the Rust gate"
echo "rust=false" >> "$GITHUB_OUTPUT"
# The web gate is needed only when the SPA itself changed.
if printf '%s\n' "${files}" | grep -q '^helexa\.ai/'; then
echo "web=true" >> "$GITHUB_OUTPUT"
echo "→ web gate: required"
else
echo "web=false" >> "$GITHUB_OUTPUT"
echo "→ web gate: skipped (nothing under helexa.ai/)"
fi
fmt:
name: Format
@@ -106,6 +141,8 @@ jobs:
name: Web (lint + typecheck + i18n + build)
timeout-minutes: 20
runs-on: fedora-43
needs: changes
if: needs.changes.outputs.web == 'true'
defaults:
run:
working-directory: helexa.ai
+47
View File
@@ -109,6 +109,40 @@ pub struct ModelLoad {
/// (#54/#137). `#[serde(default)]` for back-compat.
#[serde(default)]
pub rejected_per_principal: u64,
/// Anonymous requests currently holding a seat (#262). Read against
/// `in_flight` it says how much of this model's load is unattributable;
/// against `anon_max_in_flight`, how close that traffic is to the
/// ceiling that keeps it from starving identified callers.
#[serde(default)]
pub anon_in_flight: usize,
/// Seats anonymous traffic may hold at once (#262); `0` = refused
/// outright. `#[serde(default)]` (→ 0) for pre-#262 neurons, which
/// applied no ceiling at all — cortex skips the gauge rather than
/// publish a 0 that reads as the opposite of the truth.
#[serde(default)]
pub anon_max_in_flight: usize,
/// Cumulative anonymous requests refused for want of leftover capacity
/// (#262). Rising here with a flat `rejected_timeout` means the
/// reservation is working, not that the model is overloaded.
#[serde(default)]
pub rejected_anon_yield: u64,
/// Cumulative requests that waited for KV budget and never got it
/// (#257) — the model is saturated by long-context sequences rather
/// than by request count. `#[serde(default)]` for back-compat.
#[serde(default)]
pub rejected_kv_timeout: u64,
/// Cumulative requests whose KV reservation exceeded the model's entire
/// budget (#257) — prompts too long for this node at any load, not a
/// load-shedding signal. `#[serde(default)]` for back-compat.
#[serde(default)]
pub rejected_kv_unservable: u64,
/// This model's total KV budget in MiB and how much of it is currently
/// unreserved (#257). `kv_budget_mb == 0` means the gate is disabled
/// (CPU load, or an arch with no context profile).
#[serde(default)]
pub kv_budget_mb: u64,
#[serde(default)]
pub kv_available_mb: u64,
/// Live prefill throughput EMA in tokens/sec (#137) — prompt tokens
/// processed per second. `0.0` before the first sample. `#[serde(default)]`
/// for back-compat with pre-#137 neurons.
@@ -148,8 +182,15 @@ mod health_load_tests {
max_in_flight: 8,
max_queue_depth: 8,
rejected_queue_full: 0,
rejected_kv_timeout: 0,
rejected_kv_unservable: 0,
kv_budget_mb: 0,
kv_available_mb: 0,
rejected_timeout: 0,
rejected_per_principal: 0,
anon_in_flight: 2,
anon_max_in_flight: 7,
rejected_anon_yield: 0,
tok_s_prefill: 0.0,
tok_s_decode: 0.0,
}],
@@ -161,6 +202,8 @@ mod health_load_tests {
assert_eq!(back.models[0].queue_depth, 3);
assert_eq!(back.models[0].max_in_flight, 8);
assert_eq!(back.models[0].max_queue_depth, 8);
assert_eq!(back.models[0].anon_in_flight, 2);
assert_eq!(back.models[0].anon_max_in_flight, 7);
}
#[test]
@@ -172,6 +215,10 @@ mod health_load_tests {
assert_eq!(m.in_flight, 2);
assert_eq!(m.max_in_flight, 0);
assert_eq!(m.max_queue_depth, 0);
// Likewise a pre-#262 neuron, which enforced no anonymous ceiling.
assert_eq!(m.anon_in_flight, 0);
assert_eq!(m.anon_max_in_flight, 0);
assert_eq!(m.rejected_anon_yield, 0);
}
}
+169 -2
View File
@@ -18,8 +18,16 @@ pub struct ChatCompletionRequest {
pub temperature: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub top_p: Option<f64>,
/// The deprecated spelling of the output cap. Still what most
/// clients send; see [`ChatCompletionRequest::effective_max_tokens`].
#[serde(skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u64>,
/// OpenAI's current spelling of the output cap, which deprecated
/// `max_tokens`. Newer SDKs and clients send only this one — it has
/// to be a named field rather than falling into `extra`, or the cap
/// is silently ignored and generation runs to the server default.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_completion_tokens: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub stream: Option<bool>,
/// All other fields (tools, response_format, backend extensions, etc.)
@@ -27,20 +35,49 @@ pub struct ChatCompletionRequest {
pub extra: Value,
}
impl ChatCompletionRequest {
/// The output cap the caller asked for, under either spelling.
///
/// `max_completion_tokens` wins when both are present — it is the
/// current OpenAI field, and cortex's metering already reserves
/// against it (`metering::requested_max_output`), so preferring it
/// here keeps what we bill and what we generate in agreement.
pub fn effective_max_tokens(&self) -> Option<u64> {
self.max_completion_tokens.or(self.max_tokens)
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ChatMessage {
pub role: String,
/// Absent on an assistant turn that carries only `tool_calls` —
/// see [`MessageContent::Null`]. `#[serde(default)]` so a message
/// with no `content` key at all deserializes rather than being
/// rejected.
#[serde(default)]
pub content: MessageContent,
#[serde(flatten)]
pub extra: Value,
}
/// Content can be a simple string or an array of content parts (for vision).
#[derive(Debug, Clone, Serialize, Deserialize)]
/// Content can be a simple string, an array of content parts (for
/// vision), or absent.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(untagged)]
pub enum MessageContent {
Text(String),
Parts(Vec<Value>),
/// `"content": null`, or no `content` key at all.
///
/// This is the OpenAI-canonical shape for an assistant turn whose
/// only payload is `tool_calls`, and agentic clients replay the
/// assistant turn verbatim on the follow-up request that carries
/// the tool result — so any client doing OpenAI-native tool
/// calling sends it on its second turn. HF chat templates model it
/// the same way (`content is none` renders as empty), so it maps
/// straight through to the prompt.
#[default]
Null,
}
// ── Chat completion response (non-streaming) ─────────────────────────
@@ -170,3 +207,133 @@ pub struct ModelObject {
#[serde(flatten)]
pub extra: Value,
}
#[cfg(test)]
mod tests {
use super::*;
/// The follow-up request an OpenAI-native agentic client sends
/// after running a tool: it replays its own assistant turn, which
/// OpenAI emits with `"content": null` because the payload was
/// only `tool_calls`. Rejecting this shape breaks every such
/// client on its second turn.
#[test]
fn assistant_tool_call_turn_with_null_content_deserializes() {
let req: ChatCompletionRequest = serde_json::from_str(
r#"{
"model": "m",
"messages": [
{"role": "user", "content": "list the files"},
{"role": "assistant", "content": null, "tool_calls": [
{"id": "call_0", "type": "function",
"function": {"name": "bash", "arguments": "{\"command\":\"ls\"}"}}
]},
{"role": "tool", "content": "a.txt", "tool_call_id": "call_0"}
]
}"#,
)
.expect("null assistant content is a valid OpenAI request");
assert!(matches!(req.messages[1].content, MessageContent::Null));
// The tool calls survive in `extra`, where the chat template
// reads them from.
assert_eq!(
req.messages[1].extra["tool_calls"][0]["function"]["name"],
"bash"
);
assert_eq!(req.messages[2].extra["tool_call_id"], "call_0");
}
/// Some clients omit the key entirely rather than sending null.
#[test]
fn assistant_turn_with_no_content_key_deserializes() {
let msg: ChatMessage = serde_json::from_str(
r#"{"role": "assistant", "tool_calls": [
{"id": "c", "type": "function",
"function": {"name": "bash", "arguments": "{}"}}]}"#,
)
.expect("absent content is a valid OpenAI message");
assert!(matches!(msg.content, MessageContent::Null));
}
/// Absent content round-trips back onto the wire as null, so a
/// translated or re-serialized request stays OpenAI-shaped.
#[test]
fn null_content_round_trips() {
let msg = ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Null,
};
let v = serde_json::to_value(&msg).expect("serialize");
assert_eq!(v["content"], Value::Null);
assert!(v.get("content").is_some(), "content key must be present");
}
/// Newer OpenAI SDKs send only `max_completion_tokens`; the cap
/// must survive as a named field rather than falling into `extra`,
/// where every generation loop would miss it.
#[test]
fn max_completion_tokens_is_the_effective_cap() {
let req: ChatCompletionRequest = serde_json::from_str(
r#"{"model": "m", "max_completion_tokens": 240,
"messages": [{"role": "user", "content": "hi"}]}"#,
)
.expect("deserialize");
assert_eq!(req.max_completion_tokens, Some(240));
assert_eq!(req.max_tokens, None);
assert_eq!(req.effective_max_tokens(), Some(240));
assert!(
req.extra.get("max_completion_tokens").is_none(),
"must be a named field, not swept into extra"
);
}
/// The legacy spelling keeps working on its own.
#[test]
fn legacy_max_tokens_still_caps() {
let req: ChatCompletionRequest = serde_json::from_str(
r#"{"model": "m", "max_tokens": 128,
"messages": [{"role": "user", "content": "hi"}]}"#,
)
.expect("deserialize");
assert_eq!(req.effective_max_tokens(), Some(128));
}
/// Clients that send both for compatibility must not be rejected,
/// and the current field wins — matching what cortex meters
/// against, so billed and generated caps agree.
#[test]
fn both_spellings_resolve_to_the_current_field() {
let req: ChatCompletionRequest = serde_json::from_str(
r#"{"model": "m", "max_tokens": 99, "max_completion_tokens": 256,
"messages": [{"role": "user", "content": "hi"}]}"#,
)
.expect("sending both must not be an error");
assert_eq!(req.effective_max_tokens(), Some(256));
}
/// Neither spelling present leaves the cap to the server default.
#[test]
fn absent_cap_is_none() {
let req: ChatCompletionRequest = serde_json::from_str(
r#"{"model": "m", "messages": [{"role": "user", "content": "hi"}]}"#,
)
.expect("deserialize");
assert_eq!(req.effective_max_tokens(), None);
}
/// The string and array forms keep working unchanged.
#[test]
fn text_and_parts_content_still_deserialize() {
let text: ChatMessage =
serde_json::from_str(r#"{"role": "user", "content": "hi"}"#).expect("text");
assert!(matches!(text.content, MessageContent::Text(ref t) if t == "hi"));
let parts: ChatMessage = serde_json::from_str(
r#"{"role": "user", "content": [{"type": "text", "text": "hi"}]}"#,
)
.expect("parts");
assert!(matches!(parts.content, MessageContent::Parts(ref p) if p.len() == 1));
}
}
+3
View File
@@ -102,6 +102,8 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
temperature: req.temperature,
top_p: req.top_p,
max_tokens: Some(req.max_tokens),
// Anthropic has one spelling; it maps onto the legacy field.
max_completion_tokens: None,
stream: req.stream,
extra,
}
@@ -387,6 +389,7 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
let text = match c.message.content {
MessageContent::Text(t) => t,
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
MessageContent::Null => String::new(),
};
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
(text, stop)
+9 -1
View File
@@ -111,7 +111,15 @@ fn describe_metrics() {
);
metrics::describe_counter!(
"cortex_model_rejections_total",
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal — the load-shedding signal (#137)"
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal / anon_yield — the load-shedding signal (#137, #262)"
);
metrics::describe_gauge!(
"cortex_model_anon_in_flight",
"Anonymous (unattributable) requests holding a seat on a neuron:model (#262)"
);
metrics::describe_gauge!(
"cortex_model_anon_max_in_flight",
"Seats anonymous traffic may hold at once, so it cannot starve identified callers (#262)"
);
metrics::describe_gauge!(
"cortex_model_tok_s_decode",
+15
View File
@@ -380,6 +380,21 @@ fn export_health_metrics(node: &str, h: &HealthResponse) {
counter!("cortex_model_rejections_total",
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "per_principal")
.absolute(m.rejected_per_principal);
counter!("cortex_model_rejections_total",
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "anon_yield")
.absolute(m.rejected_anon_yield);
// How much of this model's load is unattributable (#262), and the
// ceiling that keeps it from starving identified callers. 0 for a
// pre-#262 neuron, which had no ceiling — skip rather than publish
// a 0 that reads as "anonymous is refused" when it means the
// opposite.
gauge!("cortex_model_anon_in_flight", "node" => node.to_string(), "model" => m.id.clone())
.set(m.anon_in_flight as f64);
if m.anon_max_in_flight > 0 {
gauge!("cortex_model_anon_max_in_flight",
"node" => node.to_string(), "model" => m.id.clone())
.set(m.anon_max_in_flight as f64);
}
}
for d in &h.devices {
let device = d.index.to_string();
+40 -9
View File
@@ -71,15 +71,46 @@ pub async fn forward_request(
})?;
if !response.status().is_success() {
// Streaming body — can't snippet without breaking the stream
// pass-through. Log status + URL; the client still gets the
// upstream status, just without the leaked body.
tracing::warn!(
node = %route.node_name,
url = %url,
status = response.status().as_u16(),
"proxy: upstream returned non-2xx"
);
// Buffer the error body so the journal records WHY, then hand the
// same bytes on. Streaming is what a success body needs; an error
// body is a few hundred bytes and streaming it only guarantees
// that nobody ever sees the reason.
//
// This was a real dead end: neuron answered 503 `insufficient_vram`
// with the deficit in the body, and cortex logged only
// `status=503`. Neither journal held the cause, and the guard
// responsible could only be found by reading neuron's source
// (#257). The body is upstream-generated (our own #63 envelope),
// not caller content, and it is truncated in the log.
const MAX_ERROR_BODY: usize = 64 * 1024;
let status = response.status();
let (parts, body) = response.into_parts();
return match axum::body::to_bytes(body, MAX_ERROR_BODY).await {
Ok(bytes) => {
let snippet: String = String::from_utf8_lossy(&bytes).chars().take(600).collect();
tracing::warn!(
node = %route.node_name,
url = %url,
status = status.as_u16(),
body = %snippet,
"proxy: upstream returned non-2xx"
);
Ok(Response::from_parts(parts, axum::body::Body::from(bytes)))
}
// Over the cap or a read failure: the body is consumed and
// cannot be handed on. Say so rather than passing an empty
// body off as the upstream's answer.
Err(e) => {
tracing::warn!(
node = %route.node_name,
url = %url,
status = status.as_u16(),
error = %e,
"proxy: upstream returned non-2xx (body unreadable, dropped)"
);
Ok(Response::from_parts(parts, axum::body::Body::empty()))
}
};
}
Ok(response)
@@ -49,8 +49,15 @@ async fn seed_loaded(fleet: &CortexState, node: &str, in_flight: usize, queue_de
max_in_flight: 8,
max_queue_depth: 8,
rejected_queue_full: 0,
rejected_kv_timeout: 0,
rejected_kv_unservable: 0,
kv_budget_mb: 0,
kv_available_mb: 0,
rejected_timeout: 0,
rejected_per_principal: 0,
anon_in_flight: 0,
anon_max_in_flight: 7,
rejected_anon_yield: 0,
tok_s_prefill: 0.0,
tok_s_decode: 0.0,
},
+131 -7
View File
@@ -27,7 +27,15 @@ use sqlx::Row;
use uuid::Uuid;
pub fn router(state: &AppState) -> Router<AppState> {
let protected = Router::new()
// Session-only: identity management and the full account view.
//
// Key management stays here deliberately (#264). If an inference key
// could mint keys, one leaked credential would become a foothold that
// SURVIVES revoking the original — a spending problem turning into an
// eviction problem. `/account` stays too: it carries `angel_access`,
// and whatever field is added to it next would otherwise be exposed to
// inference credentials by default.
let session_only = Router::new()
.route("/web/v1/account", get(account))
.route("/web/v1/keys", get(list_keys).post(create_key))
.route("/web/v1/keys/{id}/archive", post(archive_key))
@@ -35,11 +43,23 @@ pub fn router(state: &AppState) -> Router<AppState> {
"/web/v1/keys/{id}/limit",
axum::routing::patch(update_key_limit),
)
.layer(axum::middleware::from_fn_with_state(
state.clone(),
require_session,
));
// Session **or** API key: reading and refilling the allocation.
//
// A service that watches its own balance should not have to hold its
// owner's password — the credential that can mint and revoke keys — to
// read a number it is already entitled to spend against.
let account_auth = Router::new()
.route("/web/v1/allocation", get(allocation))
.route("/web/v1/redeem", post(redeem))
.route("/web/v1/topup/request", post(request_topup))
.layer(axum::middleware::from_fn_with_state(
state.clone(),
require_session,
require_account,
));
Router::new()
@@ -49,7 +69,8 @@ pub fn router(state: &AppState) -> Router<AppState> {
.route("/web/v1/login", post(login))
.route("/web/v1/password-reset/request", post(reset_request))
.route("/web/v1/password-reset/confirm", post(reset_confirm))
.merge(protected)
.merge(session_only)
.merge(account_auth)
}
// ── errors ──────────────────────────────────────────────────────────
@@ -142,6 +163,71 @@ async fn require_session(State(state): State<AppState>, mut req: Request, next:
}
}
/// Authenticated **account**, injected by [`require_account`]. Unlike
/// [`AuthUser`] this says nothing about *who* is calling — only which
/// account the call bills to — because an API key identifies an account
/// without identifying a person.
#[derive(Clone, Copy)]
struct AuthAccount(Uuid);
/// Accept **either** a web session or an inference API key, and resolve
/// both to an account id (#264).
///
/// Routes behind this may read and refill an allocation. They may not
/// touch keys: a service watching its own balance should not have to
/// hold the credential that can mint and revoke credentials. See the
/// router for why key management stays session-only.
///
/// The two token shapes are unambiguous — API keys carry the
/// `sk-helexa-` prefix (`crypto::generate_api_key`) and JWTs cannot — so
/// this dispatches on the prefix rather than trying to decode a session
/// and falling back, which would log a decode failure for every
/// perfectly valid key.
async fn require_account(State(state): State<AppState>, mut req: Request, next: Next) -> Response {
let Some(token) = req
.headers()
.get(header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
.map(str::trim)
.map(str::to_string)
else {
return WebError::Unauthorized.into_response();
};
let account = if token.starts_with("sk-helexa-") {
// `resolve_key` already requires both key and account to be
// `active`, so an archived key or a deactivated account
// authenticates nothing — no extra check needed here.
match crate::ledger::resolve_key(&state.pool, &sha256(&token)).await {
Ok(Some(principal)) => principal.account_id,
Ok(None) => return WebError::Unauthorized.into_response(),
Err(e) => {
tracing::error!(error = %e, "key resolve failed on account route");
return WebError::Internal.into_response();
}
}
} else {
let uid = decode::<Claims>(
&token,
&DecodingKey::from_secret(state.config.auth.jwt_secret.as_bytes()),
&Validation::default(),
)
.ok()
.and_then(|d| Uuid::parse_str(&d.claims.sub).ok());
let Some(uid) = uid else {
return WebError::Unauthorized.into_response();
};
match account_id_for(&state, uid).await {
Ok(a) => a,
Err(e) => return e.into_response(),
}
};
req.extensions_mut().insert(AuthAccount(account));
next.run(req).await
}
/// The caller's single account id.
async fn account_id_for(state: &AppState, user_id: Uuid) -> WebResult<Uuid> {
let row = sqlx::query("SELECT id FROM accounts WHERE owner_user_id = $1")
@@ -635,6 +721,46 @@ async fn account(
.into_response())
}
/// `GET /web/v1/allocation` — the account's token balance, readable with
/// **either** a session or an API key (#264).
///
/// Deliberately narrower than [`account`]: balance and top-up
/// availability, nothing else. `/account` also returns `angel_access`,
/// and whatever field is added to it next would otherwise be handed to
/// every inference key by default. A separate projection means widening
/// the account view can never widen what a key can see.
async fn allocation(
State(state): State<AppState>,
Extension(AuthAccount(acct)): Extension<AuthAccount>,
) -> WebResult<Response> {
let row = sqlx::query(
"SELECT allocation_total, allocation_spent, allocation_reserved FROM accounts WHERE id = $1",
)
.bind(acct)
.fetch_one(&state.pool)
.await?;
// Same display-only eligibility the dashboard uses, so an automated
// caller can check before asking and avoid provoking a 409. The grant
// path re-checks, so a stale read cannot conjure a top-up.
let (topup_available, topup_reason) =
match crate::topup::auto_eligibility(&state.pool, acct).await {
Ok(Ok(())) => (true, None),
Ok(Err(reason)) => (false, Some(reason.to_string())),
Err(e) => {
tracing::warn!(error = %e, "top-up eligibility check failed");
(false, None)
}
};
Ok(Json(json!({
"allocation_total": row.get::<i64, _>("allocation_total"),
"allocation_spent": row.get::<i64, _>("allocation_spent"),
"allocation_reserved": row.get::<i64, _>("allocation_reserved"),
"topup_available": topup_available,
"topup_reason": topup_reason,
}))
.into_response())
}
/// Whether this user holds any investor-portal grant.
///
/// A **boolean, and deliberately nothing more**. helexa.ai is a static
@@ -680,10 +806,9 @@ async fn has_angel_access(state: &AppState, user_id: Uuid) -> bool {
/// caller is authenticated and asking about their own account.
async fn request_topup(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Extension(AuthAccount(acct)): Extension<AuthAccount>,
) -> WebResult<Response> {
use crate::topup::AutoTopUpError;
let acct = account_id_for(&state, user.0).await?;
match crate::topup::auto_grant(&state.pool, acct).await {
Ok(grant) => {
tracing::info!(
@@ -859,10 +984,9 @@ struct RedeemReq {
/// or already-redeemed code (no oracle).
async fn redeem(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Extension(AuthAccount(acct)): Extension<AuthAccount>,
Json(req): Json<RedeemReq>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
match crate::topup::redeem(&state.pool, acct, &req.code).await {
Ok(new_total) => Ok(Json(json!({ "allocation_total": new_total })).into_response()),
Err(crate::topup::TopUpError::Invalid) => {
+154
View File
@@ -424,3 +424,157 @@ async fn topup_concurrent_double_redeem_one_winner() {
.count();
assert_eq!(wins, 1, "exactly one redemption wins the single-use code");
}
/// An API key may read its own allocation and refill it, but must never
/// reach key management (#264).
///
/// The second half is the assertion that matters. If an inference key
/// could mint keys, one leaked credential would become a foothold that
/// survives revoking the original — a spending problem turning into an
/// eviction problem. The first half is the point of the change: a service
/// watching its own balance should not need the account password.
#[tokio::test]
async fn api_key_reads_allocation_but_cannot_touch_keys() {
let Some((base, pool)) = spawn_or_skip("api_key_reads_allocation_but_cannot_touch_keys").await
else {
return;
};
let client = reqwest::Client::new();
let email = unique_email();
post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123"}),
None,
)
.await;
pool.execute(
sqlx::query("UPDATE users SET email_verified = true WHERE email = $1").bind(&email),
)
.await
.unwrap();
let session = post(
format!("{base}/web/v1/login"),
json!({"email": email, "password": "password123"}),
None,
)
.await
.json::<Value>()
.await
.unwrap()["token"]
.as_str()
.unwrap()
.to_string();
let created = post(
format!("{base}/web/v1/keys"),
json!({"label": "service"}),
Some(&session),
)
.await
.json::<Value>()
.await
.unwrap();
let raw_key = created["key"].as_str().unwrap().to_string();
let key_id = created["id"].as_str().unwrap().to_string();
// ── the key CAN read its own allocation ──────────────────────────
let r = client
.get(format!("{base}/web/v1/allocation"))
.bearer_auth(&raw_key)
.send()
.await
.unwrap();
assert_eq!(
r.status(),
200,
"an API key must be able to read allocation"
);
let body: Value = r.json().await.unwrap();
assert!(body["allocation_total"].is_i64());
assert!(body["allocation_spent"].is_i64());
assert!(body["allocation_reserved"].is_i64());
assert!(body["topup_available"].is_boolean());
// Narrower than /account on purpose: nothing about the human.
assert!(
body.get("angel_access").is_none(),
"allocation must not leak account-holder fields to an inference key: {body}"
);
// A session reaches the same route.
let r = client
.get(format!("{base}/web/v1/allocation"))
.bearer_auth(&session)
.send()
.await
.unwrap();
assert_eq!(r.status(), 200, "a session must still work here");
// ── the key CANNOT manage keys ───────────────────────────────────
let r = post(
format!("{base}/web/v1/keys"),
json!({"label": "minted-by-a-key"}),
Some(&raw_key),
)
.await;
assert_eq!(
r.status(),
401,
"an API key must not be able to mint another key"
);
let r = client
.get(format!("{base}/web/v1/keys"))
.bearer_auth(&raw_key)
.send()
.await
.unwrap();
assert_eq!(r.status(), 401, "an API key must not be able to list keys");
let r = post(
format!("{base}/web/v1/keys/{key_id}/archive"),
json!({}),
Some(&raw_key),
)
.await;
assert_eq!(
r.status(),
401,
"an API key must not be able to archive a key"
);
// /account stays session-only — it carries account-holder fields.
let r = client
.get(format!("{base}/web/v1/account"))
.bearer_auth(&raw_key)
.send()
.await
.unwrap();
assert_eq!(r.status(), 401, "/account must remain session-only");
// ── an archived key authenticates nothing ────────────────────────
let r = post(
format!("{base}/web/v1/keys/{key_id}/archive"),
json!({}),
Some(&session),
)
.await;
assert!(r.status().is_success(), "archive via session should work");
let r = client
.get(format!("{base}/web/v1/allocation"))
.bearer_auth(&raw_key)
.send()
.await
.unwrap();
assert_eq!(r.status(), 401, "an archived key must authenticate nothing");
// Garbage bearers are refused on both shapes.
for bogus in ["sk-helexa-nonsense", "not-a-jwt"] {
let r = client
.get(format!("{base}/web/v1/allocation"))
.bearer_auth(bogus)
.send()
.await
.unwrap();
assert_eq!(r.status(), 401, "bogus bearer {bogus} must be refused");
}
}
+128 -9
View File
@@ -97,7 +97,7 @@ async fn list_models(State(state): State<Arc<NeuronState>>) -> impl IntoResponse
async fn load_model(
State(state): State<Arc<NeuronState>>,
Json(spec): Json<ModelSpec>,
ApiJson(spec): ApiJson<ModelSpec>,
) -> impl IntoResponse {
// Driver/library mismatch preflight (#19): every CUDA load is
// guaranteed to fail until the host reboots. Reject up front with
@@ -168,7 +168,7 @@ fn preflight_kind(err: &PreflightError) -> &'static str {
async fn unload_model(
State(state): State<Arc<NeuronState>>,
Json(body): Json<Value>,
ApiJson(body): ApiJson<Value>,
) -> impl IntoResponse {
let model_id = match body.get("model_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
@@ -284,7 +284,7 @@ fn principal_key(headers: &axum::http::HeaderMap) -> Option<String> {
async fn chat_completions(
State(state): State<Arc<NeuronState>>,
headers: axum::http::HeaderMap,
Json(mut req): Json<ChatCompletionRequest>,
ApiJson(mut req): ApiJson<ChatCompletionRequest>,
) -> impl IntoResponse {
let Some(candle) = state.candle.as_ref().map(Arc::clone) else {
return (
@@ -294,6 +294,13 @@ async fn chat_completions(
.into_response();
};
// Collapse the two spellings of the output cap onto the one field
// the generation loops read. Clients that send only
// `max_completion_tokens` (OpenAI's current field — the DeepSeek
// Harness among them) would otherwise have their cap silently
// ignored and generate to the server default.
req.max_tokens = req.effective_max_tokens();
// Reasoning-content opt-in. Off by default → naïve clients
// (Zed's commit-message generator, vanilla OpenAI clients)
// never see `<think>` blocks. On when the caller sends
@@ -359,7 +366,7 @@ async fn chat_completions(
async fn responses(
State(state): State<Arc<NeuronState>>,
headers: axum::http::HeaderMap,
Json(req): Json<ResponsesRequest>,
ApiJson(req): ApiJson<ResponsesRequest>,
) -> impl IntoResponse {
let Some(candle) = state.candle.as_ref().map(Arc::clone) else {
return (
@@ -431,11 +438,11 @@ async fn responses(
.first()
.map(|c| match &c.message.content {
MessageContent::Text(t) => t.clone(),
MessageContent::Parts(_) => {
MessageContent::Parts(_) | MessageContent::Null => {
// Candle output is always text today;
// a Parts response would be surprising.
// Empty-string fallback is safer than
// a panic.
// a Parts or absent response would be
// surprising. Empty-string fallback is
// safer than a panic.
String::new()
}
})
@@ -483,7 +490,7 @@ async fn responses(
async fn images_generations(
State(state): State<Arc<NeuronState>>,
headers: axum::http::HeaderMap,
Json(req): Json<cortex_core::images::ImagesGenerationRequest>,
ApiJson(req): ApiJson<cortex_core::images::ImagesGenerationRequest>,
) -> impl IntoResponse {
use cortex_core::error_envelope::OpenAiError;
use cortex_core::images as wire;
@@ -644,6 +651,24 @@ fn inference_error_response(err: InferenceError) -> axum::response::Response {
.with_retry_after(5)
.with_extra("free_mb", json!(free_mb))
.with_extra("required_mb", json!(required_mb)),
// Not backpressure: an idle node would refuse this prompt too, so
// there is nothing to retry (#257). 413 rather than 400 — the
// request is well-formed, it is the payload that is too large.
InferenceError::PromptTooLongForVram {
required_mb,
budget_mb,
} => OpenAiError::new(
413,
"invalid_request_error",
"prompt_too_long_for_vram",
format!(
"prompt needs {required_mb} MiB of KV cache but this model's \
entire budget is {budget_mb} MiB. Shorten the conversation \
or use a model with more headroom."
),
)
.with_extra("required_mb", json!(required_mb))
.with_extra("budget_mb", json!(budget_mb)),
InferenceError::VisionUnsupported { model_id } => OpenAiError::new(
400,
"invalid_request_error",
@@ -704,8 +729,37 @@ fn inference_error_response(err: InferenceError) -> axum::response::Response {
"too many concurrent requests for this key; retry shortly",
)
.with_retry_after(retry_after_secs),
// Draining (#256). 503 + Retry-After, like any transient capacity
// condition: the node is coming back, and a well-behaved client
// should retry rather than surface a failure. Deliberately not
// `rate_limit_exceeded` — nothing about this is the caller's
// fault or their rate.
InferenceError::ShuttingDown => OpenAiError::new(
503,
"api_error",
"shutting_down",
"neuron is shutting down; retry shortly",
)
.with_retry_after(15),
InferenceError::Other(e) => OpenAiError::without_code(500, "api_error", format!("{e:#}")),
};
// Every failed request says why, once, here (#63). Before this the
// envelope went to the client and nothing reached the journal: a 503
// `insufficient_vram` was returned in 225 ms with no server-side
// trace, and the guard responsible could only be identified by
// reading the source and reproducing its arithmetic (#257).
//
// `extra` carries the numbers that make a rejection actionable —
// free_mb/required_mb, prompt_len/max, retry_after — so the log line
// is self-contained rather than a pointer to go and look elsewhere.
tracing::warn!(
status = env.status,
error_type = %env.error_type,
code = env.code.as_deref().unwrap_or("-"),
detail = %env.message,
extra = %serde_json::to_string(&env.extra).unwrap_or_default(),
"request rejected"
);
envelope_response(env)
}
@@ -938,3 +992,68 @@ mod error_envelope_tests {
assert!(error["message"].as_str().unwrap().contains("kaboom"));
}
}
// ── Request-body extraction (#63) ────────────────────────────────────
/// `Json<T>` that fails like the rest of our API instead of like axum.
///
/// axum's own extractor answers a malformed body with a bare string —
/// `"Failed to deserialize the JSON body into the target type: …"` —
/// at 400/422, *before any handler runs*, so nothing logs it either.
/// Both halves of that hurt: no OpenAI client can parse the body (they
/// look for `error.code` / `error.type`), and the operator sees nothing
/// at all in the journal.
///
/// That combination cost a real session (#253): a client sent a
/// spec-legal shape our types didn't model, got 422 with an unparseable
/// body, surfaced nothing to its user, and neuron logged not one line —
/// the cause was findable only by reading the wire types.
///
/// So: same #63 envelope as every other error, plus a `WARN` naming the
/// endpoint and the deserialization detail.
pub struct ApiJson<T>(pub T);
impl<T, S> axum::extract::FromRequest<S> for ApiJson<T>
where
T: serde::de::DeserializeOwned,
S: Send + Sync,
{
type Rejection = axum::response::Response;
async fn from_request(req: axum::extract::Request, state: &S) -> Result<Self, Self::Rejection> {
use axum::extract::rejection::JsonRejection;
let (method, uri) = (req.method().clone(), req.uri().clone());
match Json::<T>::from_request(req, state).await {
Ok(Json(value)) => Ok(ApiJson(value)),
Err(rejection) => {
// Distinguish "shape we don't accept" (422 — the body
// parsed as JSON but not as this request) from "not JSON
// at all" (400) and a wrong content-type (415), because
// they need different fixes from the caller.
let (status, code) = match &rejection {
JsonRejection::JsonDataError(_) => (422, "invalid_request_body"),
JsonRejection::JsonSyntaxError(_) => (400, "malformed_json"),
JsonRejection::MissingJsonContentType(_) => (415, "unsupported_media_type"),
_ => (400, "invalid_request_body"),
};
let detail = rejection.body_text();
tracing::warn!(
method = %method,
uri = %uri,
status,
code,
detail = %detail,
"request body rejected before dispatch"
);
Err(envelope_response(
cortex_core::error_envelope::OpenAiError::new(
status,
"invalid_request_error",
code,
detail,
),
))
}
}
}
}
+46 -1
View File
@@ -180,9 +180,44 @@ pub struct AdmissionConfig {
/// for any single principal (resolved from the `x-helexa-*` headers
/// cortex stamps), so one client can't monopolize the queue while others
/// wait. Over-cap → `429 rate_limit_exceeded` + `Retry-After`. `0`
/// disables the cap; anonymous requests are always exempt.
/// disables the cap.
///
/// Anonymous requests carry no principal and so cannot be capped by it.
/// They are bounded instead by `anon_max_in_flight` / `anon_max_pending`
/// below (#262).
#[serde(default = "default_admission_max_per_principal")]
pub max_per_principal: usize,
/// Ceiling on **anonymous** requests holding an in-flight seat (#262).
///
/// Anonymous traffic is served from capacity left over once identified
/// traffic is satisfied: it never takes a seat while an identified
/// request is waiting, and it may never hold every seat at once — which
/// is what this bounds, because a request already running cannot be
/// preempted. Reserving one seat is what caps how long a newly arrived
/// identified request waits: one request's duration, not all of them.
///
/// Unset (the default) derives `max_in_flight - 1`, floored at 1 so a
/// single-seat model still serves anonymous callers. `0` refuses
/// anonymous traffic outright.
#[serde(default)]
pub anon_max_in_flight: Option<usize>,
/// Ceiling on anonymous requests in flight **plus** queued (#262).
///
/// The queue needs the same reservation as the seats: without it an
/// anonymous flood fills the queue and identified callers are refused at
/// the door, before priority can help them.
///
/// Unset derives `max_in_flight + max_queue_depth - 1`, floored at 1.
#[serde(default)]
pub anon_max_pending: Option<usize>,
/// Maximum seconds a request waits for enough **KV budget** to free
/// before it is refused (#257). Deliberately far longer than
/// `max_wait_secs`: that one waits for a slot to turn over, this one
/// waits for a whole long-context sequence to finish, which on a 27B at
/// 64k context is minutes rather than seconds. Too short and the queue
/// merely converts a fast rejection into a slow one.
#[serde(default = "default_admission_kv_max_wait_secs")]
pub kv_max_wait_secs: u64,
}
impl Default for AdmissionConfig {
@@ -192,6 +227,9 @@ impl Default for AdmissionConfig {
max_queue_depth: default_admission_max_queue_depth(),
max_wait_secs: default_admission_max_wait_secs(),
max_per_principal: default_admission_max_per_principal(),
anon_max_in_flight: None,
anon_max_pending: None,
kv_max_wait_secs: default_admission_kv_max_wait_secs(),
}
}
}
@@ -328,6 +366,13 @@ fn default_bootstrap_prefill_tok_per_sec() -> f64 {
800.0
}
/// Long enough for a multi-minute long-context turn ahead of us to finish
/// and return its KV, but inside cortex's 300 s proxy timeout so the
/// caller is still listening when we answer.
fn default_admission_kv_max_wait_secs() -> u64 {
240
}
fn default_activation_headroom_mb() -> u64 {
2048
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -227,6 +227,10 @@ pub fn render_chat_template(
let content_value = match &m.content {
MessageContent::Text(s) => Value::String(s.clone()),
MessageContent::Parts(parts) => Value::Array(parts.clone()),
// Rendered as JSON null so the template's own
// `content is none` branch handles it — that is how HF
// templates model a tool-calls-only assistant turn.
MessageContent::Null => Value::Null,
};
let mut obj = serde_json::Map::new();
obj.insert("role".into(), Value::String(m.role.clone()));
@@ -417,6 +421,56 @@ mod tests {
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
}
/// The second turn of an OpenAI-native agentic loop: the client
/// replays its own assistant turn, which carries `tool_calls` and
/// `"content": null`. That must render as the tool-call block —
/// not as the literal string "none", and not as a render error
/// (which would silently drop back to the no-tools fallback
/// template and break tool calling for the rest of the session).
#[test]
fn real_template_renders_null_content_assistant_tool_call_turn() {
let template = include_str!("testdata/qwen3_6_chat_template.jinja");
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{
"id": "call_0", "type": "function",
"function": {"name": "bash", "arguments": "{\"command\":\"ls\"}"}
}]),
);
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let messages = vec![
user_msg("list the files"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text("a.txt".into()),
extra: Value::Object(tool_extras),
},
];
let out = render_chat_template(template, &messages, &Value::Null, &Value::Null)
.expect("null assistant content must render, not error");
assert!(
out.contains("<function=bash>"),
"tool call must survive the null-content turn; rendered:\n{out}"
);
assert!(
out.contains("<tool_response>\na.txt"),
"tool result must render; rendered:\n{out}"
);
assert!(
!out.contains("none"),
"null content must render as empty, not the literal \"none\"; rendered:\n{out}"
);
}
fn user_msg(text: &str) -> ChatMessage {
ChatMessage {
role: "user".into(),
@@ -850,7 +850,7 @@ fn load_gguf_inner(
tracing::info!(model = %model_id, path = ?gguf_path, "loading GGUF");
let mut file = std::fs::File::open(gguf_path).context("open GGUF file")?;
let content =
gguf_file::Content::read(&mut file).map_err(|e| anyhow::anyhow!("parse GGUF: {e}"))?;
gguf_file::Content::read(&mut file).map_err(|e| anyhow::anyhow!("parse GGUF: {e:#}"))?;
let architecture = content
.metadata
@@ -865,7 +865,7 @@ fn load_gguf_inner(
match architecture.as_str() {
"qwen3" => {
let weights = QuantizedQwen3Weights::from_gguf(content, &mut file, device)
.map_err(|e| anyhow::anyhow!("from_gguf qwen3: {e}"))?;
.map_err(|e| anyhow::anyhow!("from_gguf qwen3: {e:#}"))?;
Ok(ModelArch::Qwen3Quantized(weights))
}
"qwen3moe" => {
@@ -874,12 +874,12 @@ fn load_gguf_inner(
// accumulation precision and gives the best tokens/sec on
// consumer cards.
let weights = GGUFQWenMoE::from_gguf(content, &mut file, device, DType::F16)
.map_err(|e| anyhow::anyhow!("from_gguf qwen3_moe: {e}"))?;
.map_err(|e| anyhow::anyhow!("from_gguf qwen3_moe: {e:#}"))?;
Ok(ModelArch::Qwen3MoeQuantized(weights))
}
"llama" => {
let weights = QuantizedLlamaWeights::from_gguf(content, &mut file, device)
.map_err(|e| anyhow::anyhow!("from_gguf llama: {e}"))?;
.map_err(|e| anyhow::anyhow!("from_gguf llama: {e:#}"))?;
Ok(ModelArch::LlamaQuantized(weights))
}
other => anyhow::bail!(
@@ -943,14 +943,14 @@ fn load_dense_inner(
let cfg: qwen3_dense::Config =
serde_json::from_str(&cfg_text).context("parse Qwen3 config.json")?;
let model = qwen3_dense::ModelForCausalLM::new(&cfg, vb)
.map_err(|e| anyhow::anyhow!("build Qwen3 dense model: {e}"))?;
.map_err(|e| anyhow::anyhow!("build Qwen3 dense model: {e:#}"))?;
Ok(ModelArch::Qwen3Dense(model))
}
"qwen3_moe" => {
let cfg: qwen3_moe_dense::Config =
serde_json::from_str(&cfg_text).context("parse Qwen3 MoE config.json")?;
let model = qwen3_moe_dense::ModelForCausalLM::new(&cfg, vb)
.map_err(|e| anyhow::anyhow!("build Qwen3 MoE dense model: {e}"))?;
.map_err(|e| anyhow::anyhow!("build Qwen3 MoE dense model: {e:#}"))?;
Ok(ModelArch::Qwen3MoeDense(model))
}
"llama" => {
@@ -960,7 +960,7 @@ fn load_dense_inner(
let cache = llama_dense::Cache::new(true, dtype, &config, device)
.context("build Llama Cache")?;
let model = llama_dense::Llama::load(vb, &config)
.map_err(|e| anyhow::anyhow!("build Llama dense model: {e}"))?;
.map_err(|e| anyhow::anyhow!("build Llama dense model: {e:#}"))?;
Ok(ModelArch::LlamaDense(Box::new(
crate::harness::candle::LlamaDense::from_parts(
model,
+8 -8
View File
@@ -170,7 +170,7 @@ async fn op_step(
(BackendConfig::Single { worker, handle, .. }, ActiveSession::Single { .. }) => worker
.forward_logits_batch(*handle, tokens, prefix_lens, padded_len, step)
.await
.map_err(|e| anyhow::anyhow!("batched decode step {step}: {e}")),
.map_err(|e| anyhow::anyhow!("batched decode step {step}: {e:#}")),
#[cfg(feature = "cuda")]
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => pool
.generate_step_batch(
@@ -182,7 +182,7 @@ async fn op_step(
step,
)
.await
.map_err(|e| anyhow::anyhow!("TP batched decode step {step}: {e}")),
.map_err(|e| anyhow::anyhow!("TP batched decode step {step}: {e:#}")),
#[cfg(feature = "cuda")]
_ => anyhow::bail!("engine backend/session mismatch"),
}
@@ -204,13 +204,13 @@ async fn op_assemble(
worker
.assemble_kv_batch(*handle, seqs)
.await
.map_err(|e| anyhow::anyhow!("assemble_kv_batch: {e}"))
.map_err(|e| anyhow::anyhow!("assemble_kv_batch: {e:#}"))
}
#[cfg(feature = "cuda")]
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => pool
.assemble_kv_batch(&tp.model_id, tp.leader_handle, seqs)
.await
.map_err(|e| anyhow::anyhow!("TP assemble_kv_batch: {e}")),
.map_err(|e| anyhow::anyhow!("TP assemble_kv_batch: {e:#}")),
#[cfg(feature = "cuda")]
_ => anyhow::bail!("engine backend/session mismatch"),
}
@@ -229,7 +229,7 @@ async fn op_extract(
(BackendConfig::Single { worker, handle, .. }, ActiveSession::Single { .. }) => Ok(worker
.extract_kv_rows(*handle, rows, padded_len, steps)
.await
.map_err(|e| anyhow::anyhow!("extract_kv_rows: {e}"))?
.map_err(|e| anyhow::anyhow!("extract_kv_rows: {e:#}"))?
.into_iter()
.map(|(id, _bytes)| id.0)
.collect()),
@@ -247,7 +247,7 @@ async fn op_extract(
ids.clone(),
)
.await
.map_err(|e| anyhow::anyhow!("TP extract_kv_rows: {e}"))?;
.map_err(|e| anyhow::anyhow!("TP extract_kv_rows: {e:#}"))?;
Ok(ids)
}
#[cfg(feature = "cuda")]
@@ -262,13 +262,13 @@ async fn op_snapshot_seq(cfg: &EngineConfig, session: &mut ActiveSession) -> Res
.snapshot_kv(*handle)
.await
.map(|(id, _bytes)| id.0)
.map_err(|e| anyhow::anyhow!("snapshot after prefill: {e}")),
.map_err(|e| anyhow::anyhow!("snapshot after prefill: {e:#}")),
#[cfg(feature = "cuda")]
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => {
let id = tp.next_snapshot_id.fetch_add(1, Ordering::Relaxed);
pool.snapshot_kv_cache(&tp.model_id, tp.leader_handle, id)
.await
.map_err(|e| anyhow::anyhow!("TP snapshot after prefill: {e}"))?;
.map_err(|e| anyhow::anyhow!("TP snapshot after prefill: {e:#}"))?;
Ok(id)
}
#[cfg(feature = "cuda")]
+218 -20
View File
@@ -30,6 +30,10 @@ pub mod worker;
use anyhow::{Context, Result};
use std::path::{Path, PathBuf};
use std::process::Stdio;
#[cfg(any(feature = "cuda", test))]
use std::sync::Arc;
#[cfg(any(feature = "cuda", test))]
use std::sync::atomic::{AtomicBool, Ordering};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Lines};
use tokio::process::{Child, ChildStdin, ChildStdout, Command};
@@ -319,6 +323,87 @@ pub struct WorkerPool {
leader_comm: Option<nccl_state::SendComm>,
}
/// How long the watchdog waits for `ncclCommAbort` to return before
/// concluding the process is unrecoverable (#265). Generous relative to a
/// healthy abort (milliseconds) and short relative to a human noticing.
#[cfg(any(feature = "cuda", test))]
fn tp_abort_deadline() -> std::time::Duration {
let secs = std::env::var("NEURON_TP_ABORT_DEADLINE_S")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|s| *s > 0)
.unwrap_or(60);
std::time::Duration::from_secs(secs)
}
/// Wait for the abort thread to signal completion. `true` if it returned
/// in time (recovery proceeds normally), `false` if the deadline elapsed
/// (the process is unrecoverable).
///
/// Split out from the deadman thread so the decision is testable without
/// a GPU: the branch that ends the process is the one most worth
/// exercising, and the one hardest to reach by accident.
#[cfg(any(feature = "cuda", test))]
fn wait_for_abort(done: &AtomicBool, deadline: std::time::Duration) -> bool {
let start = std::time::Instant::now();
while start.elapsed() < deadline {
if done.load(Ordering::Acquire) {
return true;
}
std::thread::sleep(std::time::Duration::from_millis(25));
}
// One last look: the abort may have landed inside the final sleep.
done.load(Ordering::Acquire)
}
/// Exit non-zero so systemd's `Restart=on-failure` restarts us (#265).
///
/// Called only when the process cannot recover on its own: the NCCL
/// collective is wedged and the abort that would unblock it has itself
/// hung. The daemon then serves nothing — on beast 2026-08-17 it sat in
/// that state for 70 minutes with GPUs idle, weights resident, 27
/// connections queued on the listener, and `systemctl` reporting it
/// active. A process that knows it is unrecoverable should not wait for a
/// human.
///
/// **Everything diagnostic is logged first.** A restart destroys the
/// state that explains the hang, so trading a diagnosable outage for an
/// undiagnosable self-healing one would be a poor bargain.
///
/// `process::exit` deliberately skips destructors and the graceful-drain
/// path. That is normally the wrong instinct, but draining a wedged
/// process cannot work — and `TimeoutStopSec=120s` plus
/// `TimeoutStopFailureMode=abort` would add two more minutes of outage on
/// top of the wedge (#256).
#[cfg(feature = "cuda")]
fn exit_unrecoverable(model_id: &str, reason: &str) -> ! {
// Read straight from the driver rather than any cached value: the
// cache is refreshed by the very machinery that is wedged.
let vram = std::process::Command::new("nvidia-smi")
.args([
"--query-gpu=index,memory.used,memory.free,utilization.gpu",
"--format=csv,noheader",
])
.output()
.ok()
.map(|o| String::from_utf8_lossy(&o.stdout).replace('\n', " | "))
.unwrap_or_else(|| "unavailable".into());
tracing::error!(
model = %model_id,
reason = %reason,
vram = %vram,
pid = std::process::id(),
"tp watchdog: UNRECOVERABLE — exiting non-zero so systemd restarts this daemon (#265). \
The GPU state above is from immediately before the exit; the wedged collective and its \
request id are in the preceding watchdog lines."
);
// Give the logging layer a moment to flush before the process dies —
// an exit that eats its own explanation is the failure mode this
// whole change exists to avoid.
std::thread::sleep(std::time::Duration::from_millis(500));
std::process::exit(70) // EX_SOFTWARE
}
/// Per-step deadline for a TP forward (#17 Stage 2). A healthy decode
/// step or chunked prefill completes in well under a second; a wedged
/// NCCL collective never returns. Generous default so no legitimate step
@@ -349,23 +434,66 @@ impl WorkerPool {
"tp watchdog: leader forward exceeded deadline — NCCL collective wedged; \
aborting comm to unblock the leader thread for auto-recovery"
);
match &self.leader_comm {
Some(c) => match c.0.abort() {
Ok(()) => tracing::error!(
model = %model_id,
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
failing the step so the model auto-recovers (unload+reload)"
),
Err(e) => tracing::error!(
model = %model_id, error = ?e,
"tp watchdog: ncclCommAbort failed — recovery may stall until a process restart"
),
},
None => tracing::error!(
let Some(comm) = self.leader_comm.as_ref().map(|c| c.0.clone()) else {
tracing::error!(
model = %model_id,
"tp watchdog: no cached leader comm handle — cannot abort; recovery will rely \
on a process restart"
),
"tp watchdog: no cached leader comm handle — cannot abort; exiting so \
systemd restarts us (#265)"
);
exit_unrecoverable(model_id, "no leader comm handle to abort");
};
// `ncclCommAbort` can itself block forever — observed on beast
// 2026-08-17, where neither the success nor the failure branch below
// was ever reached and the daemon sat wedged for 70 minutes with 27
// connections queued on its listener (#265). So the abort runs on its
// own thread and the deadline below does NOT depend on it returning:
// an abort that hangs must not take the supervisor with it.
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
let owned_model = model_id.to_string();
std::thread::Builder::new()
.name("tp-watchdog-abort".into())
.spawn(move || {
match comm.abort() {
Ok(()) => tracing::error!(
model = %owned_model,
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
failing the step so the model auto-recovers (unload+reload)"
),
Err(e) => tracing::error!(
model = %owned_model, error = ?e,
"tp watchdog: ncclCommAbort failed — the collective is still wedged"
),
}
// Returning at all is the signal, success or failure: a
// returned abort leaves the process responsive, so
// auto-recovery can run and we must not exit under it.
signal.store(true, Ordering::Release);
})
.map(|_| ())
.unwrap_or_else(|e| {
// Cannot even spawn a thread — nothing here can recover.
tracing::error!(model = %model_id, error = %e, "tp watchdog: abort thread spawn failed");
exit_unrecoverable(model_id, "could not spawn the abort thread");
});
// Deadman. Disarmed only by the abort returning.
let deadline = tp_abort_deadline();
let owned_model = model_id.to_string();
if let Err(e) = std::thread::Builder::new()
.name("tp-watchdog-deadman".into())
.spawn(move || {
if wait_for_abort(&done, deadline) {
return; // abort came back; the normal path owns recovery
}
exit_unrecoverable(
&owned_model,
"ncclCommAbort did not return within the deadline",
);
})
{
tracing::error!(model = %model_id, error = %e, "tp watchdog: deadman spawn failed");
}
}
@@ -497,7 +625,7 @@ impl WorkerPool {
.leader_worker
.nccl_init(leader_cfg, comm_id.clone())
.await
.map_err(|e| anyhow::anyhow!("leader NCCL init via device worker: {e}"))?;
.map_err(|e| anyhow::anyhow!("leader NCCL init via device worker: {e:#}"))?;
match leader_resp {
rpc::WorkerResponse::InitOk => {}
rpc::WorkerResponse::Error { kind, message } => {
@@ -569,7 +697,7 @@ impl WorkerPool {
.leader_worker
.nccl_sanity()
.await
.map_err(|e| anyhow::anyhow!("leader NCCL sanity via device worker: {e}"))?;
.map_err(|e| anyhow::anyhow!("leader NCCL sanity via device worker: {e:#}"))?;
let expected = self.world_size;
let leader_sum = match leader_resp {
@@ -694,7 +822,7 @@ impl WorkerPool {
world_size,
)
.await
.map_err(|e| anyhow::anyhow!("leader TP shard load via device worker: {e}"))?;
.map_err(|e| anyhow::anyhow!("leader TP shard load via device worker: {e:#}"))?;
// 3. Collect worker confirmations. Anything other than
// LoadDenseShardOk aborts the whole load — the leader's
@@ -1250,7 +1378,7 @@ impl WorkerPool {
.leader_worker
.query_vram()
.await
.map_err(|e| anyhow::anyhow!("leader query_vram: {e}"))?;
.map_err(|e| anyhow::anyhow!("leader query_vram: {e:#}"))?;
let mut frees = vec![leader_free_mb];
let worker_errors = drain_workers(&mut self.workers, |r| match r {
WorkerResponse::VramInfo { free_mb, .. } => {
@@ -1442,3 +1570,73 @@ impl WorkerPool {
&self.exe
}
}
#[cfg(test)]
mod watchdog_tests {
use super::*;
use std::time::Duration;
/// An abort that returns disarms the deadman — the normal path then
/// owns recovery and the process must NOT exit.
#[test]
fn abort_returning_disarms_the_deadman() {
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
signal.store(true, Ordering::Release);
});
assert!(
wait_for_abort(&done, Duration::from_secs(5)),
"a returning abort must disarm the deadman"
);
}
/// The #265 case: `ncclCommAbort` never returns. The deadline must
/// elapse and report unrecoverable, which is what triggers the exit.
#[test]
fn abort_that_never_returns_trips_the_deadline() {
let done = Arc::new(AtomicBool::new(false)); // never set
let start = std::time::Instant::now();
assert!(
!wait_for_abort(&done, Duration::from_millis(200)),
"an abort that never returns must trip the deadline"
);
assert!(
start.elapsed() >= Duration::from_millis(200),
"must actually wait the deadline rather than failing fast"
);
}
/// An abort landing in the last instant still counts as returned —
/// exiting on a race would restart a process that had just recovered.
#[test]
fn abort_landing_at_the_deadline_still_counts() {
let done = Arc::new(AtomicBool::new(false));
let signal = Arc::clone(&done);
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(180));
signal.store(true, Ordering::Release);
});
assert!(
wait_for_abort(&done, Duration::from_millis(200)),
"an abort landing at the deadline must not be treated as a hang"
);
}
/// The deadline is operator-tunable but never zero — a zero deadline
/// would exit the moment any watchdog fired.
#[test]
fn deadline_defaults_sanely_and_rejects_zero() {
assert_eq!(tp_abort_deadline(), Duration::from_secs(60));
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "0") };
assert_eq!(
tp_abort_deadline(),
Duration::from_secs(60),
"zero must fall back, not exit instantly on every watchdog"
);
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "15") };
assert_eq!(tp_abort_deadline(), Duration::from_secs(15));
unsafe { std::env::remove_var("NEURON_TP_ABORT_DEADLINE_S") };
}
}
+4 -1
View File
@@ -186,6 +186,9 @@ async fn daemon(args: Args) -> Result<()> {
let registry = HarnessRegistry::from_configs(&cfg.harnesses, &bind_url, &cfg.harness);
discovery_result.harnesses = registry.names();
let candle = registry.candle();
// A second handle for the shutdown future: it must close admission
// before the drain begins, and `candle` is moved into NeuronState.
let shutdown_candle = candle.clone();
let health_cache = Arc::new(health::HealthCache::new());
health_cache
@@ -246,7 +249,7 @@ async fn daemon(args: Args) -> Result<()> {
}
axum::serve(listener, app)
.with_graceful_shutdown(startup::shutdown_signal())
.with_graceful_shutdown(startup::shutdown_signal(shutdown_candle))
.await?;
// Deactivation: serve has returned (graceful shutdown signal
+20 -1
View File
@@ -9,6 +9,7 @@ use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::preflight::PreflightError;
use cortex_core::harness::ModelSpec;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::signal;
@@ -165,7 +166,18 @@ fn preflight_kind(err: &PreflightError) -> &'static str {
/// Wired into `axum::serve(...).with_graceful_shutdown(shutdown_signal())`
/// so the HTTP listener stops accepting new connections, lets in-flight
/// requests drain, and then yields control back to main for cleanup.
pub async fn shutdown_signal() {
///
/// **Before returning, it closes admission and wakes every waiter (#256).**
/// Order matters: axum's graceful shutdown waits for *all* in-flight
/// requests, and a request queued behind the TP pool lock is in-flight as
/// far as axum is concerned. Those waits are unbounded — held by whichever
/// request is mid-forward — so without this the drain cannot finish and
/// systemd SIGKILLs at `TimeoutStopSec`. Observed 2026-08-15: SIGTERM at
/// 13:03:14, one queued request logging "still waiting on pool lock" every
/// two seconds, `Failed with result 'timeout'` at 13:05:24. The same node
/// drained in about a second when idle, which is what identified the
/// queued waiter as the cause.
pub async fn shutdown_signal(candle: Option<Arc<crate::harness::candle::CandleHarness>>) {
let ctrl_c = async {
signal::ctrl_c().await.ok();
};
@@ -179,6 +191,13 @@ pub async fn shutdown_signal() {
_ = ctrl_c => tracing::info!("received SIGINT, shutting down"),
_ = terminate => tracing::info!("received SIGTERM, shutting down"),
}
// Refuse new admissions and abandon the unbounded waits, so the drain
// that starts the moment this future resolves has a bounded amount of
// work left to wait for.
crate::harness::candle::begin_shutdown();
if let Some(candle) = candle {
candle.close_all_admission().await;
}
}
/// Unload every model currently registered. Called from `main.rs` after
@@ -157,6 +157,9 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
temperature: req.temperature,
top_p: req.top_p,
max_tokens: req.max_output_tokens,
// Responses spells the cap `max_output_tokens`, already
// resolved onto the legacy field above.
max_completion_tokens: None,
stream: Some(req.stream),
extra: Value::Object(extra),
})
+68
View File
@@ -709,3 +709,71 @@ async fn test_images_chat_at_image_endpoint_is_not_found_for_text_model() {
.unwrap();
assert_eq!(resp.status(), 404);
}
/// A malformed body must come back as the #63 envelope, not axum's bare
/// string (#253). An OpenAI client reads `error.code` / `error.type`; a
/// plain string leaves it with nothing to act on, which is exactly how a
/// harness ended up quitting silently mid-session.
#[tokio::test]
async fn malformed_body_returns_a_parseable_error_envelope() {
let base = spawn_neuron(fake_discovery()).await;
let client = reqwest::Client::new();
// Valid JSON, wrong shape for the request type → 422.
let resp = client
.post(format!("{base}/v1/chat/completions"))
.json(&json!({"model": "m", "messages": "not-an-array"}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 422);
let body: serde_json::Value = resp.json().await.expect("body must be JSON");
assert_eq!(body["error"]["type"], "invalid_request_error");
assert_eq!(body["error"]["code"], "invalid_request_body");
assert!(
body["error"]["message"]
.as_str()
.is_some_and(|m| !m.is_empty()),
"message must say what was wrong: {body}"
);
// Not JSON at all → 400, distinguished from the above so the caller
// knows whether to fix its serializer or its schema.
let resp = client
.post(format!("{base}/v1/chat/completions"))
.header("content-type", "application/json")
.body("{not json")
.send()
.await
.unwrap();
assert_eq!(resp.status(), 400);
let body: serde_json::Value = resp.json().await.expect("body must be JSON");
assert_eq!(body["error"]["code"], "malformed_json");
}
/// The shape from #253 itself: an assistant turn carrying only
/// `tool_calls`. It must be *accepted* now — this guards the envelope
/// work from quietly re-breaking what #253 fixed.
#[tokio::test]
async fn null_assistant_content_is_not_a_body_rejection() {
let base = spawn_neuron(fake_discovery()).await;
let resp = reqwest::Client::new()
.post(format!("{base}/v1/chat/completions"))
.json(&json!({
"model": "m",
"messages": [
{"role": "user", "content": "hi"},
{"role": "assistant", "content": null, "tool_calls": [
{"id": "c1", "type": "function",
"function": {"name": "b", "arguments": "{}"}}]},
{"role": "tool", "content": "ok", "tool_call_id": "c1"}
]
}))
.send()
.await
.unwrap();
// 503 (no candle harness in this fixture) — the point is that it got
// past body extraction rather than being rejected as malformed.
assert_ne!(resp.status(), 422, "null content must not be a body error");
assert_ne!(resp.status(), 400, "null content must not be a body error");
}
+21
View File
@@ -2,6 +2,22 @@
Description=Neuron — per-node GPU discovery and harness daemon for cortex
After=network-online.target
Wants=network-online.target
# Start-limit bound for the self-restart path (#265). When a TP forward
# wedges and the NCCL abort that would unblock it also hangs, neuron
# exits 70 so Restart=on-failure recovers it in seconds instead of the
# 70-minute outage that needed a human on 2026-08-17.
#
# That only helps if the wedge is transient. A deterministic one — a bad
# build, a sick card, a model that wedges on load — would otherwise
# restart forever, and a flapping node is WORSE than a down one: cortex
# marks it healthy between restarts and keeps routing to it, so every
# client sees intermittent failures instead of one node cleanly out.
#
# Four failures in ten minutes stops the unit and leaves it `failed`,
# where `systemctl status` and an alert can see it. Visibly broken beats
# invisibly cycling.
StartLimitIntervalSec=600
StartLimitBurst=4
[Service]
Type=simple
@@ -24,6 +40,11 @@ TimeoutStartSec=1800s
# On stop, neuron drains in-flight requests then unloads every model
# to release CUDA contexts cleanly. Allow generous time for big-model
# unloads; systemd will SIGKILL after this bound.
#
# Note this bound does NOT apply to the self-restart path (#265): a
# wedged process exits directly rather than draining, precisely because
# draining cannot work when the collective is stuck — and waiting this
# out would add two minutes to an outage that is already total.
TimeoutStopSec=120s
KillSignal=SIGTERM
@@ -19,7 +19,7 @@ gateway.
```
┌──────────────┐
clients ─────▶│ cortex │ :31313 API, :31314 metrics
clients ────▶│ cortex │ :31313 API, :31314 metrics
└──┬────┬────┬─┘
│ │ │
┌────▼┐ ┌─▼──┐ ┌▼────┐
@@ -108,6 +108,43 @@ KV cache for several concurrent long contexts is itself substantial.
`max_per_principal` is what stops one busy client consuming the whole
model.
### Anonymous callers
Fair share is keyed on the account and key cortex resolves from the
bearer token. A caller that sends no credential — or one that does not
resolve, which `require_auth = false` deliberately tolerates — has no
key to be shared out, so `max_per_principal` cannot bind it.
Anonymous traffic is therefore served from the capacity left over once
identified traffic is satisfied. It never takes a seat while an
authenticated request is waiting for one, whoever arrived first, and it
cannot hold every seat at once:
```toml
anon_max_in_flight = 7 # unset: max_in_flight - 1, floored at 1
anon_max_pending = 15 # unset: max_in_flight + max_queue_depth - 1
```
The defaults hold back one seat and one queue place, which is what
bounds how long an authenticated request can be made to wait: a request
already running cannot be preempted, so without a reserved seat an
anonymous burst arriving during an idle moment locks the model for as
long as all of it takes to finish. On a single-seat model the floor
keeps anonymous callers served — priority alone decides who wins there.
Set `anon_max_in_flight = 0` to refuse anonymous traffic outright, or
`require_auth = true` on the gateway to make attribution universal.
Anonymous callers still contend with each other, and under sustained
authenticated load they are refused rather than served slowly — the
refusal is a retryable `503`, so a client that backs off gets served
when the load passes.
Watch it with `cortex_model_anon_in_flight` against
`cortex_model_in_flight` (how much of a model's load is unattributable)
and `cortex_model_rejections_total{reason="anon_yield"}`. That last
counter rising while `reason="wait_timeout"` stays flat means the
reservation is doing its job, not that the model is overloaded.
## VRAM guards
neuron refuses work it cannot finish rather than crashing partway:
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB