Compare commits

..

58 Commits

Author SHA1 Message Date
4cb52e3144 Merge pull request 'feat(neuron): server-measured prefill/decode timing on Finish (#85)' (#100) from feat/85-prefill-decode-timing into main
Some checks failed
build-prerelease / Resolve version stamps + change detection (push) Waiting to run
build-prerelease / Lint (fmt + clippy) (push) Has been cancelled
build-prerelease / Test (push) Has been cancelled
build-prerelease / Build cortex binary (push) Has been cancelled
build-prerelease / Build helexa-bench binary (push) Has been cancelled
build-prerelease / Build helexa-upstream binary (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Package helexa-upstream RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-27 09:08:15 +00:00
6f956dfda3 fix(neuron): hoist TP prefill/decode timers out of 'work block (#85)
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m32s
CI / Clippy (push) Successful in 3m17s
CI / Test (push) Successful in 7m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 39s
CI / CUDA type-check (pull_request) Successful in 1m49s
CI / Clippy (pull_request) Successful in 2m58s
CI / Test (pull_request) Successful in 7m3s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
The TP streaming producer builds its Finish event after the 'work
labelled block exits (inside `if failure.is_none()`), but prefill_elapsed
and decode_start were declared inside that block — so the CUDA type-check
failed with E0425 (the CPU build doesn't compile this cfg(cuda) path).

Hoist `prefill_ms_measured: u32` and `decode_start: Option<Instant>`
above the block; populate them at the prefill→decode boundary inside;
read them at the terminal Finish. The worker and local producers were
unaffected (their timers already share the Finish scope).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq
2026-06-27 11:58:26 +03:00
6e0f15c888 feat(neuron): server-measured prefill/decode timing on Finish (#85)
Some checks failed
CI / CUDA type-check (push) Failing after 1m40s
CI / Format (push) Successful in 42s
CI / Clippy (push) Successful in 2m51s
CI / Test (push) Successful in 6m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The harness emitted only token counts on InferenceEvent::Finish; all
timing was client-side SSE arrival, so bench "TTFT" conflated tokenize
+ prefill and decode tok/s was a window estimate.

Add FinishTiming { prefill_ms, decode_ms, prefill_tokens } to the
Finish event, populated by all three streaming producers (TP, worker,
and local CPU paths), and surface it on the OpenAI chat
`usage.helexa_timing` extension so helexa-bench can compute true
prefill vs decode tok/s. cortex forwards usage verbatim, so the field
survives proxying. Non-streaming and Responses paths carry None for
now (bench reads the streaming chat path).

Keystone for the Performance observability epic (#83).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq
2026-06-27 11:41:17 +03:00
66eb9f558f Merge pull request 'fix(neuron): surface reasoning_tokens in non-streaming /v1/responses usage' (#82) from fix/responses-usage-reasoning-tokens into main
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 47s
build-prerelease / Build neuron-blackwell (push) Successful in 1m27s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m4s
build-prerelease / Build neuron-ampere (push) Successful in 2m14s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m14s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m41s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m57s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m52s
build-prerelease / Test (push) Successful in 6m41s
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 55s
2026-06-26 18:30:05 +00:00
f96a2e7ed3 fix(neuron): surface reasoning_tokens in non-streaming /v1/responses usage
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m38s
CI / Clippy (push) Successful in 2m42s
CI / Test (push) Successful in 6m36s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 36s
CI / CUDA type-check (pull_request) Successful in 1m35s
CI / Clippy (pull_request) Successful in 2m54s
CI / Test (pull_request) Successful in 6m33s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
The non-streaming responses handler hardcoded `output_tokens_details: None`,
so the reasoning sub-count that the chat path now computes (via
`split_off_reasoning`, which strips the `<think>` span and counts it into
`completion_tokens_details.reasoning_tokens`) never reached the Responses-API
usage object. The streaming responses path already emits it.

Carry `chat usage.completion_tokens_details.reasoning_tokens` through to
`ResponsesUsage.output_tokens_details.reasoning_tokens`, so streaming and
non-streaming `/v1/responses` report reasoning accounting identically.
`output_tokens` still counts every generated token (reasoning included);
`reasoning_tokens` is the additive sub-count, per OpenAI's shape.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K
2026-06-26 21:21:59 +03:00
b17b555a3d Merge pull request 'fix(neuron): strip &lt;think&gt; reasoning from non-streaming completions' (#81) from fix/responses-nonstreaming-reasoning-leak into main
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 46s
build-prerelease / Build neuron-blackwell (push) Successful in 1m27s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m3s
build-prerelease / Build neuron-ampere (push) Successful in 2m12s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m6s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m40s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m51s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m47s
build-prerelease / Test (push) Successful in 6m26s
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 54s
2026-06-26 17:47:45 +00:00
13daf95514 fix(neuron): strip <think> reasoning from non-streaming completions
All checks were successful
CI / Format (push) Successful in 39s
CI / CUDA type-check (push) Successful in 1m41s
CI / Clippy (push) Successful in 2m47s
CI / Test (push) Successful in 5m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 40s
CI / CUDA type-check (pull_request) Successful in 1m38s
CI / Clippy (pull_request) Successful in 2m55s
CI / Test (pull_request) Successful in 6m20s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
The non-streaming inference path (single-GPU `chat_completion` and the TP
`chat_completion_tp_inner`) returned the model's full decode — reasoning
preamble + `</think>` + answer — as the assistant `content`. The streaming
path already drops reasoning (emits it as ReasoningDelta, which the chat and
Responses projectors discard), so the two transports disagreed: a streaming
client saw the clean answer, a non-streaming client saw the chain-of-thought
glued to the front of it.

This is exactly what broke agent-zero v2.0 on `/v1/responses` (non-streaming):
the model produced a correct in-band JSON answer after `</think>`, but a0's
parser saw the `<think>` preamble first and rejected the turn as "misformat,
no valid tool request found". a0 v2.1 happens to tolerate the preamble, but
any strict non-streaming client (and a0 v2.0) does not — and reasoning tokens
leaking into `content` also misreport as visible output.

Add `split_off_reasoning(generated_ids, reasoning_pair)`: if the model
declares a reasoning marker pair and its close token (`</think>`) appears in
the output, return only the tokens after the last close marker as content and
count the rest as reasoning. The chat template injects the *opening* marker
into the prompt, so the generated tokens carry the close marker but not the
open one — splitting on the close-token id (not a decoded string) is robust to
tokenizer byte-fallback. Non-reasoning models, thinking-disabled requests, and
generations truncated mid-reasoning have no close token and pass through
unchanged.

Both non-streaming sites now decode only the answer span and populate
`completion_tokens_details.reasoning_tokens` (the streaming-only accounting
gap noted at the call sites, #64). Unit tests cover the strip, no-marker,
no-pair, close-at-end, and multiple-close cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K
2026-06-26 20:39:41 +03:00
319b01e0b2 Merge pull request 'fix(neuron): accept bare {role,content} input on /v1/responses (agent-zero)' (#80) from fix/responses-easy-message-input into main
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 49s
build-prerelease / Build neuron-blackwell (push) Successful in 1m29s
build-prerelease / Build helexa-bench binary (push) Successful in 2m21s
build-prerelease / Build cortex binary (push) Successful in 3m23s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m50s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m39s
build-prerelease / Test (push) Successful in 6m43s
build-prerelease / Package cortex RPM (push) Successful in 1m30s
build-prerelease / Build neuron-ada (push) Successful in 2m7s
build-prerelease / Build neuron-ampere (push) Successful in 2m15s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m42s
build-prerelease / Package helexa-upstream RPM (push) Successful in 1m20s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m42s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m46s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 51s
2026-06-26 16:53:08 +00:00
6731adca51 fix(neuron): accept bare {role,content} input on /v1/responses (agent-zero)
All checks were successful
CI / Format (push) Successful in 41s
CI / CUDA type-check (push) Successful in 1m40s
CI / Clippy (push) Successful in 2m45s
CI / Test (push) Successful in 6m14s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 39s
CI / CUDA type-check (pull_request) Successful in 1m36s
CI / Clippy (pull_request) Successful in 3m18s
CI / Test (pull_request) Successful in 7m6s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
agent-zero (via litellm) drives the OpenAI Responses API and sends
`input` items in the "easy input message" form — bare `{role, content}`
objects with NO `type` field. Our `ResponsesInputItem` is internally
tagged (`#[serde(tag="type")]`), so every such item failed the untagged
`ResponsesInput` deserialize and axum's `Json` extractor returned 422:

    OpenAIException - Failed to deserialize the JSON body into the target
    type: data did not match any variant of untagged enum ResponsesInput

This was a *total* failure for agent-zero (both the main model on beast
and the utility model on benjy), confirmed by on-wire capture of 15 live
requests: 36/36 input items were bare easy-messages. Other clients
(/v1/chat/completions, /v1/messages) were unaffected — only the
Responses path was exercised this strictly, for the first time.

Make `input`-item parsing match OpenAI's real tolerance, mirroring the
forward-compat `extra: Value` already at the top level of the request:

- New `ResponsesInputElement` wraps the existing typed item enum with
  two more shapes: `EasyMessage { role, content }` (bare, no type;
  `content` optional so an assistant turn with `content: null` parses)
  and `Other(Value)` — a catch-all so a single unmodeled item can never
  again 422 the whole request. The typed enum is unchanged.
- `ResponsesContentPart` gains a `#[serde(other)] Unknown` arm (e.g.
  `refusal`, audio) — dropped in translation, not rejected.
- `FunctionCallOutput.output` is now `Value` (string OR array of content
  parts, per OpenAI) so a structured tool result isn't lost.
- Translator handles all three element shapes; easy-messages translate
  exactly like typed messages, `Other` and unknown parts are dropped.

Tests cover the bare-message, null-content, unknown-item, unknown-part,
and array-tool-output shapes, validated against the 15 captured bodies.

Tools forwarding + native function_call projection on the Responses path
is deliberately a follow-up (Round 2), gated on observing how agent-zero
consumes responses once unblocked (in-band JSON vs native tool calls).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K
2026-06-26 19:45:22 +03:00
7e11a7688c Merge feat/F6-beta-polish: public-beta banner + same-origin nginx + deploy (F6)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 39s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m17s
build-prerelease / Build neuron-blackwell (push) Successful in 1m43s
build-prerelease / Build neuron-ada (push) Successful in 2m15s
build-prerelease / Build neuron-ampere (push) Successful in 2m17s
build-prerelease / Build cortex binary (push) Successful in 2m50s
build-prerelease / Build helexa-bench binary (push) Successful in 2m25s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 2m1s
build-prerelease / Build helexa-upstream binary (push) Successful in 4m8s
build-prerelease / Test (push) Successful in 6m28s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m18s
build-prerelease / Package helexa-upstream RPM (push) Successful in 1m19s
build-prerelease / Package cortex RPM (push) Successful in 1m24s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m44s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 58s
2026-06-23 12:13:35 +03:00
5600575ba2 Merge feat/B7-upstream-packaging: RPM + systemd + CI for helexa-upstream (B7)
Some checks failed
build-prerelease / Resolve version stamps + change detection (push) Has been cancelled
build-prerelease / Lint (fmt + clippy) (push) Has been cancelled
build-prerelease / Test (push) Has been cancelled
build-prerelease / Build cortex binary (push) Has been cancelled
build-prerelease / Build helexa-bench binary (push) Has been cancelled
build-prerelease / Build helexa-upstream binary (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Package helexa-upstream RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 12:13:25 +03:00
bc7476bf1b feat(F6): public-beta polish — banner, same-origin nginx, deploy notes
All checks were successful
CI / CUDA type-check (push) Successful in 1m36s
CI / Format (push) Successful in 35s
CI / Clippy (push) Successful in 3m14s
CI / Test (push) Successful in 7m14s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Final beta-readiness pass for helexa.ai.

- BetaBanner: a slim, dismissible (session-scoped) public-beta strip above
  the header, shown only when VITE_PUBLIC_BETA=true; theme-aware styling;
  beta.{tag,message,dismiss} i18n keys across all languages.
- deploy/nginx.conf: edge config serving the built SPA and reverse-proxying
  both backends on the SAME ORIGIN — `/` SPA history fallback, `/v1`+
  `/health` → helexa-router (SSE: proxy_buffering off, 300s read), `/api/`
  → helexa-upstream `/web/v1/`. No CORS; the user's key is a first-party
  bearer.
- README: a deploy section (build → /var/www → nginx) reiterating
  no-server-side-chat-history.

Validated: lint, typecheck, build, i18n:check all green. Explicit-path
commit; no node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 12:08:33 +03:00
5a8f6bc7b3 Merge feat/F5-auth-chat: authenticated chat + key usage (F5)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 35s
build-prerelease / Build neuron-blackwell (push) Successful in 1m41s
build-prerelease / Build helexa-bench binary (push) Successful in 2m7s
build-prerelease / Build neuron-ampere (push) Successful in 2m23s
build-prerelease / Build neuron-ada (push) Successful in 2m23s
build-prerelease / Build cortex binary (push) Successful in 3m12s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m30s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m19s
build-prerelease / Package cortex RPM (push) Successful in 1m23s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m52s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m53s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m53s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 12:06:13 +03:00
452d7d9b3d feat(B7): packaging + CI for helexa-upstream
All checks were successful
CI / CUDA type-check (push) Successful in 1m32s
CI / Format (push) Successful in 57s
CI / Clippy (push) Successful in 2m51s
CI / Test (push) Successful in 8m0s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Ships the mesh authority as an RPM on the same prebuilt-binary pipeline as
helexa-bench.

- rpm/helexa-upstream-prerelease.spec: wraps the CI-built binary; installs
  the systemd unit, sysusers, firewalld service (tcp/8090), and
  /etc/helexa-upstream/helexa-upstream.toml (config noreplace).
- data/helexa-upstream.{service,sysusers.conf,firewalld.xml}: Type=simple
  unit (serve --config), dedicated system user with a StateDirectory home,
  inbound 8090 (/authz/v1 + /web/v1). PostgreSQL is reached outbound; the
  schema migrates on startup.
- build-prerelease.yml: build-upstream + package-upstream jobs with
  change-detection over crates/helexa-upstream/ (UPSTREAM_RE), gated into
  publish. SQLX_OFFLINE=true is set defensively — helexa-upstream uses the
  sqlx runtime query API (no compile-time macros), so it builds with no
  database and no .sqlx cache; DB integration tests stay gated behind
  UPSTREAM_TEST_DATABASE_URL.

Validated: workflow YAML parses, rpmspec expands, and
`SQLX_OFFLINE=true cargo build --release -p helexa-upstream` succeeds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 12:00:41 +03:00
21eb211d6a Merge feat/B6-served-usage: served-usage ledger + reconciliation (B6, #58)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Package cortex RPM (push) Blocked by required conditions
build-prerelease / Package helexa-bench RPM (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 37s
build-prerelease / Build neuron-blackwell (push) Successful in 1m25s
build-prerelease / Build neuron-ada (push) Successful in 2m3s
build-prerelease / Build neuron-ampere (push) Successful in 2m10s
build-prerelease / Build helexa-bench binary (push) Successful in 2m13s
build-prerelease / Build cortex binary (push) Successful in 3m25s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m45s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m51s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m50s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m52s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 11:56:06 +03:00
508b326bf7 feat(F5): authenticated chat + key usage integration
All checks were successful
CI / Format (push) Successful in 32s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 3m22s
CI / Test (push) Successful in 7m4s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Signing in upgrades the chat workspace from anonymous to account-scoped.

- Auth context exposes accountId (resolved on login and on reload for an
  existing token) so the chat can scope its Dexie owner and the dashboard
  can query.
- Chat: when authed, owner switches to the account id (anon history was
  already re-owned via claimAnonymousData on login), the anon message cap
  is lifted (budget is enforced upstream by the account allocation), the
  full default model (VITE_DEFAULT_MODEL) is used, and the user's API key
  is sent as the bearer.
- The bearer is the raw key the user stored locally via "use for chat on
  this device" in the key-creation modal (client-side only — consistent
  with no server-side secrets). Signed in without a stored key → a banner
  prompts creating/enabling one (sending is disabled until then).
- Error mapping: insufficient_quota → top-up link (/account) when authed,
  sign-up (/register) when anon; rate_limit_exceeded → a wait-and-retry
  hint; both distinct from generic errors.
- New chat (topUp/rateLimited/needsKey/manageKeysLink) and account
  (keys.useForChat/usedForChat) i18n keys across all languages.

Validated: lint, typecheck, build, i18n:check all green. Explicit-path
commit; no node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:55:48 +03:00
0de99a8cc7 Merge feat/F4-account-dashboard: auth + account dashboard (F4)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 47s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m16s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 11:46:14 +03:00
f4117224fc feat(B6): served-usage ledger + reconciliation (#58)
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m53s
CI / Test (push) Successful in 6m41s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Operators are now metered for the tokens they serve on behalf of mesh
accounts, and the upstream rolls that up for compensation.

cortex-gateway:
- served_usage.rs: an in-process per-(account,key,UTC-day) served-token
  counter, incremented in metering::usage_sink alongside spend/settle for
  every authenticated request. A flush task (spawned in run() when
  [upstream].enabled, mirroring poller/evictor) POSTs ABSOLUTE cumulative
  counters to upstream on [upstream].served_usage_report_interval_secs.
- The no-limit infra key is the operator's local key with hard_cap=None
  (already supported) — it's metered for served-usage but never budget-
  refused and never hits upstream.

helexa-upstream:
- POST /authz/v1/served-usage: upserts rows keyed by (operator_id from the
  client bearer, account, key, period) with
  GREATEST(existing, incoming) — monotonic + idempotent, so re-sends,
  races, and a restarted cortex's lower counter never regress the total.
- reconcile.rs + `helexa-upstream reconcile` CLI: rolls up unreconciled
  served_usage per operator/period (SUM::bigint), stamps reconciled_at,
  prints the totals. Payout mechanism out of scope.

Validated against a throwaway Postgres: monotonic upsert (100→250, a 50
re-send stays 250, same-value idempotent) and reconcile rollup +
stamp-once; cortex counter unit test for per-principal accumulation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:45:57 +03:00
ce29e0c171 Merge feat/F3-anon-chat: anonymous chat landing + IndexedDB + SSE (F3)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 52s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m14s
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 6m13s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-06-23 11:32:00 +03:00
1bf3348c8c feat(F4): auth + account dashboard (mockable client)
All checks were successful
CI / Format (push) Successful in 42s
CI / CUDA type-check (push) Successful in 1m41s
CI / Clippy (push) Successful in 3m9s
CI / Test (push) Successful in 6m30s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The self-service account surface consuming helexa-upstream's /web/v1 (B4/B5),
with a mock so it works before/independent of the live backend.

- api/types.ts + api/account.ts: typed AccountApi over a same-origin `/api`
  prefix (vite-proxied in dev, nginx in prod) covering register/verify/
  login/password-reset/keys(list,create,archive,limit)/account/redeem;
  ApiError carries the backend code. MockAccountApi behind
  VITE_USE_MOCK_ACCOUNT_API (in-memory account, raw-key-once, redeem).
- auth/: context + useAuth, AuthProvider (JWT in localStorage, login fetches
  the account and runs claimAnonymousData → anon IndexedDB history is
  re-owned to the account, still client-side), RequireAuth guard
  (→ /login?next=).
- pages/auth/: Login, Register (sends the FingerprintJS visitor id →
  triggers the silent abuse detection), VerifyEmail (?token), RequestReset,
  ResetPassword (?token, matches the backend /reset?token= link).
- pages/account/: Dashboard (allocation balance + usage bar, redeem top-up,
  logout) and ApiKeys (list, create-modal showing the raw key ONCE with
  copy, per-key limit editor percent↔hardcap, archive). 401 → logout.
- App wraps AuthProvider + routes (account guarded); Header auth cluster
  reflects useAuth (Account/Sign out vs Sign in/up).
- `account` i18n namespace (53 keys) added + wired across all 32 langs.

Validated: lint, typecheck, build, i18n:check, lang-labels all green.
Explicit-path commit; no node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:31:43 +03:00
7c12b9ea98 Merge feat/F2-mission: /mission route — EU digital sovereignty (F2)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 49s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m48s
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 6m59s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-06-23 11:19:38 +03:00
c596519dbd feat(F3): anonymous chat landing — IndexedDB history + SSE + fingerprint
All checks were successful
CI / Format (push) Successful in 46s
CI / CUDA type-check (push) Successful in 1m40s
CI / Clippy (push) Successful in 3m20s
CI / Test (push) Successful in 8m3s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The beta centerpiece: a chat workspace at `/` with zero server-side
history. Everything personal lives in the browser (Dexie/IndexedDB);
inference streams from the mesh router.

- data/db.ts: Dexie schema (projects, conversations[owner:'anon'|accountId],
  messages, meta) — `owner` namespaces anonymous vs account data;
  claimAnonymousData() (repositories) re-owns anon data on login (F4),
  still client-side.
- data/repositories.ts: typed CRUD + ordered queries (projects,
  conversations, messages) used reactively via useLiveQuery.
- lib/fingerprint.ts: FingerprintJS OSS, cached in meta (best-effort, never
  auth) — namespaces anon data + a soft throttle id.
- lib/chatClient.ts: streamChatCompletion → POST /v1/chat/completions
  stream:true; parses the SSE byte stream incrementally (data:/[DONE]/
  partial frames), surfaces the OpenAI envelope error.code, AbortController
  for Stop.
- lib/useChat.ts: persists the user turn, opens a streaming assistant
  message, appends deltas to Dexie live, titles the conversation, finalizes
  on done/error.
- pages/Chat.tsx: sidebar (new chat / new project, conversations grouped by
  project + Unsorted), live-updating thread with streaming + error
  rendering, composer with send/stop. Anonymous mode: no bearer +
  VITE_ANON_MODEL + a client message cap with a sign-up nudge. Routed at `/`.
- chat i18n namespace extended (newChat/newProject/unsorted/emptyState/
  anonBanner/signUp/stop) across all 33 languages (parity holds).

Validated: npm run lint, typecheck, build all green; i18n:check consistent.
(Full browser flow exercised in F6 verify.) Explicit-path commit; no
node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:19:19 +03:00
a6b1fdc33d Merge feat/F1-theming-i18n: theming + 33-lang i18n + usage-ordered selector (F1)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 41s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m14s
build-prerelease / Test (push) Successful in 5m37s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-06-23 11:10:33 +03:00
8dd82776f1 feat(F2): /mission route — European digital sovereignty
All checks were successful
CI / Format (push) Successful in 50s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 3m31s
CI / Test (push) Successful in 7m22s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The secondary narrative page, re-focused (vs the reference placeholder) to
lead with European digital sovereignty.

- Renamed the `home` i18n namespace → `mission` (32 langs' home.json →
  mission.json; updated i18n/index.ts imports + ns list and the
  check-i18n NAMESPACES). The reference's narrative content was already
  translated, so the non-English languages carry over as mission copy.
- Rewrote en/mission.json to lead with sovereignty: data residency, a
  GDPR-native no-server-side-history stance, EU operator ownership,
  region-affine routing, and independence from US hyperscalers — same key
  structure, so cross-language parity holds.
- src/pages/Mission.tsx ported from the reference Home page (sections:
  hero/intent/whyNow/howItWorks/principles/roadAhead/joinMesh) bound to the
  `mission` namespace; routed at /mission in App.tsx (the Header link from
  F1 now resolves).

Validated: npm run lint, typecheck, build all green; i18n:check (mission
namespace consistent across all languages) and i18n:lang-labels pass.
Explicit-path commit; no node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:10:14 +03:00
8600d4fbf2 Merge feat/B5-topup-codes: single-use top-up codes + mint CLI (B5, #59)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 37s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m22s
build-prerelease / Test (push) Successful in 5m44s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-06-23 11:01:26 +03:00
7a6f252fe0 feat(F1): theming + 33-language i18n + usage-ordered language selector
All checks were successful
CI / Format (push) Successful in 40s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m50s
CI / Test (push) Successful in 6m34s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Ports the reference site's visual + i18n foundation into helexa.ai and
adds the deliberate usage-ordered language picker.

- Ported from ~/git/helexa-ai/helexa.ai: src/layout (ThemeProvider/theme,
  localStorage + data-theme, light/dark), src/App.css (cyan/hot-pink accents,
  system fonts), src/i18n (index + languages + translation-priority +
  resources for 33 languages × common/home/chat), Footer, DirectionalIcon,
  public assets, and the check-i18n-* scripts.
- getLanguageOptionsByUsage() (in translation-priority.ts): orders the
  selector by the TRANSLATION_PRIORITY ranking (≈ native-speaker usage),
  deduping repeated entries and appending any unranked supported language —
  NOT alphabetical, the marketing-driven choice that foregrounds helexa's
  international grounding. RTL preserved.
- Header: usage-ordered language dropdown (autonym + secondary label in the
  current language), theme toggle, and new nav — `/` (chat), `/mission`,
  and a Login/Register auth cluster stubbed until F4. New nav keys
  (mission/login/register/account/logout) injected into all 33 common.json
  with English placeholders so key-parity holds.
- App composes ThemeProvider → BrowserRouter → Header + routes + Footer
  (placeholders for `/` and `/mission`); main.tsx loads i18n.

Validated: npm run lint, typecheck, build all green; npm run i18n:check
reports all keys consistent across the 33 languages. (Build emits a
chunk-size advisory — code-splitting is deferred to F6 polish.) Staged with
explicit paths; no node_modules/dist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:01:04 +03:00
bb0d1e51b8 Merge feat/B3-cortex-upstream-client: cortex upstream entitlement client + chain (B3, #57)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 41s
build-prerelease / Build neuron-blackwell (push) Successful in 1m37s
build-prerelease / Build neuron-ampere (push) Successful in 2m14s
build-prerelease / Build neuron-ada (push) Successful in 2m14s
build-prerelease / Build helexa-bench binary (push) Successful in 2m45s
build-prerelease / Build cortex binary (push) Successful in 2m58s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m32s
build-prerelease / Test (push) Successful in 6m44s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m38s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m46s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m47s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m15s
build-prerelease / Package cortex RPM (push) Successful in 1m17s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 54s
2026-06-23 10:51:03 +03:00
2348cc2234 feat(B5): single-use top-up codes (redeem + mint CLI)
All checks were successful
CI / Format (push) Successful in 33s
CI / CUDA type-check (push) Successful in 1m33s
CI / Clippy (push) Successful in 3m5s
CI / Test (push) Successful in 6m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Completes the hybrid allocation model — the flat free grant (B4) plus
top-up codes that extend an account's allocation.

- topup.rs: redeem(account, raw_code) — timing-safe, single-use. A
  conditional `UPDATE … WHERE redeemed_by IS NULL RETURNING value` claims
  the code atomically (concurrent double-redeem → exactly one winner), then
  raises accounts.allocation_total in the same tx. Only sha256(code) is
  stored; a not-found code and an already-redeemed code return the SAME
  generic error via the same path (no "valid but spent" oracle). Raising
  the total automatically lifts every percent-limited key's effective cap;
  hardcap keys stay pinned (by design).
- mint(value, count, denomination) — inserts codes (hash-only), returns the
  raw codes once. Exposed as `helexa-upstream mint --value --count
  [--denomination]` (raw codes to stdout, one per line) — the seam the
  future faucet bot calls. Bot itself out of scope.
- POST /web/v1/redeem (session-protected) → {allocation_total} | generic 400.

Validated against Postgres: redeem raises the 1_000_000 free grant to
1_500_000, second redemption + unknown code both generic-400, and a
concurrent race for one code yields exactly one HTTP 200.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 10:50:43 +03:00
f2ba12bbc5 Merge feat/B4-account-api: /web/v1 account API + silent fingerprint abuse (B4, #59)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 33s
build-prerelease / Build neuron-blackwell (push) Successful in 1m36s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m34s
build-prerelease / Build neuron-ada (push) Successful in 2m5s
build-prerelease / Build neuron-ampere (push) Successful in 2m15s
build-prerelease / Build helexa-bench binary (push) Successful in 2m24s
build-prerelease / Build cortex binary (push) Successful in 2m34s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 10:42:16 +03:00
a9d7382be8 feat(B3): cortex upstream entitlement client (#57) + chained provider
All checks were successful
CI / Format (push) Successful in 40s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m56s
CI / Test (push) Successful in 6m36s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
cortex can now validate locally-unrecognised bearer keys against the
helexa-upstream authority and reserve/settle their budget there — mesh
accounts work for real inference. The EntitlementProvider trait is the
seam, so cortex's enforcement (auth.rs, metering.rs) is otherwise
unchanged.

- entitlements_upstream.rs: UpstreamEntitlementProvider over reqwest →
  B2's /authz/v1 (resolve/reserve/settle/release/snapshot), presenting the
  operator client bearer. Maps the wire contract back to the trait: granted
  → Reservation, rejected → BudgetError, 401 → InvalidKey. Fail-closed —
  unreachable resolve → AuthError::Unavailable (503, never 401);
  unreachable reserve → retryable BudgetError::RateLimited (refuse, never
  serve un-authorized). settle/release are best-effort (the upstream
  sweeper reaps a lost one).
- entitlements_chain.rs: ChainedEntitlementProvider tries local first
  (operator + infra keys, no network), falls through to upstream for
  unknown keys, and dispatches reserve/settle/release/snapshot to whichever
  backend resolved each account (local treats unknown principals as
  uncapped, so it can't be the blind default).
- cortex-core: AuthError::Unavailable{retry_after_secs}; [upstream] config
  (enabled/url/bearer/timeout). auth.rs maps Unavailable → 503 +
  Retry-After distinctly from InvalidKey → 401, regardless of require_auth.
- state.rs wires the chain when [upstream].enabled, else stays purely local.

Tests (upstream_chain.rs, 4): local key resolves without touching upstream;
unknown key falls through to a mock upstream; unknown-everywhere → 401
InvalidKey; upstream-unreachable → Unavailable (503-mapped), with local keys
still resolving. Existing gateway suites updated for the new config field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 10:41:48 +03:00
d94c62c143 feat(B4): /web/v1 account API + silent fingerprint multi-account abuse
All checks were successful
CI / Format (push) Successful in 44s
CI / CUDA type-check (push) Successful in 1m40s
CI / Clippy (push) Successful in 3m14s
CI / Test (push) Successful in 6m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The human-facing account surface the helexa.ai frontend (F4) consumes,
on top of B2's authz surface. Email+password auth with JWT sessions
(distinct from inference API keys); plain-JSON errors (the #63 envelope
stays on the authz surface).

- Auth lifecycle: register → email-verify → login → password-reset
  (request/confirm). argon2id passwords; verify/reset via single-use
  sha256-hashed email tokens; register and reset-request always return
  202 (no account enumeration). Email via a pluggable EmailSender (lettre
  Smtp + dev Log transport).
- API keys: create (sk-helexa-<base62(32 OsRng)>, raw shown once, stored
  as sha256 + non-secret prefix), list (prefix never the secret), archive,
  PATCH per-key limit (percent|hardcap). Protected by a JWT session
  middleware.
- Account balance endpoint (allocation total/spent/reserved).
- Silent fingerprint abuse: register captures the browser fingerprint;
  >= threshold (default 5) accounts sharing one fingerprint are silently
  deactivated + flagged — registration still returns a normal 202, and a
  deactivated account's key resolves as an ordinary 401 at the authz
  surface (no "banned" signal anywhere).
- crypto: argon2 hash/verify + CSPRNG token/key minting (base62). config
  gains [auth] + [email]. CORS on the app for the browser SPA.

Validated against a throwaway Postgres 16: verify-once, full lifecycle
(register→verify→login→create key→account→list→authz resolve→archive→401),
and 5-same-fingerprint → all accounts silently deactivated + no-clue 401.
8 unit + 11 gated integration tests; all skip cleanly without
UPSTREAM_TEST_DATABASE_URL so CI stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 10:28:20 +03:00
cb9e7c7c2e chore: untrack helexa.ai/node_modules + dist (B2 .gitignore slip)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 36s
build-prerelease / Build neuron-blackwell (push) Successful in 1m41s
build-prerelease / Build neuron-ada (push) Successful in 2m20s
build-prerelease / Build neuron-ampere (push) Successful in 2m21s
build-prerelease / Build cortex binary (push) Successful in 3m27s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m43s
build-prerelease / Test (push) Successful in 6m11s
build-prerelease / Package cortex RPM (push) Successful in 1m35s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m46s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m52s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m54s
build-prerelease / Build helexa-bench binary (push) Successful in 2m0s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m20s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 55s
The B2/B4 branches were cut before F0 added the helexa.ai .gitignore
entries, so a 'git add -A' on those branches swept node_modules into the
commit; it rode into main via the B2 merge. Untrack it (it stays on disk,
now correctly ignored). History still carries the blobs — acceptable for
an internal repo; can gc/filter later if size matters.
2026-06-23 10:27:37 +03:00
2604b9f134 Merge feat/B2-authz-api: /authz/v1 authority surface + client-auth + sweeper (B2)
Some checks failed
build-prerelease / Resolve version stamps + change detection (push) Successful in 45s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-23 10:25:27 +03:00
178e3092d5 Merge feat/F0-helexa-ai-scaffold: helexa.ai frontend scaffold (F0)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 53s
build-prerelease / Build neuron-blackwell (push) Successful in 1m27s
build-prerelease / Build neuron-ada (push) Successful in 2m4s
build-prerelease / Build neuron-ampere (push) Successful in 2m12s
build-prerelease / Build cortex binary (push) Successful in 2m32s
build-prerelease / Build helexa-bench binary (push) Successful in 2m41s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m49s
build-prerelease / Package cortex RPM (push) Successful in 1m16s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m35s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m36s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m29s
build-prerelease / Test (push) Successful in 6m40s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 2m2s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 55s
2026-06-23 10:12:38 +03:00
46befde4cd feat(B2): /authz/v1 authority surface + client-auth + reservation sweeper
All checks were successful
CI / Format (push) Successful in 49s
CI / CUDA type-check (push) Successful in 1m42s
CI / Clippy (push) Successful in 3m11s
CI / Test (push) Successful in 7m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The machine surface cortex's UpstreamEntitlementProvider (#57) consumes,
mirroring the EntitlementProvider trait 1:1 over the B1 ledger.

- `authz.rs`: POST /authz/v1/{resolve,reserve,settle,release,snapshot}.
  resolve → {principal, snapshot} | 401 invalid_api_key (a deactivated
  account resolves as the SAME 401 — the silent-abuse no-clue property).
  reserve returns 200 whether granted ({reservation_id}) or budget-refused
  ({rejected:{kind,...}}) — a refusal is an authoritative answer, not a
  transport failure; non-2xx means "fail closed" to the client. settle/
  release → 204 (idempotent). snapshot → {hard_cap,spent,reserved} | 404.
  Rejections use the shared #63 OpenAiError envelope (cortex-core dep).
- Client auth: shared-bearer middleware (constant-time compare via subtle)
  maps a token → operator_id (stamped into request extensions for #58
  served-usage); empty config = open dev surface (logged). mTLS deferred.
- ledger gains resolve_key (sha256 lookup, account-active-gated), snapshot,
  and sweep_stale (one data-modifying-CTE statement releasing aged-out open
  reservations and folding their reserved tokens back into accounts+keys).
- Sweeper task spawned in run(); [authz] ttl/interval + [client_auth]
  config; crypto::sha256 helper.

Validated against a throwaway Postgres 16 (fresh schema): resolve→reserve→
settle→snapshot round-trip, over-cap → 200 insufficient_quota rejection
(not retried away), deactivated account → 401 (no clue), missing/wrong
client bearer → 401 before any DB hit. 5 unit + 8 gated integration tests;
all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 10:12:19 +03:00
cf87e156c5 Merge feat/B1-helexa-upstream-skeleton: helexa-upstream skeleton + schema + ledger (B1, #59)
Some checks are pending
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 32s
build-prerelease / Build neuron-blackwell (push) Successful in 1m27s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m19s
build-prerelease / Test (push) Successful in 6m30s
build-prerelease / Build neuron-ada (push) Successful in 2m5s
build-prerelease / Build neuron-ampere (push) Successful in 2m14s
build-prerelease / Build helexa-bench binary (push) Successful in 2m15s
build-prerelease / Build cortex binary (push) Successful in 2m24s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m43s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m18s
build-prerelease / Package cortex RPM (push) Successful in 1m21s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m42s
2026-06-23 09:58:58 +03:00
79073170ec feat(F0): helexa.ai frontend scaffold + monorepo coexistence
All checks were successful
CI / Format (push) Successful in 47s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m38s
CI / Test (push) Successful in 5m24s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
New top-level `helexa.ai/` app for the public beta — Vite + React (SWC) +
TypeScript + react-bootstrap + react-router + react-i18next-ready. Not a
Cargo crate; lives beside the workspace.

- Vite with @vitejs/plugin-react-swc (standard Vite + npm, not the
  reference's rolldown/pnpm pin). `vite.config.ts` dev-proxies the mesh
  data-plane (/v1, /health → helexa-router) and account control-plane
  (/api → helexa-upstream /web/v1) same-origin, targets overridable via
  VITE_ROUTER_BASE_URL / VITE_ACCOUNT_BASE_URL.
- tsconfig (app/node, ported from the reference), eslint flat config,
  minimal index.css reset + bootstrap CSS, a placeholder App shell.
- Deps pre-declared for later phases: dexie + dexie-react-hooks (IndexedDB
  chat history), @fingerprintjs/fingerprintjs (anon throttle + register
  fingerprint), i18next/react-i18next, react-icons.
- Monorepo: root .gitignore ignores helexa.ai/{node_modules,dist} +
  .env.local (mirrors the existing /bench entries); committed
  package-lock.json for reproducible installs.

Validated: npm install resolves (vite 7 + plugin-react-swc 4 + react 19),
`npm run lint`/`typecheck`/`build` all green (344 modules via SWC →
dist/). The frontend isn't in the Cargo workspace, so the Rust CI is
unaffected. A path-filtered web CI job is deferred (needs a Node-capable
runner confirmed) and folded into a later phase.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 09:58:35 +03:00
71106afaf1 feat(B1): helexa-upstream crate skeleton + Postgres schema + ledger
All checks were successful
CI / CUDA type-check (push) Successful in 1m37s
CI / Format (push) Successful in 33s
CI / Clippy (push) Successful in 3m14s
CI / Test (push) Successful in 5m38s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
First milestone of the mesh-level account/authorization authority (#59).
New workspace crate `crates/helexa-upstream` (binary + lib), mirroring the
helexa-router skeleton: axum `build_app`/`run`, clap `serve --config`,
figment `UPSTREAM_`-prefixed config.

- **Storage:** PostgreSQL via sqlx (runtime query API — builds in CI with no
  DB or offline cache; correctness covered by gated integration tests).
  `migrations/0001_init.sql` is the full schema: users (argon2 hash slot,
  email_verified, registration_fingerprint), email_tokens, accounts
  (allocation total/spent/reserved + `accounts_no_overshoot` CHECK, silent
  `status` deactivation flag, fingerprint_flagged), api_keys (sha256 hash,
  percent|hardcap limit, cap_window, per-key ledger), reservations
  (BIGSERIAL id → maps to cortex Reservation.id u64), top_up_codes,
  served_usage, sessions. Migrations run on startup.
- **Ledger (no-overshoot core):** `ledger::{reserve,settle,release}` —
  reserve takes `SELECT … FOR UPDATE` on the account + key rows so
  concurrent reserves serialize and spent+reserved can never exceed the
  effective cap (= min(resolved key cap, remaining account allocation));
  the CHECK is the DB backstop. Settle clamps actual to [0,reserved] and is
  idempotent; release is idempotent. `resolve_abs_cap` (percent/hardcap,
  i128 math) is pure + unit-tested. Balance semantics here; rolling-window
  sub-caps + RateLimited land with the authz API (B2).
- `/health` does a DB round-trip.
- Config + example TOML ([server]/[db]/[grant] free grant/[abuse]
  fingerprint threshold).

Validated end-to-end against a throwaway Postgres 16: migration applies,
20 concurrent reserves of 100 against a 500 cap admit exactly 5 (reserved
== 500, never over), settle/release idempotent, hardcap key sub-cap binds
below the account, `/health` → db ok. CI runs the cap-math + config unit
tests; the DB integration tests skip cleanly when UPSTREAM_TEST_DATABASE_URL
is unset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 09:49:19 +03:00
d2dcdd6ebb Merge feat/74-outbound-tls-pinning: verify downstream cortex TLS certs (#74)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 32s
build-prerelease / Build neuron-blackwell (push) Successful in 1m48s
build-prerelease / Build helexa-bench binary (push) Successful in 1m59s
build-prerelease / Build neuron-ada (push) Successful in 2m23s
build-prerelease / Build neuron-ampere (push) Successful in 2m25s
build-prerelease / Build cortex binary (push) Successful in 2m58s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m8s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m22s
build-prerelease / Test (push) Successful in 5m37s
build-prerelease / Package cortex RPM (push) Successful in 1m14s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m38s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m38s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m41s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 54s
2026-06-21 21:29:15 +03:00
222c2a6116 Merge feat/75-federation-catalogue: aggregate /v1/models across operators (#75)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 40s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m33s
build-prerelease / Build cortex binary (push) Has been cancelled
build-prerelease / Build helexa-bench binary (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-21 21:23:49 +03:00
1115bb0942 feat(#74): verify downstream cortex TLS certs (outbound pinning)
All checks were successful
CI / Format (push) Successful in 41s
CI / CUDA type-check (push) Successful in 1m34s
CI / Clippy (push) Successful in 2m23s
CI / Test (push) Successful in 5m19s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The router is a TLS client to cortexes; the router->cortex hop crosses
the helexa->operator boundary carrying the client's bearer. This pins
that hop to an enrolled cert.

Trust mechanism (the open question): per-cortex enrolled trust anchor.
Each [[cortexes]] entry gets an optional `tls_ca` — a PEM CA (or
self-signed cert) the cortex's TLS cert must chain to. When set, the
router builds a client that trusts ONLY that anchor (platform roots
disabled), so the cortex must present the expected cert and a rogue
endpoint with any other (even publicly-valid) cert is rejected at the
handshake. Enrolment = the operator hands helexa the cortex's cert,
referenced by path in router config. This is the natural model for
self-hosted operators behind their own nginx/private CA, and reuses the
reqwest public API (no custom rustls verifier, no new TLS backend).

- `RouterState` now holds a per-cortex `reqwest::Client` map
  (`client_for`), replacing the single shared client; poller and dispatch
  use the per-cortex client. `build_client(tls_ca)` is the builder.
- Fail closed: a `tls_ca` that can't load omits the cortex from the
  client map — it's never polled or routed to, rather than silently
  degrading to unpinned TLS. The poller treats a missing client (and a
  rejected handshake) as a failed poll, so #72's existing reachability
  debounce excludes it.

Tests (`tls.rs`, 4): a live tokio-rustls HTTPS server proves a client
enrolled with the server's cert is accepted (200) while clients pinned to
a different cert — or using default roots — are rejected; the poller
marks a wrong-cert cortex unreachable while a correctly-enrolled one is
reachable; a missing pin file disables the cortex (fail closed); garbage
PEM is rejected at build. Existing suites updated for the per-cortex
client + new config field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 21:23:20 +03:00
63f578cb15 feat(#75): aggregate /v1/models across operators (federation catalogue)
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m38s
CI / Clippy (push) Successful in 2m18s
CI / Test (push) Successful in 4m54s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The router's /v1/models is now the deduped union of every reachable
cortex's catalogue, so an opencode client doing discovery against the
router resolves the whole federation without knowing about operators or
cortexes (resolves #61's "Router/discovery contract").

To preserve per-model limit/cost, the topology poller now retains each
cortex's full `cortex_core::node::CortexModelEntry` (was distilled to a
{loaded, feasible} bool). `entry_feasible()` replaces the dropped field;
dispatch (#73) and `cortexes_serving` use it — no routing behaviour
change.

`catalogue.rs::aggregate_models`:
- Dedupe by model id; a model served by >=1 reachable cortex appears once.
- Merge availability: `loaded` OR across operators; only feasible
  (loaded-or-cold-loadable) entries surface — a catalogue-only model no
  neuron can host is hidden.
- Re-tier to operator names: `feasible_on` becomes the cortexes that can
  serve it and `locations` the operators it's loaded on (node = cortex
  name), so the federation view doesn't leak each operator's neuron names
  or per-device VRAM.
- Conflict resolution: `limit` → tightest (smallest context, so a client
  never overflows the most-constrained operator); `cost` → cheapest
  (the federation "from" price). Richer range/region policy couples to
  #68, noted as follow-up.

Tests: 4 unit (dedupe+merge, unreachable excluded, infeasible hidden,
tightest-limit+cheapest-cost) + 1 end-to-end (two mock cortexes
overlapping on a model → GET /v1/models over HTTP asserts the merged
union). dispatch/topology suites updated for the entry-storage change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 21:08:16 +03:00
76c90fa993 Merge feat/73-capacity-aware-dispatch: capacity-aware dispatch + region affinity + failover (#73)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 32s
build-prerelease / Build neuron-blackwell (push) Successful in 1m38s
build-prerelease / Build helexa-bench binary (push) Successful in 2m13s
build-prerelease / Build neuron-ampere (push) Successful in 2m19s
build-prerelease / Build neuron-ada (push) Successful in 2m20s
build-prerelease / Build cortex binary (push) Successful in 2m24s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m6s
build-prerelease / Test (push) Successful in 5m41s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m21s
build-prerelease / Package cortex RPM (push) Successful in 1m19s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m33s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m34s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m37s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 53s
2026-06-21 19:48:13 +03:00
7984d27553 feat(#73): capacity-aware dispatch with region affinity + failover
All checks were successful
CI / Format (push) Successful in 40s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m16s
CI / Test (push) Successful in 4m53s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The router's data path. Wires the topology poller (#72) and the shared
streaming proxy (#71) into real request routing.

- `dispatch.rs`: `select_cortexes(model)` ranks reachable cortexes that
  can serve the model, best-first — loaded/warm before cold-loadable,
  region match before not, more healthy nodes before fewer, name for
  determinism. `dispatch()` extracts `model`, picks candidates, and
  forwards via `helexa_stream::forward_streaming` (bearer + bytes
  verbatim, SSE streamed back). Cortex's #63 rejections (429/400/…) pass
  through untouched; transport failures fail over to the next candidate;
  a genuine HTTP response — any status — is returned as-is, never retried
  away.
- Router-originated rejections use the #63 envelope: 404 model_not_found
  (no operator serves it), 503 service_unavailable + Retry-After (known
  but all unreachable / all candidates failed to connect), 400
  missing_model_field. `error.rs` is the router's envelope→axum adapter
  (mirrors cortex-gateway's).
- `handlers.rs`: `/v1/chat/completions`, `/v1/completions`,
  `/v1/responses`, `/v1/messages` dispatch to the same path on a chosen
  cortex. The router holds zero entitlement logic — routes on capacity,
  not budget.
- Config: optional `region` on the router and per-cortex for geo affinity.

Tests (`dispatch.rs`): routes to a serving cortex + forwards the bearer;
cortex 429 passes through and is NOT retried; transport failure fails
over to a live cortex; unknown→404, known-but-unreachable→503,
missing-model→400; ranking order (warm/region/headroom). 7 new, existing
skeleton/topology suites unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 19:40:07 +03:00
43ffffdccb Merge feat/72-router-topology-poller: router↔cortex capacity & catalogue poller (#72)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 34s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m18s
build-prerelease / Build neuron-blackwell (push) Successful in 1m43s
build-prerelease / Build helexa-bench binary (push) Successful in 2m6s
build-prerelease / Build neuron-ada (push) Successful in 2m17s
build-prerelease / Build neuron-ampere (push) Successful in 2m18s
build-prerelease / Build cortex binary (push) Successful in 2m18s
build-prerelease / Test (push) Successful in 4m50s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m19s
build-prerelease / Package cortex RPM (push) Successful in 1m25s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m41s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m40s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 55s
2026-06-21 19:09:13 +03:00
5fd7736abd feat(#72): router↔cortex topology poller (multi-operator capacity map)
All checks were successful
CI / Format (push) Successful in 39s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m14s
CI / Test (push) Successful in 4m43s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Builds the live topology the dispatcher (#73) will route on — the same
pattern as cortex↔neuron, one tier up.

- `poller.rs`: background loop polls each configured cortex's
  `GET /v1/models` (deserialised straight into the shared
  `cortex_core::node::CortexModelEntry`) and `GET /health`, on a
  configurable `poll_interval_secs` (default 10).
- `state.rs`: `RouterState` gains an `http_client`, `poll_interval`, and a
  `RwLock<HashMap<cortex_name, CortexTopology>>` pre-populated from config
  so the poller/handlers always find an entry. Per cortex: `reachable`,
  `consecutive_failures`, `last_poll`, healthy/total node counts, and a
  per-model `{loaded, feasible}` map (feasible = loaded OR cortex reports
  `feasible_on`, i.e. cold-loadable). `cortexes_serving(model)` returns the
  reachable cortexes that can serve a model — groundwork for #73.
- Debounce: a cortex flips unreachable only after
  `POLL_FAILURE_THRESHOLD` (3) consecutive failed polls, and recovers on
  the next good poll — mirrors cortex's neuron-poll debounce so a blip
  can't yank a whole operator out of routing. `/health` poll is
  best-effort and never flips reachability on its own.
- `lib.rs` spawns the poll loop in `run()`. `/health` now surfaces
  `cortexes.reachable`; `status` stays router-liveness (always `ok`).

Tests (`topology.rs`): live-map build (loaded vs catalogue-only feasible,
node counts, routing helper); unreachable→excluded→recovers across the
debounce threshold; dead endpoint never panics. Skeleton tests unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 19:01:29 +03:00
03fd4960c3 Merge fix/71-shared-streaming-proxy: shared helexa-stream SSE proxy (#71)
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 35s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m24s
build-prerelease / Build neuron-blackwell (push) Successful in 1m40s
build-prerelease / Build neuron-ampere (push) Successful in 2m17s
build-prerelease / Build helexa-bench binary (push) Successful in 2m11s
build-prerelease / Build neuron-ada (push) Successful in 2m18s
build-prerelease / Build cortex binary (push) Successful in 2m25s
build-prerelease / Test (push) Successful in 4m53s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m16s
build-prerelease / Package cortex RPM (push) Successful in 1m15s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m36s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m39s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 54s
# Conflicts:
#	Cargo.lock
#	Cargo.toml
2026-06-21 18:15:12 +03:00
5ed6bc3390 Merge feat/70-router-skeleton: helexa-router binary skeleton (#70)
Some checks failed
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 56s
build-prerelease / Build neuron-blackwell (push) Successful in 1m31s
build-prerelease / Build neuron-ada (push) Successful in 2m4s
build-prerelease / Build neuron-ampere (push) Successful in 2m11s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m29s
build-prerelease / Build cortex binary (push) Successful in 2m42s
build-prerelease / Build helexa-bench binary (push) Successful in 2m56s
build-prerelease / Package cortex RPM (push) Successful in 1m18s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m42s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m47s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 2m24s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 2m25s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-06-21 18:06:07 +03:00
cabec1d08a fix(#71): extract SSE streaming passthrough into shared helexa-stream
All checks were successful
CI / Format (push) Successful in 40s
CI / CUDA type-check (push) Successful in 1m38s
CI / Clippy (push) Successful in 2m15s
CI / Test (push) Successful in 6m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The true-streaming SSE passthrough (Body::from_stream, no full-response
buffering, with chunk-observation hooks) was cortex-only. helexa-router
(#69) needs the same mechanism to proxy a chat-completions/messages
stream verbatim to a selected cortex. Extract it once.

New `crates/helexa-stream` owns the *mechanism* (kept HTTP-free
cortex-core untouched — it would have forced axum/reqwest/futures onto
every cortex-core consumer):

- `forward_streaming(client, url, headers, body, observer)` — POST and
  stream the response back chunk-for-chunk; status-agnostic, so a
  non-2xx (e.g. cortex 429) is passed through with status+headers
  intact (the #69 backpressure-passthrough requirement).
- `ChunkObserver` trait + `ObservedStream` wrapper — feeds each chunk to
  the observer, calls `finish` exactly once on clean end or on drop
  (client disconnect).
- `BodyTail` (bounded tail accumulator) + `last_count_for` (trailing
  OpenAI `usage` extraction) — the reusable pieces an observer uses.

cortex keeps its *policy*: `proxy.rs` now supplies a `CortexMetrics`
observer (per-request token metrics + per-principal reservation settle),
its logging contract, and the error envelope, driving the shared
mechanism. `proxy::last_count_for` is re-exported so `handlers`/
`anthropic_sse` call sites are unchanged. No behaviour change — the
existing cortex `streaming.rs` tests pass as-is.

helexa-stream tests prove chunk-for-chunk incremental delivery, observer
finish-once, usage extraction, and non-2xx passthrough.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 18:05:35 +03:00
881fc85a4c feat(#70): helexa-router binary skeleton — plaintext axum server
All checks were successful
CI / Format (push) Successful in 38s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m19s
CI / Test (push) Successful in 4m40s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Foundation for epic #69 (public multi-operator ingress proxy). New
`crates/helexa-router` workspace binary: a plaintext axum server that
reuses cortex-core types and serves the two stub endpoints the rest of
#69 builds on.

- `[router] listen` + `[[cortexes]]` config via figment + `HELEXA_ROUTER_`
  env overrides, matching the cortex/neuron convention.
- `GET /health` reports the configured downstream cortex count.
- `GET /v1/models` returns an empty OpenAI list (real cross-operator
  aggregation is #75).
- No inbound TLS listener (edge nginx terminates client TLS per #69's
  posture); no auth layer — the router forwards the client bearer to
  cortex and holds zero entitlement logic (#47 stays additive).
- 3 tests: both endpoints over a real ephemeral-port server, plus
  TOML+env config load.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-21 17:52:28 +03:00
b2ed20b55a docs(CLAUDE.md): document the branch → CI → merge-on-green workflow
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 31s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
Capture the development loop in-repo so it's available to any session, not
just via personal agent memory: feature branch per change; local CI triad is
CPU-only so the branch CI's CUDA type-check is the real gate for neuron/TP
changes; push on local-green and background-watch; merge when the four
validation jobs are green (not the SRPM/COPR deploy jobs); docs-only changes
can go straight to main. Notes the core.sshCommand key-pinning gotcha.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
2026-06-21 15:05:46 +03:00
bee27e9b9c Merge fix/68-cost-schema-wire-contract: pin the /v1/models cost wire contract
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 31s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m20s
build-prerelease / Build neuron-blackwell (push) Successful in 1m42s
build-prerelease / Build helexa-bench binary (push) Successful in 2m2s
build-prerelease / Build neuron-ampere (push) Successful in 2m17s
build-prerelease / Build neuron-ada (push) Successful in 2m19s
build-prerelease / Build cortex binary (push) Successful in 2m31s
build-prerelease / Test (push) Successful in 4m49s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m22s
build-prerelease / Package cortex RPM (push) Successful in 1m20s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m44s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m45s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m47s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 54s
Closes #68. Documents ModelCost as the source-of-truth pricing field
(USD per 1M tokens, JSON numbers — models.dev/opencode shape), defines the
absent-vs-0.0 distinction (not-priced vs intentionally-free), adds a wire
test locking it, and documents cost.* in models.example.toml. The cost code
path already existed; this pins the contract. Branch CI green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
2026-06-20 12:08:23 +03:00
87d9c291ce fix(#68): pin the /v1/models cost wire contract — units + absent-vs-zero
All checks were successful
CI / Format (push) Successful in 44s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m15s
CI / Test (push) Successful in 4m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The cost code path already exists (cortex list_models populates
cost: profile.cost from the catalogue; aliases inherit it), so opencode's
$0.00 is a config gap (no cost in the live models.toml), not missing
plumbing. What was missing is the *contract*: units pinned against a wire
test, and a defined meaning for "free".

- Document ModelCost as the load-bearing source of truth: USD per 1,000,000
  tokens as JSON numbers (models.dev/opencode shape) — NOT per-token, NOT
  decimal strings (OpenRouter's pricing shape, which helexa deliberately
  does not emit). Define the absent-vs-zero distinction: cost omitted = "not
  priced / unknown"; cost present with 0.0 = "intentionally free". Note the
  advertised rate must equal what metering (#51) / reconciliation (#58/#59)
  bill against — today both read this catalogue value.
- New wire test (model_cost.rs): a priced model with cache tiers flows
  through as per-million numbers; an explicit-0.0 free model keeps its cost
  block with cache tiers omitted; an unpriced model omits `cost` entirely.
- models.example.toml: document cost.* in the field reference and show all
  three cases (priced-free explicit 0.0 vs the unpriced Qwen3-8B with no
  cost block).

Decisions recorded on #68: source of truth = operator models.toml for now
(marketplace clearing house #59 later, same value); no OpenRouter-style
`pricing` (opencode/models.dev alignment is sufficient); end-to-end
non-zero $ spent needs operators to populate cost in the live catalogue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
2026-06-20 12:02:03 +03:00
d4742467e0 Merge fix/65-text-prefill-vram-backstop: request-time length-aware VRAM backstop for text prefill
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 32s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 1m41s
build-prerelease / Build neuron-ampere (push) Successful in 2m16s
build-prerelease / Build neuron-ada (push) Successful in 2m17s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m38s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Test (push) Successful in 4m59s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m36s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m41s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 51s
Closes #65. Gives the text prefill path a request-time, length-aware
VRAM guard (reusing #67's ContextProfile KV cost against current free
VRAM), closing the poll-vs-request snapshot staleness gap and the
vision/text asymmetry. Branch CI green (fmt, clippy, test, CUDA type-check).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
2026-06-20 11:53:33 +03:00
e7f7e376fc fix(#65): request-time length-aware VRAM backstop for text prefill
All checks were successful
CI / Format (push) Successful in 33s
CI / CUDA type-check (push) Successful in 1m41s
CI / Clippy (push) Successful in 2m24s
CI / Test (push) Successful in 4m41s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Close the poll-vs-request snapshot gap #67 left open. The text prefill
guard (validate_request) only checked the static min_free_vram floor;
the derived input cap (effective_prompt_cap) is computed at /models poll
time from the tightest card's free VRAM *then*. If free VRAM drops
between that poll and the request — a co-resident model loads, a
concurrent prefill grows its KV — a prompt at-or-below the now-stale cap
clears the floor yet no longer fits, OOMing mid-prefill and poisoning the
device context (the 2026-05-26 beast incident #47 exists to eliminate).

validate_request now re-runs #67's length×KV-vs-VRAM physics against
request-time free VRAM, reusing the model's ContextProfile
(kv_bytes_per_token_per_card, full-attention-layer-only, TP-sharded)
rather than re-deriving the cost. Footprint = KV(prompt + output_reserve)
+ activation_headroom + static floor, all per card and commensurable with
the tightest-card free VRAM on both single-GPU and TP loads. Degenerate
zero-KV / no-profile models ride the existing floor check, mirroring
derive_limit's VRAM-ceiling fallback; CPU loads (vram_free_mb == 0) skip
all VRAM checks unchanged.

This closes the vision/text asymmetry: the text path now has the
live-VRAM guard validate_vision_prefill already gave the vision path.

5 unit tests incl. the acceptance staleness test: a cap derived against
ample free VRAM, applied at request time against tightened VRAM, rejects
a prompt sized at the stale cap with a clean InsufficientVram (503)
instead of an OOM. Threaded context_limit_cfg into chat_completion_tp_inner
(spawned, no &self) and used &self.context_limit_cfg at the three method
call sites.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
2026-06-20 11:45:53 +03:00
3b9a6e37f6 Merge fix/cortex-poll-debounce-retryable: poll debounce + retryable 503 for feasible-but-unhealthy node
All checks were successful
build-prerelease / Resolve version stamps + change detection (push) Successful in 58s
build-prerelease / Build neuron-blackwell (push) Successful in 1m31s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m31s
build-prerelease / Build cortex binary (push) Successful in 3m0s
build-prerelease / Test (push) Successful in 5m3s
build-prerelease / Package cortex RPM (push) Successful in 1m20s
build-prerelease / Build neuron-ada (push) Successful in 2m2s
build-prerelease / Build neuron-ampere (push) Successful in 2m12s
build-prerelease / Build helexa-bench binary (push) Successful in 2m9s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m39s
build-prerelease / Package helexa-bench RPM (push) Successful in 1m20s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m41s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 1m43s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 53s
2026-06-18 12:46:30 +03:00
526b662c5e fix(cortex): poll-failure debounce + retryable 503 for feasible-but-unhealthy node
All checks were successful
CI / Format (push) Successful in 44s
CI / CUDA type-check (push) Successful in 1m31s
CI / Clippy (push) Successful in 2m13s
CI / Test (push) Successful in 5m9s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Defense-in-depth for the agent0 NoFeasibleNeuron storm (root cause fixed in
neuron). Two cortex resilience gaps this incident exposed:

1. Brittle health flip: the poller marked a node unhealthy on a SINGLE missed
   /models poll, instantly yanking the node and all its models from routing.
   A busy neuron briefly slow to answer shouldn't be declared dead. Now
   debounced: NodeState.consecutive_poll_failures must reach
   POLL_FAILURE_THRESHOLD (3) before the node flips unhealthy (~20s at the 10s
   poll interval); any successful poll resets it. A never-healthy node stays
   unhealthy (the counter only protects an already-healthy node from blips).

2. Transient surfaced as permanent: when a catalogued model's only feasible
   neuron is momentarily unhealthy, the router returned 404 NoFeasibleNeuron —
   which litellm/clients treat as non-retryable, so agent0 hard-failed.
   pick_feasible_neuron now distinguishes "a feasible node exists but is
   unhealthy right now" → new RouteError::FeasibleNodeUnhealthy (503 +
   Retry-After: 3, retryable) from "no node could ever satisfy the topology" →
   404 NoFeasibleNeuron (permanent). Mirrors the beast case exactly: healthy
   1-GPU nodes + an unhealthy 2-GPU node → retry, don't fail.

Tests: poller test updated to assert debounce (1 miss keeps healthy, 3 flip);
new feasibility_routing tests cover transient-503 vs permanent-404. Local
fmt/clippy/test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 12:39:18 +03:00
db7e373b90 fix(neuron): decouple GET /models from the inference worker (control-plane starvation)
Some checks failed
CI / CUDA type-check (push) Successful in 1m40s
CI / Format (push) Successful in 36s
CI / Clippy (push) Successful in 2m20s
CI / Test (push) Successful in 4m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Package helexa-neuron-ada RPM (push) Blocked by required conditions
build-prerelease / Package helexa-neuron-ampere RPM (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 36s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 1m43s
build-prerelease / Build neuron-ampere (push) Successful in 2m20s
build-prerelease / Build neuron-ada (push) Successful in 2m21s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m34s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
Root cause of the agent0 `NoFeasibleNeuron` 404 storm: `GET /models` →
`LoadedHandle::derived_limit` (#67) queried free VRAM *synchronously through
the per-device worker thread* on every poll. During inference that worker is
saturated serially processing forward jobs, so the VRAM query queued behind
them and `/models` blocked for seconds. cortex's poller timed out on `/models`,
marked the (sole-feasible) node unhealthy, and the model fell out of routing →
404. Confirmed live: under load, `/version` and `/health` stayed ~4ms while
`/models` hit the 5s timeout.

Fix — the HTTP control plane never touches the inference worker:
- LoadedModel / TpLoadedModel gain `last_free_mb: AtomicU64`, a cached free-VRAM
  reading.
- `derived_limit` is now sync and reads `last_free_mb` instead of awaiting a
  worker query — so `/models` is a pure cache read regardless of inference load.
- The cache is refreshed off the request path: seeded at load (worker idle),
  then by a background `vram_cache_refresh_loop` every 5s. Single-GPU caches the
  device's free VRAM; TP caches the tightest free across ranks — the exact
  values `derived_limit` used before, just no longer on the request path. A
  transient `0` (worker gone/poisoned) never clobbers a good cached value.
- The request-path live VRAM check in `validate_request` is unchanged, so the
  real prefill OOM guard still uses fresh readings.

226 neuron unit tests pass; non-CUDA build + fmt + clippy green. CUDA/TP paths
validated by branch CI; live acceptance = `/models` stays responsive under
concurrent inference (re-run of the repro).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 12:32:15 +03:00
275 changed files with 27255 additions and 308 deletions

View File

@@ -66,6 +66,7 @@ jobs:
build_cortex: ${{ steps.changes.outputs.build_cortex }}
build_neuron: ${{ steps.changes.outputs.build_neuron }}
build_bench: ${{ steps.changes.outputs.build_bench }}
build_upstream: ${{ steps.changes.outputs.build_upstream }}
check_rust: ${{ steps.changes.outputs.check_rust }}
steps:
- uses: actions/checkout@v4
@@ -104,6 +105,7 @@ jobs:
BUILD_CORTEX=true
BUILD_NEURON=true
BUILD_BENCH=true
BUILD_UPSTREAM=true
CHECK_RUST=true
if [ "${GITHUB_EVENT_NAME}" = "push" ]; then
@@ -149,6 +151,7 @@ jobs:
NEURON_RE='^crates/neuron/|^crates/cortex-core/|^Cargo\.toml$|^Cargo\.lock$|^rpm/helexa-neuron-prerelease\.spec$|^data/neuron|^neuron\.example\.toml$|^\.gitea/workflows/build-prerelease\.yml$'
CORTEX_RE='^crates/cortex-gateway/|^crates/cortex-cli/|^crates/cortex-core/|^Cargo\.toml$|^Cargo\.lock$|^rpm/cortex-prerelease\.spec$|^data/cortex|^cortex\.example\.toml$|^models\.example\.toml$|^\.gitea/workflows/build-prerelease\.yml$'
BENCH_RE='^crates/helexa-bench/|^crates/cortex-core/|^Cargo\.toml$|^Cargo\.lock$|^rpm/helexa-bench-prerelease\.spec$|^data/helexa-bench|^helexa-bench\.example\.toml$|^\.gitea/workflows/build-prerelease\.yml$'
UPSTREAM_RE='^crates/helexa-upstream/|^crates/cortex-core/|^Cargo\.toml$|^Cargo\.lock$|^rpm/helexa-upstream-prerelease\.spec$|^data/helexa-upstream|^helexa-upstream\.example\.toml$|^\.gitea/workflows/build-prerelease\.yml$'
# Any Rust change (incl. crates not packaged here, e.g.
# helexa-acp) still needs lint+test on main.
RUST_RE='\.rs$|^crates/|Cargo\.toml$|^Cargo\.lock$'
@@ -156,10 +159,12 @@ jobs:
CORTEX_BASE=$(base_for cortex)
NEURON_BASE=$(base_for helexa-neuron-blackwell)
BENCH_BASE=$(base_for helexa-bench)
UPSTREAM_BASE=$(base_for helexa-upstream)
BUILD_CORTEX=$(decide "$CORTEX_BASE" "$CORTEX_RE")
BUILD_NEURON=$(decide "$NEURON_BASE" "$NEURON_RE")
BUILD_BENCH=$(decide "$BENCH_BASE" "$BENCH_RE")
if [ "$BUILD_CORTEX" = "true" ] || [ "$BUILD_NEURON" = "true" ] || [ "$BUILD_BENCH" = "true" ]; then
BUILD_UPSTREAM=$(decide "$UPSTREAM_BASE" "$UPSTREAM_RE")
if [ "$BUILD_CORTEX" = "true" ] || [ "$BUILD_NEURON" = "true" ] || [ "$BUILD_BENCH" = "true" ] || [ "$BUILD_UPSTREAM" = "true" ]; then
CHECK_RUST=true
else
CHECK_RUST=$(decide "$CORTEX_BASE" "$RUST_RE")
@@ -170,8 +175,9 @@ jobs:
echo "build_cortex=${BUILD_CORTEX}" >> "$GITHUB_OUTPUT"
echo "build_neuron=${BUILD_NEURON}" >> "$GITHUB_OUTPUT"
echo "build_bench=${BUILD_BENCH}" >> "$GITHUB_OUTPUT"
echo "build_upstream=${BUILD_UPSTREAM}" >> "$GITHUB_OUTPUT"
echo "check_rust=${CHECK_RUST}" >> "$GITHUB_OUTPUT"
echo "### change detection: build_cortex=${BUILD_CORTEX} build_neuron=${BUILD_NEURON} build_bench=${BUILD_BENCH} check_rust=${CHECK_RUST}"
echo "### change detection: build_cortex=${BUILD_CORTEX} build_neuron=${BUILD_NEURON} build_bench=${BUILD_BENCH} build_upstream=${BUILD_UPSTREAM} check_rust=${CHECK_RUST}"
# fmt + clippy + test moved here from ci.yml for main pushes so the
# two workflows stop queueing against each other (ci.yml's checks
@@ -303,6 +309,45 @@ jobs:
path: artifacts/helexa-bench
retention-days: 1
build-upstream:
name: Build helexa-upstream binary
timeout-minutes: 25
needs: prepare
if: needs.prepare.outputs.build_upstream == 'true'
# Pure-Rust, non-CUDA binary — same runner as cortex/bench.
runs-on: rust
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: sccache
SCCACHE_ENDPOINT: http://caveman.kosherinata.internal:9000
SCCACHE_REGION: auto
SCCACHE_S3_USE_SSL: "false"
AWS_ACCESS_KEY_ID: ${{ secrets.SCCACHE_S3_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SCCACHE_S3_SECRET_KEY }}
# helexa-upstream uses the sqlx runtime query API (no compile-time
# query macros), so it builds without a database or a .sqlx cache.
# Set OFFLINE defensively so a stray macro can never reach for a DB.
SQLX_OFFLINE: "true"
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
- name: Build helexa-upstream (release, sccache escalation)
run: script/ci-cargo-escalate.sh cargo build --release -p helexa-upstream
- name: Stage binary
run: |
mkdir --parents artifacts
cp target/release/helexa-upstream artifacts/helexa-upstream
./artifacts/helexa-upstream --version || true
- uses: actions/upload-artifact@v3
with:
name: upstream-fc43
path: artifacts/helexa-upstream
retention-days: 1
build-neuron:
name: Build neuron-${{ matrix.flavour }}
timeout-minutes: 35
@@ -459,6 +504,44 @@ jobs:
path: ~/rpmbuild/RPMS/x86_64/*.rpm
retention-days: 7
package-upstream:
name: Package helexa-upstream RPM
timeout-minutes: 20
needs: [prepare, build-upstream]
runs-on: rpm
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
- uses: actions/download-artifact@v3
with:
name: upstream-fc43
path: artifacts/
- name: Build RPM
run: |
set -eux
rm -f ~/.rpmmacros
rpmdev-setuptree
cp artifacts/helexa-upstream ~/rpmbuild/SOURCES/
cp data/helexa-upstream.service ~/rpmbuild/SOURCES/
cp data/helexa-upstream-sysusers.conf ~/rpmbuild/SOURCES/
cp data/helexa-upstream-firewalld.xml ~/rpmbuild/SOURCES/
cp helexa-upstream.example.toml ~/rpmbuild/SOURCES/
cp LICENSE ~/rpmbuild/SOURCES/
rpmbuild -bb rpm/helexa-upstream-prerelease.spec \
--define "upstream_version ${{ needs.prepare.outputs.version }}" \
--define "upstream_prerelease ${{ needs.prepare.outputs.release }}" \
--undefine dist \
--define "dist .fc43"
- uses: actions/upload-artifact@v3
with:
name: rpm-upstream-fc43
path: ~/rpmbuild/RPMS/x86_64/*.rpm
retention-days: 7
package-neuron:
name: Package helexa-neuron-${{ matrix.flavour }} RPM
timeout-minutes: 20
@@ -508,7 +591,7 @@ jobs:
publish:
name: Publish to rpm.lair.cafe (unstable)
timeout-minutes: 25
needs: [lint, test, package-cortex, package-neuron, package-bench]
needs: [lint, test, package-cortex, package-neuron, package-bench, package-upstream]
# Runs when at least one package was built and nothing failed.
# lint/test may be skipped (docs-only refs never get here because
# no packages build), but a real failure in any blocks the
@@ -518,10 +601,11 @@ jobs:
!cancelled()
&& (needs.lint.result == 'success' || needs.lint.result == 'skipped')
&& (needs.test.result == 'success' || needs.test.result == 'skipped')
&& (needs.package-cortex.result == 'success' || needs.package-neuron.result == 'success' || needs.package-bench.result == 'success')
&& (needs.package-cortex.result == 'success' || needs.package-neuron.result == 'success' || needs.package-bench.result == 'success' || needs.package-upstream.result == 'success')
&& needs.package-cortex.result != 'failure'
&& needs.package-neuron.result != 'failure'
&& needs.package-bench.result != 'failure'
&& needs.package-upstream.result != 'failure'
}}
runs-on: rpm
concurrency:

3
.gitignore vendored
View File

@@ -1,6 +1,9 @@
/target
/bench/node_modules
/bench/dist
/helexa.ai/node_modules
/helexa.ai/dist
helexa.ai/.env.local
*.swp
*.swo
.idea/

View File

@@ -185,6 +185,32 @@ Run these locally before pushing. `cargo fmt --all` fixes formatting
automatically. Clippy warnings must be resolved, not suppressed with
`#[allow(...)]` unless there is a clear rationale.
## Development workflow
Work each change on its own branch; `main` stays releasable.
1. Implement on a feature branch (`fix/<issue>-…`, `feat/<issue>-…`).
2. Run the CI triad locally (`cargo fmt --check --all`,
`cargo clippy --workspace -- -D warnings`, `cargo test --workspace`).
Local builds are **CPU-only** — the `#[cfg(feature = "cuda")]` neuron/TP
paths do NOT compile locally. The branch CI's **CUDA type-check** job is
the only thing that validates them, so for any neuron change the push to
Gitea is the real gate, not a rubber stamp.
3. Push the branch on local-green (no need to ask first), and background-watch
its CI run via the gitea-mcp `actions_run_read` tools. Start the next piece
of work meanwhile.
4. Merge to `main` when the four **validation** jobs are green — Format,
Clippy, Test, CUDA type-check. The SRPM / COPR / version-bump jobs are the
deploy pipeline (they run on `main`), not validation — don't wait on them.
5. Merging/pushing to `main` triggers the auto-deploy pipeline.
Docs-only changes (no `#[cfg(feature = "cuda")]` impact) can go straight to
`main` — there's nothing for the CUDA type-check to prove.
SSH note: the gitea remote host offers multiple agent keys and cuts the
connection before reaching the right one. This repo pins the working key via
`git config core.sshCommand "ssh -i ~/.ssh/id_grenade -o IdentitiesOnly=yes"`.
## Environment
- Targets Fedora 43 (systemd, SELinux enforcing)

873
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -7,6 +7,9 @@ members = [
"crates/neuron",
"crates/helexa-acp",
"crates/helexa-bench",
"crates/helexa-router",
"crates/helexa-stream",
"crates/helexa-upstream",
]
[workspace.package]

View File

@@ -90,3 +90,20 @@ account_id = "operator"
key_id = "infra"
# No hard_cap → uncapped operator infra key (own fleet, own use). Still
# metered for visibility.
# -- Upstream (helexa mesh) entitlements client (#57) --------------------
# When enabled, a bearer key NOT found in [[entitlements.keys]] above is
# validated against the helexa-upstream authority (mesh accounts), and its
# budget is reserved/settled there. Operator-local keys (incl. the infra
# key) never leave this process. Fail-closed: if upstream is unreachable a
# request is refused (503 + Retry-After), never served un-authorized.
# Disabled by default — a standalone operator runs purely local.
[upstream]
enabled = false
# url = "https://upstream.helexa.ai"
# Shared client bearer this cortex presents (maps to an operator_id
# upstream). Override via CORTEX_UPSTREAM__BEARER in prod.
# bearer = "replace-with-operator-client-secret"
# timeout_secs = 5
# How often to flush served-usage counters to upstream for reconciliation (#58).
# served_usage_report_interval_secs = 60

View File

@@ -22,6 +22,43 @@ pub struct GatewayConfig {
/// setups keep working until keys are configured.
#[serde(default)]
pub entitlements: EntitlementsConfig,
/// helexa-upstream client (#57). When enabled, keys not found in the
/// local `[entitlements]` config are validated against the mesh
/// authority, and budget is reserved/settled there. Disabled by default
/// — a single operator runs purely local.
#[serde(default)]
pub upstream: UpstreamClientConfig,
}
/// `[upstream]` — the helexa-upstream authority client (#57). Locally
/// unrecognised bearer keys are resolved against `url`'s `/authz/v1` surface
/// (mesh accounts); local keys (operator + infra) never leave the process.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct UpstreamClientConfig {
/// Enable the upstream fallthrough. Off → purely local entitlements.
#[serde(default)]
pub enabled: bool,
/// Base URL of helexa-upstream (e.g. "https://upstream.helexa.ai").
#[serde(default)]
pub url: String,
/// Shared client bearer this cortex presents to `/authz/v1` (maps to an
/// operator_id upstream). Sent as `Authorization: Bearer <bearer>`.
#[serde(default)]
pub bearer: String,
/// Per-call timeout (seconds) to upstream.
#[serde(default = "default_upstream_timeout")]
pub timeout_secs: u64,
/// How often (seconds) to flush served-usage counters to upstream for
/// reconciliation (#58).
#[serde(default = "default_served_usage_interval")]
pub served_usage_report_interval_secs: u64,
}
fn default_upstream_timeout() -> u64 {
5
}
fn default_served_usage_interval() -> u64 {
60
}
/// `[entitlements]` — the local/static [`crate::entitlements::EntitlementProvider`]
@@ -129,6 +166,7 @@ impl Default for GatewayConfig {
neurons: vec![],
models_config: default_models_path(),
entitlements: EntitlementsConfig::default(),
upstream: UpstreamClientConfig::default(),
}
}
}

View File

@@ -81,12 +81,19 @@ pub struct BudgetSnapshot {
pub reserved: u64,
}
/// Authentication failure — the bearer key could not be resolved. Maps to
/// `401 invalid_api_key` (#49/#63).
/// Authentication failure — the bearer key could not be resolved.
#[derive(Debug, thiserror::Error)]
pub enum AuthError {
/// The key is genuinely unknown → `401 invalid_api_key` (#49/#63).
#[error("invalid or unknown API key")]
InvalidKey,
/// The authority that could resolve the key is unreachable (e.g. the
/// helexa-upstream client failed, #57). Fail **closed** but distinctly:
/// a transient outage must surface as `503 service_unavailable` +
/// `Retry-After`, never `401` — a real key must not be rejected as
/// invalid during an upstream blip.
#[error("entitlement authority unavailable; retry in {retry_after_secs}s")]
Unavailable { retry_after_secs: u64 },
}
/// Why a reservation was refused. Carries enough for the caller to build the

View File

@@ -54,10 +54,26 @@ pub struct ModelLimit {
pub output: usize,
}
/// Operator-set pricing in USD per 1M tokens.
/// Operator-set pricing, **USD per 1,000,000 tokens, as JSON numbers**
/// (`float`) — the models.dev/opencode `cost` convention, which is what
/// helexa's primary client reads. NOT per-token, NOT decimal strings (that
/// is OpenRouter's `pricing` shape, which helexa deliberately does not emit
/// — see #68). A client must not rescale by 10⁶.
///
/// Self-hosted deployments typically leave both at `0.0`. Cache fields are
/// optional — set when the backend supports a prefix-cache discount tier.
/// `cost` is sourced from the operator's `models.toml` catalogue profile and
/// surfaced verbatim on `/v1/models`. The *absent* vs *zero* distinction is
/// intentional and load-bearing (#68):
/// - **`cost` absent** (the whole object omitted) — the model is **not
/// priced**: the operator has not declared a rate. Clients should treat
/// spend as unknown, not free.
/// - **`cost` present with `input`/`output` = `0.0`** — the model is
/// **intentionally free** (self-hosted, no charge). opencode renders `$0`.
///
/// Cache fields are optional — set them only when the backend supports a
/// prefix-cache discount tier (relevant once cache-token reporting, #64,
/// lands). The advertised rate here must equal the rate metering (#51) and
/// reconciliation (#58/#59) bill against; today both read this catalogue
/// value.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelCost {
/// USD per 1M input (prompt) tokens.
@@ -98,7 +114,8 @@ pub struct ModelInfo {
/// `None` when neither the catalogue nor the loaded model can provide it.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub limit: Option<ModelLimit>,
/// Operator-set pricing in USD per 1M tokens (0.0 = free/self-hosted).
/// Operator-set pricing — see [`ModelCost`] for units and the
/// absent (not priced) vs `0.0` (intentionally free) distinction.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cost: Option<ModelCost>,
/// `true` when the model's tokenizer contains recognised tool-call

View File

@@ -32,6 +32,12 @@ pub struct NodeState {
/// least-busy replica when a model is loaded on more than one neuron.
/// Empty until the first /health poll reports load.
pub model_load: HashMap<String, ModelLoad>,
/// Consecutive failed `/models` polls. The poller marks a node
/// unhealthy only once this crosses a threshold, so a single transient
/// miss (e.g. a neuron momentarily slow to answer while busy) doesn't
/// yank the node — and all its models — out of routing. Reset to 0 on
/// any successful poll.
pub consecutive_poll_failures: u32,
}
/// A model registered on a node, with its runtime status.
@@ -130,7 +136,9 @@ pub struct CortexModelEntry {
/// at load time. `None` when neither source provides it.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub limit: Option<ModelLimit>,
/// Operator-set pricing in USD per 1M tokens (0.0 = free/self-hosted).
/// Operator-set pricing from the catalogue profile — see
/// [`cortex_core::harness::ModelCost`] for units (USD per 1M tokens) and
/// the absent (not priced) vs `0.0` (intentionally free) distinction.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cost: Option<ModelCost>,
/// `true` when any neuron reports this model supports tool calls.

View File

@@ -116,6 +116,23 @@ pub struct Usage {
/// prompt caching lands (#11); `None` until then.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub prompt_tokens_details: Option<PromptTokensDetails>,
/// helexa extension (non-OpenAI): server-measured prefill/decode
/// timing, so the bench harness can compute true prefill vs decode
/// tok/s instead of inferring both from client-side SSE arrival
/// (#85). Additive and optional — standard OpenAI clients ignore
/// it; cortex forwards usage verbatim so it survives proxying.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub helexa_timing: Option<HelexaTiming>,
}
/// helexa extension carried on [`Usage::helexa_timing`]. Mirrors
/// neuron's internal `FinishTiming`. All fields are server-measured;
/// `prefill_tokens` is the prefill-rate denominator.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HelexaTiming {
pub prefill_ms: u64,
pub decode_ms: u64,
pub prefill_tokens: u64,
}
/// Sub-counts of `Usage::completion_tokens`.

View File

@@ -66,14 +66,48 @@ pub struct ResponsesRequest {
pub extra: Value,
}
/// `input` is either a single string or an array of typed items.
/// `input` is either a single string or an array of items.
/// `#[serde(untagged)]` so the wire shape `"input": "hi"` and
/// `"input": [{...}]` both deserialize.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(untagged)]
pub enum ResponsesInput {
Text(String),
Items(Vec<ResponsesInputItem>),
Items(Vec<ResponsesInputElement>),
}
/// One element of an `input` array.
///
/// OpenAI's Responses API accepts three shapes here, and real clients
/// use all of them — most notably agent-zero (via litellm), which
/// sends the bare "easy message" form. We must tolerate every shape,
/// because `input` is an `#[serde(untagged)]` array: a single element
/// that matches no variant fails the *entire* request with a 422
/// (`did not match any variant of untagged enum ResponsesInput`).
///
/// 1. [`Self::Typed`] — an item carrying an explicit `"type"`
/// discriminant (`message`, `function_call`, `function_call_output`,
/// `reasoning`).
/// 2. [`Self::EasyMessage`] — a bare `{role, content}` with **no**
/// `type` field. This is OpenAI's `EasyInputMessage` and what
/// litellm emits for every turn. `content` is optional so an
/// assistant turn carrying only tool calls (`content: null`) still
/// parses.
/// 3. [`Self::Other`] — anything else, captured as raw JSON and
/// dropped during translation. This is the forward-compat escape
/// hatch that mirrors [`ResponsesRequest::extra`] at the item
/// level: an unmodeled item type can never again reject the whole
/// request.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(untagged)]
pub enum ResponsesInputElement {
Typed(ResponsesInputItem),
EasyMessage {
role: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
content: Option<ResponsesMessageContent>,
},
Other(Value),
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -91,8 +125,11 @@ pub enum ResponsesInputItem {
name: String,
arguments: String,
},
/// User is feeding a tool result back into the model.
FunctionCallOutput { call_id: String, output: String },
/// User is feeding a tool result back into the model. `output`
/// is a `Value` because OpenAI allows it to be either a plain
/// string or an array of content parts; the translator renders
/// either form to text rather than losing the tool result.
FunctionCallOutput { call_id: String, output: Value },
/// Reasoning items emitted by o-series models. Accepted but
/// not forwarded to the model — neuron's candle path doesn't
/// surface reasoning separately yet.
@@ -132,6 +169,11 @@ pub enum ResponsesContentPart {
#[serde(default, skip_serializing_if = "Vec::is_empty")]
annotations: Vec<Value>,
},
/// Any content-part type we don't model (e.g. `refusal`, audio).
/// Captured as a unit so an unknown part can't reject the whole
/// request; dropped during translation.
#[serde(other)]
Unknown,
}
// ── Response (non-streaming) ─────────────────────────────────────────
@@ -277,20 +319,116 @@ mod tests {
ResponsesInput::Items(items) => {
assert_eq!(items.len(), 1);
match &items[0] {
ResponsesInputItem::Message { role, content } => {
ResponsesInputElement::Typed(ResponsesInputItem::Message { role, content }) => {
assert_eq!(role, "user");
match content {
ResponsesMessageContent::Text(t) => assert_eq!(t, "hi"),
other => panic!("expected Text content, got {other:?}"),
}
}
other => panic!("expected Message item, got {other:?}"),
other => panic!("expected typed Message item, got {other:?}"),
}
}
other => panic!("expected Items, got {other:?}"),
}
}
#[test]
fn deserialises_bare_easy_message_without_type() {
// The shape agent-zero (via litellm) actually sends: `input`
// items are bare `{role, content}` with NO `type` field. This
// is the exact payload that was returning 422.
let raw = r#"{
"model": "Qwen/Qwen3.6-27B",
"store": true,
"tools": [{"type": "function", "name": "x", "description": "d", "parameters": {}}],
"input": [
{"role": "system", "content": "you are helpful"},
{"role": "assistant", "content": "{\"tool_name\":\"response\"}"},
{"role": "user", "content": "hi"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let items = match req.input {
ResponsesInput::Items(i) => i,
other => panic!("expected Items, got {other:?}"),
};
assert_eq!(items.len(), 3);
for el in &items {
assert!(
matches!(el, ResponsesInputElement::EasyMessage { .. }),
"expected EasyMessage, got {el:?}"
);
}
// `tools` / `store` ride through `extra`, not `input`.
assert!(req.extra.get("tools").is_some());
assert_eq!(req.extra.get("store"), Some(&Value::Bool(true)));
}
#[test]
fn tolerates_null_content_and_unknown_item_types() {
// An assistant turn carrying only tool calls has `content: null`;
// and a future/unmodeled item type must not 422 the request.
let raw = r#"{
"model": "m",
"input": [
{"role": "assistant", "content": null},
{"type": "item_reference", "id": "abc"},
{"type": "function_call_output", "call_id": "c1",
"output": [{"type": "output_text", "text": "result"}]},
{"role": "user", "content": "go"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let items = match req.input {
ResponsesInput::Items(i) => i,
other => panic!("expected Items, got {other:?}"),
};
assert_eq!(items.len(), 4);
assert!(matches!(
&items[0],
ResponsesInputElement::EasyMessage { content: None, .. }
));
assert!(matches!(&items[1], ResponsesInputElement::Other(_)));
assert!(matches!(
&items[2],
ResponsesInputElement::Typed(ResponsesInputItem::FunctionCallOutput { .. })
));
assert!(matches!(
&items[3],
ResponsesInputElement::EasyMessage { .. }
));
}
#[test]
fn tolerates_unknown_content_part_type() {
// A `refusal` (or any unmodeled) content part must parse, not 422.
let raw = r#"{
"model": "m",
"input": [
{"role": "assistant", "content": [
{"type": "refusal", "refusal": "no"},
{"type": "output_text", "text": "ok"}
]}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let items = match req.input {
ResponsesInput::Items(i) => i,
other => panic!("expected Items, got {other:?}"),
};
let parts = match &items[0] {
ResponsesInputElement::EasyMessage {
content: Some(ResponsesMessageContent::Parts(p)),
..
} => p,
other => panic!("expected EasyMessage with Parts, got {other:?}"),
};
assert_eq!(parts.len(), 2);
assert!(matches!(&parts[0], ResponsesContentPart::Unknown));
assert!(matches!(&parts[1], ResponsesContentPart::OutputText { .. }));
}
#[test]
fn deserialises_input_with_image() {
let raw = r#"{
@@ -308,10 +446,10 @@ mod tests {
other => panic!("expected Items, got {other:?}"),
};
let parts = match &items[0] {
ResponsesInputItem::Message {
ResponsesInputElement::Typed(ResponsesInputItem::Message {
content: ResponsesMessageContent::Parts(p),
..
} => p,
}) => p,
other => panic!("expected Parts, got {other:?}"),
};
assert_eq!(parts.len(), 2);

View File

@@ -400,6 +400,7 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
total_tokens: 0,
completion_tokens_details: None,
prompt_tokens_details: None,
helexa_timing: None,
});
MessagesResponse {
@@ -772,6 +773,7 @@ mod stream_tests {
total_tokens: 267,
completion_tokens_details: None,
prompt_tokens_details: None,
helexa_timing: None,
});
t.on_chunk(&usage_chunk);
let fin = t.finish();

View File

@@ -6,6 +6,7 @@ license.workspace = true
[dependencies]
cortex-core.workspace = true
helexa-stream = { path = "../helexa-stream" }
async-trait.workspace = true
tokio.workspace = true
axum.workspace = true

View File

@@ -22,7 +22,7 @@ use axum::http::header::AUTHORIZATION;
use axum::http::{HeaderMap, HeaderValue};
use axum::middleware::Next;
use axum::response::Response;
use cortex_core::entitlements::{HEADER_ACCOUNT_ID, HEADER_KEY_ID};
use cortex_core::entitlements::{AuthError, HEADER_ACCOUNT_ID, HEADER_KEY_ID};
use cortex_core::error_envelope::OpenAiError;
use std::sync::Arc;
@@ -83,14 +83,25 @@ pub async fn require_principal(
req.extensions_mut().insert(principal);
next.run(req).await
}
// An unrecognized key only hard-fails when auth is *required*.
// In allow-anonymous mode (the default) we must IGNORE it and
// serve the request unauthenticated — otherwise the placeholder
// keys that OpenAI-compatible clients send by default (opencode,
// Open WebUI, Agent Zero, litellm) would all break, even though
// the operator never opted into auth. Pre-#49 the bearer was
// never inspected at all; this preserves that for require_auth=false.
Err(_) => {
// The entitlement authority is unreachable (upstream client
// blip, #57). Fail **closed but distinct**: a transient outage
// must not reject a real key as `401 invalid_api_key` — it's a
// retryable `503`. This holds regardless of require_auth: we
// can't safely serve a key we couldn't authorize.
Err(AuthError::Unavailable { retry_after_secs }) => {
envelope_response(OpenAiError::service_unavailable(
"entitlement authority temporarily unavailable",
Some(retry_after_secs),
))
}
// A genuinely unrecognized key only hard-fails when auth is
// *required*. In allow-anonymous mode (the default) we IGNORE it
// and serve unauthenticated — otherwise the placeholder keys that
// OpenAI-compatible clients send by default (opencode, Open WebUI,
// Agent Zero, litellm) would all break though the operator never
// opted into auth. Pre-#49 the bearer was never inspected; this
// preserves that for require_auth=false.
Err(AuthError::InvalidKey) => {
if fleet.require_auth {
unauthorized("invalid API key")
} else {

View File

@@ -0,0 +1,112 @@
//! Chained entitlement provider (#57): operator-local keys first, mesh
//! upstream for everything else.
//!
//! `resolve` tries the [`LocalEntitlementProvider`] (operator + infra keys —
//! never a network hop); only a locally-unknown key falls through to
//! [`UpstreamEntitlementProvider`]. Because the local provider treats an
//! unconfigured principal as uncapped, reserve/settle/release/snapshot must
//! **not** blindly hit local — they dispatch to whichever backend resolved
//! that account, remembered in a map keyed by `account_id` (populated at
//! resolve time).
use crate::entitlements_local::LocalEntitlementProvider;
use crate::entitlements_upstream::UpstreamEntitlementProvider;
use async_trait::async_trait;
use cortex_core::entitlements::{
AuthError, BudgetError, BudgetSnapshot, EntitlementProvider, Principal, Reservation,
};
use std::collections::HashMap;
use tokio::sync::RwLock;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Backend {
Local,
Upstream,
}
pub struct ChainedEntitlementProvider {
local: LocalEntitlementProvider,
upstream: UpstreamEntitlementProvider,
/// account_id → which backend owns it, learned at resolve time.
backends: RwLock<HashMap<String, Backend>>,
}
impl ChainedEntitlementProvider {
pub fn new(local: LocalEntitlementProvider, upstream: UpstreamEntitlementProvider) -> Self {
Self {
local,
upstream,
backends: RwLock::new(HashMap::new()),
}
}
async fn record(&self, account_id: &str, backend: Backend) {
self.backends
.write()
.await
.insert(account_id.to_string(), backend);
}
/// The backend that owns `account_id`. Defaults to `Upstream` for an
/// account never resolved this process-lifetime (a resolve always
/// precedes reserve in a request, so this is just a safe fallback —
/// upstream fails closed if the account is bogus).
async fn backend_for(&self, account_id: &str) -> Backend {
self.backends
.read()
.await
.get(account_id)
.copied()
.unwrap_or(Backend::Upstream)
}
}
#[async_trait]
impl EntitlementProvider for ChainedEntitlementProvider {
async fn resolve(&self, api_key: &str) -> Result<Principal, AuthError> {
match self.local.resolve(api_key).await {
Ok(p) => {
self.record(&p.account_id, Backend::Local).await;
Ok(p)
}
Err(AuthError::InvalidKey) => {
let p = self.upstream.resolve(api_key).await?;
self.record(&p.account_id, Backend::Upstream).await;
Ok(p)
}
Err(e) => Err(e),
}
}
async fn reserve(
&self,
principal: &Principal,
max_tokens: u64,
) -> Result<Reservation, BudgetError> {
match self.backend_for(&principal.account_id).await {
Backend::Local => self.local.reserve(principal, max_tokens).await,
Backend::Upstream => self.upstream.reserve(principal, max_tokens).await,
}
}
async fn settle(&self, reservation: Reservation, actual_tokens: u64) {
match self.backend_for(&reservation.principal.account_id).await {
Backend::Local => self.local.settle(reservation, actual_tokens).await,
Backend::Upstream => self.upstream.settle(reservation, actual_tokens).await,
}
}
async fn release(&self, reservation: Reservation) {
match self.backend_for(&reservation.principal.account_id).await {
Backend::Local => self.local.release(reservation).await,
Backend::Upstream => self.upstream.release(reservation).await,
}
}
async fn snapshot(&self, principal: &Principal) -> Option<BudgetSnapshot> {
match self.backend_for(&principal.account_id).await {
Backend::Local => self.local.snapshot(principal).await,
Backend::Upstream => self.upstream.snapshot(principal).await,
}
}
}

View File

@@ -0,0 +1,246 @@
//! helexa-upstream client (#57): an [`EntitlementProvider`] that resolves
//! keys and reserves/settles budget against the mesh authority's
//! `/authz/v1` surface (B2). It is "just another impl of the trait" — cortex
//! enforcement (`auth.rs`, `metering.rs`) is unchanged.
//!
//! **Fail closed.** When upstream is unreachable, `resolve` returns
//! [`AuthError::Unavailable`] (→ `503`, never `401`) and `reserve` refuses
//! with a retryable [`BudgetError::RateLimited`] — a request is never served
//! on an un-authorized key, and a real key is never rejected as invalid
//! during a blip.
use async_trait::async_trait;
use cortex_core::config::UpstreamClientConfig;
use cortex_core::entitlements::{
AuthError, BudgetError, BudgetSnapshot, EntitlementProvider, Principal, Reservation,
};
use serde::Deserialize;
use std::time::Duration;
/// Retry-After (seconds) advertised when we fail closed on an upstream
/// outage.
const FAIL_CLOSED_RETRY_SECS: u64 = 5;
pub struct UpstreamEntitlementProvider {
client: reqwest::Client,
base_url: String,
bearer: String,
}
#[derive(Deserialize)]
struct PrincipalDto {
account_id: String,
key_id: String,
}
#[derive(Deserialize)]
struct SnapshotDto {
hard_cap: Option<u64>,
spent: u64,
reserved: u64,
}
#[derive(Deserialize)]
struct ResolveResp {
principal: PrincipalDto,
#[allow(dead_code)]
snapshot: Option<SnapshotDto>,
}
#[derive(Deserialize)]
struct ReserveResp {
reservation_id: Option<i64>,
rejected: Option<Rejection>,
}
#[derive(Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
enum Rejection {
InsufficientQuota {
requested: u64,
available: u64,
},
RateLimited {
requested: u64,
available: u64,
retry_after_secs: u64,
},
}
impl UpstreamEntitlementProvider {
pub fn new(cfg: &UpstreamClientConfig) -> Self {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(cfg.timeout_secs))
.build()
.expect("failed to build upstream HTTP client");
Self {
client,
base_url: cfg.url.trim_end_matches('/').to_string(),
bearer: cfg.bearer.clone(),
}
}
fn url(&self, path: &str) -> String {
format!("{}{}", self.base_url, path)
}
}
#[async_trait]
impl EntitlementProvider for UpstreamEntitlementProvider {
async fn resolve(&self, api_key: &str) -> Result<Principal, AuthError> {
let resp = self
.client
.post(self.url("/authz/v1/resolve"))
.bearer_auth(&self.bearer)
.json(&serde_json::json!({ "api_key": api_key }))
.send()
.await;
let resp = match resp {
Ok(r) => r,
Err(e) => {
tracing::warn!(error = %e, "upstream resolve unreachable; failing closed");
return Err(AuthError::Unavailable {
retry_after_secs: FAIL_CLOSED_RETRY_SECS,
});
}
};
if resp.status().as_u16() == 401 {
return Err(AuthError::InvalidKey);
}
if !resp.status().is_success() {
return Err(AuthError::Unavailable {
retry_after_secs: FAIL_CLOSED_RETRY_SECS,
});
}
match resp.json::<ResolveResp>().await {
Ok(r) => Ok(Principal {
account_id: r.principal.account_id,
key_id: r.principal.key_id,
}),
Err(e) => {
tracing::warn!(error = %e, "upstream resolve: bad body; failing closed");
Err(AuthError::Unavailable {
retry_after_secs: FAIL_CLOSED_RETRY_SECS,
})
}
}
}
async fn reserve(
&self,
principal: &Principal,
max_tokens: u64,
) -> Result<Reservation, BudgetError> {
let fail_closed = || BudgetError::RateLimited {
requested: max_tokens,
available: 0,
retry_after_secs: FAIL_CLOSED_RETRY_SECS,
};
let resp = self
.client
.post(self.url("/authz/v1/reserve"))
.bearer_auth(&self.bearer)
.json(&serde_json::json!({
"account_id": principal.account_id,
"key_id": principal.key_id,
"max_tokens": max_tokens,
}))
.send()
.await;
let resp = match resp {
Ok(r) if r.status().is_success() => r,
Ok(r) => {
tracing::warn!(status = %r.status(), "upstream reserve non-2xx; failing closed");
return Err(fail_closed());
}
Err(e) => {
tracing::warn!(error = %e, "upstream reserve unreachable; failing closed");
return Err(fail_closed());
}
};
match resp.json::<ReserveResp>().await {
Ok(ReserveResp {
reservation_id: Some(id),
..
}) => Ok(Reservation {
id: id as u64,
principal: principal.clone(),
reserved: max_tokens,
}),
Ok(ReserveResp {
rejected:
Some(Rejection::InsufficientQuota {
requested,
available,
}),
..
}) => Err(BudgetError::InsufficientQuota {
requested,
available,
}),
Ok(ReserveResp {
rejected:
Some(Rejection::RateLimited {
requested,
available,
retry_after_secs,
}),
..
}) => Err(BudgetError::RateLimited {
requested,
available,
retry_after_secs,
}),
_ => Err(fail_closed()),
}
}
async fn settle(&self, reservation: Reservation, actual_tokens: u64) {
// Best-effort; a lost settle is reaped by the upstream sweeper (B2).
let _ = self
.client
.post(self.url("/authz/v1/settle"))
.bearer_auth(&self.bearer)
.json(&serde_json::json!({
"reservation_id": reservation.id as i64,
"actual_tokens": actual_tokens,
}))
.send()
.await
.inspect_err(
|e| tracing::warn!(error = %e, "upstream settle failed (sweeper will reap)"),
);
}
async fn release(&self, reservation: Reservation) {
let _ = self
.client
.post(self.url("/authz/v1/release"))
.bearer_auth(&self.bearer)
.json(&serde_json::json!({ "reservation_id": reservation.id as i64 }))
.send()
.await
.inspect_err(
|e| tracing::warn!(error = %e, "upstream release failed (sweeper will reap)"),
);
}
async fn snapshot(&self, principal: &Principal) -> Option<BudgetSnapshot> {
let resp = self
.client
.post(self.url("/authz/v1/snapshot"))
.bearer_auth(&self.bearer)
.json(&serde_json::json!({
"account_id": principal.account_id,
"key_id": principal.key_id,
}))
.send()
.await
.ok()?;
if !resp.status().is_success() {
return None;
}
let dto = resp.json::<SnapshotDto>().await.ok()?;
Some(BudgetSnapshot {
hard_cap: dto.hard_cap,
spent: dto.spent,
reserved: dto.reserved,
})
}
}

View File

@@ -322,7 +322,11 @@ async fn anthropic_messages(
)
.await
{
Ok(guard) => Some(crate::metering::usage_sink(principal, guard)),
Ok(guard) => Some(crate::metering::usage_sink(
principal,
guard,
std::sync::Arc::clone(&fleet.served_usage),
)),
Err(env) => return crate::error::envelope_response(env),
}
}
@@ -802,7 +806,11 @@ async fn proxy_with_metrics(
)
.await
{
Ok(guard) => Some(crate::metering::usage_sink(principal, guard)),
Ok(guard) => Some(crate::metering::usage_sink(
principal,
guard,
std::sync::Arc::clone(&fleet.served_usage),
)),
Err(env) => return crate::error::envelope_response(env),
}
}

View File

@@ -1,6 +1,8 @@
pub mod anthropic_sse;
pub mod auth;
pub mod entitlements_chain;
pub mod entitlements_local;
pub mod entitlements_upstream;
pub mod error;
pub mod evictor;
pub mod handlers;
@@ -9,6 +11,7 @@ pub mod metrics;
pub mod poller;
pub mod proxy;
pub mod router;
pub mod served_usage;
pub mod state;
use anyhow::Result;
@@ -55,6 +58,28 @@ pub async fn run(config: GatewayConfig) -> Result<()> {
evictor::eviction_loop(evictor_fleet).await;
});
// Served-usage reporter (#58): when this operator is part of the mesh,
// periodically flush absolute per-principal served-token counters to
// upstream for reconciliation.
if config.upstream.enabled {
let su_fleet = Arc::clone(&fleet);
let url = config.upstream.url.clone();
let bearer = config.upstream.bearer.clone();
let interval =
std::time::Duration::from_secs(config.upstream.served_usage_report_interval_secs);
tokio::spawn(async move {
loop {
tokio::time::sleep(interval).await;
let rows = su_fleet.served_usage.snapshot();
if let Err(e) =
served_usage::report(&su_fleet.http_client, &url, &bearer, &rows).await
{
tracing::warn!(error = %e, "served-usage report failed (will retry)");
}
}
});
}
let app = build_app(Arc::clone(&fleet));
let listen_addr = config.gateway.listen.parse::<std::net::SocketAddr>()?;

View File

@@ -117,9 +117,21 @@ impl Drop for ReservationGuard {
/// Build the completion sink for an authenticated request: record spend and
/// settle the reservation with the observed total. Dropping it unused (no
/// usage observed) releases the reservation via the guard.
pub fn usage_sink(principal: Principal, guard: ReservationGuard) -> UsageSink {
pub fn usage_sink(
principal: Principal,
guard: ReservationGuard,
served_usage: std::sync::Arc<crate::served_usage::ServedUsage>,
) -> UsageSink {
Box::new(move |prompt, completion| {
record_spend(&principal, prompt, completion);
// Per-principal served-usage tally for #58 reconciliation. Recorded
// for every metered (authenticated) request; the flush task reports
// it to upstream when the operator is part of the mesh.
served_usage.add(
&principal.account_id,
&principal.key_id,
prompt + completion,
);
guard.settle(prompt + completion);
})
}

View File

@@ -5,12 +5,29 @@ use crate::state::CortexState;
use chrono::Utc;
use cortex_core::discovery::{DiscoveryResponse, HealthResponse};
use cortex_core::harness::ModelInfo;
use cortex_core::node::{ModelEntry, ModelStatus};
use cortex_core::node::{ModelEntry, ModelStatus, NodeState};
use std::sync::Arc;
use std::time::Duration;
const POLL_INTERVAL: Duration = Duration::from_secs(10);
/// Consecutive failed `/models` polls before a node is marked unhealthy.
/// Debounces transient misses (a busy neuron briefly slow to answer) so a
/// single blip can't yank a node — and its models — out of routing. At the
/// 10s poll interval this tolerates ~20s of flapping before evicting.
const POLL_FAILURE_THRESHOLD: u32 = 3;
/// Record a failed poll for `node`, marking it unhealthy only once failures
/// reach [`POLL_FAILURE_THRESHOLD`]. Below the threshold the node keeps its
/// last-known health, riding over transient misses. A successful poll resets
/// the counter (see the success arm in `poll_once`).
fn record_poll_failure(node: &mut NodeState) {
node.consecutive_poll_failures = node.consecutive_poll_failures.saturating_add(1);
if node.consecutive_poll_failures >= POLL_FAILURE_THRESHOLD {
node.healthy = false;
}
}
/// Runs forever, polling all neurons on a fixed interval.
pub async fn poll_loop(fleet: Arc<CortexState>) {
loop {
@@ -138,13 +155,14 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
// Remove models no longer reported by the neuron.
node.models.retain(|id, _| seen.contains(id));
node.consecutive_poll_failures = 0;
node.healthy = true;
node.last_poll = Some(Utc::now());
tracing::debug!(node = name, models = models.len(), "poll ok");
}
Err(e) => {
tracing::warn!(node = name, error = %e, "failed to parse /models response");
node.healthy = false;
record_poll_failure(node);
}
}
}
@@ -154,11 +172,11 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
status = %resp.status(),
"neuron returned non-success status"
);
node.healthy = false;
record_poll_failure(node);
}
Err(e) => {
tracing::warn!(node = name, error = %e, "failed to reach neuron");
node.healthy = false;
record_poll_failure(node);
}
}

View File

@@ -1,21 +1,27 @@
//! Streaming HTTP reverse proxy to neuron backends.
//!
//! For streaming requests, SSE chunks are forwarded as they arrive.
//! The proxy captures timing information for metrics but does not
//! buffer the full response.
//! The streaming *mechanism* — forward an SSE body chunk-for-chunk without
//! buffering, observing the bytes for metrics — lives in the shared
//! [`helexa_stream`] crate (#71), so cortex and helexa-router use one
//! implementation. This module supplies cortex's *policy*: the
//! [`CortexMetrics`] observer (per-request token metrics + per-principal
//! reservation settle), cortex's logging contract, and the cortex error
//! envelope. The usage-extraction helper is re-exported from the shared
//! crate so existing call sites keep working.
use crate::router::RouteDecision;
use anyhow::Result;
use axum::body::Body;
use axum::http::{HeaderMap, StatusCode};
use axum::http::HeaderMap;
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use futures::Stream;
use futures::stream::BoxStream;
use helexa_stream::{BodyTail, ChunkObserver, StreamError};
use reqwest::Client;
use std::pin::Pin;
use std::task::{Context, Poll};
use std::time::Instant;
/// Re-export the shared usage-extraction helper. Several cortex modules
/// (`handlers`, `anthropic_sse`) pull token counts out of a buffered body
/// tail via this function; it lives in `helexa-stream` now.
pub use helexa_stream::last_count_for;
/// Proxy a request body to the resolved backend node and stream the response.
///
/// Logging contract: every call emits exactly one structured event at
@@ -42,66 +48,41 @@ pub async fn forward_request(
"proxying request"
);
let mut req_builder = client.post(&url).body(body);
let observer = CortexMetrics::new(model_id, &route.node_name, request_start, usage_sink);
// Forward relevant headers.
for (key, value) in headers.iter() {
if key == "host" || key == "content-length" {
continue; // reqwest sets these
}
req_builder = req_builder.header(key, value);
}
let response = helexa_stream::forward_streaming(client, &url, headers, body, observer)
.await
.map_err(|e| {
match &e {
StreamError::Upstream(err) => tracing::warn!(
node = %route.node_name,
url = %url,
error = %err,
"proxy: upstream request failed (network)"
),
StreamError::ResponseBuild(err) => tracing::warn!(
node = %route.node_name,
url = %url,
error = %err,
"proxy: failed to build response"
),
}
ProxyError::from(e)
})?;
let upstream_resp = match req_builder.send().await {
Ok(r) => r,
Err(e) => {
tracing::warn!(
node = %route.node_name,
url = %url,
error = %e,
"proxy: upstream request failed (network)"
);
return Err(ProxyError::Upstream(e));
}
};
let upstream_status = upstream_resp.status();
if !upstream_status.is_success() {
if !response.status().is_success() {
// Streaming body — can't snippet without breaking the stream
// pass-through. Log status + URL; the client still gets the
// upstream status, just without the leaked body.
tracing::warn!(
node = %route.node_name,
url = %url,
status = upstream_status.as_u16(),
status = response.status().as_u16(),
"proxy: upstream returned non-2xx"
);
}
let status = StatusCode::from_u16(upstream_status.as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
let resp_headers = upstream_resp.headers().clone();
let stream = TokenMetricsStream::new(
Box::pin(upstream_resp.bytes_stream()),
TokenMetrics::new(model_id, &route.node_name, request_start, usage_sink),
);
let body = Body::from_stream(stream);
let mut response = Response::builder().status(status);
for (key, value) in resp_headers.iter() {
response = response.header(key, value);
}
response.body(body).map_err(|e| {
tracing::warn!(
node = %route.node_name,
url = %url,
error = %e,
"proxy: failed to build response"
);
ProxyError::ResponseBuild(e.to_string())
})
Ok(response)
}
#[derive(Debug, thiserror::Error)]
@@ -112,6 +93,15 @@ pub enum ProxyError {
ResponseBuild(String),
}
impl From<StreamError> for ProxyError {
fn from(e: StreamError) -> Self {
match e {
StreamError::Upstream(err) => ProxyError::Upstream(err),
StreamError::ResponseBuild(msg) => ProxyError::ResponseBuild(msg),
}
}
}
impl IntoResponse for ProxyError {
fn into_response(self) -> Response {
let (status, code, message) = match &self {
@@ -139,9 +129,10 @@ impl IntoResponse for ProxyError {
//
// The proxy never buffers or re-serialises the upstream body — chunks
// are forwarded verbatim. For metrics it observes each chunk's arrival
// time and keeps a bounded tail of the body text, from which the final
// OpenAI `usage` object (present on the last SSE chunk and on
// non-streaming JSON bodies alike) yields engine-truth token counts.
// time and keeps a bounded tail of the body text (via the shared
// `helexa_stream::BodyTail`), from which the final OpenAI `usage` object
// (present on the last SSE chunk and on non-streaming JSON bodies alike)
// yields engine-truth token counts.
//
// Emitted per request, labelled {model, node}:
// cortex_time_to_first_token_seconds (histogram) — first body chunk
@@ -155,37 +146,15 @@ impl IntoResponse for ProxyError {
/// non-streaming bodies.
const TAIL_CAP_BYTES: usize = 64 * 1024;
/// Find the value of the LAST `"key": <integer>` occurrence in `tail`.
/// Pure and chunk-boundary-safe (the tail is contiguous appended text).
/// The quoted-needle form means `completion_tokens` never matches
/// `completion_tokens_details`.
pub(crate) fn last_count_for(tail: &str, key: &str) -> Option<u64> {
let needle = format!("\"{key}\"");
let mut result = None;
for (idx, _) in tail.match_indices(&needle) {
let rest = tail[idx + needle.len()..].trim_start();
let Some(rest) = rest.strip_prefix(':') else {
continue;
};
let rest = rest.trim_start();
let digits: &str = &rest[..rest
.char_indices()
.find(|(_, c)| !c.is_ascii_digit())
.map(|(i, _)| i)
.unwrap_or(rest.len())];
if let Ok(v) = digits.parse::<u64>() {
result = Some(v);
}
}
result
}
struct TokenMetrics {
/// cortex's [`ChunkObserver`]: per-request token metrics plus the
/// per-principal reservation settle. Drives cortex policy over the shared
/// streaming mechanism.
struct CortexMetrics {
labels: [(&'static str, String); 2],
request_start: Instant,
first_chunk: Option<Instant>,
last_chunk: Option<Instant>,
tail: String,
tail: BodyTail,
finished: bool,
/// Per-principal metering hook (#51). Invoked exactly once in `finish`
/// with the observed `(prompt, completion)` so the reservation can be
@@ -193,7 +162,7 @@ struct TokenMetrics {
usage_sink: Option<crate::metering::UsageSink>,
}
impl TokenMetrics {
impl CortexMetrics {
fn new(
model_id: &str,
node_name: &str,
@@ -208,26 +177,19 @@ impl TokenMetrics {
request_start,
first_chunk: None,
last_chunk: None,
tail: String::new(),
tail: BodyTail::new(TAIL_CAP_BYTES),
finished: false,
usage_sink,
}
}
}
impl ChunkObserver for CortexMetrics {
fn observe(&mut self, chunk: &[u8]) {
let now = Instant::now();
self.first_chunk.get_or_insert(now);
self.last_chunk = Some(now);
self.tail.push_str(&String::from_utf8_lossy(chunk));
if self.tail.len() > TAIL_CAP_BYTES {
// Keep the newest half; the usage object is always at the
// very end of the body. Split at a char boundary.
let mut cut = self.tail.len() - TAIL_CAP_BYTES / 2;
while !self.tail.is_char_boundary(cut) {
cut += 1;
}
self.tail.drain(..cut);
}
self.tail.push(chunk);
}
/// Emit the metrics exactly once — called on clean stream end and
@@ -239,8 +201,8 @@ impl TokenMetrics {
}
self.finished = true;
let prompt = last_count_for(&self.tail, "prompt_tokens");
let completion = last_count_for(&self.tail, "completion_tokens");
let prompt = last_count_for(self.tail.as_str(), "prompt_tokens");
let completion = last_count_for(self.tail.as_str(), "completion_tokens");
// Per-model metrics — only when body chunks actually arrived.
if let Some(first) = self.first_chunk {
@@ -280,97 +242,3 @@ impl TokenMetrics {
}
}
}
/// Pass-through stream wrapper that feeds [`TokenMetrics`]. Emits on
/// clean end-of-stream; the Drop impl covers client disconnects.
struct TokenMetricsStream {
inner: BoxStream<'static, Result<bytes::Bytes, reqwest::Error>>,
metrics: TokenMetrics,
}
impl TokenMetricsStream {
fn new(
inner: BoxStream<'static, Result<bytes::Bytes, reqwest::Error>>,
metrics: TokenMetrics,
) -> Self {
Self { inner, metrics }
}
}
impl Stream for TokenMetricsStream {
type Item = Result<bytes::Bytes, reqwest::Error>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = self.get_mut();
match this.inner.as_mut().poll_next(cx) {
Poll::Ready(Some(Ok(chunk))) => {
this.metrics.observe(&chunk);
Poll::Ready(Some(Ok(chunk)))
}
Poll::Ready(Some(Err(e))) => Poll::Ready(Some(Err(e))),
Poll::Ready(None) => {
this.metrics.finish();
Poll::Ready(None)
}
Poll::Pending => Poll::Pending,
}
}
}
impl Drop for TokenMetricsStream {
fn drop(&mut self) {
self.metrics.finish();
}
}
#[cfg(test)]
mod tests {
use super::last_count_for;
#[test]
fn extracts_counts_from_final_sse_usage_chunk() {
let tail = concat!(
"data: {\"choices\":[{\"delta\":{\"content\":\"hi\"}}]}\n\n",
"data: {\"choices\":[],\"usage\":{\"prompt_tokens\":225,",
"\"completion_tokens\":42,\"total_tokens\":267}}\n\n",
"data: [DONE]\n\n"
);
assert_eq!(last_count_for(tail, "prompt_tokens"), Some(225));
assert_eq!(last_count_for(tail, "completion_tokens"), Some(42));
}
#[test]
fn extracts_counts_from_non_streaming_body() {
let tail = "{\"choices\":[{\"message\":{\"content\":\"hi\"}}],\
\"usage\":{\"prompt_tokens\": 12, \"completion_tokens\": 7}}";
assert_eq!(last_count_for(tail, "prompt_tokens"), Some(12));
assert_eq!(last_count_for(tail, "completion_tokens"), Some(7));
}
#[test]
fn ignores_details_variants_and_takes_last_occurrence() {
// completion_tokens_details must not shadow completion_tokens,
// and the LAST usage object wins (matters when content echoes
// a usage-shaped string earlier in the stream).
let tail = concat!(
"data: {\"usage\":{\"completion_tokens\":1}}\n\n",
"data: {\"usage\":{\"completion_tokens\":99,",
"\"completion_tokens_details\":{\"reasoning_tokens\":3}}}\n\n"
);
assert_eq!(last_count_for(tail, "completion_tokens"), Some(99));
}
#[test]
fn absent_keys_yield_none() {
assert_eq!(
last_count_for("data: [DONE]\n\n", "completion_tokens"),
None
);
assert_eq!(last_count_for("", "prompt_tokens"), None);
// key present but non-numeric value
assert_eq!(
last_count_for("\"completion_tokens\": null", "completion_tokens"),
None
);
}
}

View File

@@ -50,6 +50,10 @@ pub enum RouteError {
"model '{model_id}' is in the catalogue but no healthy neuron's topology satisfies its constraints"
)]
NoFeasibleNeuron { model_id: String },
#[error(
"model '{model_id}' is feasible on a neuron that is currently unhealthy — retry shortly"
)]
FeasibleNodeUnhealthy { model_id: String },
#[error("cold-load of '{model_id}' on '{node}' failed: {message}")]
ColdLoadFailed {
model_id: String,
@@ -68,7 +72,9 @@ impl RouteError {
/// safe to retry the same request); everything else is 404.
pub fn http_status(&self) -> u16 {
match self {
RouteError::NoHealthyNodes | RouteError::ModelRecovering { .. } => 503,
RouteError::NoHealthyNodes
| RouteError::ModelRecovering { .. }
| RouteError::FeasibleNodeUnhealthy { .. } => 503,
_ => 404,
}
}
@@ -81,7 +87,8 @@ impl RouteError {
| RouteError::EndpointResolveFailed(_, _)
| RouteError::NoFeasibleNeuron { .. }
| RouteError::ColdLoadFailed { .. }
| RouteError::ModelRecovering { .. } => "api_error",
| RouteError::ModelRecovering { .. }
| RouteError::FeasibleNodeUnhealthy { .. } => "api_error",
}
}
@@ -94,6 +101,7 @@ impl RouteError {
RouteError::NoFeasibleNeuron { .. } => "service_unavailable",
RouteError::ColdLoadFailed { .. } => "service_unavailable",
RouteError::ModelRecovering { .. } => "service_unavailable",
RouteError::FeasibleNodeUnhealthy { .. } => "service_unavailable",
}
}
@@ -105,6 +113,7 @@ impl RouteError {
pub fn retry_after_secs(&self) -> Option<u64> {
match self {
RouteError::ModelRecovering { .. } => Some(2),
RouteError::FeasibleNodeUnhealthy { .. } => Some(3),
RouteError::NoHealthyNodes => Some(5),
_ => None,
}
@@ -252,11 +261,32 @@ async fn pick_feasible_neuron(
b.2.cmp(&a.2) // pinned first (true > false)
.then(a.0.cmp(&b.0))
});
let pick = candidates.into_iter().next();
pick.map(|(n, e, _)| (n, e))
.ok_or_else(|| RouteError::NoFeasibleNeuron {
if let Some((n, e, _)) = candidates.into_iter().next() {
return Ok((n, e));
}
// No *healthy* feasible neuron. Distinguish a transient outage from a
// permanent misconfiguration: if some neuron is topologically feasible
// but currently unhealthy (e.g. it briefly missed polls while busy),
// this is retryable — return 503 + Retry-After so the client backs off
// and retries instead of treating a 404 as a hard failure. Only when no
// neuron could *ever* satisfy the topology is it a permanent 404.
let feasible_but_unhealthy = nodes.values().any(|node| {
!node.healthy
&& node
.discovery
.as_ref()
.is_some_and(|disc| profile.is_feasible_on(&node.name, &disc.devices))
});
if feasible_but_unhealthy {
Err(RouteError::FeasibleNodeUnhealthy {
model_id: profile.id.clone(),
})
} else {
Err(RouteError::NoFeasibleNeuron {
model_id: profile.id.clone(),
})
}
}
/// Issue `POST {endpoint}/models/load` for this profile on this neuron,

View File

@@ -0,0 +1,105 @@
//! Served-usage ledger (#58): cortex meters, per principal and per UTC day,
//! the tokens it has served on behalf of mesh accounts, and periodically
//! reports **absolute** cumulative counters to helexa-upstream for
//! reconciliation (operators are compensated for served tokens).
//!
//! Counters are cumulative-since-process-start for the current period;
//! upstream upserts them monotonically (GREATEST), so re-sending the same
//! value is idempotent and a flush that races another is harmless. (A
//! process restart resets the in-memory counter; the monotonic upsert keeps
//! upstream from regressing — at most it under-counts the restarted window,
//! acceptable for beta. One cortex per operator token is assumed.)
use serde::Serialize;
use std::collections::HashMap;
use std::sync::Mutex;
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct ServedRow {
pub account_id: String,
pub key_id: String,
pub period: String, // YYYY-MM-DD (UTC)
pub served_tokens: u64,
}
#[derive(Default)]
pub struct ServedUsage {
inner: Mutex<HashMap<(String, String, String), u64>>,
}
impl ServedUsage {
pub fn new() -> Self {
Self::default()
}
/// Add served tokens for a principal in today's (UTC) period.
pub fn add(&self, account_id: &str, key_id: &str, tokens: u64) {
if tokens == 0 {
return;
}
let period = chrono::Utc::now().format("%Y-%m-%d").to_string();
let mut m = self.inner.lock().expect("served-usage lock");
*m.entry((account_id.to_string(), key_id.to_string(), period))
.or_insert(0) += tokens;
}
/// Absolute cumulative counters, for a flush to upstream.
pub fn snapshot(&self) -> Vec<ServedRow> {
let m = self.inner.lock().expect("served-usage lock");
m.iter()
.map(|((account_id, key_id, period), &served_tokens)| ServedRow {
account_id: account_id.clone(),
key_id: key_id.clone(),
period: period.clone(),
served_tokens,
})
.collect()
}
}
/// POST the absolute counters to upstream's `/authz/v1/served-usage`.
pub async fn report(
client: &reqwest::Client,
base_url: &str,
bearer: &str,
rows: &[ServedRow],
) -> Result<(), reqwest::Error> {
if rows.is_empty() {
return Ok(());
}
let url = format!("{}/authz/v1/served-usage", base_url.trim_end_matches('/'));
client
.post(url)
.bearer_auth(bearer)
.json(&serde_json::json!({ "rows": rows }))
.send()
.await?
.error_for_status()?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn accumulates_per_principal_and_period() {
let su = ServedUsage::new();
su.add("acct", "key", 10);
su.add("acct", "key", 5);
su.add("acct", "other", 7);
su.add("acct", "key", 0); // no-op
let mut rows = su.snapshot();
rows.sort_by(|a, b| a.key_id.cmp(&b.key_id));
assert_eq!(rows.len(), 2);
let key_row = rows.iter().find(|r| r.key_id == "key").unwrap();
assert_eq!(key_row.served_tokens, 15);
assert_eq!(
rows.iter()
.find(|r| r.key_id == "other")
.unwrap()
.served_tokens,
7
);
}
}

View File

@@ -1,4 +1,6 @@
use crate::entitlements_chain::ChainedEntitlementProvider;
use crate::entitlements_local::LocalEntitlementProvider;
use crate::entitlements_upstream::UpstreamEntitlementProvider;
use cortex_core::catalogue::ModelCatalogue;
use cortex_core::config::{EvictionSettings, GatewayConfig, NeuronEndpoint};
use cortex_core::entitlements::EntitlementProvider;
@@ -20,6 +22,9 @@ pub struct CortexState {
/// Whether to reject unauthenticated requests (#49). Read by the auth
/// middleware once it lands.
pub require_auth: bool,
/// Per-principal served-token tally (#58), reported to upstream for
/// operator reconciliation by the flush task when upstream is enabled.
pub served_usage: Arc<crate::served_usage::ServedUsage>,
}
impl CortexState {
@@ -38,14 +43,27 @@ impl CortexState {
discovery: None,
activation: None,
model_load: HashMap::new(),
consecutive_poll_failures: 0,
},
);
}
let catalogue = ModelCatalogue::load(&config.models_config);
let entitlements: Arc<dyn EntitlementProvider> =
Arc::new(LocalEntitlementProvider::from_config(&config.entitlements));
// Local provider always handles operator + infra keys. When the
// upstream client is enabled (#57), wrap it in the chain so locally
// unknown keys fall through to the mesh authority; otherwise stay
// purely local.
let local = LocalEntitlementProvider::from_config(&config.entitlements);
let entitlements: Arc<dyn EntitlementProvider> = if config.upstream.enabled {
tracing::info!(url = %config.upstream.url, "upstream entitlement client enabled");
Arc::new(ChainedEntitlementProvider::new(
local,
UpstreamEntitlementProvider::new(&config.upstream),
))
} else {
Arc::new(local)
};
Self {
nodes: RwLock::new(nodes),
@@ -58,6 +76,7 @@ impl CortexState {
.expect("failed to build HTTP client"),
entitlements,
require_auth: config.entitlements.require_auth,
served_usage: Arc::new(crate::served_usage::ServedUsage::new()),
}
}
}

View File

@@ -57,6 +57,7 @@ async fn test_alias_resolves_in_chat_completions() {
}],
models_config: models_path.to_string_lossy().to_string(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -143,6 +144,7 @@ async fn test_aliases_surface_in_v1_models() {
}],
models_config: models_path.to_string_lossy().to_string(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -232,6 +234,7 @@ async fn test_alias_falls_through_for_unmapped_model() {
}],
models_config: models_path.to_string_lossy().to_string(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));

View File

@@ -105,6 +105,7 @@ async fn spawn_gateway(neuron_url: &str, entitlements: EntitlementsConfig) -> St
}],
models_config: "/dev/null".into(),
entitlements,
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));

View File

@@ -81,6 +81,7 @@ async fn spawn_gateway(neuron_url: &str, key: ApiKeyConfig) -> (Arc<CortexState>
require_auth: true,
keys: vec![key],
},
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
{

View File

@@ -430,6 +430,7 @@ pub async fn spawn_gateway_with_state(mock_url: &str) -> (Arc<CortexState>, Stri
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));

View File

@@ -89,6 +89,7 @@ async fn error_response_no_healthy_nodes() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(cortex_gateway::state::CortexState::from_config(&config));

View File

@@ -72,6 +72,7 @@ fn make_fleet(endpoint: &str, defrag_after: u32) -> Arc<CortexState> {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
Arc::new(CortexState::from_config(&config))
}

View File

@@ -0,0 +1,125 @@
//! Router: a catalogued model whose only topologically-feasible neuron is
//! currently unhealthy is a *transient* condition (retryable 503), not a
//! permanent 404. This is the exact shape of the beast incident: benjy/
//! quadbrat (1 GPU, healthy) can't host the 27B, and beast (2 GPU) — the
//! sole feasible node — briefly drops out → clients must back off and retry,
//! not hard-fail.
use cortex_core::config::{
EvictionSettings, EvictionStrategy, GatewayConfig, GatewaySettings, NeuronEndpoint,
};
use cortex_core::discovery::{DeviceInfo, DiscoveryResponse};
use cortex_gateway::router::{self, RouteError};
use cortex_gateway::state::CortexState;
use std::sync::Arc;
fn devices(n: usize) -> Vec<DeviceInfo> {
(0..n)
.map(|i| DeviceInfo {
index: i as u32,
name: "RTX 5090".into(),
vram_total_mb: 32_768,
compute_capability: "9.0".into(),
})
.collect()
}
fn discovery(host: &str, n_devices: usize) -> DiscoveryResponse {
DiscoveryResponse {
hostname: host.into(),
os: "Linux".into(),
kernel: "7.0".into(),
cuda_version: Some("13.0".into()),
driver_version: Some("999".into()),
devices: devices(n_devices),
harnesses: vec!["candle".into()],
cuda_unavailable_reason: None,
max_prompt_tokens: 49_152,
}
}
/// Catalogue with one model needing 2 devices. Returns a temp path.
fn write_catalogue() -> std::path::PathBuf {
let toml = r#"
[[models]]
id = "big-model"
harness = "candle"
min_devices = 2
"#;
let path = std::env::temp_dir().join("cortex_test_feasibility_models.toml");
std::fs::write(&path, toml).unwrap();
path
}
async fn fleet_with(big_healthy: bool, big_devices: usize) -> Arc<CortexState> {
let cat = write_catalogue();
let config = GatewayConfig {
gateway: GatewaySettings {
listen: "127.0.0.1:0".into(),
metrics_listen: "127.0.0.1:0".into(),
},
eviction: EvictionSettings {
strategy: EvictionStrategy::Lru,
defrag_after_cycles: 0,
},
neurons: vec![
NeuronEndpoint {
name: "small".into(),
endpoint: "http://127.0.0.1:1".into(),
},
NeuronEndpoint {
name: "big".into(),
endpoint: "http://127.0.0.1:2".into(),
},
],
models_config: cat.to_string_lossy().into_owned(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
{
let mut nodes = fleet.nodes.write().await;
// "small" is healthy but only has 1 GPU → not feasible for the model.
let small = nodes.get_mut("small").unwrap();
small.healthy = true;
small.discovery = Some(discovery("small", 1));
// "big" has enough GPUs but its health is the variable under test.
let big = nodes.get_mut("big").unwrap();
big.healthy = big_healthy;
big.discovery = Some(discovery("big", big_devices));
}
fleet
}
#[tokio::test]
async fn feasible_node_unhealthy_is_transient_503() {
// big (2 GPU, the only feasible node) is unhealthy; small (1 GPU) is
// healthy but can't host the model → retryable, not a permanent 404.
let fleet = fleet_with(false, 2).await;
let err = router::resolve(&fleet, "big-model")
.await
.expect_err("model can't be served right now");
assert!(
matches!(err, RouteError::FeasibleNodeUnhealthy { .. }),
"expected FeasibleNodeUnhealthy, got {err:?}"
);
assert_eq!(err.http_status(), 503);
assert_eq!(err.retry_after_secs(), Some(3));
assert_eq!(err.code(), "service_unavailable");
}
#[tokio::test]
async fn no_node_can_ever_satisfy_is_permanent_404() {
// big is healthy but only has 1 GPU now (e.g. topology genuinely can't
// satisfy min_devices=2 anywhere) → permanent, non-retryable 404.
let fleet = fleet_with(true, 1).await;
let err = router::resolve(&fleet, "big-model")
.await
.expect_err("no feasible topology");
assert!(
matches!(err, RouteError::NoFeasibleNeuron { .. }),
"expected NoFeasibleNeuron, got {err:?}"
);
assert_eq!(err.http_status(), 404);
assert_eq!(err.retry_after_secs(), None);
}

View File

@@ -72,6 +72,7 @@ async fn two_neuron_fleet(endpoint_a: &str, endpoint_b: &str) -> Arc<CortexState
],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
Arc::new(CortexState::from_config(&config))
}

View File

@@ -53,6 +53,7 @@ async fn spawn_metered_gateway(neuron_url: &str) -> (Arc<CortexState>, String) {
window: CapWindow::Balance,
}],
},
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -158,6 +159,7 @@ async fn anonymous_request_records_no_spend() {
}],
models_config: "/dev/null".into(),
entitlements: EntitlementsConfig::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
{

View File

@@ -0,0 +1,132 @@
//! Issue #68: the `cost` wire contract on `GET /v1/models`.
//!
//! `cost` is operator-set pricing sourced from the `models.toml` catalogue
//! profile (the source of truth today; the marketplace clearing house #59
//! later — both must read the same value metering/#51 bills against). The
//! shape is the models.dev/opencode convention: **USD per 1,000,000 tokens,
//! as JSON numbers**, with optional `cache_read`/`cache_write` tiers. This
//! test pins:
//! - the units/shape (per-million floats, not per-token, not strings);
//! - that cache fields flow through when present and are omitted otherwise;
//! - the load-bearing **absent vs `0.0`** distinction (#68): a model with
//! no catalogue `cost` omits the key entirely (price unknown), distinct
//! from an explicit `0.0` (intentionally free).
//!
//! Catalogue-only models surface via Pass 1 of `list_models` even with no
//! feasible neuron, so this is hermetic — no nodes or poller needed.
use cortex_core::config::{
EvictionSettings, EvictionStrategy, GatewayConfig, GatewaySettings, NeuronEndpoint,
};
use cortex_gateway::state::CortexState;
use std::sync::Arc;
use tokio::net::TcpListener;
#[tokio::test]
async fn v1_models_cost_units_shape_and_absent_vs_zero() {
// Three catalogue models exercise the whole contract: a priced model
// with cache tiers, an intentionally-free model (explicit 0.0), and an
// unpriced model (no `cost` block at all).
let models_toml = r#"
[[models]]
id = "priced-model"
harness = "candle"
cost.input = 0.5
cost.output = 1.5
cost.cache_read = 0.05
cost.cache_write = 0.6
[[models]]
id = "free-model"
harness = "candle"
cost.input = 0.0
cost.output = 0.0
[[models]]
id = "unpriced-model"
harness = "candle"
"#;
let cat_path = std::env::temp_dir().join("cortex_test_issue68_models.toml");
std::fs::write(&cat_path, models_toml).unwrap();
let config = GatewayConfig {
gateway: GatewaySettings {
listen: "127.0.0.1:0".into(),
metrics_listen: "127.0.0.1:0".into(),
},
eviction: EvictionSettings {
strategy: EvictionStrategy::Lru,
defrag_after_cycles: 0,
},
// Never contacted: build_app does not spawn the poller, so the
// catalogue alone drives /v1/models.
neurons: vec![NeuronEndpoint {
name: "mock-node".into(),
endpoint: "http://127.0.0.1:1".into(),
}],
models_config: cat_path.to_string_lossy().into_owned(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
let app = cortex_gateway::build_app(Arc::clone(&fleet));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
let body: serde_json::Value = reqwest::Client::new()
.get(format!("http://{addr}/v1/models"))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
let data = body["data"].as_array().expect("data is an array");
let entry = |id: &str| {
data.iter()
.find(|m| m["id"] == id)
.unwrap_or_else(|| panic!("{id} present in /v1/models"))
.clone()
};
// Priced model: exact values flow through as JSON numbers (USD per 1M
// tokens). If anything rescaled by 10⁶ or stringified, these fail.
let priced = entry("priced-model");
assert_eq!(priced["cost"]["input"], 0.5);
assert_eq!(priced["cost"]["output"], 1.5);
assert_eq!(priced["cost"]["cache_read"], 0.05);
assert_eq!(priced["cost"]["cache_write"], 0.6);
assert!(
priced["cost"]["input"].is_number(),
"cost.input must be a JSON number, not a string"
);
// Intentionally free: cost present, rates explicitly 0.0. Unset cache
// tiers are omitted (skip_serializing_if), not emitted as null/0.
let free = entry("free-model");
assert_eq!(free["cost"]["input"], 0.0);
assert_eq!(free["cost"]["output"], 0.0);
assert!(
free["cost"].get("cache_read").is_none(),
"absent cache tiers must be omitted, not null"
);
assert!(free["cost"].get("cache_write").is_none());
// Unpriced: the whole `cost` object is omitted — "price unknown",
// distinct from the free model's explicit 0.0. This is the #68
// distinction opencode needs to avoid showing $0 for a model whose
// price simply hasn't been declared.
let unpriced = entry("unpriced-model");
assert!(
unpriced.get("cost").is_none(),
"a model with no catalogue cost must omit `cost` entirely, got {:?}",
unpriced.get("cost")
);
let _ = std::fs::remove_file(&cat_path);
}

View File

@@ -55,6 +55,7 @@ capabilities = ["text"]
}],
models_config: cat_path.to_string_lossy().into_owned(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));

View File

@@ -32,6 +32,7 @@ async fn test_poller_discovers_models() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -84,6 +85,7 @@ async fn test_poller_updates_gateway_models_endpoint() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -156,6 +158,7 @@ async fn test_models_endpoint_unions_capabilities_across_nodes() {
],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -219,6 +222,7 @@ async fn test_poller_marks_unreachable_node_unhealthy() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -228,10 +232,26 @@ async fn test_poller_marks_unreachable_node_unhealthy() {
nodes.get_mut("dead-node").unwrap().healthy = true;
}
// Debounce (#53 follow-up): a single missed poll must NOT evict a
// previously-healthy node — a busy neuron briefly slow to answer
// shouldn't yank its models out of routing.
cortex_gateway::poller::poll_once(&fleet).await;
assert!(
fleet.nodes.read().await.get("dead-node").unwrap().healthy,
"one failed poll should not mark a healthy node unhealthy"
);
let nodes = fleet.nodes.read().await;
assert!(!nodes.get("dead-node").unwrap().healthy);
// It flips unhealthy only after POLL_FAILURE_THRESHOLD (3) consecutive
// failures.
cortex_gateway::poller::poll_once(&fleet).await;
cortex_gateway::poller::poll_once(&fleet).await;
assert!(
!fleet.nodes.read().await.get("dead-node").unwrap().healthy,
"three consecutive failed polls should mark the node unhealthy"
);
// A subsequent successful poll would reset the counter and restore
// health; covered implicitly by the discovery tests above.
}
#[tokio::test]
@@ -257,6 +277,7 @@ async fn test_poller_removes_stale_models() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -288,6 +309,7 @@ async fn test_poller_removes_stale_models() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet2 = Arc::new(CortexState::from_config(&config2));
@@ -370,6 +392,7 @@ async fn test_poller_captures_activation_from_health() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
@@ -415,6 +438,7 @@ async fn test_poller_parses_recovering_status() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));

View File

@@ -75,6 +75,7 @@ async fn spawn_gateway(neuron: &str, context: usize) -> String {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = Arc::new(CortexState::from_config(&config));
{

View File

@@ -118,6 +118,7 @@ async fn test_no_healthy_nodes() {
}],
models_config: "/dev/null".into(),
entitlements: Default::default(),
upstream: Default::default(),
};
let fleet = std::sync::Arc::new(cortex_gateway::state::CortexState::from_config(&config));

View File

@@ -0,0 +1,106 @@
//! B3: the chained entitlement provider (local → upstream) and fail-closed
//! semantics, exercised against a mock helexa-upstream `/authz/v1`.
use axum::{Json, Router, routing::post};
use cortex_core::config::{ApiKeyConfig, EntitlementsConfig, UpstreamClientConfig};
use cortex_core::entitlements::{AuthError, EntitlementProvider};
use cortex_gateway::entitlements_chain::ChainedEntitlementProvider;
use cortex_gateway::entitlements_local::LocalEntitlementProvider;
use cortex_gateway::entitlements_upstream::UpstreamEntitlementProvider;
use serde_json::{Value, json};
use tokio::net::TcpListener;
/// Mock upstream: `mesh-key` resolves to a mesh account; anything else 401.
/// reserve always grants reservation 1.
async fn spawn_mock_upstream() -> String {
async fn resolve(Json(body): Json<Value>) -> axum::response::Response {
use axum::response::IntoResponse;
if body["api_key"] == "mesh-key" {
Json(json!({"principal": {"account_id": "mesh-acct", "key_id": "mesh-key-1"}}))
.into_response()
} else {
(
axum::http::StatusCode::UNAUTHORIZED,
Json(json!({"error": {"code": "invalid_api_key"}})),
)
.into_response()
}
}
async fn reserve() -> Json<Value> {
Json(json!({ "reservation_id": 1 }))
}
let app = Router::new()
.route("/authz/v1/resolve", post(resolve))
.route("/authz/v1/reserve", post(reserve));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
fn local_with_key() -> LocalEntitlementProvider {
let cfg = EntitlementsConfig {
require_auth: false,
keys: vec![ApiKeyConfig {
key: "local-key".into(),
account_id: "op".into(),
key_id: None,
hard_cap: None,
window: Default::default(),
}],
};
LocalEntitlementProvider::from_config(&cfg)
}
fn chain(local: LocalEntitlementProvider, url: &str) -> ChainedEntitlementProvider {
let upstream = UpstreamEntitlementProvider::new(&UpstreamClientConfig {
enabled: true,
url: url.to_string(),
bearer: "client-secret".into(),
timeout_secs: 5,
served_usage_report_interval_secs: 60,
});
ChainedEntitlementProvider::new(local, upstream)
}
#[tokio::test]
async fn local_key_resolves_locally() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("local-key").await.expect("local resolves");
assert_eq!(p.account_id, "op");
}
#[tokio::test]
async fn unknown_key_falls_through_to_upstream() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("mesh-key").await.expect("upstream resolves");
assert_eq!(p.account_id, "mesh-acct");
assert_eq!(p.key_id, "mesh-key-1");
}
#[tokio::test]
async fn unknown_everywhere_is_invalid_key() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
match c.resolve("nope").await {
Err(AuthError::InvalidKey) => {}
other => panic!("expected InvalidKey, got {other:?}"),
}
}
#[tokio::test]
async fn upstream_unreachable_fails_closed_as_unavailable() {
// No mock — point at a dead port. A locally-unknown key must surface
// Unavailable (→ 503), never InvalidKey (→ 401).
let c = chain(local_with_key(), "http://127.0.0.1:1");
match c.resolve("some-mesh-key").await {
Err(AuthError::Unavailable { retry_after_secs }) => assert!(retry_after_secs > 0),
other => panic!("expected Unavailable, got {other:?}"),
}
// A local key still resolves without touching upstream.
assert_eq!(c.resolve("local-key").await.unwrap().account_id, "op");
}

View File

@@ -0,0 +1,41 @@
[package]
name = "helexa-router"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
[[bin]]
name = "helexa-router"
path = "src/main.rs"
[lib]
name = "helexa_router"
path = "src/lib.rs"
[dependencies]
cortex-core = { workspace = true }
helexa-stream = { path = "../helexa-stream" }
tokio = { workspace = true }
axum = { workspace = true }
tower-http = { workspace = true }
reqwest = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
figment = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
chrono = { workspace = true }
[dev-dependencies]
# Jail (isolated cwd + env) for config tests.
figment = { workspace = true, features = ["test"] }
# Self-signed cert generation + a minimal HTTPS server for the outbound
# TLS-pinning tests (#74).
rcgen = "0.13"
rustls = "0.23"
tokio-rustls = "0.26"

View File

@@ -0,0 +1,243 @@
//! Federation catalogue (#75) — the router's aggregate `/v1/models`.
//!
//! Presents the **deduped union** of every reachable cortex's `/v1/models`
//! as the router's own catalogue, so an opencode client doing discovery
//! against the router resolves the whole federation without knowing about
//! operators or cortexes (resolves #61's "Router/discovery contract").
//!
//! Re-tiering: the fractal design is neuron ← cortex ← router. At the
//! router tier the "nodes" are **cortexes**, so the merged entry's
//! `feasible_on` / `locations` are rewritten to **operator names**, not the
//! neuron names a cortex reports. That keeps the federation view honest
//! ("served by these operators") without leaking each operator's internal
//! topology (neuron names, per-device VRAM) to end users.
//!
//! Conflict resolution when operators advertise the same model with
//! different enrichment:
//! - **`limit`** → the *tightest* (smallest `context`), so a client never
//! overflows the most-constrained operator that might serve it (same rule
//! cortex uses across its neurons).
//! - **`cost`** → the *cheapest* (lowest input, then output), the
//! federation "from" price. Richer policy (a range, region/price-aware
//! selection) couples to #68 and is left as a follow-up.
use crate::state::{CortexTopology, entry_feasible};
use cortex_core::harness::{ModelCost, ModelLimit};
use cortex_core::node::{CortexModelEntry, ModelLocation, ModelStatus};
use std::collections::HashMap;
/// Build the federation catalogue: the deduped union of every reachable
/// cortex's serveable models, merged across operators and sorted by id.
pub fn aggregate_models(topology: &HashMap<String, CortexTopology>) -> Vec<CortexModelEntry> {
// Iterate cortexes in name order so `feasible_on` / `locations` and the
// limit/cost tie-breaks are deterministic regardless of map ordering.
let mut cortexes: Vec<(&String, &CortexTopology)> = topology.iter().collect();
cortexes.sort_by(|a, b| a.0.cmp(b.0));
let mut merged: HashMap<String, CortexModelEntry> = HashMap::new();
for (cortex_name, t) in cortexes {
if !t.reachable {
continue;
}
for entry in t.models.values() {
// Only surface models the cortex can actually serve — a
// catalogue-only entry no neuron can host shouldn't appear in
// the federation view.
if !entry_feasible(entry) {
continue;
}
merged
.entry(entry.id.clone())
.and_modify(|acc| merge_into(acc, cortex_name, entry))
.or_insert_with(|| router_entry(cortex_name, entry));
}
}
let mut out: Vec<CortexModelEntry> = merged.into_values().collect();
out.sort_by(|a, b| a.id.cmp(&b.id));
out
}
/// Seed a federation entry from the first cortex that serves the model,
/// re-tiering `feasible_on` / `locations` to the operator name.
fn router_entry(cortex: &str, e: &CortexModelEntry) -> CortexModelEntry {
CortexModelEntry {
id: e.id.clone(),
object: "model".into(),
created: e.created,
owned_by: e.owned_by.clone(),
loaded: e.loaded,
feasible_on: vec![cortex.to_string()],
locations: loaded_location(cortex, e),
capabilities: e.capabilities.clone(),
limit: e.limit.clone(),
cost: e.cost.clone(),
tool_call: e.tool_call,
reasoning: e.reasoning,
}
}
/// Fold another cortex's view of the same model into the merged entry.
fn merge_into(acc: &mut CortexModelEntry, cortex: &str, e: &CortexModelEntry) {
acc.loaded |= e.loaded;
acc.feasible_on.push(cortex.to_string());
acc.locations.extend(loaded_location(cortex, e));
for cap in &e.capabilities {
if !acc.capabilities.contains(cap) {
acc.capabilities.push(cap.clone());
}
}
acc.tool_call |= e.tool_call;
acc.reasoning |= e.reasoning;
acc.limit = tightest_limit(acc.limit.take(), e.limit.clone());
acc.cost = cheapest_cost(acc.cost.take(), e.cost.clone());
}
/// A single cortex-tier location when the model is loaded at that operator;
/// empty when only cold-loadable. Neuron-level VRAM is deliberately dropped.
fn loaded_location(cortex: &str, e: &CortexModelEntry) -> Vec<ModelLocation> {
if e.loaded {
vec![ModelLocation {
node: cortex.to_string(),
status: ModelStatus::Loaded,
vram_estimate_mb: None,
}]
} else {
Vec::new()
}
}
/// Smaller `context` wins — never advertise more headroom than the
/// most-constrained operator can honour.
fn tightest_limit(a: Option<ModelLimit>, b: Option<ModelLimit>) -> Option<ModelLimit> {
match (a, b) {
(None, x) | (x, None) => x,
(Some(a), Some(b)) => Some(if b.context < a.context { b } else { a }),
}
}
/// Cheapest by (input, output) price — the federation "from" price.
fn cheapest_cost(a: Option<ModelCost>, b: Option<ModelCost>) -> Option<ModelCost> {
match (a, b) {
(None, x) | (x, None) => x,
(Some(a), Some(b)) => Some(if (b.input, b.output) < (a.input, a.output) {
b
} else {
a
}),
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::state::CortexTopology;
fn entry(id: &str, loaded: bool, feasible: bool) -> CortexModelEntry {
CortexModelEntry {
id: id.into(),
object: "model".into(),
created: 0,
owned_by: "helexa".into(),
loaded,
feasible_on: if feasible || loaded {
vec!["some-neuron".into()]
} else {
vec![]
},
locations: vec![],
capabilities: vec![],
limit: None,
cost: None,
tool_call: false,
reasoning: false,
}
}
fn cortex(reachable: bool, entries: Vec<CortexModelEntry>) -> CortexTopology {
CortexTopology {
reachable,
consecutive_failures: 0,
last_poll: None,
healthy_nodes: 1,
total_nodes: 1,
models: entries.into_iter().map(|e| (e.id.clone(), e)).collect(),
}
}
#[test]
fn dedupes_and_merges_availability_across_cortexes() {
let mut topo = HashMap::new();
// c-a: model loaded. c-b: same model only cold-loadable.
topo.insert("c-a".into(), cortex(true, vec![entry("m", true, true)]));
topo.insert("c-b".into(), cortex(true, vec![entry("m", false, true)]));
let out = aggregate_models(&topo);
assert_eq!(out.len(), 1, "duplicate model id collapses to one");
let m = &out[0];
assert!(m.loaded, "loaded somewhere → loaded");
// feasible_on re-tiered to operator names, both present, sorted.
assert_eq!(m.feasible_on, vec!["c-a".to_string(), "c-b".to_string()]);
// Only the loaded operator contributes a location, named by operator.
assert_eq!(m.locations.len(), 1);
assert_eq!(m.locations[0].node, "c-a");
assert_eq!(m.locations[0].vram_estimate_mb, None);
}
#[test]
fn unreachable_cortex_is_excluded() {
let mut topo = HashMap::new();
topo.insert("up".into(), cortex(true, vec![entry("m", true, true)]));
topo.insert(
"down".into(),
cortex(false, vec![entry("other", true, true)]),
);
let out = aggregate_models(&topo);
assert_eq!(out.len(), 1);
assert_eq!(out[0].id, "m");
}
#[test]
fn catalogue_only_infeasible_entries_are_hidden() {
let mut topo = HashMap::new();
topo.insert("c".into(), cortex(true, vec![entry("ghost", false, false)]));
assert!(aggregate_models(&topo).is_empty());
}
#[test]
fn preserves_tightest_limit_and_cheapest_cost() {
let mut a = entry("m", true, true);
a.limit = Some(ModelLimit {
context: 32_768,
input: None,
output: 4096,
});
a.cost = Some(ModelCost {
input: 0.50,
output: 1.50,
cache_read: None,
cache_write: None,
});
let mut b = entry("m", true, true);
b.limit = Some(ModelLimit {
context: 16_384, // tighter
input: None,
output: 4096,
});
b.cost = Some(ModelCost {
input: 0.20, // cheaper
output: 0.80,
cache_read: None,
cache_write: None,
});
let mut topo = HashMap::new();
topo.insert("c-a".into(), cortex(true, vec![a]));
topo.insert("c-b".into(), cortex(true, vec![b]));
let out = aggregate_models(&topo);
assert_eq!(out.len(), 1);
assert_eq!(out[0].limit.as_ref().unwrap().context, 16_384);
assert_eq!(out[0].cost.as_ref().unwrap().input, 0.20);
}
}

View File

@@ -0,0 +1,100 @@
use figment::{
Figment,
providers::{Env, Format, Toml},
};
use serde::{Deserialize, Serialize};
use std::path::Path;
/// Top-level `helexa-router` configuration.
///
/// Loaded from TOML with `HELEXA_ROUTER_`-prefixed env overrides (using
/// `__` as the nesting separator), matching the cortex/neuron convention.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RouterConfig {
pub router: RouterSettings,
/// Downstream cortex endpoints the router can dispatch to. The skeleton
/// (#70) only loads these; capacity/catalogue polling (#72) and
/// capacity-aware dispatch (#73) consume them later.
#[serde(default)]
pub cortexes: Vec<CortexEndpoint>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RouterSettings {
/// Address to listen on for the inbound API (e.g. "0.0.0.0:8088").
///
/// Plaintext only — operator/edge nginx terminates client TLS in front
/// of the router (see #69's TLS posture). The router never owns an
/// inbound TLS listener.
pub listen: String,
/// How often (seconds) the background poller refreshes each cortex's
/// health + `/v1/models` topology (#72). Defaults to 10s, matching the
/// cortex↔neuron poll cadence one tier down.
#[serde(default = "default_poll_interval_secs")]
pub poll_interval_secs: u64,
/// This router instance's region (e.g. "eu-west"). When set, dispatch
/// (#73) prefers cortexes whose `region` matches, before falling back to
/// any feasible cortex. `None` → no geo affinity.
#[serde(default)]
pub region: Option<String>,
}
fn default_poll_interval_secs() -> u64 {
10
}
/// One downstream cortex the router may proxy to. The router verifies the
/// cortex's outbound TLS cert (#74) and routes on capacity (#73); it holds
/// no entitlement logic of its own and forwards the client bearer verbatim.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct CortexEndpoint {
/// Human-readable label (e.g. "lair-cafe").
pub name: String,
/// Base URL of the cortex gateway (e.g. "https://cortex.example.com").
pub endpoint: String,
/// Optional region tag (e.g. "eu-west") for geo affinity in dispatch
/// (#73). `None` → no region preference applies to this cortex.
#[serde(default)]
pub region: Option<String>,
/// Path to a PEM trust anchor that **enrols** this cortex (#74): the
/// expected CA (or self-signed cert) the cortex's TLS cert must chain
/// to. When set on an `https://` endpoint, the router builds a client
/// that trusts **only** this anchor (platform roots disabled), so the
/// outbound router→cortex hop — which carries the client's bearer —
/// reaches a cert the router was told to expect, and a rogue endpoint
/// presenting any other (even publicly-valid) cert is rejected at the
/// TLS handshake. A rejected handshake surfaces as a connection error,
/// which the poller (#72) already treats as unreachable → excluded.
///
/// `None` → standard platform-root validation (use for cortexes behind
/// a publicly-trusted cert, or plaintext `http://` on a private network
/// where the WireGuard mesh is the trust boundary).
#[serde(default)]
pub tls_ca: Option<String>,
}
impl RouterConfig {
/// Load configuration from a TOML file, with environment variable
/// overrides prefixed with `HELEXA_ROUTER_` and `__` as the separator
/// (e.g. `HELEXA_ROUTER_ROUTER__LISTEN=0.0.0.0:8088`).
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
Figment::new()
.merge(Toml::file(path))
.merge(Env::prefixed("HELEXA_ROUTER_").split("__"))
.extract()
.map_err(Box::new)
}
}
impl Default for RouterConfig {
fn default() -> Self {
Self {
router: RouterSettings {
listen: "0.0.0.0:8088".into(),
poll_interval_secs: default_poll_interval_secs(),
region: None,
},
cortexes: vec![],
}
}
}

View File

@@ -0,0 +1,221 @@
//! Capacity-aware dispatch (#73) — the router's data path.
//!
//! Given an inbound request's `model`, pick a reachable cortex that can
//! serve it (preferring warm/loaded, region-affine, higher-headroom),
//! forward the client's bearer **unchanged** (auth stays at cortex), and
//! stream the response back verbatim via the shared [`helexa_stream`]
//! module. Cortex's #63-shaped rejections (`429 rate_limit_exceeded`,
//! `400 context_length_exceeded`, …) pass through untouched. Transport
//! failures fail over to the next feasible cortex; a genuine HTTP response —
//! any status — is returned as-is and never retried away.
//!
//! The router holds **no entitlement logic**: it routes on capacity, not
//! budget.
use crate::config::CortexEndpoint;
use crate::error::envelope_response;
use crate::state::RouterState;
use axum::body::Bytes;
use axum::http::HeaderMap;
use axum::response::Response;
use cortex_core::error_envelope::OpenAiError;
use helexa_stream::{ChunkObserver, StreamError};
use std::cmp::Reverse;
use std::collections::HashMap;
/// Retry-After hint (seconds) on the router's own transient rejections.
const RETRY_AFTER_SECS: u64 = 5;
/// Outcome of choosing where to send a request.
#[derive(Debug, PartialEq, Eq)]
pub enum Selection {
/// Feasible reachable cortexes, best-first (failover order).
Candidates(Vec<CortexEndpoint>),
/// Some cortex knows the model but none are reachable right now → 503.
NoReachableCapacity,
/// No configured cortex serves the model at all → 404.
UnknownModel,
}
/// Rank the reachable cortexes that can serve `model`, best-first.
///
/// Ordering (each a tie-break for the next): loaded/warm before cold-loadable
/// · region match before not · more healthy nodes before fewer · name for
/// determinism.
pub async fn select_cortexes(state: &RouterState, model: &str) -> Selection {
let topo = state.topology.read().await;
let by_name: HashMap<&str, &CortexEndpoint> = state
.cortexes
.iter()
.map(|c| (c.name.as_str(), c))
.collect();
let mut ranked: Vec<Ranked> = Vec::new();
let mut known_anywhere = false;
for (name, t) in topo.iter() {
let Some(entry) = t.models.get(model) else {
continue;
};
if !crate::state::entry_feasible(entry) {
continue;
}
// Known even via an unreachable cortex's last-good poll — lets us
// tell "temporarily down" (503) from "nobody serves it" (404).
known_anywhere = true;
if !t.reachable {
continue;
}
let Some(ep) = by_name.get(name.as_str()) else {
continue;
};
let region_match = match (&state.region, &ep.region) {
(Some(r), Some(cr)) => r == cr,
_ => false,
};
ranked.push(Ranked {
loaded: entry.loaded,
region_match,
healthy_nodes: t.healthy_nodes,
endpoint: (*ep).clone(),
});
}
if ranked.is_empty() {
return if known_anywhere {
Selection::NoReachableCapacity
} else {
Selection::UnknownModel
};
}
ranked.sort_by(|a, b| {
// false < true, so negate the "good" booleans to sort good first.
(
!a.loaded,
!a.region_match,
Reverse(a.healthy_nodes),
&a.endpoint.name,
)
.cmp(&(
!b.loaded,
!b.region_match,
Reverse(b.healthy_nodes),
&b.endpoint.name,
))
});
Selection::Candidates(ranked.into_iter().map(|r| r.endpoint).collect())
}
struct Ranked {
loaded: bool,
region_match: bool,
healthy_nodes: u32,
endpoint: CortexEndpoint,
}
/// Proxy an inbound inference request to a capacity-bearing cortex.
///
/// `path` is the inference path to forward to (same on the cortex, e.g.
/// `/v1/chat/completions`). The body is parsed only to extract `model`.
pub async fn dispatch(
state: &RouterState,
path: &str,
headers: HeaderMap,
body: Bytes,
) -> Response {
let Some(model) = extract_model(&body) else {
return envelope_response(OpenAiError::new(
400,
"invalid_request_error",
"missing_model_field",
"missing 'model' field in request body",
));
};
let candidates = match select_cortexes(state, &model).await {
Selection::Candidates(c) => c,
Selection::UnknownModel => {
return envelope_response(
OpenAiError::new(
404,
"invalid_request_error",
"model_not_found",
format!("no operator serves model '{model}'"),
)
.with_param("model"),
);
}
Selection::NoReachableCapacity => {
return envelope_response(OpenAiError::service_unavailable(
format!("model '{model}' is temporarily unavailable on all operators"),
Some(RETRY_AFTER_SECS),
));
}
};
// Try candidates in order, failing over only on transport errors. A
// genuine HTTP response (any status — including cortex's #63 429/400)
// is returned verbatim and never retried away.
for ep in &candidates {
// A candidate whose pinned TLS client failed to build (#74) is
// disabled — skip it and fail over, same as an unreachable cortex.
let Some(client) = state.client_for(&ep.name) else {
tracing::warn!(cortex = %ep.name, "no TLS client (disabled); skipping candidate");
continue;
};
let url = format!("{}{}", ep.endpoint, path);
tracing::info!(cortex = %ep.name, url = %url, model = %model, "dispatching");
match helexa_stream::forward_streaming(
client,
&url,
headers.clone(),
body.clone(),
NoopObserver,
)
.await
{
Ok(resp) => return resp,
Err(StreamError::Upstream(e)) => {
tracing::warn!(
cortex = %ep.name,
url = %url,
error = %e,
"cortex unreachable; failing over"
);
continue;
}
Err(StreamError::ResponseBuild(msg)) => {
tracing::error!(cortex = %ep.name, error = %msg, "failed to build proxied response");
return envelope_response(OpenAiError::without_code(
500,
"api_error",
"failed to build proxied response",
));
}
}
}
// Every feasible cortex failed to connect.
tracing::warn!(model = %model, tried = candidates.len(), "all feasible operators unreachable");
envelope_response(OpenAiError::service_unavailable(
format!("all operators able to serve '{model}' are unreachable"),
Some(RETRY_AFTER_SECS),
))
}
/// Pull the `model` field out of a request body without re-serialising it.
fn extract_model(body: &Bytes) -> Option<String> {
let v: serde_json::Value = serde_json::from_slice(body).ok()?;
v.get("model")?.as_str().map(str::to_string)
}
/// The router proxies bytes verbatim and keeps no per-request policy, so it
/// needs no observation hooks. (Token metrics/metering stay at cortex.)
struct NoopObserver;
impl ChunkObserver for NoopObserver {
fn observe(&mut self, _chunk: &[u8]) {}
fn finish(&mut self) {}
}

View File

@@ -0,0 +1,27 @@
//! Router adapter from the shared, axum-agnostic
//! [`cortex_core::error_envelope::OpenAiError`] (#60/#63) to an axum
//! [`Response`], setting `Retry-After` when the envelope carries one.
//!
//! cortex-core owns the envelope shape; this is the only place the router
//! crosses from that data into axum. Mirrors cortex-gateway's adapter so
//! the router's own rejections (no feasible operator, all unreachable) are
//! the same #63-shaped envelopes clients already understand — distinct from
//! cortex's rejections, which the router proxies through verbatim.
use axum::http::{HeaderValue, StatusCode, header};
use axum::response::{IntoResponse, Json, Response};
use cortex_core::error_envelope::OpenAiError;
/// Render an [`OpenAiError`] as an axum response (status + JSON envelope +
/// optional `Retry-After`).
pub fn envelope_response(err: OpenAiError) -> Response {
let status = StatusCode::from_u16(err.status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR);
let retry_after = err.retry_after_secs;
let mut response = (status, Json(err.body())).into_response();
if let Some(secs) = retry_after
&& let Ok(value) = HeaderValue::from_str(&secs.to_string())
{
response.headers_mut().insert(header::RETRY_AFTER, value);
}
response
}

View File

@@ -0,0 +1,89 @@
use crate::state::RouterState;
use crate::{catalogue, dispatch};
use axum::body::Bytes;
use axum::http::HeaderMap;
use axum::response::Response;
use axum::{Json, Router, extract::State, routing::get, routing::post};
use serde_json::{Value, json};
use std::sync::Arc;
/// Routes served by the router. Inference paths are capacity-aware-dispatched
/// (#73) to a downstream cortex; `/health` and a stub `/v1/models` are local.
pub fn api_routes() -> Router<Arc<RouterState>> {
Router::new()
.route("/v1/chat/completions", post(chat_completions))
.route("/v1/completions", post(completions))
.route("/v1/responses", post(responses))
.route("/v1/messages", post(messages))
.route("/v1/models", get(list_models))
.route("/health", get(health))
.route("/", get(health))
}
// ── Inference paths — forwarded verbatim to a chosen cortex ──────────
//
// Each handler dispatches to the same path on a capacity-bearing cortex.
// The body is parsed only to read `model`; the bearer and bytes are
// forwarded unchanged, and the SSE response streams back verbatim.
async fn chat_completions(
State(state): State<Arc<RouterState>>,
headers: HeaderMap,
body: Bytes,
) -> Response {
dispatch::dispatch(&state, "/v1/chat/completions", headers, body).await
}
async fn completions(
State(state): State<Arc<RouterState>>,
headers: HeaderMap,
body: Bytes,
) -> Response {
dispatch::dispatch(&state, "/v1/completions", headers, body).await
}
async fn responses(
State(state): State<Arc<RouterState>>,
headers: HeaderMap,
body: Bytes,
) -> Response {
dispatch::dispatch(&state, "/v1/responses", headers, body).await
}
async fn messages(
State(state): State<Arc<RouterState>>,
headers: HeaderMap,
body: Bytes,
) -> Response {
dispatch::dispatch(&state, "/v1/messages", headers, body).await
}
/// `GET /health` — router liveness plus a summary of downstream cortex
/// reachability from the topology poller (#72). `status` reflects the
/// router process itself (always `ok` if it answers); downstream health is
/// the informational `cortexes` block, so a fully-degraded fleet doesn't
/// make the router look dead to its own liveness probe.
async fn health(State(state): State<Arc<RouterState>>) -> Json<Value> {
let topo = state.topology.read().await;
let reachable = topo.values().filter(|t| t.reachable).count();
Json(json!({
"status": "ok",
"cortexes": {
"configured": state.cortexes.len(),
"reachable": reachable,
}
}))
}
/// `GET /v1/models` — the federation catalogue (#75): the deduped union of
/// every reachable cortex's `/v1/models`, so a client doing discovery
/// against the router resolves the whole federation without knowing about
/// operators or cortexes.
async fn list_models(State(state): State<Arc<RouterState>>) -> Json<Value> {
let topo = state.topology.read().await;
let data: Vec<Value> = catalogue::aggregate_models(&topo)
.iter()
.map(|e| json!(e))
.collect();
Json(json!({ "object": "list", "data": data }))
}

View File

@@ -0,0 +1,60 @@
//! helexa-router — public multi-operator ingress proxy (router.helexa.ai).
//!
//! The router is the data-plane *ingress* tier: a geo-distributed,
//! capacity-aware, OpenAI/Anthropic-compatible reverse proxy in front of
//! many operator-run cortexes ("cortex-of-cortexes"). End users configure
//! one `baseURL` and the router forwards their request to a cortex with
//! capacity, proxying #63-shaped rejections back verbatim.
//!
//! It holds **zero entitlement logic** — auth/budget stays at cortex
//! (epic #47); the router forwards the client bearer unchanged and routes
//! on capacity (epic #69). A background [`poller`] keeps a live
//! per-cortex topology (#72) that the dispatcher (#73) will route on.
pub mod catalogue;
pub mod config;
pub mod dispatch;
pub mod error;
pub mod handlers;
pub mod poller;
pub mod state;
use anyhow::Result;
use config::RouterConfig;
use std::sync::Arc;
use tower_http::cors::CorsLayer;
use tower_http::trace::TraceLayer;
/// Build the axum application: handlers + CORS + tracing. No auth layer —
/// the router asserts no identity of its own and forwards the client bearer
/// to the downstream cortex, which authenticates it (#69).
pub fn build_app(state: Arc<state::RouterState>) -> axum::Router {
axum::Router::new()
.merge(handlers::api_routes())
.layer(CorsLayer::permissive())
.layer(TraceLayer::new_for_http())
.with_state(state)
}
/// Start the router: build state from config and bind the plaintext HTTP
/// listener. TLS is terminated by edge nginx ahead of this process.
pub async fn run(config: RouterConfig) -> Result<()> {
let state = Arc::new(state::RouterState::from_config(&config));
// Background topology poller (#72): refresh each cortex's health +
// catalogue so routing decisions see live capacity.
let poller_state = Arc::clone(&state);
tokio::spawn(async move {
poller::poll_loop(poller_state).await;
});
let app = build_app(Arc::clone(&state));
let listen_addr = config.router.listen.parse::<std::net::SocketAddr>()?;
tracing::info!("helexa-router listening on {listen_addr}");
let listener = tokio::net::TcpListener::bind(listen_addr).await?;
axum::serve(listener, app).await?;
Ok(())
}

View File

@@ -0,0 +1,52 @@
use anyhow::Result;
use clap::{Parser, Subcommand};
use helexa_router::config::RouterConfig;
use tracing_subscriber::EnvFilter;
#[derive(Parser)]
#[command(name = "helexa-router")]
#[command(about = "Public multi-operator ingress proxy for helexa")]
#[command(version)]
struct Cli {
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
/// Start the router server.
Serve {
/// Path to the router config file.
#[arg(short, long, default_value = "helexa-router.toml")]
config: String,
},
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,helexa_router=debug")),
)
.init();
let cli = Cli::parse();
match cli.command {
Commands::Serve { config } => {
let cfg = RouterConfig::load(&config)
.map_err(|e| anyhow::anyhow!("failed to load config from '{config}': {e}"))?;
tracing::info!(
cortexes = cfg.cortexes.len(),
listen = %cfg.router.listen,
"starting helexa-router"
);
helexa_router::run(cfg).await?;
}
}
Ok(())
}

View File

@@ -0,0 +1,150 @@
//! Background poller that refreshes the multi-operator topology (#72).
//!
//! The same pattern as cortex↔neuron, one tier up: periodically poll each
//! configured cortex's `GET /v1/models` (catalogue × topology feasibility +
//! loaded state) and `GET /health` (coarse node-health/load), building the
//! live map the dispatcher (#73) routes on. An unreachable or erroring
//! cortex is debounced over [`POLL_FAILURE_THRESHOLD`] consecutive misses,
//! then flipped unhealthy and excluded from routing; it recovers on the
//! next successful poll.
use crate::state::RouterState;
use chrono::Utc;
use cortex_core::node::CortexModelEntry;
use serde::Deserialize;
use std::time::Duration;
/// Per-cortex HTTP timeout for each poll request.
const POLL_TIMEOUT: Duration = Duration::from_secs(5);
/// Consecutive failed polls before a cortex is marked unreachable. Mirrors
/// cortex's neuron-poll debounce: a single blip (a busy cortex briefly slow
/// to answer) can't yank it — and all its models — out of routing.
pub const POLL_FAILURE_THRESHOLD: u32 = 3;
/// cortex's `/v1/models` envelope — `{ "object": "list", "data": [...] }`.
#[derive(Debug, Deserialize)]
struct ModelsEnvelope {
#[serde(default)]
data: Vec<CortexModelEntry>,
}
/// The subset of cortex's `/health` the router reads.
#[derive(Debug, Deserialize)]
struct CortexHealth {
nodes: CortexHealthNodes,
}
#[derive(Debug, Deserialize)]
struct CortexHealthNodes {
healthy: u32,
total: u32,
}
/// Run forever, polling all cortexes on the configured interval.
pub async fn poll_loop(state: std::sync::Arc<RouterState>) {
loop {
poll_once(&state).await;
tokio::time::sleep(state.poll_interval).await;
}
}
/// Poll every configured cortex once. Public for testing.
pub async fn poll_once(state: &RouterState) {
for cortex in &state.cortexes {
poll_cortex(state, &cortex.name, &cortex.endpoint).await;
}
}
/// Poll one cortex: refresh its model map from `/v1/models`, then its node
/// health from `/health`. A `/v1/models` failure debounces toward
/// unreachable; the `/health` poll is best-effort and never flips
/// reachability on its own (a cortex serving `/v1/models` is routable even
/// if `/health` momentarily isn't).
async fn poll_cortex(state: &RouterState, name: &str, endpoint: &str) {
// A cortex whose pinned TLS client failed to build (#74) is disabled:
// there is no client to poll with, so it stays unreachable.
let Some(client) = state.client_for(name) else {
let mut topo = state.topology.write().await;
if let Some(entry) = topo.get_mut(name) {
entry.consecutive_failures = entry.consecutive_failures.saturating_add(1);
entry.reachable = false;
}
tracing::warn!(cortex = name, "no TLS client (disabled); skipping poll");
return;
};
let models = fetch_models(client, endpoint).await;
let mut topo = state.topology.write().await;
let Some(entry) = topo.get_mut(name) else {
return; // not a configured cortex (shouldn't happen)
};
match models {
Ok(models) => {
entry.models = models.into_iter().map(|m| (m.id.clone(), m)).collect();
entry.reachable = true;
entry.consecutive_failures = 0;
entry.last_poll = Some(Utc::now());
tracing::debug!(cortex = name, models = entry.models.len(), "poll ok");
}
Err(reason) => {
entry.consecutive_failures = entry.consecutive_failures.saturating_add(1);
if entry.consecutive_failures >= POLL_FAILURE_THRESHOLD {
entry.reachable = false;
}
tracing::warn!(
cortex = name,
failures = entry.consecutive_failures,
reachable = entry.reachable,
reason,
"cortex poll failed"
);
}
}
drop(topo);
// Best-effort health (node counts). Never flips reachability.
if let Some((healthy, total)) = fetch_health(client, endpoint).await {
let mut topo = state.topology.write().await;
if let Some(entry) = topo.get_mut(name) {
entry.healthy_nodes = healthy;
entry.total_nodes = total;
}
}
}
/// GET `/v1/models`, returning the parsed entries or a short failure reason.
async fn fetch_models(
client: &reqwest::Client,
endpoint: &str,
) -> Result<Vec<CortexModelEntry>, &'static str> {
let url = format!("{endpoint}/v1/models");
let resp = client
.get(&url)
.timeout(POLL_TIMEOUT)
.send()
.await
.map_err(|_| "unreachable")?;
if !resp.status().is_success() {
return Err("non-success status");
}
let envelope = resp
.json::<ModelsEnvelope>()
.await
.map_err(|_| "bad json")?;
Ok(envelope.data)
}
/// GET `/health`, returning `(healthy, total)` node counts. `None` on any
/// failure — the caller leaves the previous counts in place.
async fn fetch_health(client: &reqwest::Client, endpoint: &str) -> Option<(u32, u32)> {
let url = format!("{endpoint}/health");
let resp = client.get(&url).timeout(POLL_TIMEOUT).send().await.ok()?;
if !resp.status().is_success() {
return None;
}
let health = resp.json::<CortexHealth>().await.ok()?;
Some((health.nodes.healthy, health.nodes.total))
}

View File

@@ -0,0 +1,144 @@
use crate::config::{CortexEndpoint, RouterConfig};
use chrono::{DateTime, Utc};
use cortex_core::node::CortexModelEntry;
use std::collections::HashMap;
use std::time::Duration;
use tokio::sync::RwLock;
/// Shared router state: the configured cortex list plus the live topology
/// map the poller (#72) maintains and the dispatcher (#73) will route on.
///
/// This is the router tier of the fractal neuron ← cortex ← router design:
/// just as cortex polls each neuron for capacity/catalogue, the router
/// polls each cortex's `/health` + `/v1/models`.
#[derive(Debug)]
pub struct RouterState {
/// Downstream cortex endpoints, as configured.
pub cortexes: Vec<CortexEndpoint>,
/// Per-cortex HTTP client, keyed by cortex name (#74). A cortex enrolled
/// with a `tls_ca` gets a client that trusts only that anchor; others
/// get a default client. A cortex whose `tls_ca` failed to load is
/// **absent** here — `client_for` returns `None` and it is never
/// polled or routed to (fail closed: a misconfigured pin must not
/// silently fall back to unpinned TLS).
clients: HashMap<String, reqwest::Client>,
/// This router instance's region, for dispatch geo affinity (#73).
pub region: Option<String>,
/// How often the poller refreshes the topology.
pub poll_interval: Duration,
/// Live per-cortex topology, keyed by cortex name. Pre-populated from
/// config (every configured cortex present, `reachable = false`) so the
/// poller and handlers always find an entry; the poller flips
/// reachability and fills the model map.
pub topology: RwLock<HashMap<String, CortexTopology>>,
}
/// Live view of one downstream cortex, refreshed each poll.
#[derive(Debug, Clone, Default)]
pub struct CortexTopology {
/// Whether the cortex is currently routable. Flipped `false` only after
/// [`crate::poller::POLL_FAILURE_THRESHOLD`] consecutive failed polls
/// (debounces transient blips); restored on the next successful poll.
pub reachable: bool,
/// Consecutive failed polls; reset to 0 on success.
pub consecutive_failures: u32,
/// Timestamp of the last successful poll.
pub last_poll: Option<DateTime<Utc>>,
/// Healthy / total neuron counts from the cortex's `/health` (coarse
/// load signal; #73 refines headroom). 0/0 until first health poll.
pub healthy_nodes: u32,
pub total_nodes: u32,
/// The cortex's full `/v1/models` entries, keyed by model id. Stored
/// whole (not distilled to a loaded/feasible bool) so the federation
/// catalogue (#75) can preserve per-model `limit`/`cost`/capabilities.
pub models: HashMap<String, CortexModelEntry>,
}
/// Whether a cortex can serve this model — loaded now, or feasible to
/// cold-load (its catalogue × topology says some neuron can host it).
pub fn entry_feasible(entry: &CortexModelEntry) -> bool {
entry.loaded || !entry.feasible_on.is_empty()
}
impl RouterState {
pub fn from_config(config: &RouterConfig) -> Self {
let topology = config
.cortexes
.iter()
.map(|c| (c.name.clone(), CortexTopology::default()))
.collect();
// One client per cortex. A `tls_ca` that fails to load omits the
// cortex from the map (fail closed) rather than degrading to an
// unpinned client.
let mut clients = HashMap::new();
for c in &config.cortexes {
match build_client(c.tls_ca.as_deref()) {
Ok(client) => {
clients.insert(c.name.clone(), client);
}
Err(e) => {
tracing::error!(
cortex = %c.name,
tls_ca = c.tls_ca.as_deref().unwrap_or(""),
error = %e,
"failed to build pinned TLS client; cortex disabled (fail closed)"
);
}
}
}
Self {
cortexes: config.cortexes.clone(),
clients,
region: config.router.region.clone(),
poll_interval: Duration::from_secs(config.router.poll_interval_secs),
topology: RwLock::new(topology),
}
}
/// The HTTP client to use for `name`, or `None` if the cortex is
/// disabled (its `tls_ca` failed to load). Callers must treat `None` as
/// "not routable / not pollable".
pub fn client_for(&self, name: &str) -> Option<&reqwest::Client> {
self.clients.get(name)
}
/// Names of reachable cortexes that can serve `model_id` (loaded or
/// feasible to cold-load). Groundwork for capacity-aware dispatch (#73);
/// unreachable cortexes are excluded by construction.
pub async fn cortexes_serving(&self, model_id: &str) -> Vec<String> {
let topo = self.topology.read().await;
topo.iter()
.filter(|(_, t)| t.reachable)
.filter(|(_, t)| t.models.get(model_id).is_some_and(entry_feasible))
.map(|(name, _)| name.clone())
.collect()
}
}
/// Build a cortex HTTP client. With `tls_ca` set, the client trusts **only**
/// that PEM anchor (platform roots disabled) — pinning the router→cortex hop
/// to an enrolled cert (#74). Without it, standard platform-root validation.
pub fn build_client(tls_ca: Option<&str>) -> Result<reqwest::Client, BuildClientError> {
let mut builder = reqwest::Client::builder();
if let Some(path) = tls_ca {
let pem = std::fs::read(path).map_err(|e| BuildClientError::Read(path.to_string(), e))?;
let cert = reqwest::Certificate::from_pem(&pem).map_err(BuildClientError::Parse)?;
builder = builder
.tls_built_in_root_certs(false)
.add_root_certificate(cert);
}
builder.build().map_err(BuildClientError::Build)
}
/// Why a cortex's pinned client could not be built (→ cortex disabled).
#[derive(Debug, thiserror::Error)]
pub enum BuildClientError {
#[error("reading TLS anchor '{0}'")]
Read(String, #[source] std::io::Error),
#[error("parsing TLS anchor PEM")]
Parse(#[source] reqwest::Error),
#[error("building HTTP client")]
Build(#[source] reqwest::Error),
}

View File

@@ -0,0 +1,132 @@
//! End-to-end federation-catalogue test for #75: poll two mock cortexes
//! that overlap on a model, then `GET /v1/models` on the router and verify
//! the deduped union with merged availability and preserved limit/cost.
use axum::Router;
use axum::routing::get;
use helexa_router::config::{CortexEndpoint, RouterConfig};
use helexa_router::poller::poll_once;
use helexa_router::state::RouterState;
use serde_json::{Value, json};
use std::sync::Arc;
use tokio::net::TcpListener;
/// Spawn a mock cortex serving the given `/v1/models` `data` array.
async fn spawn_cortex(models: Value) -> String {
let models = Arc::new(models);
let app = Router::new()
.route(
"/v1/models",
get({
let models = Arc::clone(&models);
move || {
let models = Arc::clone(&models);
async move { axum::Json(json!({ "object": "list", "data": &*models })) }
}
}),
)
.route(
"/health",
get(|| async { axum::Json(json!({"status":"ok","nodes":{"healthy":1,"total":1}})) }),
);
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
/// Spawn the router (with poller) wired to the given cortex endpoints, and
/// poll once synchronously so the topology is populated before we query.
async fn spawn_router(cortexes: Vec<CortexEndpoint>) -> String {
let cfg = RouterConfig {
cortexes,
..Default::default()
};
let state = Arc::new(RouterState::from_config(&cfg));
poll_once(&state).await; // deterministic: fill topology now
let app = helexa_router::build_app(Arc::clone(&state));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
fn model(id: &str, loaded: bool, feasible_on: &[&str], ctx: u64, input_cost: f64) -> Value {
json!({
"id": id,
"object": "model",
"created": 0,
"owned_by": "helexa",
"loaded": loaded,
"feasible_on": feasible_on,
"locations": [],
"limit": { "context": ctx, "output": 4096 },
"cost": { "input": input_cost, "output": input_cost * 3.0 }
})
}
#[tokio::test]
async fn federation_catalogue_dedupes_and_preserves_limit_cost() {
// cortex A: "shared" loaded (ctx 32768, $0.50) + "only-a" loaded.
let a = spawn_cortex(json!([
model("shared", true, &["beast"], 32_768, 0.50),
model("only-a", true, &["beast"], 8_192, 1.00),
]))
.await;
// cortex B: "shared" cold-loadable, tighter ctx (16384), cheaper ($0.20).
let b = spawn_cortex(json!([model("shared", false, &["benjy"], 16_384, 0.20)])).await;
let router = spawn_router(vec![
CortexEndpoint {
name: "op-a".into(),
endpoint: a,
region: None,
tls_ca: None,
},
CortexEndpoint {
name: "op-b".into(),
endpoint: b,
region: None,
tls_ca: None,
},
])
.await;
let body: Value = reqwest::get(format!("{router}/v1/models"))
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(body["object"], "list");
let data = body["data"].as_array().unwrap();
// Deduped union: "shared" once + "only-a".
assert_eq!(data.len(), 2);
let shared = data.iter().find(|m| m["id"] == "shared").unwrap();
// Loaded somewhere (op-a) → loaded.
assert_eq!(shared["loaded"], true);
// feasible_on re-tiered to operator names, both present, sorted.
let feasible: Vec<&str> = shared["feasible_on"]
.as_array()
.unwrap()
.iter()
.map(|v| v.as_str().unwrap())
.collect();
assert_eq!(feasible, vec!["op-a", "op-b"]);
// Tightest limit (16384) and cheapest cost ($0.20) win.
assert_eq!(shared["limit"]["context"], 16_384);
assert_eq!(shared["cost"]["input"], 0.20);
// Loaded location named by operator, no neuron VRAM leaked.
let locs = shared["locations"].as_array().unwrap();
assert_eq!(locs.len(), 1);
assert_eq!(locs[0]["node"], "op-a");
assert!(data.iter().any(|m| m["id"] == "only-a"));
}

View File

@@ -0,0 +1,301 @@
//! Capacity-aware dispatch acceptance tests for #73.
//!
//! Covers: a request routes to a cortex serving the model; the client's
//! bearer reaches the cortex; cortex's #63 rejections pass through verbatim
//! and are NOT retried away; transport failure fails over to another
//! feasible cortex; unknown model → 404, no reachable capacity → 503; and
//! the selection ranking (warm/region/headroom).
use axum::body::Bytes;
use axum::extract::State;
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::post;
use axum::{Json, Router};
use cortex_core::node::CortexModelEntry;
use helexa_router::config::{CortexEndpoint, RouterConfig};
use helexa_router::dispatch::{Selection, dispatch, select_cortexes};
use helexa_router::state::{CortexTopology, RouterState};
use serde_json::{Value, json};
use std::collections::HashMap;
use tokio::net::TcpListener;
/// A minimal `CortexModelEntry` for MODEL with the given serveability.
fn model_entry(loaded: bool, feasible: bool) -> CortexModelEntry {
CortexModelEntry {
id: MODEL.into(),
object: "model".into(),
created: 0,
owned_by: "helexa".into(),
loaded,
feasible_on: if feasible || loaded {
vec!["n".into()]
} else {
vec![]
},
locations: vec![],
capabilities: vec![],
limit: None,
cost: None,
tool_call: false,
reasoning: false,
}
}
const MODEL: &str = "Qwen/Qwen3-Coder-30B";
// ── Mock cortex backend ──────────────────────────────────────────────
/// Behaviour of a mock cortex, carried in axum State.
#[derive(Clone)]
struct MockCortex {
/// Identifies which cortex answered, echoed in the 200 body.
name: &'static str,
/// When true, return a genuine #63-shaped `429 rate_limit_exceeded`.
rate_limited: bool,
}
async fn mock_handler(State(m): State<MockCortex>, headers: HeaderMap) -> Response {
if m.rate_limited {
return (
StatusCode::TOO_MANY_REQUESTS,
Json(json!({"error":{"type":"rate_limit_error","code":"rate_limit_exceeded","message":"slow down","param":null}})),
)
.into_response();
}
let auth = headers
.get("authorization")
.and_then(|v| v.to_str().ok())
.unwrap_or("")
.to_string();
Json(json!({ "served_by": m.name, "auth_seen": auth })).into_response()
}
async fn spawn_cortex(mock: MockCortex) -> String {
let app = Router::new()
.route("/v1/chat/completions", post(mock_handler))
.with_state(mock);
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
fn ok_cortex(name: &'static str) -> MockCortex {
MockCortex {
name,
rate_limited: false,
}
}
// ── Helpers to build state with a hand-set topology ──────────────────
fn state_with(cortexes: Vec<CortexEndpoint>, region: Option<String>) -> RouterState {
let cfg = RouterConfig {
cortexes,
..Default::default()
};
let mut state = RouterState::from_config(&cfg);
state.region = region;
state
}
/// Overwrite the topology entry for `name` so tests control reachability and
/// model serveability directly (no live poll).
async fn set_topology(
state: &RouterState,
name: &str,
reachable: bool,
loaded: bool,
feasible: bool,
healthy_nodes: u32,
) {
let mut topo = state.topology.write().await;
let mut models = HashMap::new();
models.insert(MODEL.to_string(), model_entry(loaded, feasible));
topo.insert(
name.to_string(),
CortexTopology {
reachable,
consecutive_failures: 0,
last_poll: None,
healthy_nodes,
total_nodes: healthy_nodes,
models,
},
);
}
fn ep(name: &str, endpoint: &str, region: Option<&str>) -> CortexEndpoint {
CortexEndpoint {
name: name.into(),
endpoint: endpoint.into(),
region: region.map(str::to_string),
tls_ca: None,
}
}
fn chat_body() -> Bytes {
Bytes::from(format!("{{\"model\":\"{MODEL}\",\"stream\":false}}"))
}
async fn body_json(resp: Response) -> (StatusCode, Value) {
let status = resp.status();
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.unwrap();
let v = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
(status, v)
}
// ── Tests ────────────────────────────────────────────────────────────
#[tokio::test]
async fn routes_to_serving_cortex_and_forwards_bearer() {
let url = spawn_cortex(ok_cortex("c1")).await;
let state = state_with(vec![ep("c1", &url, None)], None);
set_topology(&state, "c1", true, true, true, 2).await;
let mut headers = HeaderMap::new();
headers.insert("authorization", "Bearer sk-test-123".parse().unwrap());
let resp = dispatch(&state, "/v1/chat/completions", headers, chat_body()).await;
let (status, body) = body_json(resp).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["served_by"], "c1");
// Bearer reached the cortex unchanged.
assert_eq!(body["auth_seen"], "Bearer sk-test-123");
}
#[tokio::test]
async fn cortex_429_passes_through_and_is_not_retried() {
// c1 (ranked first: loaded) returns a genuine 429; c2 would return 200.
let c1 = spawn_cortex(MockCortex {
name: "c1",
rate_limited: true,
})
.await;
let c2 = spawn_cortex(ok_cortex("c2")).await;
let state = state_with(vec![ep("c1", &c1, None), ep("c2", &c2, None)], None);
// Both reachable + loaded; c1 has more headroom so it ranks first.
set_topology(&state, "c1", true, true, true, 5).await;
set_topology(&state, "c2", true, true, true, 1).await;
let resp = dispatch(
&state,
"/v1/chat/completions",
HeaderMap::new(),
chat_body(),
)
.await;
let (status, body) = body_json(resp).await;
// The genuine 4xx is returned verbatim — NOT retried to c2.
assert_eq!(status, StatusCode::TOO_MANY_REQUESTS);
assert_eq!(body["error"]["code"], "rate_limit_exceeded");
assert!(body.get("served_by").is_none(), "must not have hit c2");
}
#[tokio::test]
async fn fails_over_to_next_cortex_on_transport_error() {
// c_dead ranks first (more headroom) but its endpoint is a closed port;
// c_live is the fallback. The router must fail over and c_live serves.
let live = spawn_cortex(ok_cortex("c_live")).await;
let state = state_with(
vec![
ep("c_dead", "http://127.0.0.1:1", None),
ep("c_live", &live, None),
],
None,
);
set_topology(&state, "c_dead", true, true, true, 9).await;
set_topology(&state, "c_live", true, true, true, 1).await;
let resp = dispatch(
&state,
"/v1/chat/completions",
HeaderMap::new(),
chat_body(),
)
.await;
let (status, body) = body_json(resp).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["served_by"], "c_live");
}
#[tokio::test]
async fn unknown_model_is_404() {
let state = state_with(vec![ep("c1", "http://127.0.0.1:1", None)], None);
// Topology has no entry for MODEL at all.
let resp = dispatch(
&state,
"/v1/chat/completions",
HeaderMap::new(),
chat_body(),
)
.await;
let (status, body) = body_json(resp).await;
assert_eq!(status, StatusCode::NOT_FOUND);
assert_eq!(body["error"]["code"], "model_not_found");
}
#[tokio::test]
async fn known_but_all_unreachable_is_503() {
let state = state_with(vec![ep("c1", "http://127.0.0.1:1", None)], None);
// Cortex knows the model (from a prior good poll) but is now unreachable.
set_topology(&state, "c1", false, true, true, 2).await;
let resp = dispatch(
&state,
"/v1/chat/completions",
HeaderMap::new(),
chat_body(),
)
.await;
let (status, body) = body_json(resp).await;
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(body["error"]["code"], "service_unavailable");
}
#[tokio::test]
async fn missing_model_field_is_400() {
let state = state_with(vec![ep("c1", "http://127.0.0.1:1", None)], None);
let resp = dispatch(
&state,
"/v1/chat/completions",
HeaderMap::new(),
Bytes::from_static(b"{\"messages\":[]}"),
)
.await;
let (status, body) = body_json(resp).await;
assert_eq!(status, StatusCode::BAD_REQUEST);
assert_eq!(body["error"]["code"], "missing_model_field");
}
#[tokio::test]
async fn ranking_prefers_loaded_then_region_then_headroom() {
// Router is in eu-west. Candidates:
// warm-eu : loaded, region match, 1 node → best
// warm-us : loaded, no region, 9 nodes
// cold-eu : feasible only, region match → worst (cold)
let state = state_with(
vec![
ep("warm-eu", "http://127.0.0.1:1", Some("eu-west")),
ep("warm-us", "http://127.0.0.1:1", Some("us-east")),
ep("cold-eu", "http://127.0.0.1:1", Some("eu-west")),
],
Some("eu-west".into()),
);
set_topology(&state, "warm-eu", true, true, true, 1).await;
set_topology(&state, "warm-us", true, true, true, 9).await;
set_topology(&state, "cold-eu", true, false, true, 5).await;
let Selection::Candidates(order) = select_cortexes(&state, MODEL).await else {
panic!("expected candidates");
};
let names: Vec<&str> = order.iter().map(|e| e.name.as_str()).collect();
assert_eq!(names, vec!["warm-eu", "warm-us", "cold-eu"]);
}

View File

@@ -0,0 +1,97 @@
//! Skeleton acceptance tests for #70: the router builds, serves `/health`
//! and `/v1/models` on a plaintext port, and loads its cortex-endpoint list
//! from TOML with env overrides.
use helexa_router::config::{CortexEndpoint, RouterConfig};
use helexa_router::state::RouterState;
use std::sync::Arc;
use tokio::net::TcpListener;
/// Bind the router app on an ephemeral port and return its base URL.
async fn spawn_router(cortexes: Vec<CortexEndpoint>) -> String {
let cfg = RouterConfig {
cortexes,
..Default::default()
};
let state = Arc::new(RouterState::from_config(&cfg));
let app = helexa_router::build_app(state);
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
#[tokio::test]
async fn health_reports_configured_cortex_count() {
let base = spawn_router(vec![
CortexEndpoint {
name: "a".into(),
endpoint: "https://a.example.com".into(),
region: None,
tls_ca: None,
},
CortexEndpoint {
name: "b".into(),
endpoint: "https://b.example.com".into(),
region: None,
tls_ca: None,
},
])
.await;
let body: serde_json::Value = reqwest::get(format!("{base}/health"))
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(body["status"], "ok");
assert_eq!(body["cortexes"]["configured"], 2);
}
#[tokio::test]
async fn models_returns_empty_openai_list() {
let base = spawn_router(vec![]).await;
let resp = reqwest::get(format!("{base}/v1/models")).await.unwrap();
assert!(resp.status().is_success());
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["object"], "list");
assert_eq!(body["data"].as_array().unwrap().len(), 0);
}
#[test]
#[allow(clippy::result_large_err)]
fn config_loads_from_toml_with_env_override() {
figment::Jail::expect_with(|jail| {
jail.create_file(
"helexa-router.toml",
r#"
[router]
listen = "127.0.0.1:8088"
[[cortexes]]
name = "lair-cafe"
endpoint = "https://cortex.lair.cafe"
"#,
)?;
// Env override wins over the TOML value.
jail.set_env("HELEXA_ROUTER_ROUTER__LISTEN", "0.0.0.0:9099");
let cfg = RouterConfig::load("helexa-router.toml").expect("load config");
assert_eq!(cfg.router.listen, "0.0.0.0:9099");
assert_eq!(cfg.cortexes.len(), 1);
assert_eq!(cfg.cortexes[0].name, "lair-cafe");
assert_eq!(cfg.cortexes[0].endpoint, "https://cortex.lair.cafe");
Ok(())
});
}

View File

@@ -0,0 +1,210 @@
//! Outbound TLS pinning tests for #74.
//!
//! Proves the router, as a TLS client to cortexes, reaches a cortex
//! presenting its **enrolled** cert and rejects one presenting an
//! unexpected (or untrusted) cert — and that a rejected handshake flows
//! through the existing reachability path (#72) to exclude the cortex.
//!
//! A minimal `tokio-rustls` HTTPS server presents a self-signed cert; the
//! router's `reqwest` client (native-tls) validates against the PEM anchor
//! enrolled in config. Server (rustls) and client (native-tls) interoperate
//! at the protocol level — what matters is the trust decision.
use helexa_router::config::{CortexEndpoint, RouterConfig};
use helexa_router::poller::poll_once;
use helexa_router::state::{RouterState, build_client};
use std::io::Write;
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
use tokio_rustls::TlsAcceptor;
/// A self-signed cert: PEM (for the reqwest pin file) + DER cert/key (for
/// the rustls server).
struct TestCert {
cert_pem: String,
cert_der: rustls::pki_types::CertificateDer<'static>,
key_der: Vec<u8>,
}
fn make_cert() -> TestCert {
let key = rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).unwrap();
TestCert {
cert_pem: key.cert.pem(),
cert_der: key.cert.der().clone(),
key_der: key.key_pair.serialize_der(),
}
}
/// Write a cert PEM to a unique temp file (named by `tag`) and return the
/// path. `tag` is caller-unique (we use the bound port), so no randomness.
fn write_pem(tag: &str, pem: &str) -> String {
let path = std::env::temp_dir().join(format!("helexa-router-tls-{tag}.pem"));
let mut f = std::fs::File::create(&path).unwrap();
f.write_all(pem.as_bytes()).unwrap();
path.to_string_lossy().into_owned()
}
/// Spawn a minimal HTTPS server presenting `cert`, answering every request
/// with a canned `/v1/models`-shaped 200. Returns its `https://` base URL.
async fn spawn_https(cert: &TestCert) -> String {
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let key = rustls::pki_types::PrivateKeyDer::Pkcs8(rustls::pki_types::PrivatePkcs8KeyDer::from(
cert.key_der.clone(),
));
let config = rustls::ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(vec![cert.cert_der.clone()], key)
.unwrap();
let acceptor = TlsAcceptor::from(Arc::new(config));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((stream, _)) = listener.accept().await else {
continue;
};
let acceptor = acceptor.clone();
tokio::spawn(async move {
if let Ok(mut tls) = acceptor.accept(stream).await {
let mut buf = [0u8; 2048];
let _ = tls.read(&mut buf).await; // consume request line/headers
let body = "{\"object\":\"list\",\"data\":[]}";
let resp = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
let _ = tls.write_all(resp.as_bytes()).await;
let _ = tls.shutdown().await;
}
});
}
});
format!("https://{addr}")
}
fn tag_for(url: &str) -> String {
url.rsplit(':').next().unwrap_or("0").to_string()
}
#[tokio::test]
async fn pinned_client_accepts_enrolled_cert_and_rejects_others() {
let server_cert = make_cert();
let other_cert = make_cert();
let url = spawn_https(&server_cert).await;
let tag = tag_for(&url);
let good_pin = write_pem(&format!("{tag}-good"), &server_cert.cert_pem);
let bad_pin = write_pem(&format!("{tag}-bad"), &other_cert.cert_pem);
// Enrolled with the server's own cert → handshake trusted → 200.
let good = build_client(Some(&good_pin)).unwrap();
let resp = good.get(format!("{url}/v1/models")).send().await;
assert!(resp.is_ok(), "enrolled cert must be accepted: {resp:?}");
assert_eq!(resp.unwrap().status(), 200);
// Enrolled with a different cert → server's cert is unexpected → reject.
let bad = build_client(Some(&bad_pin)).unwrap();
assert!(
bad.get(format!("{url}/v1/models")).send().await.is_err(),
"unexpected cert must be rejected"
);
// No enrollment (default platform roots) → self-signed cert untrusted.
let default = build_client(None).unwrap();
assert!(
default
.get(format!("{url}/v1/models"))
.send()
.await
.is_err(),
"un-enrolled self-signed cert must be rejected by default roots"
);
}
#[tokio::test]
async fn poller_excludes_cortex_with_unexpected_cert() {
let server_cert = make_cert();
let other_cert = make_cert();
let url = spawn_https(&server_cert).await;
let tag = tag_for(&url);
let good_pin = write_pem(&format!("{tag}-pgood"), &server_cert.cert_pem);
let bad_pin = write_pem(&format!("{tag}-pbad"), &other_cert.cert_pem);
// Cortex A enrolled correctly → reachable. Cortex B enrolled with the
// wrong cert → TLS handshake fails → excluded.
let cfg = RouterConfig {
cortexes: vec![
CortexEndpoint {
name: "good".into(),
endpoint: url.clone(),
region: None,
tls_ca: Some(good_pin),
},
CortexEndpoint {
name: "bad".into(),
endpoint: url.clone(),
region: None,
tls_ca: Some(bad_pin),
},
],
..Default::default()
};
let state = RouterState::from_config(&cfg);
poll_once(&state).await;
let topo = state.topology.read().await;
assert!(
topo["good"].reachable,
"correctly-enrolled cortex reachable"
);
assert!(
!topo["bad"].reachable,
"cortex presenting an unexpected cert is excluded"
);
}
#[tokio::test]
async fn misconfigured_pin_disables_cortex_fail_closed() {
// A `tls_ca` pointing at a nonexistent file must NOT fall back to an
// unpinned client — the cortex is disabled entirely.
let cfg = RouterConfig {
cortexes: vec![
CortexEndpoint {
name: "broken".into(),
endpoint: "https://127.0.0.1:1".into(),
region: None,
tls_ca: Some("/no/such/anchor.pem".into()),
},
CortexEndpoint {
name: "plain".into(),
endpoint: "http://127.0.0.1:1".into(),
region: None,
tls_ca: None,
},
],
..Default::default()
};
let state = RouterState::from_config(&cfg);
assert!(
state.client_for("broken").is_none(),
"a cortex with an unloadable pin is disabled (fail closed)"
);
assert!(
state.client_for("plain").is_some(),
"an un-pinned cortex still gets a client"
);
}
#[test]
fn build_client_rejects_garbage_pem() {
let path = write_pem(
"garbage",
"-----BEGIN CERTIFICATE-----\nnope\n-----END CERTIFICATE-----",
);
assert!(build_client(Some(&path)).is_err());
}

View File

@@ -0,0 +1,172 @@
//! Topology-poller acceptance tests for #72: the router maintains a live
//! map of which cortexes serve which models, marks an unreachable/erroring
//! cortex unhealthy and excludes it from routing, and recovers it once
//! reachable again.
use axum::extract::State;
use axum::http::StatusCode;
use axum::routing::get;
use axum::{Json, Router};
use helexa_router::config::{CortexEndpoint, RouterConfig};
use helexa_router::poller::{POLL_FAILURE_THRESHOLD, poll_once};
use helexa_router::state::{RouterState, entry_feasible};
use serde_json::{Value, json};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use tokio::net::TcpListener;
/// Shared "is this mock cortex up?" flag, toggled by tests to simulate
/// outage and recovery.
#[derive(Clone)]
struct MockState {
up: Arc<AtomicBool>,
}
async fn mock_models(State(s): State<MockState>) -> Result<Json<Value>, StatusCode> {
if !s.up.load(Ordering::SeqCst) {
return Err(StatusCode::SERVICE_UNAVAILABLE);
}
Ok(Json(json!({
"object": "list",
"data": [
{
"id": "Qwen/Qwen3-Coder-30B",
"object": "model",
"created": 0,
"owned_by": "helexa",
"loaded": true,
"feasible_on": ["beast"],
"locations": [{"node": "beast", "status": "loaded", "vram_estimate_mb": 19000}]
},
{
"id": "Qwen/Qwen3-VL-8B",
"object": "model",
"created": 0,
"owned_by": "helexa",
"loaded": false,
"feasible_on": ["beast"],
"locations": []
}
]
})))
}
async fn mock_health(State(s): State<MockState>) -> Result<Json<Value>, StatusCode> {
if !s.up.load(Ordering::SeqCst) {
return Err(StatusCode::SERVICE_UNAVAILABLE);
}
Ok(Json(json!({
"status": "ok",
"nodes": { "healthy": 2, "total": 3 }
})))
}
/// Spawn a mock cortex; returns (base_url, up_flag).
async fn spawn_mock_cortex() -> (String, Arc<AtomicBool>) {
let up = Arc::new(AtomicBool::new(true));
let state = MockState { up: up.clone() };
let app = Router::new()
.route("/v1/models", get(mock_models))
.route("/health", get(mock_health))
.with_state(state);
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
(format!("http://{addr}"), up)
}
fn state_for(name: &str, endpoint: &str) -> RouterState {
let cfg = RouterConfig {
cortexes: vec![CortexEndpoint {
name: name.into(),
endpoint: endpoint.into(),
region: None,
tls_ca: None,
}],
..Default::default()
};
RouterState::from_config(&cfg)
}
#[tokio::test]
async fn poll_builds_live_topology() {
let (base, _up) = spawn_mock_cortex().await;
let state = state_for("c1", &base);
poll_once(&state).await;
let topo = state.topology.read().await;
let c1 = topo.get("c1").expect("cortex present");
assert!(c1.reachable, "should be reachable after a good poll");
assert_eq!(c1.consecutive_failures, 0);
assert!(c1.last_poll.is_some());
assert_eq!((c1.healthy_nodes, c1.total_nodes), (2, 3));
// Loaded model: loaded + feasible. Catalogue-only model: feasible only
// (not loaded, but feasible_on non-empty).
let coder = c1.models.get("Qwen/Qwen3-Coder-30B").unwrap();
assert!(coder.loaded && entry_feasible(coder));
let vl = c1.models.get("Qwen/Qwen3-VL-8B").unwrap();
assert!(!vl.loaded && entry_feasible(vl));
drop(topo);
// The routing helper sees both serveable models on the reachable cortex.
assert_eq!(
state.cortexes_serving("Qwen/Qwen3-VL-8B").await,
vec!["c1".to_string()]
);
}
#[tokio::test]
async fn unreachable_cortex_excluded_then_recovers() {
let (base, up) = spawn_mock_cortex().await;
let state = state_for("c1", &base);
// Healthy first.
poll_once(&state).await;
assert!(state.topology.read().await["c1"].reachable);
// Take it down. The first failures debounce (stay reachable) until the
// threshold; only then is it excluded.
up.store(false, Ordering::SeqCst);
for i in 1..POLL_FAILURE_THRESHOLD {
poll_once(&state).await;
assert!(
state.topology.read().await["c1"].reachable,
"still reachable after {i} failure(s) (below threshold)"
);
}
poll_once(&state).await; // crosses the threshold
{
let topo = state.topology.read().await;
assert!(!topo["c1"].reachable, "excluded after threshold failures");
assert!(topo["c1"].consecutive_failures >= POLL_FAILURE_THRESHOLD);
}
// Excluded from routing.
assert!(
state
.cortexes_serving("Qwen/Qwen3-Coder-30B")
.await
.is_empty()
);
// Bring it back: the next successful poll restores it.
up.store(true, Ordering::SeqCst);
poll_once(&state).await;
let topo = state.topology.read().await;
assert!(topo["c1"].reachable, "recovered after a good poll");
assert_eq!(topo["c1"].consecutive_failures, 0);
}
#[tokio::test]
async fn unconfigured_endpoint_is_unreachable() {
// Nothing listening on this port → polls fail; below threshold it stays
// at its initial unreachable state, and never panics.
let state = state_for("dead", "http://127.0.0.1:1");
poll_once(&state).await;
let topo = state.topology.read().await;
assert!(!topo["dead"].reachable);
assert_eq!(topo["dead"].consecutive_failures, 1);
}

View File

@@ -0,0 +1,21 @@
[package]
name = "helexa-stream"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
[lib]
name = "helexa_stream"
path = "src/lib.rs"
[dependencies]
axum = { workspace = true }
reqwest = { workspace = true }
futures = { workspace = true }
thiserror = { workspace = true }
[dev-dependencies]
tokio = { workspace = true }
tokio-stream = { workspace = true }
async-stream = "0.3"

View File

@@ -0,0 +1,290 @@
//! Shared streaming reverse-proxy mechanism (#71).
//!
//! cortex and helexa-router both need to proxy an OpenAI/Anthropic SSE
//! response from a downstream backend **verbatim** — chunks forwarded as
//! they arrive, never buffering the full body — while observing the bytes
//! for metrics/metering. This crate owns that mechanism so there is one
//! implementation, not one per tier.
//!
//! The split is mechanism vs policy:
//!
//! - **Mechanism (here):** [`forward_streaming`] POSTs to a backend and
//! streams the response body back through an [`ObservedStream`], which
//! feeds every chunk to a caller-supplied [`ChunkObserver`] and calls
//! [`ChunkObserver::finish`] exactly once on clean end-of-stream or on
//! drop (client disconnect mid-stream). [`BodyTail`] and
//! [`last_count_for`] are the reusable pieces an observer uses to pull
//! the trailing OpenAI `usage` object out of the streamed bytes.
//! - **Policy (caller):** what to *do* with the observed bytes — which
//! metric names to emit, which labels, whether to settle a per-principal
//! reservation — lives in the consumer's `ChunkObserver` impl, not here.
//!
//! The proxy is status-agnostic: a non-2xx upstream response (e.g. a
//! cortex `429 rate_limit_exceeded`) is streamed back with its status and
//! headers intact, so honest backpressure reaches the client unchanged.
//! Only a network failure or a malformed response build is an error.
use axum::body::{Body, Bytes};
use axum::http::{HeaderMap, StatusCode};
use axum::response::Response;
use futures::Stream;
use futures::stream::BoxStream;
use reqwest::Client;
use std::pin::Pin;
use std::task::{Context, Poll};
/// Observes the bytes of a streamed proxy response without altering them.
///
/// `observe` is called for each forwarded chunk; `finish` is called
/// exactly once — on clean end-of-stream or on drop — and implementations
/// must be idempotent (the [`ObservedStream`] guards against a double call,
/// but a `finish` that runs side effects should still self-guard).
pub trait ChunkObserver: Send + Unpin + 'static {
/// A body chunk has been forwarded downstream. The slice is the exact
/// bytes the client receives.
fn observe(&mut self, chunk: &[u8]);
/// The stream has ended (cleanly or via client disconnect). Called once.
fn finish(&mut self);
}
/// A bounded accumulator for the tail of a streamed body.
///
/// The OpenAI `usage` object rides on the final SSE chunk (and sits at the
/// end of a non-streaming JSON body), so retaining a generous tail is
/// enough to recover token counts via [`last_count_for`]; the cap bounds
/// memory on huge bodies. Appends are char-boundary-safe.
#[derive(Debug)]
pub struct BodyTail {
tail: String,
cap: usize,
}
impl BodyTail {
/// Create a tail retaining at most `cap` bytes.
pub fn new(cap: usize) -> Self {
Self {
tail: String::new(),
cap,
}
}
/// Append a chunk, trimming from the front past the cap. When trimming,
/// the newest half is kept (the usage object is always at the very end).
pub fn push(&mut self, chunk: &[u8]) {
self.tail.push_str(&String::from_utf8_lossy(chunk));
if self.tail.len() > self.cap {
let mut cut = self.tail.len() - self.cap / 2;
while !self.tail.is_char_boundary(cut) {
cut += 1;
}
self.tail.drain(..cut);
}
}
/// The retained tail text.
pub fn as_str(&self) -> &str {
&self.tail
}
}
/// Find the value of the LAST `"key": <integer>` occurrence in `tail`.
///
/// Pure and chunk-boundary-safe (the tail is contiguous appended text).
/// The quoted-needle form means `completion_tokens` never matches
/// `completion_tokens_details`, and taking the last occurrence means the
/// final `usage` object wins even if content earlier in the stream echoed
/// a usage-shaped string.
pub fn last_count_for(tail: &str, key: &str) -> Option<u64> {
let needle = format!("\"{key}\"");
let mut result = None;
for (idx, _) in tail.match_indices(&needle) {
let rest = tail[idx + needle.len()..].trim_start();
let Some(rest) = rest.strip_prefix(':') else {
continue;
};
let rest = rest.trim_start();
let digits: &str = &rest[..rest
.char_indices()
.find(|(_, c)| !c.is_ascii_digit())
.map(|(i, _)| i)
.unwrap_or(rest.len())];
if let Ok(v) = digits.parse::<u64>() {
result = Some(v);
}
}
result
}
/// Error from [`forward_streaming`]. Distinguishes a network/transport
/// failure reaching the backend from a failure assembling the downstream
/// response. A non-2xx upstream *status* is not an error — it is streamed
/// through verbatim.
#[derive(Debug, thiserror::Error)]
pub enum StreamError {
#[error("upstream request failed")]
Upstream(reqwest::Error),
#[error("failed to build response")]
ResponseBuild(String),
}
/// POST `body` to `url` and stream the response back verbatim through
/// `observer`.
///
/// Request headers are forwarded except `host` / `content-length` (reqwest
/// sets these). The returned [`Response`] carries the upstream status and
/// headers unchanged — including non-2xx — with a body that streams the
/// upstream bytes chunk-for-chunk, feeding each chunk to `observer`.
pub async fn forward_streaming<O: ChunkObserver>(
client: &Client,
url: &str,
headers: HeaderMap,
body: Bytes,
observer: O,
) -> Result<Response, StreamError> {
let mut req_builder = client.post(url).body(body);
for (key, value) in headers.iter() {
if key == "host" || key == "content-length" {
continue; // reqwest sets these
}
req_builder = req_builder.header(key, value);
}
let upstream = req_builder.send().await.map_err(StreamError::Upstream)?;
let status =
StatusCode::from_u16(upstream.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
let resp_headers = upstream.headers().clone();
let stream = ObservedStream::new(Box::pin(upstream.bytes_stream()), observer);
let body = Body::from_stream(stream);
let mut response = Response::builder().status(status);
for (key, value) in resp_headers.iter() {
response = response.header(key, value);
}
response
.body(body)
.map_err(|e| StreamError::ResponseBuild(e.to_string()))
}
/// Pass-through stream wrapper that feeds a [`ChunkObserver`]. Forwards
/// each chunk verbatim, calls `observe` per chunk, and `finish` once on
/// clean end-of-stream; the `Drop` impl covers client disconnects.
pub struct ObservedStream<O: ChunkObserver> {
inner: BoxStream<'static, Result<Bytes, reqwest::Error>>,
observer: O,
finished: bool,
}
impl<O: ChunkObserver> ObservedStream<O> {
/// Wrap a byte stream with an observer.
pub fn new(inner: BoxStream<'static, Result<Bytes, reqwest::Error>>, observer: O) -> Self {
Self {
inner,
observer,
finished: false,
}
}
fn finish(&mut self) {
if self.finished {
return;
}
self.finished = true;
self.observer.finish();
}
}
impl<O: ChunkObserver> Stream for ObservedStream<O> {
type Item = Result<Bytes, reqwest::Error>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = self.get_mut();
match this.inner.as_mut().poll_next(cx) {
Poll::Ready(Some(Ok(chunk))) => {
this.observer.observe(&chunk);
Poll::Ready(Some(Ok(chunk)))
}
Poll::Ready(Some(Err(e))) => Poll::Ready(Some(Err(e))),
Poll::Ready(None) => {
this.finish();
Poll::Ready(None)
}
Poll::Pending => Poll::Pending,
}
}
}
impl<O: ChunkObserver> Drop for ObservedStream<O> {
fn drop(&mut self) {
self.finish();
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_counts_from_final_sse_usage_chunk() {
let tail = concat!(
"data: {\"choices\":[{\"delta\":{\"content\":\"hi\"}}]}\n\n",
"data: {\"choices\":[],\"usage\":{\"prompt_tokens\":225,",
"\"completion_tokens\":42,\"total_tokens\":267}}\n\n",
"data: [DONE]\n\n"
);
assert_eq!(last_count_for(tail, "prompt_tokens"), Some(225));
assert_eq!(last_count_for(tail, "completion_tokens"), Some(42));
}
#[test]
fn extracts_counts_from_non_streaming_body() {
let tail = "{\"choices\":[{\"message\":{\"content\":\"hi\"}}],\
\"usage\":{\"prompt_tokens\": 12, \"completion_tokens\": 7}}";
assert_eq!(last_count_for(tail, "prompt_tokens"), Some(12));
assert_eq!(last_count_for(tail, "completion_tokens"), Some(7));
}
#[test]
fn ignores_details_variants_and_takes_last_occurrence() {
// completion_tokens_details must not shadow completion_tokens,
// and the LAST usage object wins (matters when content echoes
// a usage-shaped string earlier in the stream).
let tail = concat!(
"data: {\"usage\":{\"completion_tokens\":1}}\n\n",
"data: {\"usage\":{\"completion_tokens\":99,",
"\"completion_tokens_details\":{\"reasoning_tokens\":3}}}\n\n"
);
assert_eq!(last_count_for(tail, "completion_tokens"), Some(99));
}
#[test]
fn absent_keys_yield_none() {
assert_eq!(
last_count_for("data: [DONE]\n\n", "completion_tokens"),
None
);
assert_eq!(last_count_for("", "prompt_tokens"), None);
// key present but non-numeric value
assert_eq!(
last_count_for("\"completion_tokens\": null", "completion_tokens"),
None
);
}
#[test]
fn body_tail_retains_usage_after_cap_trim() {
// Cap small enough that the filler forces several front-trims, but
// (as in production, where cap ≫ the usage object) large enough that
// the trailing usage object survives the newest-half retention.
let mut tail = BodyTail::new(512);
for _ in 0..100 {
tail.push(b"data: {\"choices\":[{\"delta\":{\"content\":\"x\"}}]}\n\n");
}
assert!(tail.as_str().len() <= 512, "cap must bound the tail");
tail.push(b"data: {\"usage\":{\"prompt_tokens\":5,\"completion_tokens\":9}}\n\n");
assert_eq!(last_count_for(tail.as_str(), "prompt_tokens"), Some(5));
assert_eq!(last_count_for(tail.as_str(), "completion_tokens"), Some(9));
}
}

View File

@@ -0,0 +1,162 @@
//! Integration tests for the shared streaming proxy (#71): proves a backend
//! SSE response is forwarded chunk-for-chunk (no buffering), the observer
//! sees every byte and finishes once, and non-2xx is streamed through with
//! its status intact — the behaviours both cortex and helexa-router rely on.
use axum::Router;
use axum::body::Body;
use axum::http::{HeaderMap, StatusCode};
use axum::response::Response;
use axum::routing::post;
use helexa_stream::{BodyTail, ChunkObserver, forward_streaming, last_count_for};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::TcpListener;
/// Observer that records what it saw, for assertions.
#[derive(Clone, Default)]
struct RecordingObserver {
inner: Arc<Mutex<Recorded>>,
}
#[derive(Default)]
struct Recorded {
chunks: usize,
finished: usize,
tail: String,
}
impl ChunkObserver for RecordingObserver {
fn observe(&mut self, chunk: &[u8]) {
let mut r = self.inner.lock().unwrap();
r.chunks += 1;
r.tail.push_str(&String::from_utf8_lossy(chunk));
}
fn finish(&mut self) {
self.inner.lock().unwrap().finished += 1;
}
}
/// Mock backend that streams 5 SSE chunks with 30ms gaps, then a usage
/// chunk and `[DONE]`.
async fn sse_handler() -> Response {
let chunks: Vec<&'static str> = vec![
"data: {\"choices\":[{\"delta\":{\"content\":\"a\"}}]}\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"b\"}}]}\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"c\"}}]}\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"d\"}}]}\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"e\"}}]}\n\n",
"data: {\"choices\":[],\"usage\":{\"prompt_tokens\":11,\"completion_tokens\":5}}\n\n",
"data: [DONE]\n\n",
];
let stream = async_stream::stream! {
for c in chunks {
tokio::time::sleep(Duration::from_millis(30)).await;
yield Ok::<_, std::io::Error>(axum::body::Bytes::from_static(c.as_bytes()));
}
};
Response::new(Body::from_stream(stream))
}
async fn rate_limited_handler() -> Response {
Response::builder()
.status(StatusCode::TOO_MANY_REQUESTS)
.body(Body::from("{\"error\":{\"type\":\"rate_limit_exceeded\"}}"))
.unwrap()
}
async fn spawn_backend(router: Router) -> String {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, router).await.unwrap();
});
format!("http://{addr}")
}
#[tokio::test]
async fn streams_chunks_incrementally_and_observes_usage() {
let base = spawn_backend(Router::new().route("/v1/chat/completions", post(sse_handler))).await;
let observer = RecordingObserver::default();
let probe = observer.clone();
let client = reqwest::Client::new();
let resp = forward_streaming(
&client,
&format!("{base}/v1/chat/completions"),
HeaderMap::new(),
axum::body::Bytes::from_static(b"{\"model\":\"x\",\"stream\":true}"),
observer,
)
.await
.expect("forward ok");
assert_eq!(resp.status(), StatusCode::OK);
// Read the proxied body as a stream, timestamping arrivals.
let mut body = resp.into_body().into_data_stream();
let mut arrivals: Vec<Instant> = Vec::new();
let mut collected = String::new();
use futures::StreamExt;
while let Some(item) = body.next().await {
let bytes = item.unwrap();
arrivals.push(Instant::now());
collected.push_str(&String::from_utf8_lossy(&bytes));
}
// Incremental delivery: first and last chunk are meaningfully apart
// (5×30ms gaps), proving no full-response buffering.
let spread = *arrivals.last().unwrap() - arrivals[0];
assert!(
spread >= Duration::from_millis(100),
"expected incremental delivery, spread was {spread:?}"
);
// The client received the terminator and the usage object verbatim.
assert!(collected.contains("data: [DONE]"));
// The observer saw the bytes and finished exactly once.
let r = probe.inner.lock().unwrap();
assert!(r.chunks >= 5, "observer saw {} chunks", r.chunks);
assert_eq!(r.finished, 1, "finish must run exactly once");
assert_eq!(last_count_for(&r.tail, "prompt_tokens"), Some(11));
assert_eq!(last_count_for(&r.tail, "completion_tokens"), Some(5));
}
#[tokio::test]
async fn non_2xx_is_streamed_through_verbatim() {
let base =
spawn_backend(Router::new().route("/v1/chat/completions", post(rate_limited_handler)))
.await;
let observer = RecordingObserver::default();
let probe = observer.clone();
let client = reqwest::Client::new();
let resp = forward_streaming(
&client,
&format!("{base}/v1/chat/completions"),
HeaderMap::new(),
axum::body::Bytes::new(),
observer,
)
.await
.expect("forward ok");
// Backpressure status reaches the client unchanged.
assert_eq!(resp.status(), StatusCode::TOO_MANY_REQUESTS);
let body = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.unwrap();
assert!(String::from_utf8_lossy(&body).contains("rate_limit_exceeded"));
// finish still runs once even with a tiny non-streaming body.
assert_eq!(probe.inner.lock().unwrap().finished, 1);
}
#[test]
fn body_tail_smoke() {
let mut tail = BodyTail::new(128);
tail.push(b"hello ");
tail.push(b"world");
assert_eq!(tail.as_str(), "hello world");
}

View File

@@ -0,0 +1,63 @@
[package]
name = "helexa-upstream"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
[[bin]]
name = "helexa-upstream"
path = "src/main.rs"
[lib]
name = "helexa_upstream"
path = "src/lib.rs"
[dependencies]
tokio = { workspace = true }
axum = { workspace = true }
tower-http = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
figment = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
chrono = { workspace = true }
# PostgreSQL — the mesh authority's system of record. Runtime query API
# (not the compile-time `query!` macros) so the crate builds in CI without a
# live database or a committed `.sqlx` offline cache; correctness is covered
# by the gated integration tests. (Macro adoption is a later refinement once
# a dev DB + offline cache exist.)
sqlx = { version = "0.8", default-features = false, features = [
"runtime-tokio",
"tls-rustls",
"postgres",
"macros",
"migrate",
"chrono",
"uuid",
] }
uuid = { version = "1", features = ["v4", "serde"] }
sha2 = "0.10"
subtle = "2.6"
# Web auth (B4): argon2id password hashing, JWT sessions, CSPRNG secrets,
# transactional email.
argon2 = "0.5"
jsonwebtoken = "9"
rand = "0.8"
lettre = { version = "0.11", default-features = false, features = [
"tokio1-rustls-tls",
"smtp-transport",
"builder",
] }
# cortex-core for the shared #63 OpenAiError envelope on the authz surface.
cortex-core = { workspace = true }
[dev-dependencies]
figment = { workspace = true, features = ["test"] }
reqwest = { workspace = true }

View File

@@ -0,0 +1,137 @@
-- helexa-upstream initial schema (#59): accounts, keys, ledger, top-up
-- codes, served-usage. The mesh-level authority's system of record.
--
-- Token amounts are BIGINT (i64) throughout; the cortex EntitlementProvider
-- carries u64 but mesh allocations sit comfortably inside i64 and Postgres
-- has no unsigned type.
CREATE EXTENSION IF NOT EXISTS citext;
CREATE EXTENSION IF NOT EXISTS pgcrypto;
-- ── Users (web auth: email + password) ──────────────────────────────
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email CITEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL, -- argon2id PHC string
email_verified BOOLEAN NOT NULL DEFAULT FALSE,
-- Browser fingerprint captured at registration (#abuse). Best-effort,
-- client-supplied; the primary signal for silent multi-account
-- detection. NULL when the client could not produce one.
registration_fingerprint TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX users_registration_fingerprint_idx
ON users (registration_fingerprint)
WHERE registration_fingerprint IS NOT NULL;
-- Single-use email tokens for verification and password reset. Only the
-- sha256 of the emailed secret is stored.
CREATE TABLE email_tokens (
token_hash BYTEA PRIMARY KEY,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('verify', 'reset')),
expires_at TIMESTAMPTZ NOT NULL,
consumed_at TIMESTAMPTZ
);
CREATE INDEX email_tokens_user_idx ON email_tokens (user_id);
-- ── Accounts (the billable allocation ledger) ───────────────────────
CREATE TABLE accounts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
owner_user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
allocation_total BIGINT NOT NULL DEFAULT 0,
allocation_spent BIGINT NOT NULL DEFAULT 0,
allocation_reserved BIGINT NOT NULL DEFAULT 0,
-- 'deactivated' is the SILENT abuse flag: keys stop authorizing but no
-- surface ever tells the user why (see resolve → 401).
status TEXT NOT NULL DEFAULT 'active'
CHECK (status IN ('active', 'deactivated')),
-- This account shares a registration fingerprint with >= 1 other.
fingerprint_flagged BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- The no-overshoot backstop to the atomic reserve UPDATE.
CONSTRAINT accounts_no_overshoot
CHECK (allocation_spent + allocation_reserved <= allocation_total),
CONSTRAINT accounts_nonneg
CHECK (allocation_spent >= 0 AND allocation_reserved >= 0)
);
CREATE INDEX accounts_owner_idx ON accounts (owner_user_id);
-- ── API keys (Principal.key_id = api_keys.id) ───────────────────────
CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
key_hash BYTEA NOT NULL, -- sha256(raw key)
key_prefix TEXT NOT NULL, -- non-secret display prefix
label TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT 'active'
CHECK (status IN ('active', 'archived')),
-- Per-key sub-cap: 'hardcap' = absolute tokens; 'percent' = % of the
-- account's allocation_total (resolved to an absolute at reserve time).
limit_kind TEXT NOT NULL DEFAULT 'percent'
CHECK (limit_kind IN ('percent', 'hardcap')),
limit_value BIGINT NOT NULL DEFAULT 100,
-- serde of cortex_core::entitlements::CapWindow (Balance | Rolling).
cap_window JSONB NOT NULL DEFAULT '{"kind":"balance"}'::jsonb,
-- Per-key running ledger (mirrors the account ledger; Balance semantics
-- in this migration — rolling-window reset lands with the authz API).
key_spent BIGINT NOT NULL DEFAULT 0,
key_reserved BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT api_keys_key_nonneg
CHECK (key_spent >= 0 AND key_reserved >= 0)
);
-- A raw key resolves only while active; the hash is unique among active keys.
CREATE UNIQUE INDEX api_keys_active_hash_idx
ON api_keys (key_hash) WHERE status = 'active';
CREATE INDEX api_keys_account_idx ON api_keys (account_id);
-- ── Reservations (reserve → settle/release) ─────────────────────────
-- id is BIGSERIAL so it maps to the cortex Reservation.id (u64) verbatim,
-- with the Postgres sequence as the sole global authority.
CREATE TABLE reservations (
id BIGSERIAL PRIMARY KEY,
account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
key_id UUID NOT NULL REFERENCES api_keys(id) ON DELETE CASCADE,
reserved BIGINT NOT NULL,
actual BIGINT,
state TEXT NOT NULL DEFAULT 'open'
CHECK (state IN ('open', 'settled', 'released')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
settled_at TIMESTAMPTZ
);
-- The sweeper scans open reservations by age.
CREATE INDEX reservations_open_idx
ON reservations (created_at) WHERE state = 'open';
-- ── Top-up codes (hybrid allocation) ────────────────────────────────
CREATE TABLE top_up_codes (
code_hash BYTEA PRIMARY KEY, -- sha256(raw code)
value BIGINT NOT NULL, -- tokens this code grants
denomination TEXT, -- human label (e.g. "small")
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
redeemed_by UUID REFERENCES accounts(id) ON DELETE SET NULL,
redeemed_at TIMESTAMPTZ
);
-- ── Served-usage ledger (#58 reconciliation) ────────────────────────
-- Absolute per-(operator, account, key, period) served tokens, upserted by
-- each cortex; reconciliation rolls these up for operator compensation.
CREATE TABLE served_usage (
operator_id TEXT NOT NULL,
account_id UUID NOT NULL,
key_id UUID NOT NULL,
period DATE NOT NULL,
served_tokens BIGINT NOT NULL DEFAULT 0,
reconciled_at TIMESTAMPTZ,
PRIMARY KEY (operator_id, account_id, key_id, period)
);
-- ── Web sessions (DB-backed; alt/complement to stateless JWT) ───────
CREATE TABLE sessions (
token_hash BYTEA PRIMARY KEY, -- sha256(session token)
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX sessions_user_idx ON sessions (user_id);

View File

@@ -0,0 +1,340 @@
//! `/authz/v1` — the machine surface cortex's `UpstreamEntitlementProvider`
//! (#57) consumes. It mirrors the `cortex_core::entitlements::EntitlementProvider`
//! trait 1:1 (resolve / reserve / settle / release / snapshot) over the B1
//! ledger.
//!
//! Contract notes for the cortex client:
//! - A **non-2xx** response means the authority could not give an
//! authoritative answer (bad caller auth, malformed request, server
//! error) → the client should **fail closed**.
//! - `reserve` returns **200** whether granted or budget-refused: the body
//! carries either `reservation_id` or a `rejected` discriminant. A budget
//! refusal is an authoritative answer, not a transport failure.
//! - Rejections that are genuinely auth failures use the #63 `OpenAiError`
//! envelope so they can be surfaced verbatim.
use crate::crypto::sha256;
use crate::error::envelope_response;
use crate::ledger::{self, LedgerError};
use crate::state::AppState;
use axum::extract::{Request, State};
use axum::http::{StatusCode, header};
use axum::middleware::Next;
use axum::response::{IntoResponse, Response};
use axum::routing::post;
use axum::{Extension, Json, Router};
use cortex_core::error_envelope::OpenAiError;
use serde::{Deserialize, Serialize};
use subtle::ConstantTimeEq;
use uuid::Uuid;
/// The operator a validated client bearer identifies (served-usage
/// attribution, #58). Inserted into request extensions by [`client_auth`].
#[derive(Debug, Clone)]
pub struct OperatorId(pub String);
/// Build the `/authz/v1` router with the client-auth layer applied.
pub fn router(state: &AppState) -> Router<AppState> {
Router::new()
.route("/authz/v1/resolve", post(resolve))
.route("/authz/v1/reserve", post(reserve))
.route("/authz/v1/settle", post(settle))
.route("/authz/v1/release", post(release))
.route("/authz/v1/snapshot", post(snapshot))
.route("/authz/v1/served-usage", post(served_usage))
.layer(axum::middleware::from_fn_with_state(
state.clone(),
client_auth,
))
}
// ── client auth (shared bearer → operator_id) ───────────────────────
/// Validate the caller's `Authorization: Bearer` against the configured
/// client tokens (constant-time) and stamp the `operator_id`. When no tokens
/// are configured the surface is open (dev) and a synthetic operator is
/// used.
async fn client_auth(State(state): State<AppState>, mut req: Request, next: Next) -> Response {
let tokens = &state.config.client_auth.tokens;
if tokens.is_empty() {
req.extensions_mut().insert(OperatorId("dev".into()));
return next.run(req).await;
}
let presented = req
.headers()
.get(header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
.map(str::trim)
.unwrap_or("");
let matched = tokens
.iter()
.find(|t| t.token.as_bytes().ct_eq(presented.as_bytes()).into());
match matched {
Some(t) => {
req.extensions_mut()
.insert(OperatorId(t.operator_id.clone()));
next.run(req).await
}
None => envelope_response(OpenAiError::invalid_api_key(
"missing or invalid client credentials",
)),
}
}
// ── DTOs ────────────────────────────────────────────────────────────
#[derive(Deserialize)]
struct ResolveReq {
api_key: String,
}
#[derive(Serialize)]
struct PrincipalDto {
account_id: String,
key_id: String,
}
#[derive(Serialize)]
struct SnapshotDto {
hard_cap: Option<i64>,
spent: i64,
reserved: i64,
}
#[derive(Serialize)]
struct ResolveResp {
principal: PrincipalDto,
snapshot: SnapshotDto,
}
#[derive(Deserialize)]
struct ReserveReq {
account_id: String,
key_id: String,
max_tokens: i64,
}
#[derive(Serialize, Default)]
struct ReserveResp {
#[serde(skip_serializing_if = "Option::is_none")]
reservation_id: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
rejected: Option<Rejection>,
}
#[derive(Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
enum Rejection {
InsufficientQuota {
requested: i64,
available: i64,
},
// Part of the frozen wire contract so the cortex client (#57) can map it
// without a later breaking change. Not yet constructed: the B1 ledger
// implements Balance caps only; rolling-window key sub-caps (which yield
// this) land in a follow-up.
#[allow(dead_code)]
RateLimited {
requested: i64,
available: i64,
retry_after_secs: u64,
},
}
#[derive(Deserialize)]
struct SettleReq {
reservation_id: i64,
actual_tokens: i64,
}
#[derive(Deserialize)]
struct ReservationRef {
reservation_id: i64,
}
#[derive(Deserialize)]
struct SnapshotReq {
account_id: String,
key_id: String,
}
// ── handlers ────────────────────────────────────────────────────────
/// `POST /authz/v1/resolve` — bearer key → principal + snapshot, or
/// `401 invalid_api_key` (also for a deactivated account: no clue).
async fn resolve(State(state): State<AppState>, Json(req): Json<ResolveReq>) -> Response {
match ledger::resolve_key(&state.pool, &sha256(&req.api_key)).await {
Ok(Some(p)) => Json(ResolveResp {
principal: PrincipalDto {
account_id: p.account_id.to_string(),
key_id: p.key_id.to_string(),
},
snapshot: SnapshotDto {
hard_cap: Some(p.hard_cap),
spent: p.key_spent,
reserved: p.key_reserved,
},
})
.into_response(),
Ok(None) => envelope_response(OpenAiError::invalid_api_key("invalid or unknown API key")),
Err(e) => {
tracing::error!(error = %e, "resolve query failed");
envelope_response(OpenAiError::service_unavailable("authority error", Some(5)))
}
}
}
/// `POST /authz/v1/reserve` — 200 with `reservation_id` (granted) or
/// `rejected` (budget). Non-2xx only for bad input / server error.
async fn reserve(State(state): State<AppState>, Json(req): Json<ReserveReq>) -> Response {
let (Ok(account_id), Ok(key_id)) = (
Uuid::parse_str(&req.account_id),
Uuid::parse_str(&req.key_id),
) else {
return bad_request("account_id and key_id must be UUIDs");
};
match ledger::reserve(&state.pool, account_id, key_id, req.max_tokens).await {
Ok(reservation_id) => Json(ReserveResp {
reservation_id: Some(reservation_id),
rejected: None,
})
.into_response(),
Err(LedgerError::InsufficientQuota {
requested,
available,
}) => Json(ReserveResp {
reservation_id: None,
rejected: Some(Rejection::InsufficientQuota {
requested,
available,
}),
})
.into_response(),
Err(LedgerError::AccountNotFound | LedgerError::KeyNotFound) => {
// Resolve succeeded earlier; the principal vanished (archived /
// deactivated). Treat as no budget — fail closed at the client.
Json(ReserveResp {
reservation_id: None,
rejected: Some(Rejection::InsufficientQuota {
requested: req.max_tokens,
available: 0,
}),
})
.into_response()
}
Err(LedgerError::Db(e)) => {
tracing::error!(error = %e, "reserve failed");
envelope_response(OpenAiError::service_unavailable("authority error", Some(5)))
}
}
}
/// `POST /authz/v1/settle` — idempotent; `204`.
async fn settle(State(state): State<AppState>, Json(req): Json<SettleReq>) -> Response {
match ledger::settle(&state.pool, req.reservation_id, req.actual_tokens).await {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(e) => {
tracing::error!(error = %e, "settle failed");
envelope_response(OpenAiError::service_unavailable("authority error", Some(5)))
}
}
}
/// `POST /authz/v1/release` — idempotent; `204`.
async fn release(State(state): State<AppState>, Json(req): Json<ReservationRef>) -> Response {
match ledger::release(&state.pool, req.reservation_id).await {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(e) => {
tracing::error!(error = %e, "release failed");
envelope_response(OpenAiError::service_unavailable("authority error", Some(5)))
}
}
}
/// `POST /authz/v1/snapshot` — `{hard_cap, spent, reserved}` or `404`.
async fn snapshot(State(state): State<AppState>, Json(req): Json<SnapshotReq>) -> Response {
let (Ok(account_id), Ok(key_id)) = (
Uuid::parse_str(&req.account_id),
Uuid::parse_str(&req.key_id),
) else {
return bad_request("account_id and key_id must be UUIDs");
};
match ledger::snapshot(&state.pool, account_id, key_id).await {
Ok(Some((hard_cap, spent, reserved))) => Json(SnapshotDto {
hard_cap: Some(hard_cap),
spent,
reserved,
})
.into_response(),
Ok(None) => StatusCode::NOT_FOUND.into_response(),
Err(e) => {
tracing::error!(error = %e, "snapshot failed");
envelope_response(OpenAiError::service_unavailable("authority error", Some(5)))
}
}
}
// ── served-usage report (#58) ───────────────────────────────────────
#[derive(Deserialize)]
struct ServedUsageReport {
rows: Vec<ServedUsageRow>,
}
#[derive(Deserialize)]
struct ServedUsageRow {
account_id: String,
key_id: String,
period: String, // YYYY-MM-DD
served_tokens: i64,
}
/// `POST /authz/v1/served-usage` — a cortex reports the absolute served-token
/// counters it has accrued for the current period. Upsert is monotonic
/// (`GREATEST`) so re-sends and races are idempotent and never regress.
/// `operator_id` comes from the validated client bearer (request extension).
async fn served_usage(
State(state): State<AppState>,
Extension(operator): Extension<OperatorId>,
Json(req): Json<ServedUsageReport>,
) -> Response {
for row in &req.rows {
let (Ok(account_id), Ok(key_id)) = (
Uuid::parse_str(&row.account_id),
Uuid::parse_str(&row.key_id),
) else {
continue; // skip malformed ids rather than fail the whole batch
};
let Ok(period) = chrono::NaiveDate::parse_from_str(&row.period, "%Y-%m-%d") else {
continue;
};
let res = sqlx::query(
"INSERT INTO served_usage (operator_id, account_id, key_id, period, served_tokens) \
VALUES ($1, $2, $3, $4, $5) \
ON CONFLICT (operator_id, account_id, key_id, period) \
DO UPDATE SET served_tokens = GREATEST(served_usage.served_tokens, EXCLUDED.served_tokens)",
)
.bind(&operator.0)
.bind(account_id)
.bind(key_id)
.bind(period)
.bind(row.served_tokens.max(0))
.execute(&state.pool)
.await;
if let Err(e) = res {
tracing::error!(error = %e, "served-usage upsert failed");
return envelope_response(OpenAiError::service_unavailable("authority error", Some(5)));
}
}
StatusCode::NO_CONTENT.into_response()
}
fn bad_request(msg: &str) -> Response {
envelope_response(OpenAiError::new(
400,
"invalid_request_error",
"invalid_request",
msg,
))
}

View File

@@ -0,0 +1,261 @@
//! helexa-upstream configuration: loaded from `helexa-upstream.toml` with
//! figment, `UPSTREAM_`-prefixed env overrides (mirrors the cortex/router
//! convention, e.g. `UPSTREAM_SERVER__LISTEN`, `UPSTREAM_DB__URL`).
use figment::{
Figment,
providers::{Env, Format, Toml},
};
use serde::{Deserialize, Serialize};
use std::path::Path;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct UpstreamConfig {
#[serde(default)]
pub server: ServerSettings,
pub db: DbSettings,
#[serde(default)]
pub grant: GrantSettings,
#[serde(default)]
pub abuse: AbuseSettings,
#[serde(default)]
pub client_auth: ClientAuthSettings,
#[serde(default)]
pub authz: AuthzSettings,
#[serde(default)]
pub auth: AuthSettings,
#[serde(default)]
pub email: EmailSettings,
}
/// `[auth]` — web-session signing + token lifetimes (B4). Web sessions are
/// JWTs, distinct from inference API keys.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AuthSettings {
/// HMAC secret for signing session JWTs. MUST be overridden in prod
/// (env `UPSTREAM_AUTH__JWT_SECRET`); the default is dev-only.
#[serde(default = "default_jwt_secret")]
pub jwt_secret: String,
/// Session token lifetime (seconds).
#[serde(default = "default_session_ttl")]
pub session_ttl_secs: u64,
/// Email verification / password-reset token lifetime (seconds).
#[serde(default = "default_email_token_ttl")]
pub email_token_ttl_secs: u64,
/// Public base URL of the frontend, used to build verify/reset links.
#[serde(default = "default_app_base_url")]
pub app_base_url: String,
}
impl Default for AuthSettings {
fn default() -> Self {
Self {
jwt_secret: default_jwt_secret(),
session_ttl_secs: default_session_ttl(),
email_token_ttl_secs: default_email_token_ttl(),
app_base_url: default_app_base_url(),
}
}
}
/// `[email]` — transactional email transport for verify/reset.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EmailSettings {
/// `"log"` (dev — logs the link) or `"smtp"`.
#[serde(default = "default_email_provider")]
pub provider: String,
/// SMTP relay URL (e.g. "smtp://user:pass@host:587") when provider=smtp.
#[serde(default)]
pub smtp_url: Option<String>,
/// `From:` address.
#[serde(default = "default_from_addr")]
pub from_addr: String,
}
impl Default for EmailSettings {
fn default() -> Self {
Self {
provider: default_email_provider(),
smtp_url: None,
from_addr: default_from_addr(),
}
}
}
/// `[client_auth]` — credentials operators' cortexes present to `/authz/v1`.
/// Each token maps to an `operator_id` (served-usage attribution, #58). This
/// transport credential is distinct from end-user API keys (which ride in
/// the `resolve` body). v2 adds mTLS.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ClientAuthSettings {
/// When empty the authz surface is **open** (dev only; logged at warn).
#[serde(default)]
pub tokens: Vec<ClientToken>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ClientToken {
/// Shared bearer a cortex presents.
pub token: String,
/// Operator this token identifies.
pub operator_id: String,
}
/// `[authz]` — reservation lifecycle knobs.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AuthzSettings {
/// Open reservations older than this are swept (released), self-healing
/// a reservation whose settle/release from cortex was lost.
#[serde(default = "default_reservation_ttl")]
pub reservation_ttl_secs: u64,
/// How often the sweeper runs.
#[serde(default = "default_sweep_interval")]
pub sweep_interval_secs: u64,
}
impl Default for AuthzSettings {
fn default() -> Self {
Self {
reservation_ttl_secs: default_reservation_ttl(),
sweep_interval_secs: default_sweep_interval(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerSettings {
/// Address to listen on (e.g. "0.0.0.0:8090"). Plaintext — edge nginx
/// terminates TLS, consistent with the rest of the stack.
#[serde(default = "default_listen")]
pub listen: String,
}
impl Default for ServerSettings {
fn default() -> Self {
Self {
listen: default_listen(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DbSettings {
/// PostgreSQL connection URL (e.g. "postgres://user:pass@host/helexa").
pub url: String,
/// Max pool connections.
#[serde(default = "default_max_connections")]
pub max_connections: u32,
}
/// `[grant]` — the flat free token grant every email-verified account
/// receives (the floor of the hybrid allocation model; top-up codes extend
/// it).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GrantSettings {
#[serde(default = "default_free_grant")]
pub free_token_grant: i64,
}
impl Default for GrantSettings {
fn default() -> Self {
Self {
free_token_grant: default_free_grant(),
}
}
}
/// `[abuse]` — silent multi-account abuse detection. When at least
/// `fingerprint_account_threshold` accounts share one registration
/// fingerprint, all of them are silently deactivated (no notice to the
/// user; deactivation only surfaces as ordinary inference rejections).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AbuseSettings {
#[serde(default = "default_fingerprint_threshold")]
pub fingerprint_account_threshold: i64,
}
impl Default for AbuseSettings {
fn default() -> Self {
Self {
fingerprint_account_threshold: default_fingerprint_threshold(),
}
}
}
fn default_listen() -> String {
"0.0.0.0:8090".into()
}
fn default_max_connections() -> u32 {
16
}
fn default_free_grant() -> i64 {
1_000_000
}
fn default_fingerprint_threshold() -> i64 {
5
}
fn default_reservation_ttl() -> u64 {
120
}
fn default_sweep_interval() -> u64 {
60
}
fn default_jwt_secret() -> String {
"dev-insecure-change-me".into()
}
fn default_session_ttl() -> u64 {
7 * 24 * 3600
}
fn default_email_token_ttl() -> u64 {
24 * 3600
}
fn default_app_base_url() -> String {
"http://localhost:5173".into()
}
fn default_email_provider() -> String {
"log".into()
}
fn default_from_addr() -> String {
"helexa <no-reply@helexa.ai>".into()
}
impl UpstreamConfig {
/// Load from a TOML file with `UPSTREAM_`-prefixed env overrides
/// (`__` nesting separator).
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
Figment::new()
.merge(Toml::file(path))
.merge(Env::prefixed("UPSTREAM_").split("__"))
.extract()
.map_err(Box::new)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
#[allow(clippy::result_large_err)]
fn loads_toml_with_env_override_and_defaults() {
figment::Jail::expect_with(|jail| {
jail.create_file(
"helexa-upstream.toml",
r#"
[db]
url = "postgres://localhost/helexa"
"#,
)?;
jail.set_env("UPSTREAM_SERVER__LISTEN", "127.0.0.1:9099");
let cfg = UpstreamConfig::load("helexa-upstream.toml").expect("load");
assert_eq!(cfg.server.listen, "127.0.0.1:9099");
assert_eq!(cfg.db.url, "postgres://localhost/helexa");
// Defaults applied when sections omitted.
assert_eq!(cfg.grant.free_token_grant, 1_000_000);
assert_eq!(cfg.abuse.fingerprint_account_threshold, 5);
assert_eq!(cfg.db.max_connections, 16);
Ok(())
});
}
}

View File

@@ -0,0 +1,113 @@
//! Hashing + secret-generation helpers.
//!
//! - **Passwords** (low-entropy) → argon2id PHC strings.
//! - **API keys / top-up codes / email + session tokens** (high-entropy
//! secrets minted here) → stored only as their sha256; sha256 is the fast,
//! sufficient choice for high-entropy material.
use argon2::Argon2;
use argon2::password_hash::rand_core::OsRng as ArgonOsRng;
use argon2::password_hash::{PasswordHash, PasswordHasher, PasswordVerifier, SaltString};
use rand::RngCore;
use sha2::{Digest, Sha256};
/// sha256 of `input`, as raw bytes (matches the `BYTEA` columns).
pub fn sha256(input: &str) -> Vec<u8> {
let mut h = Sha256::new();
h.update(input.as_bytes());
h.finalize().to_vec()
}
/// Hash a password with argon2id, returning a PHC string for storage.
pub fn hash_password(password: &str) -> Result<String, argon2::password_hash::Error> {
let salt = SaltString::generate(&mut ArgonOsRng);
Ok(Argon2::default()
.hash_password(password.as_bytes(), &salt)?
.to_string())
}
/// Verify a password against a stored PHC hash. `false` on any mismatch or
/// malformed hash (never panics).
pub fn verify_password(password: &str, phc: &str) -> bool {
match PasswordHash::new(phc) {
Ok(parsed) => Argon2::default()
.verify_password(password.as_bytes(), &parsed)
.is_ok(),
Err(_) => false,
}
}
/// A fresh URL-safe high-entropy secret (256 bits) for email/session/reset
/// tokens. The caller stores only `sha256` of this and emails/returns the
/// raw value.
pub fn random_token() -> String {
let mut bytes = [0u8; 32];
rand::rngs::OsRng.fill_bytes(&mut bytes);
base62(&bytes)
}
/// Mint a new API key: `(raw, prefix)`. `raw` is shown to the user once;
/// only `sha256(raw)` is stored. The prefix is a non-secret display tag.
pub fn generate_api_key() -> (String, String) {
let mut bytes = [0u8; 32];
rand::rngs::OsRng.fill_bytes(&mut bytes);
let raw = format!("sk-helexa-{}", base62(&bytes));
// Non-secret prefix for the dashboard list (scheme + first few chars).
let prefix: String = raw.chars().take(14).collect();
(raw, prefix)
}
/// base62 encode (0-9A-Za-z) — URL/clipboard friendly, no padding.
fn base62(bytes: &[u8]) -> String {
const ALPHABET: &[u8] = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
// Treat the bytes as a big-endian integer and base62 it. 32 bytes → ~43
// chars. Simple repeated-division over a big-uint built from the bytes.
let mut digits: Vec<u8> = vec![0];
for &byte in bytes {
let mut carry = byte as u32;
for d in digits.iter_mut() {
let v = (*d as u32) * 256 + carry;
*d = (v % 62) as u8;
carry = v / 62;
}
while carry > 0 {
digits.push((carry % 62) as u8);
carry /= 62;
}
}
digits
.iter()
.rev()
.map(|&d| ALPHABET[d as usize] as char)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn password_round_trips_and_rejects_wrong() {
let phc = hash_password("correct horse").unwrap();
assert!(verify_password("correct horse", &phc));
assert!(!verify_password("wrong", &phc));
assert!(!verify_password("correct horse", "not-a-phc-string"));
}
#[test]
fn api_key_has_scheme_prefix_and_unique_body() {
let (raw, prefix) = generate_api_key();
assert!(raw.starts_with("sk-helexa-"));
assert!(prefix.starts_with("sk-helexa-"));
let (raw2, _) = generate_api_key();
assert_ne!(raw, raw2, "keys are unique");
}
#[test]
fn random_tokens_are_unique_and_nonempty() {
let a = random_token();
let b = random_token();
assert!(!a.is_empty());
assert_ne!(a, b);
}
}

View File

@@ -0,0 +1,20 @@
//! PostgreSQL pool + embedded migrations.
use anyhow::{Context, Result};
use sqlx::postgres::{PgPool, PgPoolOptions};
/// Connect to Postgres and run embedded migrations (`./migrations`).
pub async fn connect_and_migrate(url: &str, max_connections: u32) -> Result<PgPool> {
let pool = PgPoolOptions::new()
.max_connections(max_connections)
.connect(url)
.await
.with_context(|| "connecting to PostgreSQL")?;
sqlx::migrate!("./migrations")
.run(&pool)
.await
.with_context(|| "running migrations")?;
Ok(pool)
}

View File

@@ -0,0 +1,64 @@
//! Transactional email for verification + password-reset links.
//!
//! Two transports: `Log` (dev — writes the link to the log so flows are
//! testable without a relay) and `Smtp` (lettre over rustls). Built from
//! `[email]` config.
use crate::config::EmailSettings;
use anyhow::{Context, Result};
use lettre::message::Mailbox;
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
#[derive(Clone)]
pub enum EmailSender {
/// Dev: log the message instead of sending.
Log { from: String },
Smtp {
from: String,
transport: AsyncSmtpTransport<Tokio1Executor>,
},
}
impl EmailSender {
pub fn from_config(cfg: &EmailSettings) -> Result<Self> {
match cfg.provider.as_str() {
"smtp" => {
let url = cfg
.smtp_url
.as_deref()
.context("[email].smtp_url required when provider = \"smtp\"")?;
let transport = AsyncSmtpTransport::<Tokio1Executor>::from_url(url)
.context("parsing [email].smtp_url")?
.build();
Ok(EmailSender::Smtp {
from: cfg.from_addr.clone(),
transport,
})
}
_ => Ok(EmailSender::Log {
from: cfg.from_addr.clone(),
}),
}
}
/// Send a plaintext email. Errors are returned but the caller treats
/// send failures as non-fatal to the request (the user can re-request).
pub async fn send(&self, to: &str, subject: &str, body: &str) -> Result<()> {
match self {
EmailSender::Log { from } => {
tracing::info!(%from, %to, %subject, body, "EMAIL (log transport)");
Ok(())
}
EmailSender::Smtp { from, transport } => {
let msg = Message::builder()
.from(from.parse::<Mailbox>().context("parsing from_addr")?)
.to(to.parse::<Mailbox>().context("parsing recipient")?)
.subject(subject)
.body(body.to_string())
.context("building message")?;
transport.send(msg).await.context("sending email")?;
Ok(())
}
}
}
}

View File

@@ -0,0 +1,21 @@
//! Adapter from the shared, axum-agnostic
//! [`cortex_core::error_envelope::OpenAiError`] (#60/#63) to an axum
//! response, with `Retry-After`. The `/authz/v1` surface speaks the #63
//! envelope so cortex (an OpenAI-compatible proxy) can forward rejections
//! verbatim. (The future `/web/v1` surface uses a plain JSON error shape.)
use axum::http::{HeaderValue, StatusCode, header};
use axum::response::{IntoResponse, Json, Response};
use cortex_core::error_envelope::OpenAiError;
pub fn envelope_response(err: OpenAiError) -> Response {
let status = StatusCode::from_u16(err.status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR);
let retry_after = err.retry_after_secs;
let mut response = (status, Json(err.body())).into_response();
if let Some(secs) = retry_after
&& let Ok(value) = HeaderValue::from_str(&secs.to_string())
{
response.headers_mut().insert(header::RETRY_AFTER, value);
}
response
}

View File

@@ -0,0 +1,21 @@
//! HTTP handlers. B1 ships `/health`; the authz (`/authz/v1`) and web
//! (`/web/v1`) surfaces land in later phases.
use crate::state::AppState;
use axum::{Json, Router, extract::State, routing::get};
use serde_json::{Value, json};
pub fn routes() -> Router<AppState> {
Router::new()
.route("/health", get(health))
.route("/", get(health))
}
/// `GET /health` — liveness + a database round-trip (`SELECT 1`).
async fn health(State(state): State<AppState>) -> Json<Value> {
let db_ok = sqlx::query("SELECT 1").execute(&state.pool).await.is_ok();
Json(json!({
"status": if db_ok { "ok" } else { "degraded" },
"db": if db_ok { "ok" } else { "unreachable" },
}))
}

View File

@@ -0,0 +1,338 @@
//! The allocation ledger: reserve → settle/release with the no-overshoot
//! guarantee enforced by a row-locked transaction.
//!
//! Each reserve takes `SELECT … FOR UPDATE` on the account (and key) row, so
//! concurrent reserves from many cortexes serialize and `spent + reserved`
//! can never exceed the effective cap. The `accounts_no_overshoot` CHECK is
//! the DB-level backstop. Settle/release are idempotent (they only act on a
//! reservation still in `open`).
//!
//! Per-key effective cap = `min(resolved key cap, remaining account
//! allocation)`. The key cap is resolved from its `limit_kind`:
//! `hardcap` → the value verbatim; `percent` → that % of the account's
//! `allocation_total`.
//!
//! Cap-window semantics: this module implements **Balance** (non-resetting)
//! caps. Rolling-window key sub-caps (and the `RateLimited` rejection that
//! rides them) land with the authz API (B2); today an over-cap is always
//! `InsufficientQuota`.
use sqlx::postgres::PgPool;
use uuid::Uuid;
/// A bearer key resolved to its principal + a budget snapshot.
#[derive(Debug, Clone)]
pub struct ResolvedPrincipal {
pub account_id: Uuid,
pub key_id: Uuid,
/// Effective per-key absolute cap (the key sub-cap; the account cap
/// still binds at reserve time).
pub hard_cap: i64,
pub key_spent: i64,
pub key_reserved: i64,
}
/// Resolve a key by its `sha256` hash to its principal, or `None` when the
/// key is unknown/archived **or its account is deactivated** (the silent
/// abuse flag — indistinguishable from an unknown key, by design: no clue).
pub async fn resolve_key(
pool: &PgPool,
key_hash: &[u8],
) -> Result<Option<ResolvedPrincipal>, sqlx::Error> {
let row = sqlx::query(
"SELECT k.id AS key_id, k.account_id, k.limit_kind, k.limit_value, \
k.key_spent, k.key_reserved, a.allocation_total \
FROM api_keys k JOIN accounts a ON a.id = k.account_id \
WHERE k.key_hash = $1 AND k.status = 'active' AND a.status = 'active'",
)
.bind(key_hash)
.fetch_optional(pool)
.await?;
Ok(row.map(|r| {
let total: i64 = sqlx::Row::get(&r, "allocation_total");
let limit_kind: String = sqlx::Row::get(&r, "limit_kind");
let limit_value: i64 = sqlx::Row::get(&r, "limit_value");
ResolvedPrincipal {
account_id: sqlx::Row::get(&r, "account_id"),
key_id: sqlx::Row::get(&r, "key_id"),
hard_cap: resolve_abs_cap(&limit_kind, limit_value, total),
key_spent: sqlx::Row::get(&r, "key_spent"),
key_reserved: sqlx::Row::get(&r, "key_reserved"),
}
}))
}
/// Per-key budget snapshot `(hard_cap, spent, reserved)`, or `None` if the
/// key/account isn't an active pair.
pub async fn snapshot(
pool: &PgPool,
account_id: Uuid,
key_id: Uuid,
) -> Result<Option<(i64, i64, i64)>, sqlx::Error> {
let row = sqlx::query(
"SELECT k.limit_kind, k.limit_value, k.key_spent, k.key_reserved, a.allocation_total \
FROM api_keys k JOIN accounts a ON a.id = k.account_id \
WHERE k.id = $1 AND k.account_id = $2 AND k.status = 'active' AND a.status = 'active'",
)
.bind(key_id)
.bind(account_id)
.fetch_optional(pool)
.await?;
Ok(row.map(|r| {
let total: i64 = sqlx::Row::get(&r, "allocation_total");
let limit_kind: String = sqlx::Row::get(&r, "limit_kind");
let limit_value: i64 = sqlx::Row::get(&r, "limit_value");
let cap = resolve_abs_cap(&limit_kind, limit_value, total);
(
cap,
sqlx::Row::get::<i64, _>(&r, "key_spent"),
sqlx::Row::get::<i64, _>(&r, "key_reserved"),
)
}))
}
/// Release every `open` reservation older than `max_age_secs`, returning
/// each one's reserved tokens to its account and key in a single statement.
/// The lost-settle self-heal. Returns the number swept.
pub async fn sweep_stale(pool: &PgPool, max_age_secs: i64) -> Result<u64, sqlx::Error> {
// Data-modifying CTEs: release stale rows, then fold their reserved sums
// back into accounts and api_keys. All in one atomic statement.
let result = sqlx::query(
"WITH stale AS ( \
UPDATE reservations SET state = 'released', settled_at = now() \
WHERE state = 'open' AND created_at < now() - make_interval(secs => $1) \
RETURNING account_id, key_id, reserved \
), acct AS ( \
UPDATE accounts a SET allocation_reserved = allocation_reserved - s.total \
FROM (SELECT account_id, SUM(reserved) AS total FROM stale GROUP BY account_id) s \
WHERE a.id = s.account_id \
) \
UPDATE api_keys k SET key_reserved = key_reserved - s.total \
FROM (SELECT key_id, SUM(reserved) AS total FROM stale GROUP BY key_id) s \
WHERE k.id = s.key_id",
)
.bind(max_age_secs as f64)
.execute(pool)
.await?;
Ok(result.rows_affected())
}
/// Resolve a key's per-key cap to an absolute token count.
///
/// `percent` is `floor(allocation_total * limit_value / 100)`; `hardcap` is
/// `limit_value` verbatim. Computed in i128 to avoid overflow, floored at 0.
pub fn resolve_abs_cap(limit_kind: &str, limit_value: i64, allocation_total: i64) -> i64 {
let cap = match limit_kind {
"percent" => (allocation_total as i128 * limit_value as i128) / 100,
_ => limit_value as i128, // "hardcap" (and any unknown → treat as absolute)
};
cap.clamp(0, i64::MAX as i128) as i64
}
#[derive(Debug, thiserror::Error)]
pub enum LedgerError {
#[error("account not found")]
AccountNotFound,
#[error("api key not found or not active")]
KeyNotFound,
/// Account balance or a Balance-window key sub-cap is exhausted.
#[error("insufficient quota: requested {requested}, available {available}")]
InsufficientQuota { requested: i64, available: i64 },
#[error(transparent)]
Db(#[from] sqlx::Error),
}
/// Reserve `max_tokens` against `account_id`/`key_id`. Returns the
/// reservation id (the `BIGSERIAL`, mapped to the cortex `Reservation.id`).
pub async fn reserve(
pool: &PgPool,
account_id: Uuid,
key_id: Uuid,
max_tokens: i64,
) -> Result<i64, LedgerError> {
let mut tx = pool.begin().await?;
// Lock the account row — serializes concurrent reserves on this account.
let acct = sqlx::query(
"SELECT allocation_total, allocation_spent, allocation_reserved \
FROM accounts WHERE id = $1 AND status = 'active' FOR UPDATE",
)
.bind(account_id)
.fetch_optional(&mut *tx)
.await?;
let Some(acct) = acct else {
return Err(LedgerError::AccountNotFound);
};
let total: i64 = sqlx::Row::get(&acct, "allocation_total");
let spent: i64 = sqlx::Row::get(&acct, "allocation_spent");
let reserved: i64 = sqlx::Row::get(&acct, "allocation_reserved");
let account_avail = total - spent - reserved;
// Lock the key row and resolve its absolute sub-cap.
let key = sqlx::query(
"SELECT limit_kind, limit_value, key_spent, key_reserved \
FROM api_keys WHERE id = $1 AND account_id = $2 AND status = 'active' FOR UPDATE",
)
.bind(key_id)
.bind(account_id)
.fetch_optional(&mut *tx)
.await?;
let Some(key) = key else {
return Err(LedgerError::KeyNotFound);
};
let limit_kind: String = sqlx::Row::get(&key, "limit_kind");
let limit_value: i64 = sqlx::Row::get(&key, "limit_value");
let key_spent: i64 = sqlx::Row::get(&key, "key_spent");
let key_reserved: i64 = sqlx::Row::get(&key, "key_reserved");
let key_cap = resolve_abs_cap(&limit_kind, limit_value, total);
let key_avail = key_cap - key_spent - key_reserved;
let available = account_avail.min(key_avail).max(0);
if max_tokens > available {
// tx rolls back on drop
return Err(LedgerError::InsufficientQuota {
requested: max_tokens,
available,
});
}
let id: i64 = sqlx::Row::get(
&sqlx::query(
"INSERT INTO reservations (account_id, key_id, reserved, state) \
VALUES ($1, $2, $3, 'open') RETURNING id",
)
.bind(account_id)
.bind(key_id)
.bind(max_tokens)
.fetch_one(&mut *tx)
.await?,
"id",
);
sqlx::query("UPDATE accounts SET allocation_reserved = allocation_reserved + $1 WHERE id = $2")
.bind(max_tokens)
.bind(account_id)
.execute(&mut *tx)
.await?;
sqlx::query("UPDATE api_keys SET key_reserved = key_reserved + $1 WHERE id = $2")
.bind(max_tokens)
.bind(key_id)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(id)
}
/// Settle a reservation with the actual tokens used (clamped to
/// `[0, reserved]`). Idempotent: a second settle (or settle after release)
/// is a no-op.
pub async fn settle(
pool: &PgPool,
reservation_id: i64,
actual_tokens: i64,
) -> Result<(), LedgerError> {
let mut tx = pool.begin().await?;
let row = sqlx::query(
"UPDATE reservations SET state = 'settled', settled_at = now(), \
actual = LEAST(GREATEST($2, 0), reserved) \
WHERE id = $1 AND state = 'open' \
RETURNING reserved, account_id, key_id, actual",
)
.bind(reservation_id)
.bind(actual_tokens)
.fetch_optional(&mut *tx)
.await?;
let Some(row) = row else {
return Ok(()); // already settled/released, or unknown → idempotent no-op
};
let reserved: i64 = sqlx::Row::get(&row, "reserved");
let actual: i64 = sqlx::Row::get(&row, "actual");
let account_id: Uuid = sqlx::Row::get(&row, "account_id");
let key_id: Uuid = sqlx::Row::get(&row, "key_id");
sqlx::query(
"UPDATE accounts SET allocation_reserved = allocation_reserved - $1, \
allocation_spent = allocation_spent + $2 WHERE id = $3",
)
.bind(reserved)
.bind(actual)
.bind(account_id)
.execute(&mut *tx)
.await?;
sqlx::query(
"UPDATE api_keys SET key_reserved = key_reserved - $1, key_spent = key_spent + $2 WHERE id = $3",
)
.bind(reserved)
.bind(actual)
.bind(key_id)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(())
}
/// Release a reservation, returning its full reserved amount to the
/// allocation. Idempotent.
pub async fn release(pool: &PgPool, reservation_id: i64) -> Result<(), LedgerError> {
let mut tx = pool.begin().await?;
let row = sqlx::query(
"UPDATE reservations SET state = 'released', settled_at = now() \
WHERE id = $1 AND state = 'open' \
RETURNING reserved, account_id, key_id",
)
.bind(reservation_id)
.fetch_optional(&mut *tx)
.await?;
let Some(row) = row else {
return Ok(());
};
let reserved: i64 = sqlx::Row::get(&row, "reserved");
let account_id: Uuid = sqlx::Row::get(&row, "account_id");
let key_id: Uuid = sqlx::Row::get(&row, "key_id");
sqlx::query("UPDATE accounts SET allocation_reserved = allocation_reserved - $1 WHERE id = $2")
.bind(reserved)
.bind(account_id)
.execute(&mut *tx)
.await?;
sqlx::query("UPDATE api_keys SET key_reserved = key_reserved - $1 WHERE id = $2")
.bind(reserved)
.bind(key_id)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::resolve_abs_cap;
#[test]
fn hardcap_is_verbatim() {
assert_eq!(resolve_abs_cap("hardcap", 50_000, 1_000_000), 50_000);
}
#[test]
fn percent_is_fraction_of_allocation() {
assert_eq!(resolve_abs_cap("percent", 25, 1_000_000), 250_000);
assert_eq!(resolve_abs_cap("percent", 100, 1_000_000), 1_000_000);
// floor
assert_eq!(resolve_abs_cap("percent", 33, 10), 3);
}
#[test]
fn percent_does_not_overflow_on_large_allocation() {
// total * value would overflow i64 if not widened to i128.
let cap = resolve_abs_cap("percent", 100, i64::MAX);
assert_eq!(cap, i64::MAX);
}
#[test]
fn negative_or_zero_clamps_to_zero() {
assert_eq!(resolve_abs_cap("hardcap", -5, 100), 0);
assert_eq!(resolve_abs_cap("percent", 0, 1_000_000), 0);
}
}

View File

@@ -0,0 +1,90 @@
//! helexa-upstream — the mesh-level account/authorization authority (#59).
//!
//! The clearing house above cortex: it issues accounts and API keys, holds
//! the real token-allocation ledger, authorizes inference in real time
//! (reserve → settle, fail-closed), and tracks served usage for operator
//! reconciliation. cortex's `UpstreamEntitlementProvider` (#57) is a client
//! of the `/authz/v1` surface; the helexa.ai frontend is a client of the
//! `/web/v1` surface.
//!
//! Landed so far: B1 — schema + reserve→settle [`ledger`] (no-overshoot) +
//! `/health`. B2 — the `/authz/v1` [`authz`] surface (resolve/reserve/
//! settle/release/snapshot) with shared-bearer client auth and a
//! stale-reservation sweeper.
pub mod authz;
pub mod config;
pub mod crypto;
pub mod db;
pub mod email;
pub mod error;
pub mod handlers;
pub mod ledger;
pub mod reconcile;
pub mod state;
pub mod topup;
pub mod web;
use anyhow::Result;
use config::UpstreamConfig;
use email::EmailSender;
use state::AppState;
use std::time::Duration;
use tower_http::cors::CorsLayer;
use tower_http::trace::TraceLayer;
/// Build the axum application.
pub fn build_app(state: AppState) -> axum::Router {
axum::Router::new()
.merge(handlers::routes())
.merge(authz::router(&state))
.merge(web::router(&state))
// The /web/v1 surface is called cross-origin by the browser SPA in
// dev; same-origin behind nginx in prod. Permissive is fine — these
// endpoints authenticate via bearer/JWT, not cookies.
.layer(CorsLayer::permissive())
.layer(TraceLayer::new_for_http())
.with_state(state)
}
/// Start the service: connect Postgres, run migrations, spawn the
/// reservation sweeper, bind the listener.
pub async fn run(config: UpstreamConfig) -> Result<()> {
let pool = db::connect_and_migrate(&config.db.url, config.db.max_connections).await?;
let email = EmailSender::from_config(&config.email)?;
let listen = config.server.listen.clone();
let state = AppState::new(pool, config, email);
if state.config.client_auth.tokens.is_empty() {
tracing::warn!(
"no [client_auth] tokens configured — the /authz/v1 surface is OPEN (dev only)"
);
}
// Stale-reservation sweeper: releases open reservations whose
// settle/release from cortex was lost, self-healing allocation_reserved.
spawn_sweeper(&state);
let addr = listen.parse::<std::net::SocketAddr>()?;
tracing::info!("helexa-upstream listening on {addr}");
let listener = tokio::net::TcpListener::bind(addr).await?;
axum::serve(listener, build_app(state)).await?;
Ok(())
}
fn spawn_sweeper(state: &AppState) {
let pool = state.pool.clone();
let ttl = state.config.authz.reservation_ttl_secs as i64;
let interval = Duration::from_secs(state.config.authz.sweep_interval_secs);
tokio::spawn(async move {
loop {
tokio::time::sleep(interval).await;
match ledger::sweep_stale(&pool, ttl).await {
Ok(n) if n > 0 => tracing::info!(swept = n, "released stale reservations"),
Ok(_) => {}
Err(e) => tracing::warn!(error = %e, "reservation sweep failed"),
}
}
});
}

View File

@@ -0,0 +1,99 @@
use anyhow::Result;
use clap::{Parser, Subcommand};
use helexa_upstream::config::UpstreamConfig;
use tracing_subscriber::EnvFilter;
#[derive(Parser)]
#[command(name = "helexa-upstream")]
#[command(about = "Mesh-level account & authorization authority for helexa")]
#[command(version)]
struct Cli {
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
/// Start the upstream server.
Serve {
/// Path to the config file.
#[arg(short, long, default_value = "helexa-upstream.toml")]
config: String,
},
/// Mint single-use top-up codes and print them (one per line). The raw
/// codes are shown only here — only their hash is stored. (The future
/// faucet bot calls the same path.)
Mint {
#[arg(short, long, default_value = "helexa-upstream.toml")]
config: String,
/// Tokens each code grants.
#[arg(long)]
value: i64,
/// How many codes to mint.
#[arg(long, default_value_t = 1)]
count: u32,
/// Optional human label (e.g. "small", "beta-launch").
#[arg(long)]
denomination: Option<String>,
},
/// Roll up not-yet-reconciled served usage per operator/period (#58),
/// stamp it reconciled, and print the totals. Payout is out of scope.
Reconcile {
#[arg(short, long, default_value = "helexa-upstream.toml")]
config: String,
},
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,helexa_upstream=debug")),
)
.init();
let cli = Cli::parse();
match cli.command {
Commands::Serve { config } => {
let cfg = UpstreamConfig::load(&config)
.map_err(|e| anyhow::anyhow!("failed to load config from '{config}': {e}"))?;
tracing::info!(listen = %cfg.server.listen, "starting helexa-upstream");
helexa_upstream::run(cfg).await?;
}
Commands::Mint {
config,
value,
count,
denomination,
} => {
let cfg = UpstreamConfig::load(&config)
.map_err(|e| anyhow::anyhow!("failed to load config from '{config}': {e}"))?;
let pool =
helexa_upstream::db::connect_and_migrate(&cfg.db.url, cfg.db.max_connections)
.await?;
let codes =
helexa_upstream::topup::mint(&pool, value, count, denomination.as_deref()).await?;
// Raw codes to stdout (one per line) for the operator to distribute;
// logs/diagnostics go to stderr via tracing.
for code in codes {
println!("{code}");
}
}
Commands::Reconcile { config } => {
let cfg = UpstreamConfig::load(&config)
.map_err(|e| anyhow::anyhow!("failed to load config from '{config}': {e}"))?;
let pool =
helexa_upstream::db::connect_and_migrate(&cfg.db.url, cfg.db.max_connections)
.await?;
let rollup = helexa_upstream::reconcile::reconcile(&pool).await?;
for r in &rollup {
println!("{}\t{}\t{}", r.operator_id, r.period, r.total_served_tokens);
}
tracing::info!(operators_periods = rollup.len(), "reconciliation complete");
}
}
Ok(())
}

View File

@@ -0,0 +1,43 @@
//! Reconciliation rollup (#58): aggregate the served-usage ledger per
//! operator and period for operator compensation, stamping rows
//! `reconciled_at` so each window is settled once. The payout mechanism
//! itself is out of scope — this produces the authoritative per-operator
//! totals a settlement process consumes.
use sqlx::Row;
use sqlx::postgres::PgPool;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RollupRow {
pub operator_id: String,
pub period: chrono::NaiveDate,
pub total_served_tokens: i64,
}
/// Roll up all not-yet-reconciled served-usage into per-(operator, period)
/// totals, then stamp those rows `reconciled_at`. Returns the rollup.
/// Idempotent: a second run finds nothing unreconciled and returns empty.
pub async fn reconcile(pool: &PgPool) -> Result<Vec<RollupRow>, sqlx::Error> {
let mut tx = pool.begin().await?;
let rows = sqlx::query(
// SUM(bigint) is numeric in Postgres — cast back to bigint for i64.
"SELECT operator_id, period, SUM(served_tokens)::bigint AS total \
FROM served_usage WHERE reconciled_at IS NULL \
GROUP BY operator_id, period ORDER BY operator_id, period",
)
.fetch_all(&mut *tx)
.await?;
let rollup: Vec<RollupRow> = rows
.iter()
.map(|r| RollupRow {
operator_id: r.get("operator_id"),
period: r.get("period"),
total_served_tokens: r.get::<i64, _>("total"),
})
.collect();
sqlx::query("UPDATE served_usage SET reconciled_at = now() WHERE reconciled_at IS NULL")
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(rollup)
}

View File

@@ -0,0 +1,23 @@
//! Shared application state.
use crate::config::UpstreamConfig;
use crate::email::EmailSender;
use sqlx::postgres::PgPool;
use std::sync::Arc;
#[derive(Clone)]
pub struct AppState {
pub pool: PgPool,
pub config: Arc<UpstreamConfig>,
pub email: EmailSender,
}
impl AppState {
pub fn new(pool: PgPool, config: UpstreamConfig, email: EmailSender) -> Self {
Self {
pool,
config: Arc::new(config),
email,
}
}
}

View File

@@ -0,0 +1,82 @@
//! Single-use top-up codes (#B5) — the second half of the hybrid allocation
//! model. Each code grants `value` tokens to the account that redeems it,
//! raising `accounts.allocation_total`. Minting codes is operator/CLI side
//! (the future faucet bot calls the same `mint` path); redemption is a
//! `/web/v1` action.
//!
//! Security: only `sha256(code)` is stored. Redemption is **timing-safe and
//! single-use** — a conditional `UPDATE … WHERE redeemed_by IS NULL` does
//! the claim atomically (concurrent double-redeem → exactly one winner), and
//! a not-found code and an already-redeemed code return the **same** generic
//! failure with the same code path (no oracle for "valid but spent").
use crate::crypto::{random_token, sha256};
use sqlx::Row;
use sqlx::postgres::PgPool;
use uuid::Uuid;
#[derive(Debug, thiserror::Error)]
pub enum TopUpError {
/// Code unknown OR already redeemed — deliberately indistinguishable.
#[error("invalid or already-redeemed code")]
Invalid,
#[error(transparent)]
Db(#[from] sqlx::Error),
}
/// Redeem `raw_code` for `account_id`, raising the account's
/// `allocation_total` by the code's value. Returns the new total.
pub async fn redeem(pool: &PgPool, account_id: Uuid, raw_code: &str) -> Result<i64, TopUpError> {
let mut tx = pool.begin().await?;
// Atomic single-use claim. `redeemed_by IS NULL` is the guarantee: under
// concurrent redemption exactly one UPDATE touches the row.
let claimed = sqlx::query(
"UPDATE top_up_codes SET redeemed_by = $1, redeemed_at = now() \
WHERE code_hash = $2 AND redeemed_by IS NULL RETURNING value",
)
.bind(account_id)
.bind(sha256(raw_code))
.fetch_optional(&mut *tx)
.await?;
let Some(row) = claimed else {
// Not found or already redeemed — same path, same error.
return Err(TopUpError::Invalid);
};
let value: i64 = row.get("value");
let new_total: i64 = sqlx::query(
"UPDATE accounts SET allocation_total = allocation_total + $1 WHERE id = $2 \
RETURNING allocation_total",
)
.bind(value)
.bind(account_id)
.fetch_one(&mut *tx)
.await?
.get("allocation_total");
tx.commit().await?;
Ok(new_total)
}
/// Mint `count` codes each worth `value` tokens, optionally tagged with a
/// `denomination` label. Returns the raw codes (shown once — only their
/// hash is stored). The CLI prints these; the future faucet bot calls this.
pub async fn mint(
pool: &PgPool,
value: i64,
count: u32,
denomination: Option<&str>,
) -> Result<Vec<String>, sqlx::Error> {
let mut codes = Vec::with_capacity(count as usize);
for _ in 0..count {
let raw = format!("helexa-topup-{}", random_token());
sqlx::query(
"INSERT INTO top_up_codes (code_hash, value, denomination) VALUES ($1, $2, $3)",
)
.bind(sha256(&raw))
.bind(value)
.bind(denomination)
.execute(pool)
.await?;
codes.push(raw);
}
Ok(codes)
}

View File

@@ -0,0 +1,594 @@
//! `/web/v1` — the human-facing account API the helexa.ai frontend (#F4)
//! consumes: email+password auth (register / verify / login / reset),
//! API-key CRUD with per-key limits, and the account balance. Web sessions
//! are JWTs, **distinct** from inference API keys.
//!
//! Errors use a plain JSON shape `{ "error": { "message", "code" } }` (web
//! clients, not OpenAI clients — the #63 envelope is the authz surface).
//!
//! Silent fingerprint abuse (no clue to the abuser): registration captures
//! the browser fingerprint and always succeeds; when ≥ threshold accounts
//! share one fingerprint, all are silently `deactivated` (keys then resolve
//! as ordinary `401`s at the authz surface — never a "banned" signal).
use crate::crypto::{generate_api_key, hash_password, random_token, sha256, verify_password};
use crate::state::AppState;
use axum::extract::{Path, Request, State};
use axum::http::{StatusCode, header};
use axum::middleware::Next;
use axum::response::{IntoResponse, Json, Response};
use axum::routing::{get, post};
use axum::{Extension, Router};
use chrono::{DateTime, Duration, Utc};
use jsonwebtoken::{DecodingKey, EncodingKey, Header, Validation, decode, encode};
use serde::{Deserialize, Serialize};
use serde_json::json;
use sqlx::Row;
use uuid::Uuid;
pub fn router(state: &AppState) -> Router<AppState> {
let protected = Router::new()
.route("/web/v1/account", get(account))
.route("/web/v1/keys", get(list_keys).post(create_key))
.route("/web/v1/keys/{id}/archive", post(archive_key))
.route(
"/web/v1/keys/{id}/limit",
axum::routing::patch(update_key_limit),
)
.route("/web/v1/redeem", post(redeem))
.layer(axum::middleware::from_fn_with_state(
state.clone(),
require_session,
));
Router::new()
.route("/web/v1/register", post(register))
.route("/web/v1/verify", post(verify))
.route("/web/v1/login", post(login))
.route("/web/v1/password-reset/request", post(reset_request))
.route("/web/v1/password-reset/confirm", post(reset_confirm))
.merge(protected)
}
// ── errors ──────────────────────────────────────────────────────────
enum WebError {
BadRequest(&'static str),
Unauthorized,
Internal,
}
impl IntoResponse for WebError {
fn into_response(self) -> Response {
let (status, code, message) = match self {
WebError::BadRequest(m) => (StatusCode::BAD_REQUEST, "bad_request", m),
WebError::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized", "unauthorized"),
WebError::Internal => (
StatusCode::INTERNAL_SERVER_ERROR,
"internal_error",
"internal error",
),
};
(
status,
Json(json!({"error": {"message": message, "code": code}})),
)
.into_response()
}
}
impl From<sqlx::Error> for WebError {
fn from(e: sqlx::Error) -> Self {
tracing::error!(error = %e, "web db error");
WebError::Internal
}
}
type WebResult<T> = Result<T, WebError>;
// ── sessions (JWT) ──────────────────────────────────────────────────
#[derive(Serialize, Deserialize)]
struct Claims {
sub: String, // user id
exp: usize,
}
fn mint_session(state: &AppState, user_id: Uuid) -> WebResult<String> {
let exp = (Utc::now() + Duration::seconds(state.config.auth.session_ttl_secs as i64))
.timestamp() as usize;
let claims = Claims {
sub: user_id.to_string(),
exp,
};
encode(
&Header::default(),
&claims,
&EncodingKey::from_secret(state.config.auth.jwt_secret.as_bytes()),
)
.map_err(|_| WebError::Internal)
}
/// Authenticated user id, injected by [`require_session`].
#[derive(Clone)]
struct AuthUser(Uuid);
async fn require_session(State(state): State<AppState>, mut req: Request, next: Next) -> Response {
let token = req
.headers()
.get(header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
.map(str::trim);
let Some(token) = token else {
return WebError::Unauthorized.into_response();
};
let decoded = decode::<Claims>(
token,
&DecodingKey::from_secret(state.config.auth.jwt_secret.as_bytes()),
&Validation::default(),
);
match decoded
.ok()
.and_then(|d| Uuid::parse_str(&d.claims.sub).ok())
{
Some(uid) => {
req.extensions_mut().insert(AuthUser(uid));
next.run(req).await
}
None => WebError::Unauthorized.into_response(),
}
}
/// The caller's single account id.
async fn account_id_for(state: &AppState, user_id: Uuid) -> WebResult<Uuid> {
let row = sqlx::query("SELECT id FROM accounts WHERE owner_user_id = $1")
.bind(user_id)
.fetch_optional(&state.pool)
.await?;
row.map(|r| r.get::<Uuid, _>("id"))
.ok_or(WebError::Internal)
}
// ── auth lifecycle ──────────────────────────────────────────────────
#[derive(Deserialize)]
struct RegisterReq {
email: String,
password: String,
#[serde(default)]
fingerprint: Option<String>,
}
/// `POST /web/v1/register` — always returns `202`, regardless of whether the
/// email was new, already taken, or fingerprint-flagged (no enumeration, no
/// abuse clue).
async fn register(State(state): State<AppState>, Json(req): Json<RegisterReq>) -> Response {
match register_inner(&state, req).await {
Ok(()) | Err(WebError::BadRequest(_)) => {}
Err(e) => return e.into_response(),
}
// Generic 202 whatever happened above (except hard server errors).
StatusCode::ACCEPTED.into_response()
}
async fn register_inner(state: &AppState, req: RegisterReq) -> WebResult<()> {
if !req.email.contains('@') {
return Err(WebError::BadRequest("invalid email"));
}
if req.password.len() < 8 {
return Err(WebError::BadRequest("password too short (min 8)"));
}
let phc = hash_password(&req.password).map_err(|_| WebError::Internal)?;
// Insert the user; a duplicate email silently no-ops (no enumeration).
let user_id: Option<Uuid> = sqlx::query(
"INSERT INTO users (email, password_hash, registration_fingerprint) \
VALUES ($1, $2, $3) ON CONFLICT (email) DO NOTHING RETURNING id",
)
.bind(&req.email)
.bind(&phc)
.bind(&req.fingerprint)
.fetch_optional(&state.pool)
.await?
.map(|r| r.get("id"));
let Some(user_id) = user_id else {
return Ok(()); // email already registered — say nothing
};
// Account with the flat free grant.
sqlx::query("INSERT INTO accounts (owner_user_id, allocation_total) VALUES ($1, $2)")
.bind(user_id)
.bind(state.config.grant.free_token_grant)
.execute(&state.pool)
.await?;
// Silent fingerprint abuse handling.
if let Some(fp) = req.fingerprint.as_deref().filter(|f| !f.is_empty()) {
apply_fingerprint_policy(state, fp).await?;
}
// Email verification link.
let token = random_token();
let expires: DateTime<Utc> =
Utc::now() + Duration::seconds(state.config.auth.email_token_ttl_secs as i64);
sqlx::query(
"INSERT INTO email_tokens (token_hash, user_id, kind, expires_at) \
VALUES ($1, $2, 'verify', $3)",
)
.bind(sha256(&token))
.bind(user_id)
.bind(expires)
.execute(&state.pool)
.await?;
let link = format!("{}/verify?token={token}", state.config.auth.app_base_url);
let _ = state
.email
.send(
&req.email,
"Verify your helexa account",
&format!("Welcome to helexa. Verify your email:\n\n{link}\n"),
)
.await;
Ok(())
}
/// Count accounts sharing `fp`; flag them, and silently deactivate all once
/// the count reaches the configured threshold. No response difference — the
/// abuser gets no signal.
async fn apply_fingerprint_policy(state: &AppState, fp: &str) -> WebResult<()> {
let count: i64 =
sqlx::query_scalar("SELECT count(*) FROM users WHERE registration_fingerprint = $1")
.bind(fp)
.fetch_one(&state.pool)
.await?;
if count > 1 {
sqlx::query(
"UPDATE accounts SET fingerprint_flagged = true \
WHERE owner_user_id IN (SELECT id FROM users WHERE registration_fingerprint = $1)",
)
.bind(fp)
.execute(&state.pool)
.await?;
}
if count >= state.config.abuse.fingerprint_account_threshold {
let res = sqlx::query(
"UPDATE accounts SET status = 'deactivated' \
WHERE owner_user_id IN (SELECT id FROM users WHERE registration_fingerprint = $1)",
)
.bind(fp)
.execute(&state.pool)
.await?;
tracing::warn!(
fingerprint = fp,
accounts = res.rows_affected(),
"silently deactivated fingerprint-abusing accounts"
);
}
Ok(())
}
#[derive(Deserialize)]
struct TokenReq {
token: String,
}
/// `POST /web/v1/verify` — consume a verification token, mark verified.
async fn verify(State(state): State<AppState>, Json(req): Json<TokenReq>) -> WebResult<Response> {
let row = sqlx::query(
"UPDATE email_tokens SET consumed_at = now() \
WHERE token_hash = $1 AND kind = 'verify' AND consumed_at IS NULL AND expires_at > now() \
RETURNING user_id",
)
.bind(sha256(&req.token))
.fetch_optional(&state.pool)
.await?;
let Some(row) = row else {
return Err(WebError::BadRequest("invalid or expired token"));
};
let user_id: Uuid = row.get("user_id");
sqlx::query("UPDATE users SET email_verified = true WHERE id = $1")
.bind(user_id)
.execute(&state.pool)
.await?;
Ok(StatusCode::OK.into_response())
}
#[derive(Deserialize)]
struct LoginReq {
email: String,
password: String,
}
/// `POST /web/v1/login` — verify password + email-verified → session JWT.
async fn login(State(state): State<AppState>, Json(req): Json<LoginReq>) -> WebResult<Response> {
let row = sqlx::query("SELECT id, password_hash, email_verified FROM users WHERE email = $1")
.bind(&req.email)
.fetch_optional(&state.pool)
.await?;
// Generic 401 for every failure mode (no enumeration).
let Some(row) = row else {
return Err(WebError::Unauthorized);
};
let phc: String = row.get("password_hash");
let verified: bool = row.get("email_verified");
if !verify_password(&req.password, &phc) || !verified {
return Err(WebError::Unauthorized);
}
let user_id: Uuid = row.get("id");
let token = mint_session(&state, user_id)?;
Ok(Json(json!({
"token": token,
"expires_in": state.config.auth.session_ttl_secs,
}))
.into_response())
}
#[derive(Deserialize)]
struct EmailReq {
email: String,
}
/// `POST /web/v1/password-reset/request` — always `202` (no enumeration);
/// mints + emails a reset token only if the account exists.
async fn reset_request(State(state): State<AppState>, Json(req): Json<EmailReq>) -> Response {
// The inner only ever yields `Internal` (DB failure); a missing email is
// Ok(()) so there's no enumeration. Surface 500 on a real error, else 202.
match reset_request_inner(&state, &req.email).await {
Ok(()) => StatusCode::ACCEPTED.into_response(),
Err(e) => e.into_response(),
}
}
async fn reset_request_inner(state: &AppState, email: &str) -> WebResult<()> {
let row = sqlx::query("SELECT id FROM users WHERE email = $1")
.bind(email)
.fetch_optional(&state.pool)
.await?;
let Some(row) = row else { return Ok(()) };
let user_id: Uuid = row.get("id");
let token = random_token();
let expires: DateTime<Utc> =
Utc::now() + Duration::seconds(state.config.auth.email_token_ttl_secs as i64);
sqlx::query(
"INSERT INTO email_tokens (token_hash, user_id, kind, expires_at) \
VALUES ($1, $2, 'reset', $3)",
)
.bind(sha256(&token))
.bind(user_id)
.bind(expires)
.execute(&state.pool)
.await?;
let link = format!("{}/reset?token={token}", state.config.auth.app_base_url);
let _ = state
.email
.send(
email,
"Reset your helexa password",
&format!("Reset your password:\n\n{link}\n"),
)
.await;
Ok(())
}
#[derive(Deserialize)]
struct ResetConfirmReq {
token: String,
new_password: String,
}
/// `POST /web/v1/password-reset/confirm` — consume reset token, rotate hash.
async fn reset_confirm(
State(state): State<AppState>,
Json(req): Json<ResetConfirmReq>,
) -> WebResult<Response> {
if req.new_password.len() < 8 {
return Err(WebError::BadRequest("password too short (min 8)"));
}
let row = sqlx::query(
"UPDATE email_tokens SET consumed_at = now() \
WHERE token_hash = $1 AND kind = 'reset' AND consumed_at IS NULL AND expires_at > now() \
RETURNING user_id",
)
.bind(sha256(&req.token))
.fetch_optional(&state.pool)
.await?;
let Some(row) = row else {
return Err(WebError::BadRequest("invalid or expired token"));
};
let user_id: Uuid = row.get("user_id");
let phc = hash_password(&req.new_password).map_err(|_| WebError::Internal)?;
sqlx::query("UPDATE users SET password_hash = $1 WHERE id = $2")
.bind(phc)
.bind(user_id)
.execute(&state.pool)
.await?;
Ok(StatusCode::OK.into_response())
}
// ── account + keys (protected) ──────────────────────────────────────
async fn account(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
let row = sqlx::query(
"SELECT allocation_total, allocation_spent, allocation_reserved FROM accounts WHERE id = $1",
)
.bind(acct)
.fetch_one(&state.pool)
.await?;
Ok(Json(json!({
"account_id": acct.to_string(),
"allocation_total": row.get::<i64, _>("allocation_total"),
"allocation_spent": row.get::<i64, _>("allocation_spent"),
"allocation_reserved": row.get::<i64, _>("allocation_reserved"),
}))
.into_response())
}
async fn list_keys(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
let rows = sqlx::query(
"SELECT id, key_prefix, label, status, limit_kind, limit_value, key_spent, key_reserved, \
created_at \
FROM api_keys WHERE account_id = $1 ORDER BY created_at DESC",
)
.bind(acct)
.fetch_all(&state.pool)
.await?;
let keys: Vec<_> = rows
.iter()
.map(|r| {
json!({
"id": r.get::<Uuid, _>("id").to_string(),
"prefix": r.get::<String, _>("key_prefix"),
"label": r.get::<String, _>("label"),
"status": r.get::<String, _>("status"),
"limit_kind": r.get::<String, _>("limit_kind"),
"limit_value": r.get::<i64, _>("limit_value"),
"spent": r.get::<i64, _>("key_spent"),
"reserved": r.get::<i64, _>("key_reserved"),
"created_at": r.get::<DateTime<Utc>, _>("created_at").to_rfc3339(),
})
})
.collect();
Ok(Json(json!({ "keys": keys })).into_response())
}
#[derive(Deserialize)]
struct CreateKeyReq {
#[serde(default)]
label: String,
/// "percent" | "hardcap" (default percent=100 → full allocation).
#[serde(default)]
limit_kind: Option<String>,
#[serde(default)]
limit_value: Option<i64>,
}
async fn create_key(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Json(req): Json<CreateKeyReq>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
let limit_kind = match req.limit_kind.as_deref() {
Some("hardcap") => "hardcap",
_ => "percent",
};
let limit_value = req.limit_value.unwrap_or(100).max(0);
let (raw, prefix) = generate_api_key();
let id: Uuid = sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix, label, limit_kind, limit_value) \
VALUES ($1, $2, $3, $4, $5, $6) RETURNING id",
)
.bind(acct)
.bind(sha256(&raw))
.bind(&prefix)
.bind(&req.label)
.bind(limit_kind)
.bind(limit_value)
.fetch_one(&state.pool)
.await?
.get("id");
// The raw key is shown exactly once.
Ok((
StatusCode::CREATED,
Json(json!({
"id": id.to_string(),
"key": raw,
"prefix": prefix,
"limit_kind": limit_kind,
"limit_value": limit_value,
})),
)
.into_response())
}
async fn archive_key(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Path(id): Path<Uuid>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
let res = sqlx::query(
"UPDATE api_keys SET status = 'archived' WHERE id = $1 AND account_id = $2 AND status = 'active'",
)
.bind(id)
.bind(acct)
.execute(&state.pool)
.await?;
if res.rows_affected() == 0 {
return Err(WebError::BadRequest("no such active key"));
}
Ok(StatusCode::NO_CONTENT.into_response())
}
#[derive(Deserialize)]
struct UpdateLimitReq {
limit_kind: String,
limit_value: i64,
}
async fn update_key_limit(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Path(id): Path<Uuid>,
Json(req): Json<UpdateLimitReq>,
) -> WebResult<Response> {
if req.limit_kind != "percent" && req.limit_kind != "hardcap" {
return Err(WebError::BadRequest(
"limit_kind must be percent or hardcap",
));
}
if req.limit_value < 0 {
return Err(WebError::BadRequest("limit_value must be >= 0"));
}
let acct = account_id_for(&state, user.0).await?;
let res = sqlx::query(
"UPDATE api_keys SET limit_kind = $1, limit_value = $2 WHERE id = $3 AND account_id = $4",
)
.bind(&req.limit_kind)
.bind(req.limit_value)
.bind(id)
.bind(acct)
.execute(&state.pool)
.await?;
if res.rows_affected() == 0 {
return Err(WebError::BadRequest("no such key"));
}
Ok(StatusCode::NO_CONTENT.into_response())
}
#[derive(Deserialize)]
struct RedeemReq {
code: String,
}
/// `POST /web/v1/redeem` — redeem a single-use top-up code, raising the
/// account's allocation. Returns the new total. Generic 400 for an invalid
/// or already-redeemed code (no oracle).
async fn redeem(
State(state): State<AppState>,
Extension(user): Extension<AuthUser>,
Json(req): Json<RedeemReq>,
) -> WebResult<Response> {
let acct = account_id_for(&state, user.0).await?;
match crate::topup::redeem(&state.pool, acct, &req.code).await {
Ok(new_total) => Ok(Json(json!({ "allocation_total": new_total })).into_response()),
Err(crate::topup::TopUpError::Invalid) => {
Err(WebError::BadRequest("invalid or already-redeemed code"))
}
Err(crate::topup::TopUpError::Db(e)) => {
tracing::error!(error = %e, "redeem db error");
Err(WebError::Internal)
}
}
}

View File

@@ -0,0 +1,243 @@
//! Integration tests for the `/authz/v1` surface against a real Postgres,
//! driving the built axum app over HTTP. Gated on `UPSTREAM_TEST_DATABASE_URL`
//! (skips cleanly when unset, so CI stays green without a DB):
//!
//! UPSTREAM_TEST_DATABASE_URL=postgres://helexa:helexa@localhost/helexa_test \
//! cargo test -p helexa-upstream --test authz_pg
use helexa_upstream::config::{ClientToken, UpstreamConfig};
use helexa_upstream::crypto::sha256;
use helexa_upstream::db::connect_and_migrate;
use helexa_upstream::state::AppState;
use serde_json::{Value, json};
use sqlx::Executor;
use sqlx::Row;
use sqlx::postgres::PgPool;
use uuid::Uuid;
const CLIENT_TOKEN: &str = "test-operator-token";
async fn spawn_or_skip(test: &str) -> Option<(String, PgPool)> {
let Ok(url) = std::env::var("UPSTREAM_TEST_DATABASE_URL") else {
eprintln!("skipping {test}: UPSTREAM_TEST_DATABASE_URL not set");
return None;
};
let pool = connect_and_migrate(&url, 16).await.expect("migrate");
let mut config = UpstreamConfig {
server: Default::default(),
db: helexa_upstream::config::DbSettings {
url,
max_connections: 16,
},
grant: Default::default(),
abuse: Default::default(),
client_auth: Default::default(),
authz: Default::default(),
auth: Default::default(),
email: Default::default(),
};
config.client_auth.tokens.push(ClientToken {
token: CLIENT_TOKEN.into(),
operator_id: "op-test".into(),
});
let email = helexa_upstream::email::EmailSender::from_config(&config.email).unwrap();
let state = AppState::new(pool.clone(), config, email);
let app = helexa_upstream::build_app(state);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
Some((format!("http://{addr}"), pool))
}
/// Seed an account with `total` allocation and an active key with raw value
/// `raw` (percent=100). Optionally deactivate the account. Returns
/// (account_id, key_id).
async fn seed_key(pool: &PgPool, total: i64, raw: &str, deactivated: bool) -> (Uuid, Uuid) {
let user_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO users (email, password_hash, email_verified) VALUES ($1,'x',true) RETURNING id",
)
.bind(format!("u-{}@t.local", Uuid::new_v4())),
)
.await
.unwrap()
.get("id");
let status = if deactivated { "deactivated" } else { "active" };
let account_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO accounts (owner_user_id, allocation_total, status) VALUES ($1,$2,$3) RETURNING id",
)
.bind(user_id)
.bind(total)
.bind(status),
)
.await
.unwrap()
.get("id");
let key_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix, limit_kind, limit_value) \
VALUES ($1,$2,'sk-test','percent',100) RETURNING id",
)
.bind(account_id)
.bind(sha256(raw)),
)
.await
.unwrap()
.get("id");
(account_id, key_id)
}
fn client() -> reqwest::Client {
reqwest::Client::new()
}
async fn post(
c: &reqwest::Client,
url: String,
body: Value,
bearer: Option<&str>,
) -> reqwest::Response {
let mut req = c.post(url).json(&body);
if let Some(b) = bearer {
req = req.bearer_auth(b);
}
req.send().await.unwrap()
}
#[tokio::test]
async fn resolve_reserve_settle_round_trip() {
let Some((base, pool)) = spawn_or_skip("resolve_reserve_settle_round_trip").await else {
return;
};
let raw = format!("sk-{}", Uuid::new_v4());
let (account_id, key_id) = seed_key(&pool, 1000, &raw, false).await;
let c = client();
// resolve
let r = post(
&c,
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 200);
let body: Value = r.json().await.unwrap();
assert_eq!(body["principal"]["account_id"], account_id.to_string());
assert_eq!(body["principal"]["key_id"], key_id.to_string());
assert_eq!(body["snapshot"]["hard_cap"], 1000);
// reserve 400
let r = post(
&c,
format!("{base}/authz/v1/reserve"),
json!({"account_id": account_id, "key_id": key_id, "max_tokens": 400}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 200);
let body: Value = r.json().await.unwrap();
let rid = body["reservation_id"].as_i64().expect("granted");
// settle 150
let r = post(
&c,
format!("{base}/authz/v1/settle"),
json!({"reservation_id": rid, "actual_tokens": 150}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 204);
// snapshot reflects spend
let r = post(
&c,
format!("{base}/authz/v1/snapshot"),
json!({"account_id": account_id, "key_id": key_id}),
Some(CLIENT_TOKEN),
)
.await;
let body: Value = r.json().await.unwrap();
assert_eq!(body["spent"], 150);
assert_eq!(body["reserved"], 0);
}
#[tokio::test]
async fn over_cap_reserve_is_rejected_not_errored() {
let Some((base, pool)) = spawn_or_skip("over_cap_reserve_is_rejected_not_errored").await else {
return;
};
let raw = format!("sk-{}", Uuid::new_v4());
let (account_id, key_id) = seed_key(&pool, 100, &raw, false).await;
let c = client();
let r = post(
&c,
format!("{base}/authz/v1/reserve"),
json!({"account_id": account_id, "key_id": key_id, "max_tokens": 999}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 200, "budget refusal is an authoritative 200");
let body: Value = r.json().await.unwrap();
assert!(body["reservation_id"].is_null());
assert_eq!(body["rejected"]["kind"], "insufficient_quota");
assert_eq!(body["rejected"]["available"], 100);
}
#[tokio::test]
async fn deactivated_account_resolves_as_invalid_no_clue() {
let Some((base, pool)) = spawn_or_skip("deactivated_account_resolves_as_invalid_no_clue").await
else {
return;
};
let raw = format!("sk-{}", Uuid::new_v4());
seed_key(&pool, 1000, &raw, true).await; // deactivated
let c = client();
let r = post(
&c,
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw}),
Some(CLIENT_TOKEN),
)
.await;
// Indistinguishable from an unknown key.
assert_eq!(r.status(), 401);
let body: Value = r.json().await.unwrap();
assert_eq!(body["error"]["code"], "invalid_api_key");
}
#[tokio::test]
async fn missing_client_auth_is_401_before_db() {
let Some((base, pool)) = spawn_or_skip("missing_client_auth_is_401_before_db").await else {
return;
};
let raw = format!("sk-{}", Uuid::new_v4());
seed_key(&pool, 1000, &raw, false).await;
let c = client();
// No bearer → rejected by client_auth.
let r = post(
&c,
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw}),
None,
)
.await;
assert_eq!(r.status(), 401);
// Wrong bearer → also rejected.
let r = post(
&c,
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw}),
Some("wrong"),
)
.await;
assert_eq!(r.status(), 401);
}

View File

@@ -0,0 +1,204 @@
//! Integration tests for the allocation ledger against a real PostgreSQL.
//!
//! Gated on `UPSTREAM_TEST_DATABASE_URL` — when unset (CI's generic runner,
//! local builds without a DB), every test logs a skip and returns, so
//! `cargo test --workspace` stays green without Postgres. Point the env var
//! at a throwaway database to exercise the no-overshoot guarantee and
//! settle/release idempotency:
//!
//! UPSTREAM_TEST_DATABASE_URL=postgres://helexa:helexa@localhost/helexa_test \
//! cargo test -p helexa-upstream --test ledger_pg
use helexa_upstream::db::connect_and_migrate;
use helexa_upstream::ledger::{self, LedgerError};
use sqlx::Executor;
use sqlx::Row;
use sqlx::postgres::PgPool;
use uuid::Uuid;
/// Returns a migrated pool, or `None` (with a skip log) when the env var is
/// unset.
async fn pool_or_skip(test: &str) -> Option<PgPool> {
let Ok(url) = std::env::var("UPSTREAM_TEST_DATABASE_URL") else {
eprintln!("skipping {test}: UPSTREAM_TEST_DATABASE_URL not set");
return None;
};
Some(
connect_and_migrate(&url, 16)
.await
.expect("connect + migrate"),
)
}
/// Seed a verified user + account (with `total` allocation) + an active key
/// (percent=100 so the account cap binds). Returns (account_id, key_id).
async fn seed(pool: &PgPool, total: i64) -> (Uuid, Uuid) {
let user_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO users (email, password_hash, email_verified) \
VALUES ($1, 'x', true) RETURNING id",
)
.bind(format!("u-{}@test.local", Uuid::new_v4())),
)
.await
.unwrap()
.get("id");
let account_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO accounts (owner_user_id, allocation_total) \
VALUES ($1, $2) RETURNING id",
)
.bind(user_id)
.bind(total),
)
.await
.unwrap()
.get("id");
let key_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix, limit_kind, limit_value) \
VALUES ($1, $2, 'sk-test', 'percent', 100) RETURNING id",
)
.bind(account_id)
.bind(Uuid::new_v4().as_bytes().to_vec()),
)
.await
.unwrap()
.get("id");
(account_id, key_id)
}
async fn account_cols(pool: &PgPool, account_id: Uuid) -> (i64, i64) {
let row = pool
.fetch_one(
sqlx::query("SELECT allocation_spent, allocation_reserved FROM accounts WHERE id = $1")
.bind(account_id),
)
.await
.unwrap();
(row.get("allocation_spent"), row.get("allocation_reserved"))
}
#[tokio::test]
async fn concurrent_reserves_never_overshoot() {
let Some(pool) = pool_or_skip("concurrent_reserves_never_overshoot").await else {
return;
};
// Allocation admits exactly 5 reservations of 100 (cap 500).
let (account_id, key_id) = seed(&pool, 500).await;
let mut handles = Vec::new();
for _ in 0..20 {
let pool = pool.clone();
handles.push(tokio::spawn(async move {
ledger::reserve(&pool, account_id, key_id, 100).await
}));
}
let mut ok = 0;
let mut quota = 0;
for h in handles {
match h.await.unwrap() {
Ok(_) => ok += 1,
Err(LedgerError::InsufficientQuota { .. }) => quota += 1,
Err(e) => panic!("unexpected error: {e}"),
}
}
assert_eq!(ok, 5, "exactly 5 reserves of 100 fit in a 500 allocation");
assert_eq!(quota, 15);
let (spent, reserved) = account_cols(&pool, account_id).await;
assert_eq!(spent, 0);
assert_eq!(reserved, 500, "reserved exactly the cap, never over");
}
#[tokio::test]
async fn settle_is_idempotent_and_reconciles_spend() {
let Some(pool) = pool_or_skip("settle_is_idempotent_and_reconciles_spend").await else {
return;
};
let (account_id, key_id) = seed(&pool, 1000).await;
let rid = ledger::reserve(&pool, account_id, key_id, 400)
.await
.unwrap();
// Settle actual=150 (< reserved 400): spent=150, reserved back to 0.
ledger::settle(&pool, rid, 150).await.unwrap();
let (spent, reserved) = account_cols(&pool, account_id).await;
assert_eq!((spent, reserved), (150, 0));
// Second settle is a no-op.
ledger::settle(&pool, rid, 999).await.unwrap();
let (spent2, reserved2) = account_cols(&pool, account_id).await;
assert_eq!((spent2, reserved2), (150, 0), "settle is idempotent");
}
#[tokio::test]
async fn release_returns_reservation_and_is_idempotent() {
let Some(pool) = pool_or_skip("release_returns_reservation_and_is_idempotent").await else {
return;
};
let (account_id, key_id) = seed(&pool, 1000).await;
let rid = ledger::reserve(&pool, account_id, key_id, 300)
.await
.unwrap();
assert_eq!(account_cols(&pool, account_id).await, (0, 300));
ledger::release(&pool, rid).await.unwrap();
assert_eq!(account_cols(&pool, account_id).await, (0, 0));
// Idempotent; settle-after-release also a no-op.
ledger::release(&pool, rid).await.unwrap();
ledger::settle(&pool, rid, 100).await.unwrap();
assert_eq!(account_cols(&pool, account_id).await, (0, 0));
}
#[tokio::test]
async fn hardcap_key_subcap_binds_below_account() {
let Some(pool) = pool_or_skip("hardcap_key_subcap_binds_below_account").await else {
return;
};
// Account has 1000 but the key is hard-capped at 200.
let user_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO users (email, password_hash, email_verified) \
VALUES ($1, 'x', true) RETURNING id",
)
.bind(format!("u-{}@test.local", Uuid::new_v4())),
)
.await
.unwrap()
.get("id");
let account_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO accounts (owner_user_id, allocation_total) VALUES ($1, 1000) RETURNING id",
)
.bind(user_id),
)
.await
.unwrap()
.get("id");
let key_id: Uuid = pool
.fetch_one(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix, limit_kind, limit_value) \
VALUES ($1, $2, 'sk-test', 'hardcap', 200) RETURNING id",
)
.bind(account_id)
.bind(Uuid::new_v4().as_bytes().to_vec()),
)
.await
.unwrap()
.get("id");
ledger::reserve(&pool, account_id, key_id, 200)
.await
.unwrap();
match ledger::reserve(&pool, account_id, key_id, 1).await {
Err(LedgerError::InsufficientQuota { available, .. }) => assert_eq!(available, 0),
other => panic!("expected InsufficientQuota, got {other:?}"),
}
}

View File

@@ -0,0 +1,116 @@
//! Integration test for the served-usage report (#58): the idempotent,
//! monotonic upsert and the reconcile rollup. Gated on
//! UPSTREAM_TEST_DATABASE_URL (skips cleanly when unset).
use helexa_upstream::config::{ClientToken, UpstreamConfig};
use helexa_upstream::db::connect_and_migrate;
use helexa_upstream::email::EmailSender;
use helexa_upstream::reconcile::reconcile;
use helexa_upstream::state::AppState;
use serde_json::{Value, json};
use sqlx::Row;
use sqlx::postgres::PgPool;
use uuid::Uuid;
const CLIENT_TOKEN: &str = "su-test-token";
const OPERATOR: &str = "op-su-test";
async fn spawn_or_skip(test: &str) -> Option<(String, PgPool)> {
let Ok(url) = std::env::var("UPSTREAM_TEST_DATABASE_URL") else {
eprintln!("skipping {test}: UPSTREAM_TEST_DATABASE_URL not set");
return None;
};
let pool = connect_and_migrate(&url, 16).await.expect("migrate");
let mut config = UpstreamConfig {
server: Default::default(),
db: helexa_upstream::config::DbSettings {
url,
max_connections: 16,
},
grant: Default::default(),
abuse: Default::default(),
client_auth: Default::default(),
authz: Default::default(),
auth: Default::default(),
email: Default::default(),
};
config.client_auth.tokens.push(ClientToken {
token: CLIENT_TOKEN.into(),
operator_id: OPERATOR.into(),
});
let email = EmailSender::from_config(&config.email).unwrap();
let state = AppState::new(pool.clone(), config, email);
let app = helexa_upstream::build_app(state);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
Some((format!("http://{addr}"), pool))
}
async fn report(base: &str, rows: Value) -> u16 {
reqwest::Client::new()
.post(format!("{base}/authz/v1/served-usage"))
.bearer_auth(CLIENT_TOKEN)
.json(&json!({ "rows": rows }))
.send()
.await
.unwrap()
.status()
.as_u16()
}
async fn stored(pool: &PgPool, account: Uuid, key: Uuid) -> i64 {
sqlx::query(
"SELECT served_tokens FROM served_usage WHERE operator_id = $1 AND account_id = $2 AND key_id = $3",
)
.bind(OPERATOR)
.bind(account)
.bind(key)
.fetch_one(pool)
.await
.unwrap()
.get("served_tokens")
}
#[tokio::test]
async fn served_usage_upsert_is_monotonic_and_reconciles() {
let Some((base, pool)) = spawn_or_skip("served_usage_upsert_is_monotonic_and_reconciles").await
else {
return;
};
let account = Uuid::new_v4();
let key = Uuid::new_v4();
let period = "2026-06-23";
let row = |n: i64| json!([{"account_id": account, "key_id": key, "period": period, "served_tokens": n}]);
// First report.
assert_eq!(report(&base, row(100)).await, 204);
assert_eq!(stored(&pool, account, key).await, 100);
// Re-send a higher absolute value → advances.
assert_eq!(report(&base, row(250)).await, 204);
assert_eq!(stored(&pool, account, key).await, 250);
// A lower value (e.g. a restarted cortex) must NOT regress (GREATEST).
assert_eq!(report(&base, row(50)).await, 204);
assert_eq!(stored(&pool, account, key).await, 250);
// Re-sending the same value is idempotent.
assert_eq!(report(&base, row(250)).await, 204);
assert_eq!(stored(&pool, account, key).await, 250);
// Reconcile rolls it up and stamps reconciled_at; a second run is empty.
let rollup = reconcile(&pool).await.unwrap();
let mine = rollup
.iter()
.find(|r| r.operator_id == OPERATOR)
.expect("operator in rollup");
assert!(mine.total_served_tokens >= 250);
let again = reconcile(&pool).await.unwrap();
assert!(
again.iter().all(|r| r.operator_id != OPERATOR),
"already reconciled"
);
}

View File

@@ -0,0 +1,425 @@
//! Integration tests for the `/web/v1` account API + the silent fingerprint
//! abuse policy, driving the built app over HTTP against a real Postgres.
//! Gated on `UPSTREAM_TEST_DATABASE_URL` (skips cleanly when unset).
use helexa_upstream::config::{ClientToken, UpstreamConfig};
use helexa_upstream::crypto::sha256;
use helexa_upstream::db::connect_and_migrate;
use helexa_upstream::email::EmailSender;
use helexa_upstream::state::AppState;
use serde_json::{Value, json};
use sqlx::Executor;
use sqlx::Row;
use sqlx::postgres::PgPool;
const CLIENT_TOKEN: &str = "web-test-operator-token";
async fn spawn_or_skip(test: &str) -> Option<(String, PgPool)> {
let Ok(url) = std::env::var("UPSTREAM_TEST_DATABASE_URL") else {
eprintln!("skipping {test}: UPSTREAM_TEST_DATABASE_URL not set");
return None;
};
let pool = connect_and_migrate(&url, 16).await.expect("migrate");
let mut config = UpstreamConfig {
server: Default::default(),
db: helexa_upstream::config::DbSettings {
url,
max_connections: 16,
},
grant: Default::default(),
abuse: Default::default(),
client_auth: Default::default(),
authz: Default::default(),
auth: Default::default(),
email: Default::default(), // Log transport
};
config.client_auth.tokens.push(ClientToken {
token: CLIENT_TOKEN.into(),
operator_id: "op-web".into(),
});
let email = EmailSender::from_config(&config.email).unwrap();
let state = AppState::new(pool.clone(), config, email);
let app = helexa_upstream::build_app(state);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
Some((format!("http://{addr}"), pool))
}
fn unique_email() -> String {
format!("u-{}@test.local", uuid::Uuid::new_v4())
}
async fn post(url: String, body: Value, bearer: Option<&str>) -> reqwest::Response {
let c = reqwest::Client::new();
let mut req = c.post(url).json(&body);
if let Some(b) = bearer {
req = req.bearer_auth(b);
}
req.send().await.unwrap()
}
#[tokio::test]
async fn verify_endpoint_consumes_token_once() {
let Some((base, pool)) = spawn_or_skip("verify_endpoint_consumes_token_once").await else {
return;
};
let email = unique_email();
// Register, then mint a verify token directly (the raw token is only in
// the email; here we insert a known one to drive the endpoint).
assert_eq!(
post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123"}),
None
)
.await
.status(),
202
);
let user_id: uuid::Uuid = pool
.fetch_one(sqlx::query("SELECT id FROM users WHERE email = $1").bind(&email))
.await
.unwrap()
.get("id");
let raw = "verify-raw-token-xyz";
pool.execute(
sqlx::query(
"INSERT INTO email_tokens (token_hash, user_id, kind, expires_at) \
VALUES ($1, $2, 'verify', now() + interval '1 hour')",
)
.bind(sha256(raw))
.bind(user_id),
)
.await
.unwrap();
assert_eq!(
post(format!("{base}/web/v1/verify"), json!({"token": raw}), None)
.await
.status(),
200
);
// Consumed → second attempt fails.
assert_eq!(
post(format!("{base}/web/v1/verify"), json!({"token": raw}), None)
.await
.status(),
400
);
let verified: bool = pool
.fetch_one(sqlx::query("SELECT email_verified FROM users WHERE id = $1").bind(user_id))
.await
.unwrap()
.get("email_verified");
assert!(verified);
}
#[tokio::test]
async fn account_lifecycle_and_key_resolves_then_archives() {
let Some((base, pool)) =
spawn_or_skip("account_lifecycle_and_key_resolves_then_archives").await
else {
return;
};
let email = unique_email();
post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123"}),
None,
)
.await;
// Bypass the email step for the login/key portion.
pool.execute(
sqlx::query("UPDATE users SET email_verified = true WHERE email = $1").bind(&email),
)
.await
.unwrap();
// login → session JWT
let r = post(
format!("{base}/web/v1/login"),
json!({"email": email, "password": "password123"}),
None,
)
.await;
assert_eq!(r.status(), 200);
let token = r.json::<Value>().await.unwrap()["token"]
.as_str()
.unwrap()
.to_string();
// create key (raw shown once)
let r = post(
format!("{base}/web/v1/keys"),
json!({"label": "laptop"}),
Some(&token),
)
.await;
assert_eq!(r.status(), 201);
let body: Value = r.json().await.unwrap();
let raw_key = body["key"].as_str().unwrap().to_string();
let key_id = body["id"].as_str().unwrap().to_string();
assert!(raw_key.starts_with("sk-helexa-"));
// account balance reflects the free grant
let r = reqwest::Client::new()
.get(format!("{base}/web/v1/account"))
.bearer_auth(&token)
.send()
.await
.unwrap();
assert_eq!(
r.json::<Value>().await.unwrap()["allocation_total"],
1_000_000
);
// list keys shows the prefix, never the raw secret
let r = reqwest::Client::new()
.get(format!("{base}/web/v1/keys"))
.bearer_auth(&token)
.send()
.await
.unwrap();
let listed = r.json::<Value>().await.unwrap();
let k = &listed["keys"][0];
assert_eq!(k["id"], key_id);
assert!(k.get("key").is_none(), "raw secret never listed");
assert!(k["prefix"].as_str().unwrap().starts_with("sk-helexa-"));
// the key authorizes at the authz surface
let r = post(
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw_key}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 200);
// archive → the key no longer resolves
let r = post(
format!("{base}/web/v1/keys/{key_id}/archive"),
json!({}),
Some(&token),
)
.await;
assert_eq!(r.status(), 204);
let r = post(
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw_key}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(r.status(), 401);
}
#[tokio::test]
async fn fingerprint_abuse_silently_deactivates_all_no_clue() {
let Some((base, pool)) =
spawn_or_skip("fingerprint_abuse_silently_deactivates_all_no_clue").await
else {
return;
};
let fp = format!("fp-{}", uuid::Uuid::new_v4());
// 5 registrations sharing one fingerprint — every one returns a normal 202.
let mut emails = Vec::new();
for _ in 0..5 {
let email = unique_email();
let r = post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123", "fingerprint": fp}),
None,
)
.await;
assert_eq!(r.status(), 202, "registration always looks successful");
emails.push(email);
}
// Silent effect: all 5 accounts are deactivated + flagged.
let (deactivated, flagged): (i64, i64) = {
let row = pool
.fetch_one(
sqlx::query(
"SELECT \
count(*) FILTER (WHERE a.status = 'deactivated') AS d, \
count(*) FILTER (WHERE a.fingerprint_flagged) AS f \
FROM accounts a JOIN users u ON u.id = a.owner_user_id \
WHERE u.registration_fingerprint = $1",
)
.bind(&fp),
)
.await
.unwrap();
(row.get("d"), row.get("f"))
};
assert_eq!(deactivated, 5, "all sharing accounts silently deactivated");
assert_eq!(flagged, 5);
// No clue at the authz surface: a key on a deactivated account resolves
// as an ordinary 401, indistinguishable from an unknown key.
let acct: uuid::Uuid = pool
.fetch_one(
sqlx::query(
"SELECT a.id FROM accounts a JOIN users u ON u.id = a.owner_user_id \
WHERE u.registration_fingerprint = $1 LIMIT 1",
)
.bind(&fp),
)
.await
.unwrap()
.get("id");
let raw = "sk-helexa-deactivated-probe";
pool.execute(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix) VALUES ($1, $2, 'sk-helexa-')",
)
.bind(acct)
.bind(sha256(raw)),
)
.await
.unwrap();
let r = post(
format!("{base}/authz/v1/resolve"),
json!({"api_key": raw}),
Some(CLIENT_TOKEN),
)
.await;
assert_eq!(
r.status(),
401,
"deactivated account's key looks like any invalid key"
);
}
#[tokio::test]
async fn topup_redeem_raises_allocation_single_use() {
let Some((base, pool)) = spawn_or_skip("topup_redeem_raises_allocation_single_use").await
else {
return;
};
let email = unique_email();
post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123"}),
None,
)
.await;
pool.execute(
sqlx::query("UPDATE users SET email_verified = true WHERE email = $1").bind(&email),
)
.await
.unwrap();
let token = post(
format!("{base}/web/v1/login"),
json!({"email": email, "password": "password123"}),
None,
)
.await
.json::<Value>()
.await
.unwrap()["token"]
.as_str()
.unwrap()
.to_string();
// Mint a code worth 500_000 (mint path used by the CLI/faucet).
let codes = helexa_upstream::topup::mint(&pool, 500_000, 1, Some("test"))
.await
.unwrap();
let code = &codes[0];
// Redeem → allocation_total rises from the 1_000_000 free grant.
let r = post(
format!("{base}/web/v1/redeem"),
json!({"code": code}),
Some(&token),
)
.await;
assert_eq!(r.status(), 200);
assert_eq!(
r.json::<Value>().await.unwrap()["allocation_total"],
1_500_000
);
// Single-use: a second redemption fails generically (no oracle).
let r = post(
format!("{base}/web/v1/redeem"),
json!({"code": code}),
Some(&token),
)
.await;
assert_eq!(r.status(), 400);
// Unknown code: same generic 400.
let r = post(
format!("{base}/web/v1/redeem"),
json!({"code": "helexa-topup-does-not-exist"}),
Some(&token),
)
.await;
assert_eq!(r.status(), 400);
}
#[tokio::test]
async fn topup_concurrent_double_redeem_one_winner() {
let Some((base, pool)) = spawn_or_skip("topup_concurrent_double_redeem_one_winner").await
else {
return;
};
// Two verified accounts.
let mut tokens = Vec::new();
for _ in 0..2 {
let email = unique_email();
post(
format!("{base}/web/v1/register"),
json!({"email": email, "password": "password123"}),
None,
)
.await;
pool.execute(
sqlx::query("UPDATE users SET email_verified = true WHERE email = $1").bind(&email),
)
.await
.unwrap();
let t = post(
format!("{base}/web/v1/login"),
json!({"email": email, "password": "password123"}),
None,
)
.await
.json::<Value>()
.await
.unwrap()["token"]
.as_str()
.unwrap()
.to_string();
tokens.push(t);
}
let code = helexa_upstream::topup::mint(&pool, 100, 1, None)
.await
.unwrap()
.remove(0);
// Both accounts race to redeem the same code; exactly one wins.
let (a, b) = tokio::join!(
post(
format!("{base}/web/v1/redeem"),
json!({"code": code}),
Some(&tokens[0])
),
post(
format!("{base}/web/v1/redeem"),
json!({"code": code}),
Some(&tokens[1])
),
);
let wins = [a.status(), b.status()]
.iter()
.filter(|s| s.as_u16() == 200)
.count();
assert_eq!(wins, 1, "exactly one redemption wins the single-use code");
}

View File

@@ -16,7 +16,7 @@ use cortex_core::discovery::{DiscoveryResponse, HealthResponse};
use cortex_core::entitlements::{HEADER_ACCOUNT_ID, HEADER_KEY_ID};
use cortex_core::harness::ModelSpec;
use cortex_core::openai::{ChatCompletionRequest, MessageContent};
use cortex_core::responses::{ResponsesRequest, ResponsesUsage};
use cortex_core::responses::{OutputTokensDetails, ResponsesRequest, ResponsesUsage};
use futures::stream::{self, StreamExt};
use serde_json::{Value, json};
use std::convert::Infallible;
@@ -418,8 +418,14 @@ async fn responses(
input_tokens: u.prompt_tokens,
output_tokens: u.completion_tokens,
total_tokens: u.prompt_tokens + u.completion_tokens,
// Non-streaming reasoning accounting deferred (#64).
output_tokens_details: None,
// Carry the reasoning sub-count through from the chat
// usage — the non-streaming path now splits off the
// `<think>` span and counts it (see `split_off_reasoning`).
output_tokens_details: u.completion_tokens_details.as_ref().map(|d| {
OutputTokensDetails {
reasoning_tokens: d.reasoning_tokens,
}
}),
input_tokens_details: None,
});
let meta = openai_responses::ResponseMeta {

View File

@@ -23,17 +23,17 @@ use candle_transformers::models::qwen3_moe as qwen3_moe_dense;
use cortex_core::harness::{Harness, HarnessHealth, ModelInfo, ModelSpec};
use cortex_core::openai::{
ChatCompletionChoice, ChatCompletionChunk, ChatCompletionRequest, ChatCompletionResponse,
ChatMessage, MessageContent, Usage,
ChatMessage, CompletionTokensDetails, MessageContent, Usage,
};
use crate::wire::{
FinishReason, InferenceEvent, ReasoningTokenPair, ToolCallTokenPair,
FinishReason, FinishTiming, InferenceEvent, ReasoningTokenPair, ToolCallTokenPair,
detect_reasoning_token_pair, detect_tool_call_token_pair, openai_chat as wire_chat,
};
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
#[cfg(feature = "cuda")]
use std::time::Duration;
use std::time::{SystemTime, UNIX_EPOCH};
@@ -205,23 +205,50 @@ impl LoadedHandle {
/// `NEURON_MAX_PROMPT_TOKENS`, when explicitly set, is applied as a
/// clamp-only upper bound on the derived `context` — a backstop, not
/// the authority. Unset → no clamp; the derivation stands alone.
pub async fn derived_limit(
/// Refresh the cached free-VRAM reading used by [`Self::derived_limit`]
/// (#53). Queries the device worker — so it MUST run off the request
/// path (background refresher / load-time seed), never from a control
/// endpoint, since the query queues behind inference on the worker.
/// Single-GPU caches the device's free VRAM; TP caches the tightest
/// free across ranks (the same value `derived_limit` used pre-cache).
pub async fn refresh_free_mb(&self) {
let free = match self {
LoadedHandle::Single(m) => m.query_vram().await.0,
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => m.query_vram_tightest_free_mb().await,
};
// Don't clobber a good cached value with a transient `0`
// (worker gone/poisoned sentinel).
if free > 0 {
match self {
LoadedHandle::Single(m) => m.last_free_mb.store(free, Ordering::Release),
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => m.last_free_mb.store(free, Ordering::Release),
}
}
}
pub fn derived_limit(
&self,
cfg: &crate::config::ContextLimitConfig,
) -> Option<cortex_core::harness::ModelLimit> {
if !cfg.enabled {
return None;
}
// Read the *cached* free VRAM — never query the device worker here.
// This runs on `GET /models`; a live query would queue behind
// inference on the worker thread and stall the control plane (#53).
// The cache is refreshed off the request path (load + background task).
let (profile, free_mb, rate) = match self {
LoadedHandle::Single(m) => (
m.context_profile?,
m.query_vram().await.0,
m.last_free_mb.load(Ordering::Acquire),
m.prefill_rate.get(),
),
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => (
m.context_profile?,
m.query_vram_tightest_free_mb().await,
m.last_free_mb.load(Ordering::Acquire),
m.prefill_rate.get(),
),
};
@@ -391,6 +418,13 @@ pub struct LoadedModel {
/// request-path enforcement reads this — `0` means "not derived yet"
/// → fall back to the static `NEURON_MAX_PROMPT_TOKENS`.
pub derived_input_cap: AtomicUsize,
/// Cached free VRAM (MiB) for the control plane (#53). `derived_limit`
/// (served by `GET /models`) reads this instead of querying the device
/// worker, which during inference is saturated processing forward jobs —
/// a live query would queue behind them and stall `/models`, tripping
/// cortex's health poller into marking the node unhealthy. Refreshed off
/// the request path: seeded at load, then by a background task.
pub last_free_mb: AtomicU64,
}
impl LoadedModel {
@@ -503,6 +537,10 @@ pub struct TpLoadedModel {
/// Mint for pool-wide snapshot ids. Plain counter; uniqueness only
/// needs to hold per model lifetime (snapshots die with the model).
pub next_snapshot_id: std::sync::atomic::AtomicU64,
/// Cached tightest free VRAM (MiB) for the control plane (#53) — see
/// [`LoadedModel::last_free_mb`]. Read by `derived_limit` so `GET /models`
/// never fans a VRAM query out to the (inference-saturated) TP workers.
pub last_free_mb: AtomicU64,
}
#[cfg(feature = "cuda")]
@@ -746,6 +784,39 @@ impl ModelArch {
}
}
/// Split a non-streaming completion's generated tokens into the
/// visible answer and the leading reasoning span.
///
/// Reasoning models (Qwen3 `<think>`, DeepSeek-R1, …) emit their
/// chain-of-thought *before* the answer, and the chat template injects
/// the **opening** marker into the prompt — so the generated tokens look
/// like `…reasoning… </think> …answer…` with no opening marker present
/// in the output. The streaming path drops reasoning as
/// [`InferenceEvent::ReasoningDelta`]; the non-streaming path has to do
/// the equivalent post-hoc or the chain-of-thought leaks into the
/// assistant `content` (which broke agent-zero v2.0, whose parser
/// expected the bare JSON answer, not a `<think>` preamble).
///
/// Returns `(content_ids, reasoning_token_count)`. Strategy: if the model
/// declares a reasoning marker pair and its **close** token appears in
/// `generated_ids`, everything up to and including the last close token is
/// reasoning and only the tail is the answer. Otherwise (non-reasoning
/// model, thinking disabled, or a generation truncated mid-reasoning) the
/// tokens are returned unchanged. Splitting on the token id — not a
/// decoded `</think>` string — keeps this robust against tokenizer
/// byte-fallback and special-token handling.
fn split_off_reasoning<'a>(
generated_ids: &'a [u32],
reasoning: Option<&ReasoningTokenPair>,
) -> (&'a [u32], u64) {
if let Some(pair) = reasoning
&& let Some(idx) = generated_ids.iter().rposition(|&t| t == pair.close_id)
{
return (&generated_ids[idx + 1..], (idx + 1) as u64);
}
(generated_ids, 0)
}
/// Squeeze any leading singleton dims off the logits tensor so the
/// caller gets a rank-1 `[vocab_size]` slice ready for sampling. Bails
/// on a non-singleton leading dim (would mean a batched forward, which
@@ -1109,6 +1180,32 @@ fn debug_poison_armed(model_id: &str) -> bool {
armed && !FIRED.swap(true, Ordering::Relaxed)
}
/// Background control-plane VRAM cache refresher (#53). Every few seconds,
/// refreshes each loaded model's `last_free_mb` so `derived_limit` (served
/// by `GET /models`) reads a cached value and never queries the device
/// worker on the request path — a live query would queue behind inference
/// forward jobs on the worker thread, stalling `/models` for seconds and
/// tripping cortex's health poller into evicting the node from routing.
/// Holds a `Weak` so a shutting-down harness lets the task exit. The query
/// itself may queue behind inference, but that only delays this background
/// refresh — no request-path caller is ever blocked.
async fn vram_cache_refresh_loop(weak: std::sync::Weak<CandleHarness>) {
const REFRESH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(5);
loop {
tokio::time::sleep(REFRESH_INTERVAL).await;
let Some(this) = weak.upgrade() else {
return; // harness dropped — exit
};
// Snapshot handles, then release the read lock before awaiting the
// (possibly slow) worker queries so we never hold it across an await.
let handles: Vec<LoadedHandle> = this.models.read().await.values().cloned().collect();
drop(this);
for handle in handles {
handle.refresh_free_mb().await;
}
}
}
/// Background auto-recovery task (#17). Drains poisoned model ids and
/// rebuilds each via [`CandleHarness::recover_one`]. Holds a `Weak` so a
/// shutting-down harness lets the task exit; processes one id at a time,
@@ -1282,25 +1379,67 @@ fn validate_vision_prefill(prompt_len: usize, vram_free_mb: u64) -> Result<(), I
/// the caller as `max`), or if free VRAM is below the floor. Enforcing
/// the *derived* cap means a VRAM-tight host rejects a prompt that
/// wouldn't fit, instead of accepting it and OOMing mid-prefill.
///
/// The third VRAM check — the length-aware backstop (#65) — closes the
/// poll-vs-request snapshot gap #67 leaves open. `max` is
/// `effective_prompt_cap()`, the input budget derived at **/models poll
/// time** from the tightest card's free VRAM *then*. If free VRAM has
/// since dropped (a co-resident model loaded, a concurrent prefill grew
/// its KV), a prompt at-or-below that now-stale cap still clears the
/// static floor yet no longer fits — and OOMs mid-prefill, poisoning the
/// device context (the 2026-05-26 beast incident the #47 work exists to
/// eliminate). So we re-run the same length×KV-vs-VRAM physics #67 uses
/// for the cap, but against **request-time** free VRAM, reusing the
/// model's [`ContextProfile`] rather than re-deriving the KV cost. This
/// gives the text path the live-VRAM guard the vision path already has
/// (`validate_vision_prefill`). `profile`/`kv_bytes_per_token_per_card`
/// are per-card and `vram_free_mb` is the tightest card's free VRAM, so
/// the two are commensurable on both single-GPU and TP loads.
fn validate_request(
prompt_len: usize,
vram_free_mb: u64,
max: usize,
profile: Option<&super::context_limit::ContextProfile>,
cfg: &crate::config::ContextLimitConfig,
) -> Result<(), InferenceError> {
if prompt_len > max {
return Err(InferenceError::PromptTooLong { prompt_len, max });
}
// VRAM check is skipped on CPU loads (vram_free_mb == 0 sentinel)
// VRAM checks are skipped on CPU loads (vram_free_mb == 0 sentinel)
// because the (0, 0) reply from `query_vram` is also what a missing
// worker returns. The CPU path has no per-GPU memory limit anyway —
// host RAM is bounded by the OOM killer, not this check.
if vram_free_mb == 0 {
return Ok(());
}
let min = min_free_vram_mb();
if vram_free_mb != 0 && vram_free_mb < min {
if vram_free_mb < min {
return Err(InferenceError::InsufficientVram {
free_mb: vram_free_mb,
required_mb: min,
});
}
// Length-aware backstop (#65): KV the whole sequence (prompt +
// generation reserve) will occupy, plus the prefill activation
// headroom, plus the static floor as an additive cushion — all per
// card. A degenerate zero-KV profile (no full-attention layers) or a
// model with no captured profile skips this and rides the floor
// check above, mirroring `derive_limit`'s VRAM-ceiling fallback.
if let Some(profile) = profile
&& profile.kv_bytes_per_token_per_card > 0
{
let tokens = (prompt_len as u64).saturating_add(cfg.output_reserve_tokens as u64);
let kv_mb = profile.kv_bytes_per_token_per_card.saturating_mul(tokens) / (1024 * 1024);
let required_mb = kv_mb
.saturating_add(cfg.activation_headroom_mb)
.saturating_add(min);
if required_mb > vram_free_mb {
return Err(InferenceError::InsufficientVram {
free_mb: vram_free_mb,
required_mb,
});
}
}
Ok(())
}
@@ -1595,6 +1734,11 @@ impl CandleHarness {
if tokio::runtime::Handle::try_current().is_ok() {
let weak = Arc::downgrade(&this);
tokio::spawn(recovery_loop(weak, recovery_rx));
// Control-plane VRAM cache refresher (#53): keeps each loaded
// model's `last_free_mb` current off the request path, so
// `derived_limit` / `GET /models` never query the device worker
// (which is saturated during inference) and never stall.
tokio::spawn(vram_cache_refresh_loop(Arc::downgrade(&this)));
}
this
}
@@ -2181,7 +2325,13 @@ impl CandleHarness {
"chat_completion: starting"
);
validate_request(prompt_len, vram_free_mb, loaded.effective_prompt_cap())?;
validate_request(
prompt_len,
vram_free_mb,
loaded.effective_prompt_cap(),
loaded.context_profile.as_ref(),
&self.context_limit_cfg,
)?;
if vision_route.is_some() {
validate_vision_prefill(prompt_len, vram_free_mb)?;
}
@@ -2337,21 +2487,36 @@ impl CandleHarness {
)));
};
// Strip the leading `<think>` span so the chain-of-thought
// doesn't leak into `content` (the streaming path drops it
// as ReasoningDelta; this is the non-streaming equivalent).
let (content_ids, reasoning_tokens) =
split_off_reasoning(&generated_ids, loaded.reasoning_tokens.as_ref());
let completion_text = loaded
.tokenizer
.decode(&generated_ids, true)
.decode(content_ids, true)
.map_err(|e| InferenceError::Other(anyhow::anyhow!("detokenize: {e}")))?;
// The first answer token after `</think>` is usually a
// newline pair; trim it so `content` starts at the answer.
let completion_text = if reasoning_tokens > 0 {
completion_text.trim_start().to_string()
} else {
completion_text
};
let usage = Usage {
prompt_tokens: prompt_len as u64,
completion_tokens: generated_ids.len() as u64,
total_tokens: (prompt_len + generated_ids.len()) as u64,
// Reasoning accounting is streaming-only: the
// non-streaming path doesn't track `in_reasoning`
// (would require post-hoc <think> span parsing).
// Deferred — see #64.
completion_tokens_details: None,
// `reasoning_tokens` is an additive sub-count of
// `completion_tokens` (which still counts every
// generated token, reasoning included).
completion_tokens_details: (reasoning_tokens > 0)
.then_some(CompletionTokensDetails { reasoning_tokens }),
prompt_tokens_details: None,
// Non-streaming path: prefill/decode split is only
// surfaced on the streaming Finish event today (#85).
helexa_timing: None,
};
tracing::info!(
@@ -2635,7 +2800,13 @@ impl CandleHarness {
);
}
validate_request(prompt_len, vram_free_mb, loaded.effective_prompt_cap())?;
validate_request(
prompt_len,
vram_free_mb,
loaded.effective_prompt_cap(),
loaded.context_profile.as_ref(),
&self.context_limit_cfg,
)?;
if vision_route.is_some() {
validate_vision_prefill(prompt_len, vram_free_mb)?;
}
@@ -2959,7 +3130,7 @@ impl Harness for CandleHarness {
// physics + live free VRAM + measured prefill rate. `None`
// for arches without a context profile. `cost` stays
// operator-set in the catalogue, filled by the gateway.
let limit = h.derived_limit(&self.context_limit_cfg).await;
let limit = h.derived_limit(&self.context_limit_cfg);
out.push(ModelInfo {
id: h.model_id().into(),
harness: "candle".into(),
@@ -3209,6 +3380,7 @@ impl Harness for CandleHarness {
context_profile,
prefill_rate: super::context_limit::PrefillRateEma::new(),
derived_input_cap: AtomicUsize::new(0),
last_free_mb: AtomicU64::new(0),
});
if loaded.prefix_cache.is_some() {
tracing::info!(
@@ -3219,6 +3391,14 @@ impl Harness for CandleHarness {
);
}
// Seed the control-plane VRAM cache (#53) while the worker is idle
// (load just finished), so `/models` has a value before the
// background refresher's first tick and never queries the worker.
let (free_mb, _) = loaded.query_vram().await;
if free_mb > 0 {
loaded.last_free_mb.store(free_mb, Ordering::Release);
}
let mut models = self.models.write().await;
models.insert(spec.model_id.clone(), LoadedHandle::Single(loaded));
tracing::info!(model = %spec.model_id, "model loaded");
@@ -3469,6 +3649,7 @@ impl CandleHarness {
),
prefill_rate: super::context_limit::PrefillRateEma::new(),
derived_input_cap: AtomicUsize::new(0),
last_free_mb: AtomicU64::new(0),
next_snapshot_id: std::sync::atomic::AtomicU64::new(1),
});
if tp_loaded.prefix_cache.is_some() {
@@ -3480,6 +3661,14 @@ impl CandleHarness {
);
}
// Seed the control-plane VRAM cache (#53) — tightest free across
// ranks, while the workers are idle post-load — so `/models` never
// fans a query out to the inference-busy TP workers.
let free_mb = tp_loaded.query_vram_tightest_free_mb().await;
if free_mb > 0 {
tp_loaded.last_free_mb.store(free_mb, Ordering::Release);
}
let mut models = self.models.write().await;
models.insert(spec.model_id.clone(), LoadedHandle::Tp(tp_loaded));
tracing::info!(
@@ -3546,8 +3735,11 @@ impl CandleHarness {
}
let tp_for_marker = Arc::clone(&tp);
let handle =
tokio::spawn(chat_completion_tp_inner(tp, request, principal).instrument(span.clone()));
let context_limit_cfg = self.context_limit_cfg.clone();
let handle = tokio::spawn(
chat_completion_tp_inner(tp, request, principal, context_limit_cfg)
.instrument(span.clone()),
);
match handle.await {
Ok(Ok(resp)) => Ok(resp),
Ok(Err(e)) => {
@@ -3759,7 +3951,13 @@ impl CandleHarness {
"TP chat_completion (stream): starting"
);
validate_request(prompt_len, vram_free_mb, tp.effective_prompt_cap())?;
validate_request(
prompt_len,
vram_free_mb,
tp.effective_prompt_cap(),
tp.context_profile.as_ref(),
&self.context_limit_cfg,
)?;
if vision_route.is_some() {
validate_vision_prefill(prompt_len, vram_free_mb)?;
}
@@ -3807,6 +4005,11 @@ impl CandleHarness {
// call — promotes the terminal finish_reason to ToolCalls
// so Anthropic clients see stop_reason: tool_use.
let mut emitted_tool_call = false;
// Prefill/decode split timers (#85). Declared outside 'work
// so the terminal Finish — built after the block exits — can
// read them; populated at the prefill→decode boundary inside.
let mut prefill_ms_measured: u32 = 0;
let mut decode_start: Option<std::time::Instant> = None;
'work: {
// Prefix-cache decision (#11): vision requests
@@ -3934,14 +4137,16 @@ impl CandleHarness {
break 'work;
}
};
let prefill_elapsed = prefill_start.elapsed();
prefill_ms_measured = prefill_elapsed.as_millis() as u32;
tp_for_task
.prefill_rate
.record(prompt_len, prefill_start.elapsed());
.record(prompt_len, prefill_elapsed);
let (post_prefill_vram_free_mb, _) = tp_for_task.query_vram().await;
tracing::info!(
model = %model_id,
prompt_len,
prefill_ms = prefill_start.elapsed().as_millis(),
prefill_ms = prefill_elapsed.as_millis(),
vram_free_mb = post_prefill_vram_free_mb,
"TP chat_completion (stream): prefill complete"
);
@@ -3966,6 +4171,8 @@ impl CandleHarness {
break 'work;
}
};
// Decode-phase timer for the Finish prefill/decode split (#85).
decode_start = Some(std::time::Instant::now());
if Some(next_token) == eos_id {
finish_reason = FinishReason::Stop;
@@ -4243,6 +4450,13 @@ impl CandleHarness {
prompt_tokens: prompt_len as u32,
completion_tokens: all_tokens.len() as u32,
reasoning_tokens: reasoning_token_count,
timing: Some(FinishTiming {
prefill_ms: prefill_ms_measured,
decode_ms: decode_start
.map(|d| d.elapsed().as_millis() as u32)
.unwrap_or(0),
prefill_tokens: prompt_len as u32,
}),
})
.await;
}
@@ -4280,6 +4494,7 @@ async fn chat_completion_tp_inner(
tp: Arc<TpLoadedModel>,
request: ChatCompletionRequest,
principal: Option<String>,
context_limit_cfg: crate::config::ContextLimitConfig,
) -> Result<ChatCompletionResponse, InferenceError> {
let req_start = std::time::Instant::now();
let model_id = request.model.clone();
@@ -4363,7 +4578,13 @@ async fn chat_completion_tp_inner(
"TP chat_completion: starting"
);
validate_request(prompt_len, vram_free_mb, tp.effective_prompt_cap())?;
validate_request(
prompt_len,
vram_free_mb,
tp.effective_prompt_cap(),
tp.context_profile.as_ref(),
&context_limit_cfg,
)?;
if vision_route.is_some() {
validate_vision_prefill(prompt_len, vram_free_mb)?;
}
@@ -4565,19 +4786,31 @@ async fn chat_completion_tp_inner(
}
drop(pool);
// Strip the leading `<think>` span (see `split_off_reasoning` and the
// single-GPU path) so the chain-of-thought doesn't leak into `content`.
let (content_ids, reasoning_tokens) =
split_off_reasoning(&generated, tp.reasoning_tokens.as_ref());
let completion_text = tp
.tokenizer
.decode(&generated, true)
.decode(content_ids, true)
.map_err(|e| InferenceError::Other(anyhow::anyhow!("detokenize: {e}")))?;
let completion_text = if reasoning_tokens > 0 {
completion_text.trim_start().to_string()
} else {
completion_text
};
let usage = Usage {
prompt_tokens: prompt_len as u64,
completion_tokens: generated.len() as u64,
total_tokens: (prompt_len + generated.len()) as u64,
// Reasoning accounting is streaming-only (non-streaming TP path
// doesn't track `in_reasoning`). Deferred — see #64.
completion_tokens_details: None,
// `reasoning_tokens` is an additive sub-count of `completion_tokens`.
completion_tokens_details: (reasoning_tokens > 0)
.then_some(CompletionTokensDetails { reasoning_tokens }),
prompt_tokens_details: None,
// Non-streaming path: prefill/decode split is only surfaced on
// the streaming Finish event today (#85).
helexa_timing: None,
};
tracing::info!(
@@ -5907,7 +6140,8 @@ async fn stream_inference_via_worker(
}
}
};
prefill_rate.record(prefill_prompt_len, prefill_start.elapsed());
let prefill_elapsed = prefill_start.elapsed();
prefill_rate.record(prefill_prompt_len, prefill_elapsed);
let logits = Tensor::new(logits_vec.as_slice(), &Device::Cpu)?;
let mut next_token = match sample_with_penalty(&logits, &all_tokens, &mut logits_processor) {
Ok(t) => t,
@@ -5920,6 +6154,8 @@ async fn stream_inference_via_worker(
return Err(e);
}
};
// Decode-phase timer for the Finish prefill/decode split (#85).
let decode_start = std::time::Instant::now();
// Per-token routing. `tokenizers::DecodeStream` carries five
// generic parameters (`M, N, PT, PP, D`) which makes naming
@@ -6064,6 +6300,11 @@ async fn stream_inference_via_worker(
prompt_tokens: prompt_tokens.len() as u32,
completion_tokens: all_tokens.len() as u32,
reasoning_tokens: reasoning_token_count,
timing: Some(FinishTiming {
prefill_ms: prefill_elapsed.as_millis() as u32,
decode_ms: decode_start.elapsed().as_millis() as u32,
prefill_tokens: prefill_prompt_len as u32,
}),
})
.await;
@@ -6198,6 +6439,10 @@ fn run_inference_streaming(
// See `inference_tp_stream`: promotes finish_reason to ToolCalls.
let mut emitted_tool_call = false;
// Time prefill and decode separately so the Finish event can carry
// a server-measured prefill/decode split (#85) instead of leaving
// the client to infer both from SSE chunk arrival.
let prefill_start = std::time::Instant::now();
let reused = restore_or_clear_local(arch, prefix_cache, prompt_tokens)?;
// Two-stage prefill around the retokenization-stable snapshot
// boundary — see `run_inference_via_worker`.
@@ -6216,6 +6461,8 @@ fn run_inference_streaming(
None => chunked_prefill_local(arch, device, prompt_tokens, reused)?,
};
let mut next_token = sample_with_penalty(&logits, &all_tokens, &mut logits_processor)?;
let prefill_elapsed = prefill_start.elapsed();
let decode_start = std::time::Instant::now();
// Per-token routing block, used at both the prefill-sample
// tail and the decode loop. Macros are ugly but Rust's
@@ -6324,6 +6571,11 @@ fn run_inference_streaming(
prompt_tokens: prompt_tokens.len() as u32,
completion_tokens: all_tokens.len() as u32,
reasoning_tokens: reasoning_token_count,
timing: Some(FinishTiming {
prefill_ms: prefill_elapsed.as_millis() as u32,
decode_ms: decode_start.elapsed().as_millis() as u32,
prefill_tokens: prompt_tokens.len() as u32,
}),
});
Ok(())
}
@@ -6348,6 +6600,60 @@ mod tests {
const IM_START: u32 = 999;
fn think_pair() -> ReasoningTokenPair {
ReasoningTokenPair {
open_id: 100,
close_id: 200,
open_text: "<think>".into(),
close_text: "</think>".into(),
}
}
#[test]
fn split_off_reasoning_strips_up_to_close_marker() {
// [reasoning_a, reasoning_b, </think>, answer_x, answer_y]
let ids = [10, 11, 200, 42, 43];
let (content, reasoning) = split_off_reasoning(&ids, Some(&think_pair()));
assert_eq!(content, &[42, 43]);
assert_eq!(reasoning, 3); // two reasoning tokens + the close marker
}
#[test]
fn split_off_reasoning_no_close_marker_returns_all() {
// Thinking disabled / model never closed the span: return as-is.
let ids = [42, 43, 44];
let (content, reasoning) = split_off_reasoning(&ids, Some(&think_pair()));
assert_eq!(content, &ids);
assert_eq!(reasoning, 0);
}
#[test]
fn split_off_reasoning_no_marker_pair_is_noop() {
let ids = [1, 2, 3];
let (content, reasoning) = split_off_reasoning(&ids, None);
assert_eq!(content, &ids);
assert_eq!(reasoning, 0);
}
#[test]
fn split_off_reasoning_close_at_end_yields_empty_content() {
// All reasoning, answer truncated to nothing after the marker.
let ids = [10, 11, 200];
let (content, reasoning) = split_off_reasoning(&ids, Some(&think_pair()));
assert!(content.is_empty());
assert_eq!(reasoning, 3);
}
#[test]
fn split_off_reasoning_splits_on_last_close_marker() {
// Defensive: if the model emits its own <think></think> pair plus
// the prompt-injected one, split on the LAST close marker.
let ids = [200, 10, 200, 42];
let (content, reasoning) = split_off_reasoning(&ids, Some(&think_pair()));
assert_eq!(content, &[42]);
assert_eq!(reasoning, 3);
}
#[test]
fn stable_snapshot_cut_lands_after_last_im_start() {
// ChatML shape: [im_start, "system", ..., im_start, "user",
@@ -6680,6 +6986,110 @@ mod tests {
assert!(validate_vision_prefill(12_960, 12_445).is_ok());
}
// ── #65: request-time length-aware VRAM backstop (text prefill) ──
/// A beast-like profile: 16 full-attn layers, 4 kv heads, head_dim
/// 256, f16, TP=2 → 32 KiB/token/card (same numbers as the
/// `context_limit` unit tests). At defaults this makes the
/// length-aware footprint `(prompt_len + 8192)/32 + 2048 + 1500` MiB
/// per card.
fn backstop_profile() -> super::super::context_limit::ContextProfile {
super::super::context_limit::ContextProfile {
max_position_embeddings: 262_144,
kv_bytes_per_token_per_card: super::super::context_limit::kv_bytes_per_token(
16, 4, 256, 2, 2,
),
world_size: 2,
}
}
/// A prompt under the cap with ample free VRAM passes; the same
/// prompt over the cap is `PromptTooLong` before any VRAM math.
#[test]
fn validate_request_cap_and_fit() {
let cfg = crate::config::ContextLimitConfig::default();
let profile = backstop_profile();
// Under cap, 40 GB free → fits.
assert!(validate_request(8_000, 40_000, 100_000, Some(&profile), &cfg).is_ok());
// Over the cap → PromptTooLong, independent of VRAM.
assert!(matches!(
validate_request(100_001, 40_000, 100_000, Some(&profile), &cfg),
Err(InferenceError::PromptTooLong { .. })
));
}
/// The CPU sentinel (`vram_free_mb == 0`) skips every VRAM check,
/// including the new length-aware one — host RAM is the OOM killer's
/// problem, not this guard's.
#[test]
fn validate_request_cpu_sentinel_skips_vram() {
let cfg = crate::config::ContextLimitConfig::default();
let profile = backstop_profile();
assert!(validate_request(1_000_000, 0, 2_000_000, Some(&profile), &cfg).is_ok());
}
/// The static floor remains a backstop: free VRAM below
/// `min_free_vram_mb()` is rejected before the length-aware estimate
/// even runs (so `required_mb` is the floor, not the KV footprint).
#[test]
fn validate_request_static_floor_still_binds() {
let cfg = crate::config::ContextLimitConfig::default();
let profile = backstop_profile();
assert!(matches!(
validate_request(10, 800, 100_000, Some(&profile), &cfg),
Err(InferenceError::InsufficientVram {
free_mb: 800,
required_mb: 1500
})
));
}
/// A model with no captured profile (non-qwen3_5 arch) has no
/// length-aware physics to apply, so it rides only the static floor —
/// a fitting prompt with VRAM above the floor passes.
#[test]
fn validate_request_no_profile_rides_floor() {
let cfg = crate::config::ContextLimitConfig::default();
assert!(validate_request(500_000, 5_000, 1_000_000, None, &cfg).is_ok());
}
/// The acceptance test (#65): a cap derived against *ample* free VRAM
/// is later applied at request time against *tightened* free VRAM. A
/// prompt sized exactly at the now-stale `effective_prompt_cap()`
/// clears the cap and the static floor, yet no longer fits — the
/// length-aware backstop catches it with a clean `InsufficientVram`
/// instead of an OOM-poisoned context. Same prompt with the original
/// ample VRAM still passes, proving the guard only bites on staleness.
#[test]
fn validate_request_catches_poll_vs_request_staleness() {
let cfg = crate::config::ContextLimitConfig::default();
let profile = backstop_profile();
// Cap derived at /models poll time with 40 GB free on the tightest
// card — throughput binds, giving input = 87040 (the issue's
// worked beast figure).
let limit = super::super::context_limit::derive_limit(&profile, 40_000, 800.0, None, &cfg);
let cap = limit.input.expect("input budget derived");
assert_eq!(cap, 87_040);
// With that same ample VRAM, a prompt at the cap still fits.
assert!(validate_request(cap, 40_000, cap, Some(&profile), &cfg).is_ok());
// Now free VRAM has dropped to 5 GB between the poll and the
// request (a co-resident model loaded). The prompt is still ≤ cap
// and clears the 1500 MiB floor, but its footprint —
// (87040 + 8192)/32 + 2048 + 1500 = 6524 MiB — exceeds 5000 MiB.
let err = validate_request(cap, 5_000, cap, Some(&profile), &cfg)
.expect_err("stale cap must not let an over-VRAM prompt through");
assert!(matches!(
err,
InferenceError::InsufficientVram {
free_mb: 5_000,
required_mb: 6_524
}
));
}
// ── Tool-call body parsing ───────────────────────────────────────
fn weather_schemas() -> ToolSchemas {

View File

@@ -100,9 +100,9 @@ pub const KV_CACHE_DTYPE_BYTES: usize = 2;
/// state, not a growing cache). Sharded across the TP world: per-rank
/// KV-head count is `n_kv_heads / world_size`.
///
/// `2 ×` accounts for K and V. Shared by the limit derivation here and
/// the per-rank load-time logging in the TP paths (and, in future, by
/// #65's length-aware pre-flight guard).
/// `2 ×` accounts for K and V. Shared by the limit derivation here, the
/// per-rank load-time logging in the TP paths, and #65's request-time
/// length-aware pre-flight guard (`candle::validate_request`).
pub fn kv_bytes_per_token(
n_full_attn_layers: usize,
n_kv_heads: usize,

View File

@@ -84,9 +84,38 @@ pub enum InferenceEvent {
/// `output_tokens_details.reasoning_tokens` (responses).
/// Zero for non-reasoning models.
reasoning_tokens: u32,
/// Server-measured prefill/decode timing for the request, or
/// `None` on paths that don't measure it (CPU fallback that
/// doesn't instrument, tests). Streaming projectors surface
/// this as a `helexa_timing` extension on the OpenAI `usage`
/// object so the bench harness can compute true prefill vs
/// decode tok/s instead of inferring both from client-side
/// SSE arrival (#85).
timing: Option<FinishTiming>,
},
}
/// Server-measured timing for one completed inference, attached to
/// [`InferenceEvent::Finish`]. The whole point is to separate the two
/// phases the client cannot tell apart from chunk-arrival timing:
/// prefill (tokenize + prompt forward pass, ending at the first
/// sampled token) and decode (every subsequent token through EOS /
/// `max_tokens`).
#[derive(Debug, Clone, Copy)]
pub struct FinishTiming {
/// Wall-clock of the prefill phase in milliseconds: from the start
/// of the prompt forward pass(es) to the first sampled token.
pub prefill_ms: u32,
/// Wall-clock of the decode phase in milliseconds: from the first
/// sampled token to stream end.
pub decode_ms: u32,
/// Prompt tokens submitted to the prefill forward pass — the
/// denominator for prefill tok/s. With prefix-KV-cache hits (#11)
/// the elapsed `prefill_ms` drops while this stays the full prompt
/// length, so a high implied rate is itself the cache-hit signal.
pub prefill_tokens: u32,
}
/// Why a stream stopped. Stays small on purpose — anything that
/// doesn't map cleanly to one of these collapses to [`Stop`].
///

View File

@@ -22,6 +22,6 @@ pub mod openai_chat;
pub mod openai_responses;
pub use event::{
FinishReason, InferenceEvent, ReasoningTokenPair, ToolCallTokenPair,
FinishReason, FinishTiming, InferenceEvent, ReasoningTokenPair, ToolCallTokenPair,
detect_reasoning_token_pair, detect_tool_call_token_pair,
};

View File

@@ -26,11 +26,13 @@
//! producer blocks on its own send. The bounded channels
//! propagate without us writing any logic.
use cortex_core::openai::{ChatCompletionChunk, ChunkChoice, CompletionTokensDetails, Usage};
use cortex_core::openai::{
ChatCompletionChunk, ChunkChoice, CompletionTokensDetails, HelexaTiming, Usage,
};
use serde_json::json;
use tokio::sync::mpsc;
use super::event::{FinishReason, InferenceEvent, ReasoningTokenPair};
use super::event::{FinishReason, FinishTiming, InferenceEvent, ReasoningTokenPair};
/// Output channel buffer size. Mirrors the input side's bound; one
/// event maps to at most one chunk, so equal capacity keeps the
@@ -193,12 +195,14 @@ pub fn project_chat_stream_with(
prompt_tokens,
completion_tokens,
reasoning_tokens,
timing,
} => {
// The finish_reason chunk, then an OpenAI-style
// usage-only chunk (`choices: []`, `usage` populated).
// Clients (opencode) read this to track context size;
// cortex's Anthropic translator also picks `usage` up
// for its `message_delta`.
// for its `message_delta`. `timing` rides along as the
// `helexa_timing` usage extension for the bench harness (#85).
vec![
final_chunk(&id, created, &model_id, reason),
usage_chunk(
@@ -208,6 +212,7 @@ pub fn project_chat_stream_with(
prompt_tokens,
completion_tokens,
reasoning_tokens,
timing,
),
]
}
@@ -334,6 +339,7 @@ fn usage_chunk(
prompt_tokens: u32,
completion_tokens: u32,
reasoning_tokens: u32,
timing: Option<FinishTiming>,
) -> ChatCompletionChunk {
ChatCompletionChunk {
id: id.into(),
@@ -351,6 +357,14 @@ fn usage_chunk(
reasoning_tokens: reasoning_tokens as u64,
}),
prompt_tokens_details: None,
// helexa extension (#85): server-measured prefill/decode
// timing for the bench harness. Omitted on paths that don't
// measure it so standard clients see unchanged JSON.
helexa_timing: timing.map(|t| HelexaTiming {
prefill_ms: t.prefill_ms as u64,
decode_ms: t.decode_ms as u64,
prefill_tokens: t.prefill_tokens as u64,
}),
}),
extra: serde_json::Value::Object(Default::default()),
}
@@ -391,6 +405,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -413,6 +428,45 @@ mod tests {
}
}
#[tokio::test]
async fn finish_timing_surfaces_on_usage_chunk() {
// O1 (#85) wire contract: a Finish carrying FinishTiming must
// surface as `usage.helexa_timing` on the trailing usage chunk,
// which is what the bench harness reads to compute true prefill
// vs decode tok/s. Absent timing must leave it None.
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
let out_rx = project_chat_stream(rx, "id-1".into(), 1700, "m".into());
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 128,
completion_tokens: 64,
reasoning_tokens: 0,
timing: Some(FinishTiming {
prefill_ms: 200,
decode_ms: 1500,
prefill_tokens: 128,
}),
})
.await
.unwrap();
drop(tx);
let out = collect(out_rx).await;
let usage = out
.iter()
.find_map(|c| c.usage.as_ref())
.expect("usage chunk present");
let timing = usage
.helexa_timing
.as_ref()
.expect("helexa_timing populated when Finish carried timing");
assert_eq!(timing.prefill_ms, 200);
assert_eq!(timing.decode_ms, 1500);
assert_eq!(timing.prefill_tokens, 128);
}
#[tokio::test]
async fn empty_text_delta_is_dropped() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(4);
@@ -434,6 +488,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -496,6 +551,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -547,6 +603,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -592,6 +649,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -635,6 +693,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -662,6 +721,7 @@ mod tests {
prompt_tokens: 42,
completion_tokens: 5,
reasoning_tokens: 2,
timing: None,
})
.await
.unwrap();
@@ -695,6 +755,7 @@ mod tests {
prompt_tokens: 10,
completion_tokens: 7,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();

View File

@@ -29,9 +29,9 @@
use cortex_core::openai::{ChatCompletionRequest, ChatMessage, MessageContent};
use cortex_core::responses::{
OutputTokensDetails, ResponsesContentPart, ResponsesInput, ResponsesInputItem,
ResponsesMessageContent, ResponsesOutputContent, ResponsesOutputItem, ResponsesRequest,
ResponsesResponse, ResponsesUsage, events,
OutputTokensDetails, ResponsesContentPart, ResponsesInput, ResponsesInputElement,
ResponsesInputItem, ResponsesMessageContent, ResponsesOutputContent, ResponsesOutputItem,
ResponsesRequest, ResponsesResponse, ResponsesUsage, events,
};
use serde_json::{Value, json};
use tokio::sync::mpsc;
@@ -109,8 +109,26 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
});
}
ResponsesInput::Items(items) => {
for item in items {
if let Some(msg) = input_item_to_chat(item) {
for element in items {
let msg = match element {
ResponsesInputElement::Typed(item) => input_item_to_chat(item),
// Bare `{role, content}` (OpenAI EasyInputMessage —
// what litellm/agent-zero emit). `content: null`
// (e.g. an assistant turn carrying only tool calls)
// collapses to an empty string so the turn is kept.
ResponsesInputElement::EasyMessage { role, content } => Some(ChatMessage {
role,
content: content
.map(message_content_to_chat)
.unwrap_or_else(|| MessageContent::Text(String::new())),
extra: Value::Object(Default::default()),
}),
// Forward-compat: an item shape we don't model.
// Dropped rather than rejected (see
// `ResponsesInputElement::Other`).
ResponsesInputElement::Other(_) => None,
};
if let Some(msg) = msg {
messages.push(msg);
}
}
@@ -159,11 +177,18 @@ fn input_item_to_chat(item: ResponsesInputItem) -> Option<ChatMessage> {
})
}
ResponsesInputItem::FunctionCallOutput { call_id, output } => {
// `output` is either a plain string or an array of content
// parts. Render a string as-is; anything else to compact
// JSON so the tool result text reaches the model intact.
let output_text = match output {
Value::String(s) => s,
other => other.to_string(),
};
let mut extra = serde_json::Map::new();
extra.insert("tool_call_id".into(), Value::String(call_id));
Some(ChatMessage {
role: "tool".into(),
content: MessageContent::Text(output),
content: MessageContent::Text(output_text),
extra: Value::Object(extra),
})
}
@@ -192,7 +217,9 @@ fn message_content_to_chat(content: ResponsesMessageContent) -> MessageContent {
.filter_map(|p| match p {
ResponsesContentPart::InputText { text }
| ResponsesContentPart::OutputText { text, .. } => Some(text),
ResponsesContentPart::InputImage { .. } => None,
ResponsesContentPart::InputImage { .. } | ResponsesContentPart::Unknown => {
None
}
})
.collect::<Vec<_>>()
.join("\n\n");
@@ -211,6 +238,7 @@ fn message_content_to_chat(content: ResponsesMessageContent) -> MessageContent {
"image_url": { "url": image_url },
}));
}
ResponsesContentPart::Unknown => {}
}
}
MessageContent::Parts(out)
@@ -309,6 +337,9 @@ async fn run_projection(
prompt_tokens,
completion_tokens,
reasoning_tokens,
// Responses-side `helexa_timing` surfacing not wired yet;
// the bench harness reads timing off the chat path (#85).
timing: _,
} => {
finish = Some(reason);
// Surface usage on the streaming `response.completed`
@@ -535,6 +566,18 @@ mod tests {
use super::*;
use cortex_core::openai::MessageContent;
/// Wrap typed items as `input` elements. Most translator tests
/// exercise the typed path; the bare easy-message and unknown-item
/// paths have dedicated tests below.
fn typed_items(items: Vec<ResponsesInputItem>) -> ResponsesInput {
ResponsesInput::Items(
items
.into_iter()
.map(ResponsesInputElement::Typed)
.collect(),
)
}
fn meta() -> ResponseMeta {
ResponseMeta {
response_id: "resp_1".into(),
@@ -614,7 +657,7 @@ mod tests {
fn translates_input_items_to_chat_messages() {
let req = ResponsesRequest {
model: "m".into(),
input: ResponsesInput::Items(vec![
input: typed_items(vec![
ResponsesInputItem::Message {
role: "user".into(),
content: ResponsesMessageContent::Text("first".into()),
@@ -646,7 +689,7 @@ mod tests {
fn image_input_translates_to_chat_parts_array() {
let req = ResponsesRequest {
model: "m".into(),
input: ResponsesInput::Items(vec![ResponsesInputItem::Message {
input: typed_items(vec![ResponsesInputItem::Message {
role: "user".into(),
content: ResponsesMessageContent::Parts(vec![
ResponsesContentPart::InputText {
@@ -687,7 +730,7 @@ mod tests {
// it's dropped — but it must not break translation.
let req = ResponsesRequest {
model: "m".into(),
input: ResponsesInput::Items(vec![ResponsesInputItem::Message {
input: typed_items(vec![ResponsesInputItem::Message {
role: "user".into(),
content: ResponsesMessageContent::Parts(vec![
ResponsesContentPart::InputText {
@@ -729,7 +772,7 @@ mod tests {
fn text_only_parts_collapse_to_string() {
let req = ResponsesRequest {
model: "m".into(),
input: ResponsesInput::Items(vec![ResponsesInputItem::Message {
input: typed_items(vec![ResponsesInputItem::Message {
role: "user".into(),
content: ResponsesMessageContent::Parts(vec![
ResponsesContentPart::InputText {
@@ -759,7 +802,7 @@ mod tests {
fn reasoning_items_are_silently_dropped() {
let req = ResponsesRequest {
model: "m".into(),
input: ResponsesInput::Items(vec![
input: typed_items(vec![
ResponsesInputItem::Reasoning { content: vec![] },
ResponsesInputItem::Message {
role: "user".into(),
@@ -779,6 +822,74 @@ mod tests {
assert_eq!(chat.messages[0].role, "user");
}
#[test]
fn bare_easy_messages_translate_like_typed_messages() {
// The agent-zero / litellm shape: bare `{role, content}` items
// with no `type`. Deserialize from raw JSON (not hand-built)
// so this exercises the real parse path end to end.
let raw = r#"{
"model": "Qwen/Qwen3.6-27B",
"store": true,
"input": [
{"role": "system", "content": "be terse"},
{"role": "assistant", "content": "{\"tool_name\":\"response\"}"},
{"role": "user", "content": "alpha"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let chat = request_to_chat(req).unwrap();
let roles: Vec<&str> = chat.messages.iter().map(|m| m.role.as_str()).collect();
assert_eq!(roles, vec!["system", "assistant", "user"]);
assert!(matches!(
&chat.messages[2].content,
MessageContent::Text(t) if t == "alpha"
));
}
#[test]
fn null_content_and_unknown_items_survive_translation() {
// An assistant turn with `content: null` is kept (empty text);
// an unmodeled item type is dropped, not rejected.
let raw = r#"{
"model": "m",
"input": [
{"role": "assistant", "content": null},
{"type": "item_reference", "id": "x"},
{"role": "user", "content": "go"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let chat = request_to_chat(req).unwrap();
// assistant(null) kept, item_reference dropped, user kept.
let roles: Vec<&str> = chat.messages.iter().map(|m| m.role.as_str()).collect();
assert_eq!(roles, vec!["assistant", "user"]);
assert!(matches!(
&chat.messages[0].content,
MessageContent::Text(t) if t.is_empty()
));
}
#[test]
fn function_call_output_array_renders_to_text() {
// OpenAI allows `function_call_output.output` to be an array of
// content parts; the tool result must reach the model as text.
let raw = r#"{
"model": "m",
"input": [
{"type": "function_call_output", "call_id": "c1",
"output": [{"type": "output_text", "text": "42"}]}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).unwrap();
let chat = request_to_chat(req).unwrap();
assert_eq!(chat.messages.len(), 1);
assert_eq!(chat.messages[0].role, "tool");
match &chat.messages[0].content {
MessageContent::Text(t) => assert!(t.contains("42"), "got {t:?}"),
other => panic!("expected text, got {other:?}"),
}
}
// ── streaming projector ─────────────────────────────────────────
async fn collect(mut rx: mpsc::Receiver<ResponseStreamFrame>) -> Vec<ResponseStreamFrame> {
@@ -806,6 +917,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -856,6 +968,7 @@ mod tests {
prompt_tokens: 30,
completion_tokens: 12,
reasoning_tokens: 4,
timing: None,
})
.await
.unwrap();
@@ -886,6 +999,7 @@ mod tests {
prompt_tokens: 8,
completion_tokens: 3,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -910,6 +1024,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();
@@ -956,6 +1071,7 @@ mod tests {
prompt_tokens: 0,
completion_tokens: 0,
reasoning_tokens: 0,
timing: None,
})
.await
.unwrap();

View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<service>
<short>helexa-upstream</short>
<description>helexa-upstream — mesh account + budget authority API (/authz/v1 for cortex, /web/v1 for the frontend)</description>
<port protocol="tcp" port="8090"/>
</service>

View File

@@ -0,0 +1,3 @@
g helexa-upstream - -
u helexa-upstream - "helexa-upstream authority" /var/lib/helexa-upstream /sbin/nologin
m helexa-upstream helexa-upstream

View File

@@ -0,0 +1,22 @@
[Unit]
Description=helexa-upstream — mesh account + budget authority (accounts, API keys, allocation ledger, top-up codes)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/helexa-upstream serve --config /etc/helexa-upstream/helexa-upstream.toml
# HTTP authority for cortex (/authz/v1) and the frontend (/web/v1); restart
# unconditionally if it ever exits. It connects out to PostgreSQL (the
# system of record) and runs schema migrations on startup.
Restart=always
RestartSec=10
User=helexa-upstream
Group=helexa-upstream
# Service user home; no local state (PostgreSQL holds everything), but
# StateDirectory gives the user a writable, correctly-owned home.
StateDirectory=helexa-upstream
StateDirectoryMode=0755
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,39 @@
# helexa-router.example.toml — example configuration
#
# Copy to helexa-router.toml and adjust for your environment.
#
# Environment variable overrides use the HELEXA_ROUTER_ prefix with __
# separators:
# HELEXA_ROUTER_ROUTER__LISTEN=0.0.0.0:8088
[router]
# Plaintext listener. Operator/edge nginx terminates client TLS in front of
# the router — the router never owns an inbound TLS listener.
listen = "0.0.0.0:8088"
# How often (seconds) to refresh each cortex's health + /v1/models topology.
# poll_interval_secs = 10
# -- Downstream cortexes -------------------------------------------------
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
# to. The router forwards the client's bearer verbatim (auth stays at
# cortex) and routes on capacity (preferring matching `region`).
#
# Outbound TLS pinning (optional): set `tls_ca` to a PEM trust anchor that
# enrols this cortex — the CA (or self-signed cert) its TLS cert must chain
# to. The router then trusts ONLY that anchor for this cortex (platform
# roots disabled), so the router->cortex hop (which carries the client's
# bearer) reaches the cert you expect and a rogue endpoint presenting any
# other cert is rejected at the handshake. A cortex whose `tls_ca` fails to
# load is disabled (fail closed). Omit `tls_ca` for a publicly-trusted cert
# or plaintext http:// on a private (e.g. WireGuard) network.
# [[cortexes]]
# name = "lair-cafe"
# endpoint = "https://cortex.lair.cafe"
# region = "eu-west"
# tls_ca = "/etc/helexa-router/pins/lair-cafe.pem"
# [[cortexes]]
# name = "example-operator"
# endpoint = "https://cortex.example.com"

View File

@@ -0,0 +1,52 @@
# helexa-upstream.example.toml — mesh-level account/authorization authority
#
# Copy to helexa-upstream.toml and adjust. Env overrides use the UPSTREAM_
# prefix with __ separators, e.g. UPSTREAM_DB__URL=postgres://...
[server]
# Plaintext listener; edge nginx terminates TLS (consistent with the stack).
listen = "0.0.0.0:8090"
[db]
# PostgreSQL connection URL. Required.
url = "postgres://helexa:helexa@localhost/helexa_upstream"
# max_connections = 16
[grant]
# Flat free token grant every email-verified account receives (the floor of
# the hybrid allocation; single-use top-up codes extend it).
# free_token_grant = 1000000
[abuse]
# When this many accounts share one registration fingerprint, all are
# silently deactivated (no notice to the user).
# fingerprint_account_threshold = 5
# -- Client auth: credentials operators' cortexes present to /authz/v1.
# Each token maps to an operator_id (served-usage attribution). When no
# tokens are configured the authz surface is OPEN (dev only). Distinct from
# end-user API keys, which ride inside the resolve request body.
# [[client_auth.tokens]]
# token = "replace-with-a-strong-shared-secret"
# operator_id = "lair-cafe"
[authz]
# Open reservations older than this are swept (released), self-healing a
# reservation whose settle/release from a cortex was lost.
# reservation_ttl_secs = 120
# sweep_interval_secs = 60
[auth]
# HMAC secret for signing web-session JWTs. MUST be overridden in prod via
# UPSTREAM_AUTH__JWT_SECRET; the built-in default is dev-only.
# jwt_secret = "change-me"
# session_ttl_secs = 604800 # 7 days
# email_token_ttl_secs = 86400 # 24 hours
# Frontend base URL used to build verify/reset links in emails.
app_base_url = "https://helexa.ai"
[email]
# "log" (dev: logs the link) or "smtp".
provider = "log"
# smtp_url = "smtp://user:pass@smtp.example.com:587"
from_addr = "helexa <no-reply@helexa.ai>"

20
helexa.ai/.env.example Normal file
View File

@@ -0,0 +1,20 @@
# helexa.ai frontend env. Copy to .env.local for local dev (gitignored).
# Mesh data-plane (helexa-router, OpenAI-compatible inference). In dev,
# vite proxies /v1 and /health here.
VITE_ROUTER_BASE_URL=http://localhost:8088
# Account control-plane (helexa-upstream). In dev, vite proxies /api here
# (rewritten to /web/v1).
VITE_ACCOUNT_BASE_URL=http://localhost:8090
# Public-beta banner.
VITE_PUBLIC_BETA=true
# Models for the chat workspace (F3+).
# VITE_ANON_MODEL=...
# VITE_DEFAULT_MODEL=...
# Develop the account dashboard (F4) against an in-browser mock before the
# upstream account API ships.
# VITE_USE_MOCK_ACCOUNT_API=true

6
helexa.ai/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
node_modules
dist
*.local
.env.local
.env.*.local
*.tsbuildinfo

53
helexa.ai/README.md Normal file
View File

@@ -0,0 +1,53 @@
# helexa.ai
The public-beta frontend for the helexa mesh: a chat-first landing experience
(anonymous + authenticated, with all chat history kept client-side in
IndexedDB — no server-side history), a `/mission` page on European digital
sovereignty, and full account self-service (register, recover, manage API
keys, set per-key limits, redeem top-up codes) against `helexa-upstream`.
Vite + React (SWC) + TypeScript + react-bootstrap + react-router + react-i18next.
Lives as a top-level folder in the cortex monorepo; it is **not** a Cargo crate.
## Develop
```sh
cd helexa.ai
npm install
cp .env.example .env.local # adjust backend URLs
npm run dev # vite dev server, proxies /v1+/health → router, /api → upstream
```
Other scripts: `npm run build` (`tsc -b && vite build``dist/`), `npm run
preview`, `npm run lint`, `npm run typecheck`.
In dev, `vite.config.ts` proxies the mesh data-plane (helexa-router) and the
account control-plane (helexa-upstream) same-origin. Run a local router
(`cargo run -p helexa-router`) for the chat path and a local helexa-upstream
for the account path.
## Status
F0 scaffold. Theming + i18n (33 languages, usage-ordered selector), the
`/mission` page, the chat workspace (Dexie + streaming), and the account
dashboard land in subsequent phases — see
`~/.claude/plans/we-need-to-plan-modular-graham.md`.
## Deploy (public beta)
Build the SPA and serve it from edge nginx on the **same origin** as the
two backends — so the browser makes no cross-origin request (no CORS) and
the user's API key rides as a first-party bearer.
```sh
npm ci && npm run build # → dist/
sudo cp -r dist/* /var/www/helexa.ai/
sudo cp deploy/nginx.conf /etc/nginx/conf.d/helexa.ai.conf # adjust upstreams + TLS
sudo nginx -t && sudo systemctl reload nginx
```
`deploy/nginx.conf` routes `/` → SPA (history fallback), `/v1` + `/health`
→ helexa-router, and `/api/` → helexa-upstream `/web/v1/`. Set
`VITE_PUBLIC_BETA=true` at build time for the beta banner. There is **no
server-side chat history**: conversations live only in the browser
(IndexedDB).

View File

@@ -0,0 +1,60 @@
# helexa.ai — edge nginx for the public beta.
#
# Serves the built SPA (helexa.ai/dist) and reverse-proxies the two
# backends on the SAME ORIGIN, so the browser never makes a cross-origin
# request: no CORS, and the user's API key rides as a first-party bearer.
#
# / → static SPA (history fallback to index.html)
# /v1, /health → helexa-router (OpenAI-compatible inference data-plane)
# /api/ → helexa-upstream /web/v1/ (account control-plane)
#
# TLS is terminated here (certs omitted — wire up certbot / your CA). The
# upstream hosts below are examples; point them at your router/upstream.
upstream helexa_router { server 127.0.0.1:8088; }
upstream helexa_upstream { server 127.0.0.1:8090; }
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name helexa.ai;
# ssl_certificate /etc/letsencrypt/live/helexa.ai/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/helexa.ai/privkey.pem;
root /var/www/helexa.ai;
index index.html;
# Long-cache fingerprinted assets; never cache the HTML shell.
location /assets/ {
expires 1y;
add_header Cache-Control "public, immutable";
}
# Inference data-plane → router. Streaming (SSE): disable buffering so
# tokens reach the browser as they arrive.
location /v1/ {
proxy_pass http://helexa_router;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "";
proxy_buffering off;
proxy_read_timeout 300s;
}
location = /health {
proxy_pass http://helexa_router;
}
# Account control-plane → upstream /web/v1/ (strip the /api prefix).
location /api/ {
proxy_pass http://helexa_upstream/web/v1/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# SPA history fallback: anything else serves index.html.
location / {
try_files $uri $uri/ /index.html;
}
}

View File

@@ -0,0 +1,23 @@
import js from "@eslint/js";
import globals from "globals";
import reactHooks from "eslint-plugin-react-hooks";
import reactRefresh from "eslint-plugin-react-refresh";
import tseslint from "typescript-eslint";
import { defineConfig, globalIgnores } from "eslint/config";
export default defineConfig([
globalIgnores(["dist"]),
{
files: ["**/*.{ts,tsx}"],
extends: [
js.configs.recommended,
tseslint.configs.recommended,
reactHooks.configs.flat.recommended,
reactRefresh.configs.vite,
],
languageOptions: {
ecmaVersion: 2020,
globals: globals.browser,
},
},
]);

24
helexa.ai/index.html Normal file
View File

@@ -0,0 +1,24 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>helexa.ai</title>
<meta name="title" content="helexa.ai" />
<meta
name="description"
content="helexa — near-frontier AI on a sovereign, operator-run mesh. Chat now; bring your own key."
/>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://helexa.ai/" />
<meta property="og:title" content="helexa.ai" />
<meta
property="og:description"
content="helexa — near-frontier AI on a sovereign, operator-run mesh."
/>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

4115
helexa.ai/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

43
helexa.ai/package.json Normal file
View File

@@ -0,0 +1,43 @@
{
"name": "helexa.ai",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"preview": "vite preview",
"lint": "eslint .",
"typecheck": "tsc -b",
"i18n:check": "node ./scripts/check-i18n-keys.mjs",
"i18n:meta": "node ./scripts/check-i18n-metadata.mjs",
"i18n:lang-labels": "node ./scripts/check-i18n-lang-labels.mjs"
},
"dependencies": {
"@fingerprintjs/fingerprintjs": "^4.6.2",
"bootstrap": "^5.3.8",
"dexie": "^4.2.0",
"dexie-react-hooks": "^4.2.0",
"i18next": "^25.7.1",
"react": "^19.2.0",
"react-bootstrap": "^2.10.10",
"react-dom": "^19.2.0",
"react-i18next": "^16.4.0",
"react-icons": "^5.5.0",
"react-router-dom": "^7.10.1"
},
"devDependencies": {
"@eslint/js": "^9.39.1",
"@types/node": "^24.10.1",
"@types/react": "^19.2.5",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react-swc": "^4.2.0",
"eslint": "^9.39.1",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0"
}
}

BIN
helexa.ai/public/banner.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 304 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Some files were not shown because too many files have changed in this diff Show More