243 Commits

Author	SHA1	Message	Date
rob thijssen	140f9124ee	feat(helexa-bench): capability probe scenario + quality scoring (#91) ... The six perf scenarios measure speed/resources; this measures the axis they miss — reasoning/planning quality — so the frontier A/B (F3) can pick on capability, not just throughput. Per the chosen approach: store the artifact always, with schema for BOTH a manual score and a future LLM-judge; start manual. - scenario: CapabilityScenario (capability:<name>) runs a fixed prompt and captures the full output text (stream_and_measure gains a capture_text path); opt-in via config.capability_probes (empty default — long outputs, deliberate). - store: three additive columns (artifact, quality_score, scorer); capability_runs(unscored_only) worklist + set_score(id, score, scorer). Drill-down RunRow omits the large artifact column. - cli: `helexa-bench score --id <n> --score <x> [--scorer ...]` (manual); `report --capability` (per-model median score + per-run artifact snippets); GET /api/capability. LLM-judge deferred (schema ready). - example config documents an implementation-planning probe. Tests: artifact storage + scoring lifecycle, capability scenario built from config, capability markdown (median + snippet). Part of the Performance observability epic (#83), O7 — completes the milestone. Feeds the F3 frontier A/B decision gate (#94). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 15:03:13 +03:00
rob thijssen	842caf5d4d	feat(helexa-bench): cold-load / model-swap cost measurement (#90) ... The vision cold-swap policy (F4e) needs real numbers for unload→reload latency and the cold first-request after a reload — not guesses. Adds a deliberate `swap-cost` subcommand (NOT the continuous sweep, since unloading takes the live model offline for the reload). For each neuron target's warm models: unload, time the synchronous reload, then time a cold first request. Recorded under scenario "swap" so it tracks per build like everything else. - client: unload_model, load_model, and spec_from_info (reconstructs a ModelSpec from /models — TP inferred from device count, quant left None for neuron to resolve). - sweep: Sweeper::swap_cost_once + measure_swap (build_record gains a swap-timing param; existing sweep call sites pass None). - scenario: cold_probe — a single timed request reusing the SSE core. - store: two additive columns (swap_unload_ms, swap_load_ms) + a swap_costs() pivot (reload latency + cold first-request medians). - report: `report --swap` view + render_swap_json; GET /api/swap. - cli: `helexa-bench swap-cost` with an explicit offline warning. Tests: swap_costs pivot + scenario exclusion, swap markdown render. Part of the Performance observability epic (#83), O6. Feeds the vision cold-swap policy (#99 / F4e). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 14:15:42 +03:00
rob thijssen	d472b6428a	feat(helexa-bench): context-length scaling view (#88) ... The chat:<n> cells already capture prefill & decode tok/s per context (via #85/#86); this pivots them into a per-(target,model) scaling curve and computes decode-flatness — decode tok/s at the largest context ÷ the smallest. ~1.0 confirms the Gated-DeltaNet O(1)-in-sequence-length decode; a sharp drop locates where the model stops scaling for free. - store: Store::scaling() pivots the latest-build chat:<n> report cells into ScalingCurve/ScalingPoint, ordered by context, with the flatness ratio (concurrency:<n> and other scenarios are excluded). - report: render_scaling_markdown (one block per model: prefill/decode tok/s vs ctx + a flatness verdict) and render_scaling_json. - cli: `helexa-bench report --scaling` selects the view. - api: GET /api/scaling for the bench UI. - example config documents widening prompt_sizes into a scaling ladder. No new request shape — reuses the chat-latency measurement points, so a denser curve is just more prompt_sizes entries (operators widen the ladder deliberately; large contexts cost more per sample). Tests: scaling pivot + flatness + scenario exclusion, markdown render. Part of the Performance observability epic (#83), O4. Validates the long-context property that makes the 80B-A3B frontier model (#84) viable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 13:56:07 +03:00
rob thijssen	209bf58207	feat(helexa-bench): concurrency / agentic-load scenario (#89) ... Batch-1 single-request timing can't characterize the real workload — a0/hermes/opencode each fan out many agentic requests per user turn. Add a ConcurrencyScenario (concurrency:<n>) that fires N simultaneous streams and records, per burst: aggregate node throughput (total tokens / burst window), within-burst TTFT p95 (the tail under load), median admission queue-wait — derived as TTFT − server prefill_ms (#85), no /health poll needed — and the count of streams shed by admission (429/503). On a batch-1 server, throughput stays flat while queue-wait and p95 inflate with N; that gap is the evidence for/against continuous batching (#98). - scenario: ConcurrencyScenario + shared chat_payload helper; reuses stream_and_measure per stream via join_all. - config: concurrency_levels (default empty — opt-in, a burst loads the fleet) + concurrency_prompt_tokens (512); example config documents it. - store: four additive columns (same migration pattern); aggregate surfaces concurrency, within-burst p95, queue-wait, rejected medians. - report: markdown gains conc / queue ms / rej columns; JSON the full set. Tests: median/percentile/admission-detect helpers, config builds concurrency scenarios, store surfaces burst metrics. Part of the Performance observability epic (#83), O5. Produces the evidence base for continuous batching (#98 / F4d). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 13:17:10 +03:00
rob thijssen	a415454962	feat(helexa-bench): per-run VRAM high-water + GPU util/temp from /health (#87) ... Bench had no measured VRAM headroom — only the anecdotal "~2/3 used". Sample neuron's GET /health (DeviceHealth: vram_used/free, util%, temp) right after each measured run and record node-sum VRAM used plus the hottest device's utilization and temperature. - client: fetch_health (neuron only; soft None on transport failure so a flaky /health never fails a measurement). - sweep: HealthAgg folds a /health snapshot to node totals; sampled post-run (the recent decode-VRAM peak) and threaded into the record. /health is ~5s-cached neuron-side, so this is a coarse high-water proxy, documented as such. - store: three additive columns (same PRAGMA-guarded migration); aggregate emits vram_used median + node vram_total (summed from the discovery devices in gpus_json) for real headroom, plus util/temp medians. - report: markdown gains a "VRAM (GB)" used/total column; JSON gains the used/total + util/temp medians. Tests: VRAM/headroom aggregation, gpu_total_vram summation. Part of the Performance observability epic (#83). Completes the p1-now trio (O1-O3); unblocks the enriched-27B baseline capture on beast. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 12:50:53 +03:00
rob thijssen	afc1f7a706	feat(helexa-bench): percentiles + prefill/decode split in store & report (#86) ... Bench reported only a single median per cell, hiding tail latency and unable to record the server-measured prefill/decode split now emitted on `usage.helexa_timing` (#85). - scenario: parse `usage.helexa_timing` into ScenarioMetrics (prefill_ms, decode_ms, prefill_tokens). - store: persist the three columns (additive PRAGMA-guarded migration via ensure_columns, so pre-#85 DBs backfill as NULL); aggregate now emits p50/p95/p99 for TTFT and total (nearest-rank) plus a prefill-tok/s median derived from the split. - report: markdown gains prefill tok/s, TTFT p95, total p95 columns; JSON gains p95/p99 + prefill_ms/decode_ms/prefill_tps medians. Tests: nearest-rank percentile, idempotent backfill migration, and a report cell asserting percentiles + prefill split. Part of the Performance observability epic (#83). Stacked on #85. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:58:36 +03:00
rob thijssen	6f956dfda3	fix(neuron): hoist TP prefill/decode timers out of 'work block (#85) ... The TP streaming producer builds its Finish event after the 'work labelled block exits (inside `if failure.is_none()`), but prefill_elapsed and decode_start were declared inside that block — so the CUDA type-check failed with E0425 (the CPU build doesn't compile this cfg(cuda) path). Hoist `prefill_ms_measured: u32` and `decode_start: Option<Instant>` above the block; populate them at the prefill→decode boundary inside; read them at the terminal Finish. The worker and local producers were unaffected (their timers already share the Finish scope). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:58:26 +03:00
rob thijssen	6e0f15c888	feat(neuron): server-measured prefill/decode timing on Finish (#85) ... The harness emitted only token counts on InferenceEvent::Finish; all timing was client-side SSE arrival, so bench "TTFT" conflated tokenize + prefill and decode tok/s was a window estimate. Add FinishTiming { prefill_ms, decode_ms, prefill_tokens } to the Finish event, populated by all three streaming producers (TP, worker, and local CPU paths), and surface it on the OpenAI chat `usage.helexa_timing` extension so helexa-bench can compute true prefill vs decode tok/s. cortex forwards usage verbatim, so the field survives proxying. Non-streaming and Responses paths carry None for now (bench reads the streaming chat path). Keystone for the Performance observability epic (#83). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:41:17 +03:00
rob thijssen	f96a2e7ed3	fix(neuron): surface reasoning_tokens in non-streaming /v1/responses usage ... The non-streaming responses handler hardcoded `output_tokens_details: None`, so the reasoning sub-count that the chat path now computes (via `split_off_reasoning`, which strips the `<think>` span and counts it into `completion_tokens_details.reasoning_tokens`) never reached the Responses-API usage object. The streaming responses path already emits it. Carry `chat usage.completion_tokens_details.reasoning_tokens` through to `ResponsesUsage.output_tokens_details.reasoning_tokens`, so streaming and non-streaming `/v1/responses` report reasoning accounting identically. `output_tokens` still counts every generated token (reasoning included); `reasoning_tokens` is the additive sub-count, per OpenAI's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 21:21:59 +03:00
rob thijssen	13daf95514	fix(neuron): strip <think> reasoning from non-streaming completions ... The non-streaming inference path (single-GPU `chat_completion` and the TP `chat_completion_tp_inner`) returned the model's full decode — reasoning preamble + `</think>` + answer — as the assistant `content`. The streaming path already drops reasoning (emits it as ReasoningDelta, which the chat and Responses projectors discard), so the two transports disagreed: a streaming client saw the clean answer, a non-streaming client saw the chain-of-thought glued to the front of it. This is exactly what broke agent-zero v2.0 on `/v1/responses` (non-streaming): the model produced a correct in-band JSON answer after `</think>`, but a0's parser saw the `<think>` preamble first and rejected the turn as "misformat, no valid tool request found". a0 v2.1 happens to tolerate the preamble, but any strict non-streaming client (and a0 v2.0) does not — and reasoning tokens leaking into `content` also misreport as visible output. Add `split_off_reasoning(generated_ids, reasoning_pair)`: if the model declares a reasoning marker pair and its close token (`</think>`) appears in the output, return only the tokens after the last close marker as content and count the rest as reasoning. The chat template injects the *opening* marker into the prompt, so the generated tokens carry the close marker but not the open one — splitting on the close-token id (not a decoded string) is robust to tokenizer byte-fallback. Non-reasoning models, thinking-disabled requests, and generations truncated mid-reasoning have no close token and pass through unchanged. Both non-streaming sites now decode only the answer span and populate `completion_tokens_details.reasoning_tokens` (the streaming-only accounting gap noted at the call sites, #64). Unit tests cover the strip, no-marker, no-pair, close-at-end, and multiple-close cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 20:39:41 +03:00
rob thijssen	6731adca51	fix(neuron): accept bare {role,content} input on /v1/responses (agent-zero) ... agent-zero (via litellm) drives the OpenAI Responses API and sends `input` items in the "easy input message" form — bare `{role, content}` objects with NO `type` field. Our `ResponsesInputItem` is internally tagged (`#[serde(tag="type")]`), so every such item failed the untagged `ResponsesInput` deserialize and axum's `Json` extractor returned 422: OpenAIException - Failed to deserialize the JSON body into the target type: data did not match any variant of untagged enum ResponsesInput This was a *total* failure for agent-zero (both the main model on beast and the utility model on benjy), confirmed by on-wire capture of 15 live requests: 36/36 input items were bare easy-messages. Other clients (/v1/chat/completions, /v1/messages) were unaffected — only the Responses path was exercised this strictly, for the first time. Make `input`-item parsing match OpenAI's real tolerance, mirroring the forward-compat `extra: Value` already at the top level of the request: - New `ResponsesInputElement` wraps the existing typed item enum with two more shapes: `EasyMessage { role, content }` (bare, no type; `content` optional so an assistant turn with `content: null` parses) and `Other(Value)` — a catch-all so a single unmodeled item can never again 422 the whole request. The typed enum is unchanged. - `ResponsesContentPart` gains a `#[serde(other)] Unknown` arm (e.g. `refusal`, audio) — dropped in translation, not rejected. - `FunctionCallOutput.output` is now `Value` (string OR array of content parts, per OpenAI) so a structured tool result isn't lost. - Translator handles all three element shapes; easy-messages translate exactly like typed messages, `Other` and unknown parts are dropped. Tests cover the bare-message, null-content, unknown-item, unknown-part, and array-tool-output shapes, validated against the 15 captured bodies. Tools forwarding + native function_call projection on the Responses path is deliberately a follow-up (Round 2), gated on observing how agent-zero consumes responses once unblocked (in-band JSON vs native tool calls). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 19:45:22 +03:00
rob thijssen	f4117224fc	feat(B6): served-usage ledger + reconciliation (#58) ... Operators are now metered for the tokens they serve on behalf of mesh accounts, and the upstream rolls that up for compensation. cortex-gateway: - served_usage.rs: an in-process per-(account,key,UTC-day) served-token counter, incremented in metering::usage_sink alongside spend/settle for every authenticated request. A flush task (spawned in run() when [upstream].enabled, mirroring poller/evictor) POSTs ABSOLUTE cumulative counters to upstream on [upstream].served_usage_report_interval_secs. - The no-limit infra key is the operator's local key with hard_cap=None (already supported) — it's metered for served-usage but never budget- refused and never hits upstream. helexa-upstream: - POST /authz/v1/served-usage: upserts rows keyed by (operator_id from the client bearer, account, key, period) with GREATEST(existing, incoming) — monotonic + idempotent, so re-sends, races, and a restarted cortex's lower counter never regress the total. - reconcile.rs + `helexa-upstream reconcile` CLI: rolls up unreconciled served_usage per operator/period (SUM::bigint), stamps reconciled_at, prints the totals. Payout mechanism out of scope. Validated against a throwaway Postgres: monotonic upsert (100→250, a 50 re-send stays 250, same-value idempotent) and reconcile rollup + stamp-once; cortex counter unit test for per-principal accumulation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 11:45:57 +03:00
rob thijssen	8600d4fbf2	Merge feat/B5-topup-codes: single-use top-up codes + mint CLI (B5, #59)	2026-06-23 11:01:26 +03:00
rob thijssen	bb0d1e51b8	Merge feat/B3-cortex-upstream-client: cortex upstream entitlement client + chain (B3, #57)	2026-06-23 10:51:03 +03:00
rob thijssen	2348cc2234	feat(B5): single-use top-up codes (redeem + mint CLI) ... Completes the hybrid allocation model — the flat free grant (B4) plus top-up codes that extend an account's allocation. - topup.rs: redeem(account, raw_code) — timing-safe, single-use. A conditional `UPDATE … WHERE redeemed_by IS NULL RETURNING value` claims the code atomically (concurrent double-redeem → exactly one winner), then raises accounts.allocation_total in the same tx. Only sha256(code) is stored; a not-found code and an already-redeemed code return the SAME generic error via the same path (no "valid but spent" oracle). Raising the total automatically lifts every percent-limited key's effective cap; hardcap keys stay pinned (by design). - mint(value, count, denomination) — inserts codes (hash-only), returns the raw codes once. Exposed as `helexa-upstream mint --value --count [--denomination]` (raw codes to stdout, one per line) — the seam the future faucet bot calls. Bot itself out of scope. - POST /web/v1/redeem (session-protected) → {allocation_total} | generic 400. Validated against Postgres: redeem raises the 1_000_000 free grant to 1_500_000, second redemption + unknown code both generic-400, and a concurrent race for one code yields exactly one HTTP 200. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:50:43 +03:00
rob thijssen	a9d7382be8	feat(B3): cortex upstream entitlement client (#57) + chained provider ... cortex can now validate locally-unrecognised bearer keys against the helexa-upstream authority and reserve/settle their budget there — mesh accounts work for real inference. The EntitlementProvider trait is the seam, so cortex's enforcement (auth.rs, metering.rs) is otherwise unchanged. - entitlements_upstream.rs: UpstreamEntitlementProvider over reqwest → B2's /authz/v1 (resolve/reserve/settle/release/snapshot), presenting the operator client bearer. Maps the wire contract back to the trait: granted → Reservation, rejected → BudgetError, 401 → InvalidKey. Fail-closed — unreachable resolve → AuthError::Unavailable (503, never 401); unreachable reserve → retryable BudgetError::RateLimited (refuse, never serve un-authorized). settle/release are best-effort (the upstream sweeper reaps a lost one). - entitlements_chain.rs: ChainedEntitlementProvider tries local first (operator + infra keys, no network), falls through to upstream for unknown keys, and dispatches reserve/settle/release/snapshot to whichever backend resolved each account (local treats unknown principals as uncapped, so it can't be the blind default). - cortex-core: AuthError::Unavailable{retry_after_secs}; [upstream] config (enabled/url/bearer/timeout). auth.rs maps Unavailable → 503 + Retry-After distinctly from InvalidKey → 401, regardless of require_auth. - state.rs wires the chain when [upstream].enabled, else stays purely local. Tests (upstream_chain.rs, 4): local key resolves without touching upstream; unknown key falls through to a mock upstream; unknown-everywhere → 401 InvalidKey; upstream-unreachable → Unavailable (503-mapped), with local keys still resolving. Existing gateway suites updated for the new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:41:48 +03:00
rob thijssen	d94c62c143	feat(B4): /web/v1 account API + silent fingerprint multi-account abuse ... The human-facing account surface the helexa.ai frontend (F4) consumes, on top of B2's authz surface. Email+password auth with JWT sessions (distinct from inference API keys); plain-JSON errors (the #63 envelope stays on the authz surface). - Auth lifecycle: register → email-verify → login → password-reset (request/confirm). argon2id passwords; verify/reset via single-use sha256-hashed email tokens; register and reset-request always return 202 (no account enumeration). Email via a pluggable EmailSender (lettre Smtp + dev Log transport). - API keys: create (sk-helexa-<base62(32 OsRng)>, raw shown once, stored as sha256 + non-secret prefix), list (prefix never the secret), archive, PATCH per-key limit (percent|hardcap). Protected by a JWT session middleware. - Account balance endpoint (allocation total/spent/reserved). - Silent fingerprint abuse: register captures the browser fingerprint; >= threshold (default 5) accounts sharing one fingerprint are silently deactivated + flagged — registration still returns a normal 202, and a deactivated account's key resolves as an ordinary 401 at the authz surface (no "banned" signal anywhere). - crypto: argon2 hash/verify + CSPRNG token/key minting (base62). config gains [auth] + [email]. CORS on the app for the browser SPA. Validated against a throwaway Postgres 16: verify-once, full lifecycle (register→verify→login→create key→account→list→authz resolve→archive→401), and 5-same-fingerprint → all accounts silently deactivated + no-clue 401. 8 unit + 11 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:28:20 +03:00
rob thijssen	46befde4cd	feat(B2): /authz/v1 authority surface + client-auth + reservation sweeper ... The machine surface cortex's UpstreamEntitlementProvider (#57) consumes, mirroring the EntitlementProvider trait 1:1 over the B1 ledger. - `authz.rs`: POST /authz/v1/{resolve,reserve,settle,release,snapshot}. resolve → {principal, snapshot} | 401 invalid_api_key (a deactivated account resolves as the SAME 401 — the silent-abuse no-clue property). reserve returns 200 whether granted ({reservation_id}) or budget-refused ({rejected:{kind,...}}) — a refusal is an authoritative answer, not a transport failure; non-2xx means "fail closed" to the client. settle/ release → 204 (idempotent). snapshot → {hard_cap,spent,reserved} | 404. Rejections use the shared #63 OpenAiError envelope (cortex-core dep). - Client auth: shared-bearer middleware (constant-time compare via subtle) maps a token → operator_id (stamped into request extensions for #58 served-usage); empty config = open dev surface (logged). mTLS deferred. - ledger gains resolve_key (sha256 lookup, account-active-gated), snapshot, and sweep_stale (one data-modifying-CTE statement releasing aged-out open reservations and folding their reserved tokens back into accounts+keys). - Sweeper task spawned in run(); [authz] ttl/interval + [client_auth] config; crypto::sha256 helper. Validated against a throwaway Postgres 16 (fresh schema): resolve→reserve→ settle→snapshot round-trip, over-cap → 200 insufficient_quota rejection (not retried away), deactivated account → 401 (no clue), missing/wrong client bearer → 401 before any DB hit. 5 unit + 8 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:12:19 +03:00
rob thijssen	71106afaf1	feat(B1): helexa-upstream crate skeleton + Postgres schema + ledger ... First milestone of the mesh-level account/authorization authority (#59). New workspace crate `crates/helexa-upstream` (binary + lib), mirroring the helexa-router skeleton: axum `build_app`/`run`, clap `serve --config`, figment `UPSTREAM_`-prefixed config. - **Storage:** PostgreSQL via sqlx (runtime query API — builds in CI with no DB or offline cache; correctness covered by gated integration tests). `migrations/0001_init.sql` is the full schema: users (argon2 hash slot, email_verified, registration_fingerprint), email_tokens, accounts (allocation total/spent/reserved + `accounts_no_overshoot` CHECK, silent `status` deactivation flag, fingerprint_flagged), api_keys (sha256 hash, percent|hardcap limit, cap_window, per-key ledger), reservations (BIGSERIAL id → maps to cortex Reservation.id u64), top_up_codes, served_usage, sessions. Migrations run on startup. - **Ledger (no-overshoot core):** `ledger::{reserve,settle,release}` — reserve takes `SELECT … FOR UPDATE` on the account + key rows so concurrent reserves serialize and spent+reserved can never exceed the effective cap (= min(resolved key cap, remaining account allocation)); the CHECK is the DB backstop. Settle clamps actual to [0,reserved] and is idempotent; release is idempotent. `resolve_abs_cap` (percent/hardcap, i128 math) is pure + unit-tested. Balance semantics here; rolling-window sub-caps + RateLimited land with the authz API (B2). - `/health` does a DB round-trip. - Config + example TOML ([server]/[db]/[grant] free grant/[abuse] fingerprint threshold). Validated end-to-end against a throwaway Postgres 16: migration applies, 20 concurrent reserves of 100 against a 500 cap admit exactly 5 (reserved == 500, never over), settle/release idempotent, hardcap key sub-cap binds below the account, `/health` → db ok. CI runs the cap-math + config unit tests; the DB integration tests skip cleanly when UPSTREAM_TEST_DATABASE_URL is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 09:49:19 +03:00
rob thijssen	1115bb0942	feat(#74): verify downstream cortex TLS certs (outbound pinning) ... The router is a TLS client to cortexes; the router->cortex hop crosses the helexa->operator boundary carrying the client's bearer. This pins that hop to an enrolled cert. Trust mechanism (the open question): per-cortex enrolled trust anchor. Each [[cortexes]] entry gets an optional `tls_ca` — a PEM CA (or self-signed cert) the cortex's TLS cert must chain to. When set, the router builds a client that trusts ONLY that anchor (platform roots disabled), so the cortex must present the expected cert and a rogue endpoint with any other (even publicly-valid) cert is rejected at the handshake. Enrolment = the operator hands helexa the cortex's cert, referenced by path in router config. This is the natural model for self-hosted operators behind their own nginx/private CA, and reuses the reqwest public API (no custom rustls verifier, no new TLS backend). - `RouterState` now holds a per-cortex `reqwest::Client` map (`client_for`), replacing the single shared client; poller and dispatch use the per-cortex client. `build_client(tls_ca)` is the builder. - Fail closed: a `tls_ca` that can't load omits the cortex from the client map — it's never polled or routed to, rather than silently degrading to unpinned TLS. The poller treats a missing client (and a rejected handshake) as a failed poll, so #72's existing reachability debounce excludes it. Tests (`tls.rs`, 4): a live tokio-rustls HTTPS server proves a client enrolled with the server's cert is accepted (200) while clients pinned to a different cert — or using default roots — are rejected; the poller marks a wrong-cert cortex unreachable while a correctly-enrolled one is reachable; a missing pin file disables the cortex (fail closed); garbage PEM is rejected at build. Existing suites updated for the per-cortex client + new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:23:20 +03:00
rob thijssen	63f578cb15	feat(#75): aggregate /v1/models across operators (federation catalogue) ... The router's /v1/models is now the deduped union of every reachable cortex's catalogue, so an opencode client doing discovery against the router resolves the whole federation without knowing about operators or cortexes (resolves #61's "Router/discovery contract"). To preserve per-model limit/cost, the topology poller now retains each cortex's full `cortex_core::node::CortexModelEntry` (was distilled to a {loaded, feasible} bool). `entry_feasible()` replaces the dropped field; dispatch (#73) and `cortexes_serving` use it — no routing behaviour change. `catalogue.rs::aggregate_models`: - Dedupe by model id; a model served by >=1 reachable cortex appears once. - Merge availability: `loaded` OR across operators; only feasible (loaded-or-cold-loadable) entries surface — a catalogue-only model no neuron can host is hidden. - Re-tier to operator names: `feasible_on` becomes the cortexes that can serve it and `locations` the operators it's loaded on (node = cortex name), so the federation view doesn't leak each operator's neuron names or per-device VRAM. - Conflict resolution: `limit` → tightest (smallest context, so a client never overflows the most-constrained operator); `cost` → cheapest (the federation "from" price). Richer range/region policy couples to #68, noted as follow-up. Tests: 4 unit (dedupe+merge, unreachable excluded, infeasible hidden, tightest-limit+cheapest-cost) + 1 end-to-end (two mock cortexes overlapping on a model → GET /v1/models over HTTP asserts the merged union). dispatch/topology suites updated for the entry-storage change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:08:16 +03:00
rob thijssen	7984d27553	feat(#73): capacity-aware dispatch with region affinity + failover ... The router's data path. Wires the topology poller (#72) and the shared streaming proxy (#71) into real request routing. - `dispatch.rs`: `select_cortexes(model)` ranks reachable cortexes that can serve the model, best-first — loaded/warm before cold-loadable, region match before not, more healthy nodes before fewer, name for determinism. `dispatch()` extracts `model`, picks candidates, and forwards via `helexa_stream::forward_streaming` (bearer + bytes verbatim, SSE streamed back). Cortex's #63 rejections (429/400/…) pass through untouched; transport failures fail over to the next candidate; a genuine HTTP response — any status — is returned as-is, never retried away. - Router-originated rejections use the #63 envelope: 404 model_not_found (no operator serves it), 503 service_unavailable + Retry-After (known but all unreachable / all candidates failed to connect), 400 missing_model_field. `error.rs` is the router's envelope→axum adapter (mirrors cortex-gateway's). - `handlers.rs`: `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages` dispatch to the same path on a chosen cortex. The router holds zero entitlement logic — routes on capacity, not budget. - Config: optional `region` on the router and per-cortex for geo affinity. Tests (`dispatch.rs`): routes to a serving cortex + forwards the bearer; cortex 429 passes through and is NOT retried; transport failure fails over to a live cortex; unknown→404, known-but-unreachable→503, missing-model→400; ranking order (warm/region/headroom). 7 new, existing skeleton/topology suites unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:40:07 +03:00
rob thijssen	5fd7736abd	feat(#72): router↔cortex topology poller (multi-operator capacity map) ... Builds the live topology the dispatcher (#73) will route on — the same pattern as cortex↔neuron, one tier up. - `poller.rs`: background loop polls each configured cortex's `GET /v1/models` (deserialised straight into the shared `cortex_core::node::CortexModelEntry`) and `GET /health`, on a configurable `poll_interval_secs` (default 10). - `state.rs`: `RouterState` gains an `http_client`, `poll_interval`, and a `RwLock<HashMap<cortex_name, CortexTopology>>` pre-populated from config so the poller/handlers always find an entry. Per cortex: `reachable`, `consecutive_failures`, `last_poll`, healthy/total node counts, and a per-model `{loaded, feasible}` map (feasible = loaded OR cortex reports `feasible_on`, i.e. cold-loadable). `cortexes_serving(model)` returns the reachable cortexes that can serve a model — groundwork for #73. - Debounce: a cortex flips unreachable only after `POLL_FAILURE_THRESHOLD` (3) consecutive failed polls, and recovers on the next good poll — mirrors cortex's neuron-poll debounce so a blip can't yank a whole operator out of routing. `/health` poll is best-effort and never flips reachability on its own. - `lib.rs` spawns the poll loop in `run()`. `/health` now surfaces `cortexes.reachable`; `status` stays router-liveness (always `ok`). Tests (`topology.rs`): live-map build (loaded vs catalogue-only feasible, node counts, routing helper); unreachable→excluded→recovers across the debounce threshold; dead endpoint never panics. Skeleton tests unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:01:29 +03:00
rob thijssen	03fd4960c3	Merge fix/71-shared-streaming-proxy: shared helexa-stream SSE proxy (#71) ... # Conflicts: # Cargo.lock # Cargo.toml	2026-06-21 18:15:12 +03:00
rob thijssen	cabec1d08a	fix(#71): extract SSE streaming passthrough into shared helexa-stream ... The true-streaming SSE passthrough (Body::from_stream, no full-response buffering, with chunk-observation hooks) was cortex-only. helexa-router (#69) needs the same mechanism to proxy a chat-completions/messages stream verbatim to a selected cortex. Extract it once. New `crates/helexa-stream` owns the *mechanism* (kept HTTP-free cortex-core untouched — it would have forced axum/reqwest/futures onto every cortex-core consumer): - `forward_streaming(client, url, headers, body, observer)` — POST and stream the response back chunk-for-chunk; status-agnostic, so a non-2xx (e.g. cortex 429) is passed through with status+headers intact (the #69 backpressure-passthrough requirement). - `ChunkObserver` trait + `ObservedStream` wrapper — feeds each chunk to the observer, calls `finish` exactly once on clean end or on drop (client disconnect). - `BodyTail` (bounded tail accumulator) + `last_count_for` (trailing OpenAI `usage` extraction) — the reusable pieces an observer uses. cortex keeps its *policy*: `proxy.rs` now supplies a `CortexMetrics` observer (per-request token metrics + per-principal reservation settle), its logging contract, and the error envelope, driving the shared mechanism. `proxy::last_count_for` is re-exported so `handlers`/ `anthropic_sse` call sites are unchanged. No behaviour change — the existing cortex `streaming.rs` tests pass as-is. helexa-stream tests prove chunk-for-chunk incremental delivery, observer finish-once, usage extraction, and non-2xx passthrough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 18:05:35 +03:00
rob thijssen	881fc85a4c	feat(#70): helexa-router binary skeleton — plaintext axum server ... Foundation for epic #69 (public multi-operator ingress proxy). New `crates/helexa-router` workspace binary: a plaintext axum server that reuses cortex-core types and serves the two stub endpoints the rest of #69 builds on. - `[router] listen` + `[[cortexes]]` config via figment + `HELEXA_ROUTER_` env overrides, matching the cortex/neuron convention. - `GET /health` reports the configured downstream cortex count. - `GET /v1/models` returns an empty OpenAI list (real cross-operator aggregation is #75). - No inbound TLS listener (edge nginx terminates client TLS per #69's posture); no auth layer — the router forwards the client bearer to cortex and holds zero entitlement logic (#47 stays additive). - 3 tests: both endpoints over a real ephemeral-port server, plus TOML+env config load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 17:52:28 +03:00
rob thijssen	87d9c291ce	fix(#68): pin the /v1/models cost wire contract — units + absent-vs-zero ... The cost code path already exists (cortex list_models populates cost: profile.cost from the catalogue; aliases inherit it), so opencode's $0.00 is a config gap (no cost in the live models.toml), not missing plumbing. What was missing is the *contract*: units pinned against a wire test, and a defined meaning for "free". - Document ModelCost as the load-bearing source of truth: USD per 1,000,000 tokens as JSON numbers (models.dev/opencode shape) — NOT per-token, NOT decimal strings (OpenRouter's pricing shape, which helexa deliberately does not emit). Define the absent-vs-zero distinction: cost omitted = "not priced / unknown"; cost present with 0.0 = "intentionally free". Note the advertised rate must equal what metering (#51) / reconciliation (#58/#59) bill against — today both read this catalogue value. - New wire test (model_cost.rs): a priced model with cache tiers flows through as per-million numbers; an explicit-0.0 free model keeps its cost block with cache tiers omitted; an unpriced model omits `cost` entirely. - models.example.toml: document cost.* in the field reference and show all three cases (priced-free explicit 0.0 vs the unpriced Qwen3-8B with no cost block). Decisions recorded on #68: source of truth = operator models.toml for now (marketplace clearing house #59 later, same value); no OpenRouter-style `pricing` (opencode/models.dev alignment is sufficient); end-to-end non-zero $ spent needs operators to populate cost in the live catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 12:02:03 +03:00
rob thijssen	e7f7e376fc	fix(#65): request-time length-aware VRAM backstop for text prefill ... Close the poll-vs-request snapshot gap #67 left open. The text prefill guard (validate_request) only checked the static min_free_vram floor; the derived input cap (effective_prompt_cap) is computed at /models poll time from the tightest card's free VRAM *then*. If free VRAM drops between that poll and the request — a co-resident model loads, a concurrent prefill grows its KV — a prompt at-or-below the now-stale cap clears the floor yet no longer fits, OOMing mid-prefill and poisoning the device context (the 2026-05-26 beast incident #47 exists to eliminate). validate_request now re-runs #67's length×KV-vs-VRAM physics against request-time free VRAM, reusing the model's ContextProfile (kv_bytes_per_token_per_card, full-attention-layer-only, TP-sharded) rather than re-deriving the cost. Footprint = KV(prompt + output_reserve) + activation_headroom + static floor, all per card and commensurable with the tightest-card free VRAM on both single-GPU and TP loads. Degenerate zero-KV / no-profile models ride the existing floor check, mirroring derive_limit's VRAM-ceiling fallback; CPU loads (vram_free_mb == 0) skip all VRAM checks unchanged. This closes the vision/text asymmetry: the text path now has the live-VRAM guard validate_vision_prefill already gave the vision path. 5 unit tests incl. the acceptance staleness test: a cap derived against ample free VRAM, applied at request time against tightened VRAM, rejects a prompt sized at the stale cap with a clean InsufficientVram (503) instead of an OOM. Threaded context_limit_cfg into chat_completion_tp_inner (spawned, no &self) and used &self.context_limit_cfg at the three method call sites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 11:45:53 +03:00
rob thijssen	3b9a6e37f6	Merge fix/cortex-poll-debounce-retryable: poll debounce + retryable 503 for feasible-but-unhealthy node	2026-06-18 12:46:30 +03:00
rob thijssen	526b662c5e	fix(cortex): poll-failure debounce + retryable 503 for feasible-but-unhealthy node ... Defense-in-depth for the agent0 NoFeasibleNeuron storm (root cause fixed in neuron). Two cortex resilience gaps this incident exposed: 1. Brittle health flip: the poller marked a node unhealthy on a SINGLE missed /models poll, instantly yanking the node and all its models from routing. A busy neuron briefly slow to answer shouldn't be declared dead. Now debounced: NodeState.consecutive_poll_failures must reach POLL_FAILURE_THRESHOLD (3) before the node flips unhealthy (~20s at the 10s poll interval); any successful poll resets it. A never-healthy node stays unhealthy (the counter only protects an already-healthy node from blips). 2. Transient surfaced as permanent: when a catalogued model's only feasible neuron is momentarily unhealthy, the router returned 404 NoFeasibleNeuron — which litellm/clients treat as non-retryable, so agent0 hard-failed. pick_feasible_neuron now distinguishes "a feasible node exists but is unhealthy right now" → new RouteError::FeasibleNodeUnhealthy (503 + Retry-After: 3, retryable) from "no node could ever satisfy the topology" → 404 NoFeasibleNeuron (permanent). Mirrors the beast case exactly: healthy 1-GPU nodes + an unhealthy 2-GPU node → retry, don't fail. Tests: poller test updated to assert debounce (1 miss keeps healthy, 3 flip); new feasibility_routing tests cover transient-503 vs permanent-404. Local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:39:18 +03:00
rob thijssen	db7e373b90	fix(neuron): decouple GET /models from the inference worker (control-plane starvation) ... Root cause of the agent0 `NoFeasibleNeuron` 404 storm: `GET /models` → `LoadedHandle::derived_limit` (#67) queried free VRAM *synchronously through the per-device worker thread* on every poll. During inference that worker is saturated serially processing forward jobs, so the VRAM query queued behind them and `/models` blocked for seconds. cortex's poller timed out on `/models`, marked the (sole-feasible) node unhealthy, and the model fell out of routing → 404. Confirmed live: under load, `/version` and `/health` stayed ~4ms while `/models` hit the 5s timeout. Fix — the HTTP control plane never touches the inference worker: - LoadedModel / TpLoadedModel gain `last_free_mb: AtomicU64`, a cached free-VRAM reading. - `derived_limit` is now sync and reads `last_free_mb` instead of awaiting a worker query — so `/models` is a pure cache read regardless of inference load. - The cache is refreshed off the request path: seeded at load (worker idle), then by a background `vram_cache_refresh_loop` every 5s. Single-GPU caches the device's free VRAM; TP caches the tightest free across ranks — the exact values `derived_limit` used before, just no longer on the request path. A transient `0` (worker gone/poisoned) never clobbers a good cached value. - The request-path live VRAM check in `validate_request` is unchanged, so the real prefill OOM guard still uses fresh readings. 226 neuron unit tests pass; non-CUDA build + fmt + clippy green. CUDA/TP paths validated by branch CI; live acceptance = `/models` stays responsive under concurrent inference (re-run of the repro). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:32:15 +03:00
rob thijssen	5c1623a817	fix(#49): allow-anonymous mode must ignore unrecognized keys, not 401 ... Regression from #49: the auth middleware rejected ANY present-but- unresolvable bearer token with 401 invalid_api_key, even when require_auth=false. But OpenAI-compatible clients (opencode, Open WebUI, Agent Zero, litellm) send a placeholder bearer by default — so enabling the build broke every existing client even though the operator never opted into auth. Pre-#49 the bearer was never inspected at all. Fix: in allow-anonymous mode (require_auth=false, the default) an unrecognized key is now ignored and the request is served anonymously, restoring pre-#49 behaviour. A bad key only 401s when require_auth=true. A valid key is still resolved + metered in both modes. Test renamed/split: unrecognized_key_is_ignored_when_auth_not_required (now 200, served anonymously) + invalid_key_is_401_when_auth_required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 21:40:34 +03:00
rob thijssen	3b60dd7a31	Merge #56 (phase 3): fail-fast prompt pre-validation + advisory hints	2026-06-17 20:57:55 +03:00
rob thijssen	4feaaf1cfb	Merge #55 (phase 2d): cortex load-aware routing across replicas	2026-06-17 20:51:26 +03:00
rob thijssen	057bc71e80	feat(#47 #56 phase 3): fail-fast prompt pre-validation + advisory hints ... Stage 3 (DX): A0 burned an hour then failed deep in litellm with prompt_too_long (35544 > 32768). cortex knows each model's real context window (#62/#67) and can pre-empt that at the edge. - Pre-validate the prompt against the model's advertised limit.context before dispatch (in proxy_with_metrics, covering chat/completions/ responses). Over → 400 context_length_exceeded in the #60 envelope — the same shape neuron emits on overflow, just earlier and without burning a cold-load/queue slot. cortex has no tokenizer, so estimate_prompt_tokens under-counts (~4 chars/token over message text); neuron stays the exact wall and we only catch gross overages. Skipped when no limit is known. - Advisory X-Helexa-Advice header: fingerprints User-Agent (litellm / Agent-Zero / Zed) and attaches client-specific guidance. Strictly advisory — header only, never in the error envelope, behaviour never depends on it; unknown clients get nothing. 3 integration tests: over-long prompt → 400 context_length_exceeded with the advice header, refused before neuron is hit; within-context passes through; unknown client gets a clean 400 with no advice header. cortex-side (no CUDA); local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:50:38 +03:00
rob thijssen	dd31c3cd49	feat(#47 #55 phase 2d): cortex load-aware routing across replicas ... Stage 2 completes: when a model is loaded on more than one healthy neuron, the router picks the least-busy replica instead of always taking the first, and neuron backpressure propagates to the client intact. - NodeState.model_load: per-model admission load (in_flight + queue_depth), stashed by the poller from neuron's /health (#53/#2b). - router::resolve collects all loaded replicas and picks the one with the lowest in_flight+queue_depth (ties break by node name for determinism), replacing the previous first-match-wins. - Backpressure passthrough: the existing streaming proxy already forwards the upstream status + all headers verbatim, so a neuron 503/429 + Retry-After + #60 envelope reaches the client unmodified — now covered by a regression test so a future change can't silently unwrap it. Tests (tests/load_routing.rs): routes to the idle replica and follows the lighter load when it flips; ties break by name; a saturated neuron's 503 + Retry-After + envelope propagates through the gateway intact. All cortex-side (no CUDA); local fmt/clippy/test green. Retry-route-to-another-replica-on-backpressure (the issue's stretch goal) is deferred — least-busy spread + honest passthrough is the substantive win. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:45:50 +03:00
rob thijssen	c83f1eb98c	feat(#47 #54 phase 2c): neuron per-principal in-flight cap (fair-share) ... Budget caps total spend over time (#52); this caps instantaneous starvation so one principal's burst can't monopolize a model while others wait. - AdmissionController gains per-principal accounting (moved from a lone atomic to a Mutex<AdmissionState> holding the overall pending count + a per-principal map). enter(principal) now also fast-rejects when a principal already has max_per_principal requests in flight/queued → AdmissionRejection::PrincipalCap. Anonymous (None) requests are exempt. - Config [harness.candle.admission].max_per_principal (default 2 = one running + one queued; 0 disables). A bursting principal's overflow is refused while a different principal still gets a queue slot. - The principal (account/key) is reconstructed on the neuron side from the x-helexa-account-id/key-id headers cortex stamps (#49) — trusted over WireGuard, never from the request body — and threaded explicitly through all inference entry points (chat_completion, *_stream(_with), responses_stream, and the TP variants) to the admission gate. - InferenceError::PerPrincipalLimit → 429 rate_limit_exceeded + Retry-After (distinct from load-shedding's 503 Overloaded); opencode/AI SDK self-pace. Tests: fair-share unit test (A floods → A's 2nd is PrincipalCap, B still queues + is served) + the existing admission tests adapted to enter(None). Non-CUDA build green locally; TP entry points (cuda-gated) validated by CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:40:25 +03:00
rob thijssen	a60c9f1075	feat(#47 #53 phase 2b): expose per-model admission load in GET /health ... Completes #53: the bounded scheduler's lock-free counters are now visible to the fleet, which is what cortex's load-aware router (#55) consumes to spread traffic across replicas and propagate honest backpressure. - cortex-core::discovery: HealthResponse gains `models: Vec<ModelLoad>` (#[serde(default)] — back-compatible; older gateways/neurons interop). ModelLoad { id, in_flight, queue_depth }. - LoadedHandle::load() → (in_flight, queue_depth), lock-free for both single-GPU and TP; CandleHarness::load_snapshot() enumerates resident models; the /health handler overlays it from the candle harness. Tests: /health always exposes a models array (api integration test); a pre-#53 payload without `models` still deserializes, and ModelLoad round-trips (cortex-core serde tests). Local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:13:07 +03:00
rob thijssen	b2bd86bfa5	feat(#47 #53 phase 2a): neuron admission control — bounded queue + backpressure ... Replaces the per-model unbounded, untimed FIFO of inference-lock waiters (a busy model made new requests hang ~300s until the client gave up with an opaque error) with an explicit bounded scheduler. - harness::admission::AdmissionController: batch-1 scheduler — max_in_flight running (1) + a bounded queue (max_queue_depth) with a max_wait. enter() fast-rejects when the queue is full (QueueFull) or the wait elapses (Timeout); the returned AdmissionPermit is held for the request and frees both slots on drop. Pure async (no CUDA), lock-free in_flight/queue_depth counters for future /health reporting. Configurable via [harness.candle.admission] (max_in_flight=1, max_queue_depth=8, max_wait_secs=30). - Gated at all four inference entry points before the inference_lock/pool lock: single-GPU non-streaming + streaming, TP non-streaming + streaming. The streaming paths acquire the permit before opening the SSE (so a rejection is a clean error, not a half-open stream) and move it into the inference task. - InferenceError::Overloaded { retry_after_secs } → 503 rate_limit_exceeded + Retry-After via the #60/#63 envelope: a fast, retryable "busy" signal opencode/AI SDK back off on, not a stall. Scope: this branch is the admission *core* (the hang→backpressure fix). Exposing in_flight/queue_depth in GET /health (consumed by cortex load-aware routing #55) is the next focused branch under #53. 4 unit tests (admit/report load, queue-full reject, wait-timeout reject) + Overloaded envelope mapping test. Non-CUDA build green locally; the CUDA + TP sites are validated by branch CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:03:07 +03:00
rob thijssen	cdf87284af	feat(#47 phase 1d): budget enforcement — hard caps, reserve→settle, 429 ... Stage 1 complete: the A0 seatbelt (#52). Flips the metering-only reserve(0) from #51 to the request's real upper-bound cost and refuses over-cap requests *before* neuron is hit. - metering::reservation_estimate: prompt estimate (~4 chars/token over the body — cortex has no tokenizer, so a conservative over-estimate; neuron stays the exact context wall) + max output. Max output comes from max_completion_tokens / legacy max_tokens, else the model's advertised limit.output (#62), else FALLBACK_MAX_OUTPUT. Over-reserving is safe — settle reconciles to actual. - metering::reserve_or_reject: reserve the estimate; on BudgetError map to the #63 envelope and the caller refuses before dispatch — rolling window → 429 rate_limit_exceeded + Retry-After (until reset); hard balance → 429 insufficient_quota (no Retry-After). Never 402. - Wired into both the OpenAI proxy path (proxy_with_metrics) and the Anthropic path (estimate from the translated body). advertised_output_limit reads the loaded model's limit.output from fleet state. - Reservation prevents overshoot under concurrency: a successful reserve gates on spent+reserved+estimate ≤ cap, and settle records actual ≤ reserved, so spend can never exceed the hard cap. 4 integration tests with a hit-counting mock neuron: balance over-cap → 429 insufficient_quota (no Retry-After, not dispatched); rolling over-cap → 429 rate_limit_exceeded + Retry-After (not dispatched); within-cap served; **A0 repro** — a capped key's 20-request fan-out drains the cap, then is refused, neuron only saw the served ones, and spend never exceeds the cap. Plus 5 metering unit tests. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:35:04 +03:00
rob thijssen	4f16b8c541	feat(#47 phase 1c): per-request token metering + spend ledger ... Stage 1 accounting (#51): capture real per-request usage and feed it to the spend ledger + per-principal metrics. Establishes the reserve→settle lifecycle that budget enforcement (#52) will tighten. - cortex-gateway::metering: ReservationGuard makes reservation leaks impossible — settle() records actual spend + releases the remainder; dropping an un-settled guard releases the whole reservation, so any early return / error / dropped stream resolves it. UsageSink is the completion hook; principal_from_headers reconstructs the principal from the middleware-stamped headers (uniform across all proxy paths, no handler-signature churn); record_spend emits per-principal counters. - proxy::TokenMetrics gains an optional usage_sink, invoked exactly once in finish() with the observed (prompt, completion) — restructured so it always runs (even when no body/usage arrived → settle 0 → release), while preserving the existing per-model metric emissions unchanged. - All proxy paths metered: chat/completions/responses via proxy_with_metrics (reserve 0 → forward_request → settle in finish); Anthropic non-streaming settles from the buffered body; Anthropic streaming (anthropic_sse) now scans the upstream frames for the usage object (#48) — it captured none before — and settles at pump end. - This phase reserves 0 tokens (metering only, no enforcement); #52 flips the reserved amount to prompt+max_output and surfaces BudgetError. The settle/release plumbing is identical, so that change is localized. - New Prometheus counters: cortex_spend_tokens_total (+ prompt/completion splits), labelled by account/key. 2 integration tests: cumulative per-key spend after N requests with reservations settled to zero outstanding; anonymous requests record no spend. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:29:51 +03:00
rob thijssen	486d7e9a8f	feat(#47 phase 1b): API-key auth + principal resolution ... Stage 1 identity (#49): cortex now knows who a request is for. Identity rides standard bearer auth only (Authorization: Bearer <key>) — no custom required headers or body fields — which is what keeps every tier OpenAI-compatible by construction. - cortex-gateway::auth: `require_principal` axum middleware (from_fn_with_state), wired in build_app outer-to-inner as trace → CORS → auth → handlers (CORS outer so preflight short-circuits). It resolves the bearer key via the EntitlementProvider, inserts the typed Principal into request extensions (for metering #51 / enforcement #52), and stamps internal x-helexa-account-id / x-helexa-key-id headers so the principal reaches neuron, which trusts cortex over WireGuard (#54). - Anti-spoofing: client-supplied principal headers are stripped before the authoritative value is stamped — a client can never assert a principal it didn't authenticate as. - Rejection contract (#63): missing key under require_auth, or any present but unresolvable key, → 401 invalid_api_key in the #60 envelope. /health and / stay public. require_auth=false (default) allows anonymous through but still 401s a present-but-invalid key. - Header-name constants (HEADER_ACCOUNT_ID/KEY_ID) live in cortex-core so neuron (#54) shares them. The chat/completions/responses paths forward the stamped headers automatically via proxy::forward_request; the Anthropic streaming + non-streaming paths forward them explicitly via auth::forward_principal_headers (they build their own upstream requests). 5 integration tests: missing-key 401, invalid-key 401 (even when auth not required, not dispatched), valid key reaches neuron with principal headers + spoofed header stripped, anonymous allowed when not required, /health public. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:07:10 +03:00
rob thijssen	bc74e0e95f	feat(#47 phase 1a): EntitlementProvider trait + local/static provider ... Stage 1's build seam (#50): the interface auth, metering, and budget enforcement all hang off, with a local/static provider so the A0 amplification fix can land before any upstream clearing house exists. The future helexa-upstream client (#57) is just another impl. - cortex-core::entitlements: Principal {account_id, key_id}, CapWindow (Balance | Rolling{seconds}), Reservation handle, BudgetSnapshot, AuthError/BudgetError, and the async EntitlementProvider trait (resolve / reserve / settle / release / snapshot). BudgetError carries the window semantics so callers pick the #63 code (rate_limit_exceeded + Retry-After vs insufficient_quota) without the provider touching HTTP. - cortex-core::config: [entitlements] section on GatewayConfig (require_auth + [[entitlements.keys]] with account_id, optional key_id, hard_cap, window). Additive + serde(default) — anonymous/uncapped when omitted, so existing setups are unaffected. - cortex-gateway::entitlements_local: LocalEntitlementProvider. Budget math serialized under one Mutex so spent+reserved can never exceed a hard cap under concurrency (the #52 guarantee); rolling windows reset lazily; uncapped keys (no hard_cap) always reserve but still meter. - CortexState gains Arc<dyn EntitlementProvider> + require_auth, built in from_config. Not yet consumed by the request path — auth middleware is 1b (#49), enforcement is 1d (#52). - cortex.example.toml documents the section; test GatewayConfig literals updated for the new field. 6 provider unit tests (resolve, unknown-key, round-trip, balance/rolling over-cap codes, uncapped infra key). Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:00:05 +03:00
rob thijssen	f22d83df14	feat(#47 phase 0): centralize OpenAI error envelope + add Retry-After ... The rejection contract (#63) requires every "no" path to speak the OpenAI envelope with standard codes and, for retryable conditions, a Retry-After header. Two gaps remained despite #63 being closed: Retry-After was implemented nowhere, and the envelope was hand-built inline in four places (gateway handlers/proxy/router, neuron api) with no shared source of truth — exactly the inconsistency #63 set out to prevent, and a foundation every Stage 1-2 rejection (401/429/503) needs. - cortex-core: new `error_envelope::OpenAiError` — an axum-agnostic builder carrying status, type, code, message, param, optional retry_after, and diagnostic extras. Named constructors encode the #63 codes (invalid_api_key, rate_limit_exceeded, insufficient_quota, context_length_exceeded, service_unavailable) and which carry Retry-After. cortex-core stays a pure types crate; each HTTP crate owns a thin `envelope_response` adapter that sets the header. - cortex-gateway: route error_response, ProxyError, and RouteError through the shared builder; RouteError::retry_after_secs wires Retry-After on the transient NoHealthyNodes (5s) / ModelRecovering (2s) variants. - neuron: route inference_error_response through the shared builder; InsufficientVram (transient 503) now advertises Retry-After: 5. Behaviour for existing paths is unchanged (same status/type/code/extras); only the new Retry-After headers are added. Tests cover the builder wire shape and Retry-After presence/absence on both sides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 18:46:56 +03:00
rob thijssen	4b28a64b34	feat(#67 phase 5b): enforce the derived input as the prompt cap ... The request path now rejects prompts above the model's self-derived input budget, not the static NEURON_MAX_PROMPT_TOKENS — so a VRAM-tight host (where the VRAM ceiling binds below the static cap) rejects an over-budget prompt up front instead of accepting it and OOMing mid-prefill. - derived_input_cap: AtomicUsize on LoadedModel + TpLoadedModel; refreshed by LoadedHandle::derived_limit (runs on every /models poll). 0 = not derived yet. - effective_prompt_cap(): cached derived input when >0, else the static max_prompt_tokens() (cold-start / no-profile fallback). - validate_request takes the cap as a param; all 4 call sites (chat_completion, inference_stream, inference_tp_stream, TP chat_completion) pass the in-scope model's effective_prompt_cap(). - doc/context-limits.md: enforcement note updated from "remaining" to landed. Reads the cap lock-free from the sync validate path (no per-request VRAM query); the cap tracks live state via the poll-driven derivation. With this, advertise and enforce agree and both track the resident model. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:26:37 +03:00
rob thijssen	dd65eedb24	feat(#67 phase 5a): NEURON_MAX_PROMPT_TOKENS becomes a clamp-only backstop; docs ... Demotes the static per-host prompt cap from authority to an optional upper-bound clamp on the self-derived limit, and rewrites the context-limits doc around the computed model. - max_prompt_tokens_clamp(): reads NEURON_MAX_PROMPT_TOKENS directly so "explicitly set" is distinct from the 16384 default; returns None when unset (no clamp). Applied as derive_limit's hard_ceiling in LoadedHandle::derived_limit, so the advertised context is clamped only when an operator set a backstop — the derivation is otherwise authoritative and binds below it in practice. - doc/context-limits.md: intro + "After #62" rewritten as "After #67 — the neuron computes its own limit" (formula, live signals, config block, opencode note, NEURON_MAX_PROMPT_TOKENS demotion). Remaining (phase 5b, follow-up): enforce the *derived* input as the prompt cap (reject above computed input, not the static NEURON_MAX_PROMPT_TOKENS) so VRAM-tight hosts can't accept an OOM-inducing prompt. Needs a per-model cached cap read from the sync validate path; scoped separately. Until then the static cap remains the enforced backstop (advertised <= enforced holds when the env is set). fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:14:34 +03:00
rob thijssen	8b2e01a072	feat(#67 phase 4): advertise neuron-computed limit on /models; drop catalogue override ... The neuron now self-derives and advertises limit{context,input,output} per loaded model; cortex forwards it and stops consulting the operator-declared catalogue limit (which can't track hot-swapped models or live capacity). Operator-set `cost` still flows from the catalogue. neuron: - CandleHarness gains context_limit_cfg (from [harness.candle.context_limit]). - LoadedHandle::derived_limit(): profile + live tightest-card free VRAM (single: query_vram; TP: query_vram_tightest_free_mb) + prefill-rate EMA (bootstrap until first sample) → derive_limit. None for arches without a context profile. No operator clamp here (advertise the honest derived value; the clamp is an enforcement-side backstop). - list_models() fills ModelInfo.limit from derived_limit (was None). - derive_limit treats free_tightest_mb == 0 (unknown/CPU sentinel) as "no VRAM ceiling" instead of collapsing to zero. cortex: - ModelEntry gains `limit`, copied from ModelInfo.limit by the poller. - /v1/models: catalogue `limit` no longer flows (Pass 1 sets None); Pass 2 adopts the neuron's limit, taking the tightest across neurons via tightest_limit(). cost unchanged. - model_limits.rs rewritten: catalogue limit (999999) is ignored; the neuron's ModelEntry.limit is advertised; cost still from catalogue. - All ModelEntry literals updated with the new field. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:10:20 +03:00
rob thijssen	464b6b0db9	feat(neuron): self-measured prefill tok/s EMA on streaming paths (#67 phase 3) ... Refs #67. Feeds the throughput ceiling a live, per-model prefill rate instead of only the configured bootstrap estimate, so the advertised limit tracks real prefill speed and rises automatically as prefix caching (#11) reduces effective prefill cost. - context_limit::PrefillRateEma: lock-free f64-bits EMA (alpha 0.3), ignores degenerate samples, None before the first sample. Unit-tested. - prefill_rate field on LoadedModel + TpLoadedModel. - Recorded as total-prompt-tokens / prefill-elapsed in the two streaming serving paths (TP: inference_tp_stream via tp_for_task; single-GPU: stream_inference_via_worker via a new &prefill_rate param threaded from loaded_for_task). Measuring total prompt (not just the divergent suffix) means a prefix-cache hit shrinks elapsed while the prompt stays large, so the effective rate — and the ceiling — rises toward the VRAM ceiling, exactly the #11 payoff. Per the agreed scope, non-streaming + CPU paths fall back to the bootstrap estimate (opencode streams; those paths rarely carry the fleet). fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:02:02 +03:00
rob thijssen	f2e05d96ec	feat(neuron): capture ContextProfile at load + per-rank VRAM fan-out (#67 phase 2) ... Refs #67. Captures the per-model context physics at load and adds the live free-VRAM signal the derivation needs — the tightest card across TP ranks, not just the leader. - ContextProfile captured at load: - single-GPU dense CUDA path (world_size 1) via context_limit::profile_from_qwen3_5_config(config_path, ..); - TP path (world_size = tp_size) at TpLoadedModel construction. GGUF/CPU/non-qwen3_5 → None (fall back to the static prompt cap). New `context_profile` field on LoadedModel + TpLoadedModel. - profile_from_qwen3_5_config(): reads config.json (mirrors VisionMeta::from_config_path), counts full_attention layers (layer_types authoritative, full_attention_interval fallback), builds the per-card KV cost via the shared helper. - Folded the inline per-rank KV-bytes math in tp_qwen3.rs (both cuda/non-cuda log_construction_complete) and tp_qwen3_5.rs onto context_limit::kv_bytes_per_token + KV_CACHE_DTYPE_BYTES. - Per-rank VRAM fan-out (tightest card): - WorkerRequest::QueryVram + WorkerResponse::VramInfo { free_mb, total_mb }; - worker.rs handle_query_vram (cuda: mem_get_info; non-cuda: error); - WorkerPool::query_vram_tightest_free_mb fans out to every rank (leader via its device worker, subprocess ranks via RPC) → min free; - TpLoadedModel::query_vram_tightest_free_mb convenience wrapper. No advertise/enforce yet (phases 4/5). fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 13:18:27 +03:00
rob thijssen	4f05a87449	feat(neuron): self-derived context-limit core — physics + policy (#67 phase 1) ... Refs #67. The correct limit{context,input,output} for a deployment is a computed function of model architecture + live free VRAM + a coherence/throughput trade-off, not an operator-declared static fact that goes stale on model swap. This lands the arch-agnostic derivation core; later phases capture per-model physics at load, measure throughput, and advertise/enforce the computed limit. - crates/neuron/src/harness/context_limit.rs (new): - kv_bytes_per_token(): shared per-card KV cost (counts only full-attention layers; sharded by TP world size). The TP load paths' inline math folds onto this in phase 2. - ContextProfile: per-model physics snapshot (max_position_embeddings, kv_bytes_per_token_per_card, world_size). - derive_limit(): context = min(max_pos, vram_ceiling, throughput_ceiling) clamped by an optional backstop; input = context − output; rounded to 1024. 6 unit tests. - config.rs: [harness.candle.context_limit] block (mirrors prefix_cache): target_prefill_latency_secs, bootstrap_prefill_tok_per_sec, activation_headroom_mb, min_free_floor_mb, output_reserve_tokens. - neuron.example.toml: documented the new block. No runtime behaviour change yet. fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 13:00:52 +03:00