363 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
rob thijssen	c596519dbd	feat(F3): anonymous chat landing — IndexedDB history + SSE + fingerprint ... The beta centerpiece: a chat workspace at `/` with zero server-side history. Everything personal lives in the browser (Dexie/IndexedDB); inference streams from the mesh router. - data/db.ts: Dexie schema (projects, conversations[owner:'anon'|accountId], messages, meta) — `owner` namespaces anonymous vs account data; claimAnonymousData() (repositories) re-owns anon data on login (F4), still client-side. - data/repositories.ts: typed CRUD + ordered queries (projects, conversations, messages) used reactively via useLiveQuery. - lib/fingerprint.ts: FingerprintJS OSS, cached in meta (best-effort, never auth) — namespaces anon data + a soft throttle id. - lib/chatClient.ts: streamChatCompletion → POST /v1/chat/completions stream:true; parses the SSE byte stream incrementally (data:/[DONE]/ partial frames), surfaces the OpenAI envelope error.code, AbortController for Stop. - lib/useChat.ts: persists the user turn, opens a streaming assistant message, appends deltas to Dexie live, titles the conversation, finalizes on done/error. - pages/Chat.tsx: sidebar (new chat / new project, conversations grouped by project + Unsorted), live-updating thread with streaming + error rendering, composer with send/stop. Anonymous mode: no bearer + VITE_ANON_MODEL + a client message cap with a sign-up nudge. Routed at `/`. - chat i18n namespace extended (newChat/newProject/unsorted/emptyState/ anonBanner/signUp/stop) across all 33 languages (parity holds). Validated: npm run lint, typecheck, build all green; i18n:check consistent. (Full browser flow exercised in F6 verify.) Explicit-path commit; no node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 11:19:19 +03:00
rob thijssen	8dd82776f1	feat(F2): /mission route — European digital sovereignty ... The secondary narrative page, re-focused (vs the reference placeholder) to lead with European digital sovereignty. - Renamed the `home` i18n namespace → `mission` (32 langs' home.json → mission.json; updated i18n/index.ts imports + ns list and the check-i18n NAMESPACES). The reference's narrative content was already translated, so the non-English languages carry over as mission copy. - Rewrote en/mission.json to lead with sovereignty: data residency, a GDPR-native no-server-side-history stance, EU operator ownership, region-affine routing, and independence from US hyperscalers — same key structure, so cross-language parity holds. - src/pages/Mission.tsx ported from the reference Home page (sections: hero/intent/whyNow/howItWorks/principles/roadAhead/joinMesh) bound to the `mission` namespace; routed at /mission in App.tsx (the Header link from F1 now resolves). Validated: npm run lint, typecheck, build all green; i18n:check (mission namespace consistent across all languages) and i18n:lang-labels pass. Explicit-path commit; no node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 11:10:14 +03:00
rob thijssen	7a6f252fe0	feat(F1): theming + 33-language i18n + usage-ordered language selector ... Ports the reference site's visual + i18n foundation into helexa.ai and adds the deliberate usage-ordered language picker. - Ported from ~/git/helexa-ai/helexa.ai: src/layout (ThemeProvider/theme, localStorage + data-theme, light/dark), src/App.css (cyan/hot-pink accents, system fonts), src/i18n (index + languages + translation-priority + resources for 33 languages × common/home/chat), Footer, DirectionalIcon, public assets, and the check-i18n-* scripts. - getLanguageOptionsByUsage() (in translation-priority.ts): orders the selector by the TRANSLATION_PRIORITY ranking (≈ native-speaker usage), deduping repeated entries and appending any unranked supported language — NOT alphabetical, the marketing-driven choice that foregrounds helexa's international grounding. RTL preserved. - Header: usage-ordered language dropdown (autonym + secondary label in the current language), theme toggle, and new nav — `/` (chat), `/mission`, and a Login/Register auth cluster stubbed until F4. New nav keys (mission/login/register/account/logout) injected into all 33 common.json with English placeholders so key-parity holds. - App composes ThemeProvider → BrowserRouter → Header + routes + Footer (placeholders for `/` and `/mission`); main.tsx loads i18n. Validated: npm run lint, typecheck, build all green; npm run i18n:check reports all keys consistent across the 33 languages. (Build emits a chunk-size advisory — code-splitting is deferred to F6 polish.) Staged with explicit paths; no node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 11:01:04 +03:00
rob thijssen	bb0d1e51b8	Merge feat/B3-cortex-upstream-client: cortex upstream entitlement client + chain (B3, #57)	2026-06-23 10:51:03 +03:00
rob thijssen	f2ba12bbc5	Merge feat/B4-account-api: /web/v1 account API + silent fingerprint abuse (B4, #59)	2026-06-23 10:42:16 +03:00
rob thijssen	a9d7382be8	feat(B3): cortex upstream entitlement client (#57) + chained provider ... cortex can now validate locally-unrecognised bearer keys against the helexa-upstream authority and reserve/settle their budget there — mesh accounts work for real inference. The EntitlementProvider trait is the seam, so cortex's enforcement (auth.rs, metering.rs) is otherwise unchanged. - entitlements_upstream.rs: UpstreamEntitlementProvider over reqwest → B2's /authz/v1 (resolve/reserve/settle/release/snapshot), presenting the operator client bearer. Maps the wire contract back to the trait: granted → Reservation, rejected → BudgetError, 401 → InvalidKey. Fail-closed — unreachable resolve → AuthError::Unavailable (503, never 401); unreachable reserve → retryable BudgetError::RateLimited (refuse, never serve un-authorized). settle/release are best-effort (the upstream sweeper reaps a lost one). - entitlements_chain.rs: ChainedEntitlementProvider tries local first (operator + infra keys, no network), falls through to upstream for unknown keys, and dispatches reserve/settle/release/snapshot to whichever backend resolved each account (local treats unknown principals as uncapped, so it can't be the blind default). - cortex-core: AuthError::Unavailable{retry_after_secs}; [upstream] config (enabled/url/bearer/timeout). auth.rs maps Unavailable → 503 + Retry-After distinctly from InvalidKey → 401, regardless of require_auth. - state.rs wires the chain when [upstream].enabled, else stays purely local. Tests (upstream_chain.rs, 4): local key resolves without touching upstream; unknown key falls through to a mock upstream; unknown-everywhere → 401 InvalidKey; upstream-unreachable → Unavailable (503-mapped), with local keys still resolving. Existing gateway suites updated for the new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:41:48 +03:00
rob thijssen	d94c62c143	feat(B4): /web/v1 account API + silent fingerprint multi-account abuse ... The human-facing account surface the helexa.ai frontend (F4) consumes, on top of B2's authz surface. Email+password auth with JWT sessions (distinct from inference API keys); plain-JSON errors (the #63 envelope stays on the authz surface). - Auth lifecycle: register → email-verify → login → password-reset (request/confirm). argon2id passwords; verify/reset via single-use sha256-hashed email tokens; register and reset-request always return 202 (no account enumeration). Email via a pluggable EmailSender (lettre Smtp + dev Log transport). - API keys: create (sk-helexa-<base62(32 OsRng)>, raw shown once, stored as sha256 + non-secret prefix), list (prefix never the secret), archive, PATCH per-key limit (percent|hardcap). Protected by a JWT session middleware. - Account balance endpoint (allocation total/spent/reserved). - Silent fingerprint abuse: register captures the browser fingerprint; >= threshold (default 5) accounts sharing one fingerprint are silently deactivated + flagged — registration still returns a normal 202, and a deactivated account's key resolves as an ordinary 401 at the authz surface (no "banned" signal anywhere). - crypto: argon2 hash/verify + CSPRNG token/key minting (base62). config gains [auth] + [email]. CORS on the app for the browser SPA. Validated against a throwaway Postgres 16: verify-once, full lifecycle (register→verify→login→create key→account→list→authz resolve→archive→401), and 5-same-fingerprint → all accounts silently deactivated + no-clue 401. 8 unit + 11 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:28:20 +03:00
rob thijssen	cb9e7c7c2e	chore: untrack helexa.ai/node_modules + dist (B2 .gitignore slip) ... The B2/B4 branches were cut before F0 added the helexa.ai .gitignore entries, so a 'git add -A' on those branches swept node_modules into the commit; it rode into main via the B2 merge. Untrack it (it stays on disk, now correctly ignored). History still carries the blobs — acceptable for an internal repo; can gc/filter later if size matters.	2026-06-23 10:27:37 +03:00
rob thijssen	2604b9f134	Merge feat/B2-authz-api: /authz/v1 authority surface + client-auth + sweeper (B2)	2026-06-23 10:25:27 +03:00
rob thijssen	178e3092d5	Merge feat/F0-helexa-ai-scaffold: helexa.ai frontend scaffold (F0)	2026-06-23 10:12:38 +03:00
rob thijssen	46befde4cd	feat(B2): /authz/v1 authority surface + client-auth + reservation sweeper ... The machine surface cortex's UpstreamEntitlementProvider (#57) consumes, mirroring the EntitlementProvider trait 1:1 over the B1 ledger. - `authz.rs`: POST /authz/v1/{resolve,reserve,settle,release,snapshot}. resolve → {principal, snapshot} | 401 invalid_api_key (a deactivated account resolves as the SAME 401 — the silent-abuse no-clue property). reserve returns 200 whether granted ({reservation_id}) or budget-refused ({rejected:{kind,...}}) — a refusal is an authoritative answer, not a transport failure; non-2xx means "fail closed" to the client. settle/ release → 204 (idempotent). snapshot → {hard_cap,spent,reserved} | 404. Rejections use the shared #63 OpenAiError envelope (cortex-core dep). - Client auth: shared-bearer middleware (constant-time compare via subtle) maps a token → operator_id (stamped into request extensions for #58 served-usage); empty config = open dev surface (logged). mTLS deferred. - ledger gains resolve_key (sha256 lookup, account-active-gated), snapshot, and sweep_stale (one data-modifying-CTE statement releasing aged-out open reservations and folding their reserved tokens back into accounts+keys). - Sweeper task spawned in run(); [authz] ttl/interval + [client_auth] config; crypto::sha256 helper. Validated against a throwaway Postgres 16 (fresh schema): resolve→reserve→ settle→snapshot round-trip, over-cap → 200 insufficient_quota rejection (not retried away), deactivated account → 401 (no clue), missing/wrong client bearer → 401 before any DB hit. 5 unit + 8 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:12:19 +03:00
rob thijssen	cf87e156c5	Merge feat/B1-helexa-upstream-skeleton: helexa-upstream skeleton + schema + ledger (B1, #59)	2026-06-23 09:58:58 +03:00
rob thijssen	79073170ec	feat(F0): helexa.ai frontend scaffold + monorepo coexistence ... New top-level `helexa.ai/` app for the public beta — Vite + React (SWC) + TypeScript + react-bootstrap + react-router + react-i18next-ready. Not a Cargo crate; lives beside the workspace. - Vite with @vitejs/plugin-react-swc (standard Vite + npm, not the reference's rolldown/pnpm pin). `vite.config.ts` dev-proxies the mesh data-plane (/v1, /health → helexa-router) and account control-plane (/api → helexa-upstream /web/v1) same-origin, targets overridable via VITE_ROUTER_BASE_URL / VITE_ACCOUNT_BASE_URL. - tsconfig (app/node, ported from the reference), eslint flat config, minimal index.css reset + bootstrap CSS, a placeholder App shell. - Deps pre-declared for later phases: dexie + dexie-react-hooks (IndexedDB chat history), @fingerprintjs/fingerprintjs (anon throttle + register fingerprint), i18next/react-i18next, react-icons. - Monorepo: root .gitignore ignores helexa.ai/{node_modules,dist} + .env.local (mirrors the existing /bench entries); committed package-lock.json for reproducible installs. Validated: npm install resolves (vite 7 + plugin-react-swc 4 + react 19), `npm run lint`/`typecheck`/`build` all green (344 modules via SWC → dist/). The frontend isn't in the Cargo workspace, so the Rust CI is unaffected. A path-filtered web CI job is deferred (needs a Node-capable runner confirmed) and folded into a later phase. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 09:58:35 +03:00
rob thijssen	71106afaf1	feat(B1): helexa-upstream crate skeleton + Postgres schema + ledger ... First milestone of the mesh-level account/authorization authority (#59). New workspace crate `crates/helexa-upstream` (binary + lib), mirroring the helexa-router skeleton: axum `build_app`/`run`, clap `serve --config`, figment `UPSTREAM_`-prefixed config. - **Storage:** PostgreSQL via sqlx (runtime query API — builds in CI with no DB or offline cache; correctness covered by gated integration tests). `migrations/0001_init.sql` is the full schema: users (argon2 hash slot, email_verified, registration_fingerprint), email_tokens, accounts (allocation total/spent/reserved + `accounts_no_overshoot` CHECK, silent `status` deactivation flag, fingerprint_flagged), api_keys (sha256 hash, percent|hardcap limit, cap_window, per-key ledger), reservations (BIGSERIAL id → maps to cortex Reservation.id u64), top_up_codes, served_usage, sessions. Migrations run on startup. - **Ledger (no-overshoot core):** `ledger::{reserve,settle,release}` — reserve takes `SELECT … FOR UPDATE` on the account + key rows so concurrent reserves serialize and spent+reserved can never exceed the effective cap (= min(resolved key cap, remaining account allocation)); the CHECK is the DB backstop. Settle clamps actual to [0,reserved] and is idempotent; release is idempotent. `resolve_abs_cap` (percent/hardcap, i128 math) is pure + unit-tested. Balance semantics here; rolling-window sub-caps + RateLimited land with the authz API (B2). - `/health` does a DB round-trip. - Config + example TOML ([server]/[db]/[grant] free grant/[abuse] fingerprint threshold). Validated end-to-end against a throwaway Postgres 16: migration applies, 20 concurrent reserves of 100 against a 500 cap admit exactly 5 (reserved == 500, never over), settle/release idempotent, hardcap key sub-cap binds below the account, `/health` → db ok. CI runs the cap-math + config unit tests; the DB integration tests skip cleanly when UPSTREAM_TEST_DATABASE_URL is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 09:49:19 +03:00
rob thijssen	d2dcdd6ebb	Merge feat/74-outbound-tls-pinning: verify downstream cortex TLS certs (#74)	2026-06-21 21:29:15 +03:00
rob thijssen	222c2a6116	Merge feat/75-federation-catalogue: aggregate /v1/models across operators (#75)	2026-06-21 21:23:49 +03:00
rob thijssen	1115bb0942	feat(#74): verify downstream cortex TLS certs (outbound pinning) ... The router is a TLS client to cortexes; the router->cortex hop crosses the helexa->operator boundary carrying the client's bearer. This pins that hop to an enrolled cert. Trust mechanism (the open question): per-cortex enrolled trust anchor. Each [[cortexes]] entry gets an optional `tls_ca` — a PEM CA (or self-signed cert) the cortex's TLS cert must chain to. When set, the router builds a client that trusts ONLY that anchor (platform roots disabled), so the cortex must present the expected cert and a rogue endpoint with any other (even publicly-valid) cert is rejected at the handshake. Enrolment = the operator hands helexa the cortex's cert, referenced by path in router config. This is the natural model for self-hosted operators behind their own nginx/private CA, and reuses the reqwest public API (no custom rustls verifier, no new TLS backend). - `RouterState` now holds a per-cortex `reqwest::Client` map (`client_for`), replacing the single shared client; poller and dispatch use the per-cortex client. `build_client(tls_ca)` is the builder. - Fail closed: a `tls_ca` that can't load omits the cortex from the client map — it's never polled or routed to, rather than silently degrading to unpinned TLS. The poller treats a missing client (and a rejected handshake) as a failed poll, so #72's existing reachability debounce excludes it. Tests (`tls.rs`, 4): a live tokio-rustls HTTPS server proves a client enrolled with the server's cert is accepted (200) while clients pinned to a different cert — or using default roots — are rejected; the poller marks a wrong-cert cortex unreachable while a correctly-enrolled one is reachable; a missing pin file disables the cortex (fail closed); garbage PEM is rejected at build. Existing suites updated for the per-cortex client + new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:23:20 +03:00
rob thijssen	63f578cb15	feat(#75): aggregate /v1/models across operators (federation catalogue) ... The router's /v1/models is now the deduped union of every reachable cortex's catalogue, so an opencode client doing discovery against the router resolves the whole federation without knowing about operators or cortexes (resolves #61's "Router/discovery contract"). To preserve per-model limit/cost, the topology poller now retains each cortex's full `cortex_core::node::CortexModelEntry` (was distilled to a {loaded, feasible} bool). `entry_feasible()` replaces the dropped field; dispatch (#73) and `cortexes_serving` use it — no routing behaviour change. `catalogue.rs::aggregate_models`: - Dedupe by model id; a model served by >=1 reachable cortex appears once. - Merge availability: `loaded` OR across operators; only feasible (loaded-or-cold-loadable) entries surface — a catalogue-only model no neuron can host is hidden. - Re-tier to operator names: `feasible_on` becomes the cortexes that can serve it and `locations` the operators it's loaded on (node = cortex name), so the federation view doesn't leak each operator's neuron names or per-device VRAM. - Conflict resolution: `limit` → tightest (smallest context, so a client never overflows the most-constrained operator); `cost` → cheapest (the federation "from" price). Richer range/region policy couples to #68, noted as follow-up. Tests: 4 unit (dedupe+merge, unreachable excluded, infeasible hidden, tightest-limit+cheapest-cost) + 1 end-to-end (two mock cortexes overlapping on a model → GET /v1/models over HTTP asserts the merged union). dispatch/topology suites updated for the entry-storage change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:08:16 +03:00
rob thijssen	76c90fa993	Merge feat/73-capacity-aware-dispatch: capacity-aware dispatch + region affinity + failover (#73)	2026-06-21 19:48:13 +03:00
rob thijssen	7984d27553	feat(#73): capacity-aware dispatch with region affinity + failover ... The router's data path. Wires the topology poller (#72) and the shared streaming proxy (#71) into real request routing. - `dispatch.rs`: `select_cortexes(model)` ranks reachable cortexes that can serve the model, best-first — loaded/warm before cold-loadable, region match before not, more healthy nodes before fewer, name for determinism. `dispatch()` extracts `model`, picks candidates, and forwards via `helexa_stream::forward_streaming` (bearer + bytes verbatim, SSE streamed back). Cortex's #63 rejections (429/400/…) pass through untouched; transport failures fail over to the next candidate; a genuine HTTP response — any status — is returned as-is, never retried away. - Router-originated rejections use the #63 envelope: 404 model_not_found (no operator serves it), 503 service_unavailable + Retry-After (known but all unreachable / all candidates failed to connect), 400 missing_model_field. `error.rs` is the router's envelope→axum adapter (mirrors cortex-gateway's). - `handlers.rs`: `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages` dispatch to the same path on a chosen cortex. The router holds zero entitlement logic — routes on capacity, not budget. - Config: optional `region` on the router and per-cortex for geo affinity. Tests (`dispatch.rs`): routes to a serving cortex + forwards the bearer; cortex 429 passes through and is NOT retried; transport failure fails over to a live cortex; unknown→404, known-but-unreachable→503, missing-model→400; ranking order (warm/region/headroom). 7 new, existing skeleton/topology suites unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:40:07 +03:00
rob thijssen	43ffffdccb	Merge feat/72-router-topology-poller: router↔cortex capacity & catalogue poller (#72)	2026-06-21 19:09:13 +03:00
rob thijssen	5fd7736abd	feat(#72): router↔cortex topology poller (multi-operator capacity map) ... Builds the live topology the dispatcher (#73) will route on — the same pattern as cortex↔neuron, one tier up. - `poller.rs`: background loop polls each configured cortex's `GET /v1/models` (deserialised straight into the shared `cortex_core::node::CortexModelEntry`) and `GET /health`, on a configurable `poll_interval_secs` (default 10). - `state.rs`: `RouterState` gains an `http_client`, `poll_interval`, and a `RwLock<HashMap<cortex_name, CortexTopology>>` pre-populated from config so the poller/handlers always find an entry. Per cortex: `reachable`, `consecutive_failures`, `last_poll`, healthy/total node counts, and a per-model `{loaded, feasible}` map (feasible = loaded OR cortex reports `feasible_on`, i.e. cold-loadable). `cortexes_serving(model)` returns the reachable cortexes that can serve a model — groundwork for #73. - Debounce: a cortex flips unreachable only after `POLL_FAILURE_THRESHOLD` (3) consecutive failed polls, and recovers on the next good poll — mirrors cortex's neuron-poll debounce so a blip can't yank a whole operator out of routing. `/health` poll is best-effort and never flips reachability on its own. - `lib.rs` spawns the poll loop in `run()`. `/health` now surfaces `cortexes.reachable`; `status` stays router-liveness (always `ok`). Tests (`topology.rs`): live-map build (loaded vs catalogue-only feasible, node counts, routing helper); unreachable→excluded→recovers across the debounce threshold; dead endpoint never panics. Skeleton tests unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:01:29 +03:00
rob thijssen	03fd4960c3	Merge fix/71-shared-streaming-proxy: shared helexa-stream SSE proxy (#71) ... # Conflicts: # Cargo.lock # Cargo.toml	2026-06-21 18:15:12 +03:00
rob thijssen	5ed6bc3390	Merge feat/70-router-skeleton: helexa-router binary skeleton (#70)	2026-06-21 18:06:07 +03:00
rob thijssen	cabec1d08a	fix(#71): extract SSE streaming passthrough into shared helexa-stream ... The true-streaming SSE passthrough (Body::from_stream, no full-response buffering, with chunk-observation hooks) was cortex-only. helexa-router (#69) needs the same mechanism to proxy a chat-completions/messages stream verbatim to a selected cortex. Extract it once. New `crates/helexa-stream` owns the *mechanism* (kept HTTP-free cortex-core untouched — it would have forced axum/reqwest/futures onto every cortex-core consumer): - `forward_streaming(client, url, headers, body, observer)` — POST and stream the response back chunk-for-chunk; status-agnostic, so a non-2xx (e.g. cortex 429) is passed through with status+headers intact (the #69 backpressure-passthrough requirement). - `ChunkObserver` trait + `ObservedStream` wrapper — feeds each chunk to the observer, calls `finish` exactly once on clean end or on drop (client disconnect). - `BodyTail` (bounded tail accumulator) + `last_count_for` (trailing OpenAI `usage` extraction) — the reusable pieces an observer uses. cortex keeps its *policy*: `proxy.rs` now supplies a `CortexMetrics` observer (per-request token metrics + per-principal reservation settle), its logging contract, and the error envelope, driving the shared mechanism. `proxy::last_count_for` is re-exported so `handlers`/ `anthropic_sse` call sites are unchanged. No behaviour change — the existing cortex `streaming.rs` tests pass as-is. helexa-stream tests prove chunk-for-chunk incremental delivery, observer finish-once, usage extraction, and non-2xx passthrough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 18:05:35 +03:00
rob thijssen	881fc85a4c	feat(#70): helexa-router binary skeleton — plaintext axum server ... Foundation for epic #69 (public multi-operator ingress proxy). New `crates/helexa-router` workspace binary: a plaintext axum server that reuses cortex-core types and serves the two stub endpoints the rest of #69 builds on. - `[router] listen` + `[[cortexes]]` config via figment + `HELEXA_ROUTER_` env overrides, matching the cortex/neuron convention. - `GET /health` reports the configured downstream cortex count. - `GET /v1/models` returns an empty OpenAI list (real cross-operator aggregation is #75). - No inbound TLS listener (edge nginx terminates client TLS per #69's posture); no auth layer — the router forwards the client bearer to cortex and holds zero entitlement logic (#47 stays additive). - 3 tests: both endpoints over a real ephemeral-port server, plus TOML+env config load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 17:52:28 +03:00
rob thijssen	b2ed20b55a	docs(CLAUDE.md): document the branch → CI → merge-on-green workflow ... Capture the development loop in-repo so it's available to any session, not just via personal agent memory: feature branch per change; local CI triad is CPU-only so the branch CI's CUDA type-check is the real gate for neuron/TP changes; push on local-green and background-watch; merge when the four validation jobs are green (not the SRPM/COPR deploy jobs); docs-only changes can go straight to main. Notes the core.sshCommand key-pinning gotcha. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-21 15:05:46 +03:00
rob thijssen	bee27e9b9c	Merge fix/68-cost-schema-wire-contract: pin the /v1/models cost wire contract ... Closes #68. Documents ModelCost as the source-of-truth pricing field (USD per 1M tokens, JSON numbers — models.dev/opencode shape), defines the absent-vs-0.0 distinction (not-priced vs intentionally-free), adds a wire test locking it, and documents cost.* in models.example.toml. The cost code path already existed; this pins the contract. Branch CI green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 12:08:23 +03:00
rob thijssen	87d9c291ce	fix(#68): pin the /v1/models cost wire contract — units + absent-vs-zero ... The cost code path already exists (cortex list_models populates cost: profile.cost from the catalogue; aliases inherit it), so opencode's $0.00 is a config gap (no cost in the live models.toml), not missing plumbing. What was missing is the *contract*: units pinned against a wire test, and a defined meaning for "free". - Document ModelCost as the load-bearing source of truth: USD per 1,000,000 tokens as JSON numbers (models.dev/opencode shape) — NOT per-token, NOT decimal strings (OpenRouter's pricing shape, which helexa deliberately does not emit). Define the absent-vs-zero distinction: cost omitted = "not priced / unknown"; cost present with 0.0 = "intentionally free". Note the advertised rate must equal what metering (#51) / reconciliation (#58/#59) bill against — today both read this catalogue value. - New wire test (model_cost.rs): a priced model with cache tiers flows through as per-million numbers; an explicit-0.0 free model keeps its cost block with cache tiers omitted; an unpriced model omits `cost` entirely. - models.example.toml: document cost.* in the field reference and show all three cases (priced-free explicit 0.0 vs the unpriced Qwen3-8B with no cost block). Decisions recorded on #68: source of truth = operator models.toml for now (marketplace clearing house #59 later, same value); no OpenRouter-style `pricing` (opencode/models.dev alignment is sufficient); end-to-end non-zero $ spent needs operators to populate cost in the live catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 12:02:03 +03:00
rob thijssen	d4742467e0	Merge fix/65-text-prefill-vram-backstop: request-time length-aware VRAM backstop for text prefill ... Closes #65. Gives the text prefill path a request-time, length-aware VRAM guard (reusing #67's ContextProfile KV cost against current free VRAM), closing the poll-vs-request snapshot staleness gap and the vision/text asymmetry. Branch CI green (fmt, clippy, test, CUDA type-check). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 11:53:33 +03:00
rob thijssen	e7f7e376fc	fix(#65): request-time length-aware VRAM backstop for text prefill ... Close the poll-vs-request snapshot gap #67 left open. The text prefill guard (validate_request) only checked the static min_free_vram floor; the derived input cap (effective_prompt_cap) is computed at /models poll time from the tightest card's free VRAM *then*. If free VRAM drops between that poll and the request — a co-resident model loads, a concurrent prefill grows its KV — a prompt at-or-below the now-stale cap clears the floor yet no longer fits, OOMing mid-prefill and poisoning the device context (the 2026-05-26 beast incident #47 exists to eliminate). validate_request now re-runs #67's length×KV-vs-VRAM physics against request-time free VRAM, reusing the model's ContextProfile (kv_bytes_per_token_per_card, full-attention-layer-only, TP-sharded) rather than re-deriving the cost. Footprint = KV(prompt + output_reserve) + activation_headroom + static floor, all per card and commensurable with the tightest-card free VRAM on both single-GPU and TP loads. Degenerate zero-KV / no-profile models ride the existing floor check, mirroring derive_limit's VRAM-ceiling fallback; CPU loads (vram_free_mb == 0) skip all VRAM checks unchanged. This closes the vision/text asymmetry: the text path now has the live-VRAM guard validate_vision_prefill already gave the vision path. 5 unit tests incl. the acceptance staleness test: a cap derived against ample free VRAM, applied at request time against tightened VRAM, rejects a prompt sized at the stale cap with a clean InsufficientVram (503) instead of an OOM. Threaded context_limit_cfg into chat_completion_tp_inner (spawned, no &self) and used &self.context_limit_cfg at the three method call sites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 11:45:53 +03:00
rob thijssen	3b9a6e37f6	Merge fix/cortex-poll-debounce-retryable: poll debounce + retryable 503 for feasible-but-unhealthy node	2026-06-18 12:46:30 +03:00
rob thijssen	526b662c5e	fix(cortex): poll-failure debounce + retryable 503 for feasible-but-unhealthy node ... Defense-in-depth for the agent0 NoFeasibleNeuron storm (root cause fixed in neuron). Two cortex resilience gaps this incident exposed: 1. Brittle health flip: the poller marked a node unhealthy on a SINGLE missed /models poll, instantly yanking the node and all its models from routing. A busy neuron briefly slow to answer shouldn't be declared dead. Now debounced: NodeState.consecutive_poll_failures must reach POLL_FAILURE_THRESHOLD (3) before the node flips unhealthy (~20s at the 10s poll interval); any successful poll resets it. A never-healthy node stays unhealthy (the counter only protects an already-healthy node from blips). 2. Transient surfaced as permanent: when a catalogued model's only feasible neuron is momentarily unhealthy, the router returned 404 NoFeasibleNeuron — which litellm/clients treat as non-retryable, so agent0 hard-failed. pick_feasible_neuron now distinguishes "a feasible node exists but is unhealthy right now" → new RouteError::FeasibleNodeUnhealthy (503 + Retry-After: 3, retryable) from "no node could ever satisfy the topology" → 404 NoFeasibleNeuron (permanent). Mirrors the beast case exactly: healthy 1-GPU nodes + an unhealthy 2-GPU node → retry, don't fail. Tests: poller test updated to assert debounce (1 miss keeps healthy, 3 flip); new feasibility_routing tests cover transient-503 vs permanent-404. Local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:39:18 +03:00
rob thijssen	db7e373b90	fix(neuron): decouple GET /models from the inference worker (control-plane starvation) ... Root cause of the agent0 `NoFeasibleNeuron` 404 storm: `GET /models` → `LoadedHandle::derived_limit` (#67) queried free VRAM *synchronously through the per-device worker thread* on every poll. During inference that worker is saturated serially processing forward jobs, so the VRAM query queued behind them and `/models` blocked for seconds. cortex's poller timed out on `/models`, marked the (sole-feasible) node unhealthy, and the model fell out of routing → 404. Confirmed live: under load, `/version` and `/health` stayed ~4ms while `/models` hit the 5s timeout. Fix — the HTTP control plane never touches the inference worker: - LoadedModel / TpLoadedModel gain `last_free_mb: AtomicU64`, a cached free-VRAM reading. - `derived_limit` is now sync and reads `last_free_mb` instead of awaiting a worker query — so `/models` is a pure cache read regardless of inference load. - The cache is refreshed off the request path: seeded at load (worker idle), then by a background `vram_cache_refresh_loop` every 5s. Single-GPU caches the device's free VRAM; TP caches the tightest free across ranks — the exact values `derived_limit` used before, just no longer on the request path. A transient `0` (worker gone/poisoned) never clobbers a good cached value. - The request-path live VRAM check in `validate_request` is unchanged, so the real prefill OOM guard still uses fresh readings. 226 neuron unit tests pass; non-CUDA build + fmt + clippy green. CUDA/TP paths validated by branch CI; live acceptance = `/models` stays responsive under concurrent inference (re-run of the repro). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:32:15 +03:00
rob thijssen	5c1623a817	fix(#49): allow-anonymous mode must ignore unrecognized keys, not 401 ... Regression from #49: the auth middleware rejected ANY present-but- unresolvable bearer token with 401 invalid_api_key, even when require_auth=false. But OpenAI-compatible clients (opencode, Open WebUI, Agent Zero, litellm) send a placeholder bearer by default — so enabling the build broke every existing client even though the operator never opted into auth. Pre-#49 the bearer was never inspected at all. Fix: in allow-anonymous mode (require_auth=false, the default) an unrecognized key is now ignored and the request is served anonymously, restoring pre-#49 behaviour. A bad key only 401s when require_auth=true. A valid key is still resolved + metered in both modes. Test renamed/split: unrecognized_key_is_ignored_when_auth_not_required (now 200, served anonymously) + invalid_key_is_401_when_auth_required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 21:40:34 +03:00
rob thijssen	3b60dd7a31	Merge #56 (phase 3): fail-fast prompt pre-validation + advisory hints	2026-06-17 20:57:55 +03:00
rob thijssen	4feaaf1cfb	Merge #55 (phase 2d): cortex load-aware routing across replicas	2026-06-17 20:51:26 +03:00
rob thijssen	057bc71e80	feat(#47 #56 phase 3): fail-fast prompt pre-validation + advisory hints ... Stage 3 (DX): A0 burned an hour then failed deep in litellm with prompt_too_long (35544 > 32768). cortex knows each model's real context window (#62/#67) and can pre-empt that at the edge. - Pre-validate the prompt against the model's advertised limit.context before dispatch (in proxy_with_metrics, covering chat/completions/ responses). Over → 400 context_length_exceeded in the #60 envelope — the same shape neuron emits on overflow, just earlier and without burning a cold-load/queue slot. cortex has no tokenizer, so estimate_prompt_tokens under-counts (~4 chars/token over message text); neuron stays the exact wall and we only catch gross overages. Skipped when no limit is known. - Advisory X-Helexa-Advice header: fingerprints User-Agent (litellm / Agent-Zero / Zed) and attaches client-specific guidance. Strictly advisory — header only, never in the error envelope, behaviour never depends on it; unknown clients get nothing. 3 integration tests: over-long prompt → 400 context_length_exceeded with the advice header, refused before neuron is hit; within-context passes through; unknown client gets a clean 400 with no advice header. cortex-side (no CUDA); local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:50:38 +03:00
rob thijssen	dd31c3cd49	feat(#47 #55 phase 2d): cortex load-aware routing across replicas ... Stage 2 completes: when a model is loaded on more than one healthy neuron, the router picks the least-busy replica instead of always taking the first, and neuron backpressure propagates to the client intact. - NodeState.model_load: per-model admission load (in_flight + queue_depth), stashed by the poller from neuron's /health (#53/#2b). - router::resolve collects all loaded replicas and picks the one with the lowest in_flight+queue_depth (ties break by node name for determinism), replacing the previous first-match-wins. - Backpressure passthrough: the existing streaming proxy already forwards the upstream status + all headers verbatim, so a neuron 503/429 + Retry-After + #60 envelope reaches the client unmodified — now covered by a regression test so a future change can't silently unwrap it. Tests (tests/load_routing.rs): routes to the idle replica and follows the lighter load when it flips; ties break by name; a saturated neuron's 503 + Retry-After + envelope propagates through the gateway intact. All cortex-side (no CUDA); local fmt/clippy/test green. Retry-route-to-another-replica-on-backpressure (the issue's stretch goal) is deferred — least-busy spread + honest passthrough is the substantive win. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:45:50 +03:00
rob thijssen	c83f1eb98c	feat(#47 #54 phase 2c): neuron per-principal in-flight cap (fair-share) ... Budget caps total spend over time (#52); this caps instantaneous starvation so one principal's burst can't monopolize a model while others wait. - AdmissionController gains per-principal accounting (moved from a lone atomic to a Mutex<AdmissionState> holding the overall pending count + a per-principal map). enter(principal) now also fast-rejects when a principal already has max_per_principal requests in flight/queued → AdmissionRejection::PrincipalCap. Anonymous (None) requests are exempt. - Config [harness.candle.admission].max_per_principal (default 2 = one running + one queued; 0 disables). A bursting principal's overflow is refused while a different principal still gets a queue slot. - The principal (account/key) is reconstructed on the neuron side from the x-helexa-account-id/key-id headers cortex stamps (#49) — trusted over WireGuard, never from the request body — and threaded explicitly through all inference entry points (chat_completion, *_stream(_with), responses_stream, and the TP variants) to the admission gate. - InferenceError::PerPrincipalLimit → 429 rate_limit_exceeded + Retry-After (distinct from load-shedding's 503 Overloaded); opencode/AI SDK self-pace. Tests: fair-share unit test (A floods → A's 2nd is PrincipalCap, B still queues + is served) + the existing admission tests adapted to enter(None). Non-CUDA build green locally; TP entry points (cuda-gated) validated by CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:40:25 +03:00
rob thijssen	a60c9f1075	feat(#47 #53 phase 2b): expose per-model admission load in GET /health ... Completes #53: the bounded scheduler's lock-free counters are now visible to the fleet, which is what cortex's load-aware router (#55) consumes to spread traffic across replicas and propagate honest backpressure. - cortex-core::discovery: HealthResponse gains `models: Vec<ModelLoad>` (#[serde(default)] — back-compatible; older gateways/neurons interop). ModelLoad { id, in_flight, queue_depth }. - LoadedHandle::load() → (in_flight, queue_depth), lock-free for both single-GPU and TP; CandleHarness::load_snapshot() enumerates resident models; the /health handler overlays it from the candle harness. Tests: /health always exposes a models array (api integration test); a pre-#53 payload without `models` still deserializes, and ModelLoad round-trips (cortex-core serde tests). Local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:13:07 +03:00
rob thijssen	b2bd86bfa5	feat(#47 #53 phase 2a): neuron admission control — bounded queue + backpressure ... Replaces the per-model unbounded, untimed FIFO of inference-lock waiters (a busy model made new requests hang ~300s until the client gave up with an opaque error) with an explicit bounded scheduler. - harness::admission::AdmissionController: batch-1 scheduler — max_in_flight running (1) + a bounded queue (max_queue_depth) with a max_wait. enter() fast-rejects when the queue is full (QueueFull) or the wait elapses (Timeout); the returned AdmissionPermit is held for the request and frees both slots on drop. Pure async (no CUDA), lock-free in_flight/queue_depth counters for future /health reporting. Configurable via [harness.candle.admission] (max_in_flight=1, max_queue_depth=8, max_wait_secs=30). - Gated at all four inference entry points before the inference_lock/pool lock: single-GPU non-streaming + streaming, TP non-streaming + streaming. The streaming paths acquire the permit before opening the SSE (so a rejection is a clean error, not a half-open stream) and move it into the inference task. - InferenceError::Overloaded { retry_after_secs } → 503 rate_limit_exceeded + Retry-After via the #60/#63 envelope: a fast, retryable "busy" signal opencode/AI SDK back off on, not a stall. Scope: this branch is the admission *core* (the hang→backpressure fix). Exposing in_flight/queue_depth in GET /health (consumed by cortex load-aware routing #55) is the next focused branch under #53. 4 unit tests (admit/report load, queue-full reject, wait-timeout reject) + Overloaded envelope mapping test. Non-CUDA build green locally; the CUDA + TP sites are validated by branch CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 20:03:07 +03:00
rob thijssen	cdf87284af	feat(#47 phase 1d): budget enforcement — hard caps, reserve→settle, 429 ... Stage 1 complete: the A0 seatbelt (#52). Flips the metering-only reserve(0) from #51 to the request's real upper-bound cost and refuses over-cap requests *before* neuron is hit. - metering::reservation_estimate: prompt estimate (~4 chars/token over the body — cortex has no tokenizer, so a conservative over-estimate; neuron stays the exact context wall) + max output. Max output comes from max_completion_tokens / legacy max_tokens, else the model's advertised limit.output (#62), else FALLBACK_MAX_OUTPUT. Over-reserving is safe — settle reconciles to actual. - metering::reserve_or_reject: reserve the estimate; on BudgetError map to the #63 envelope and the caller refuses before dispatch — rolling window → 429 rate_limit_exceeded + Retry-After (until reset); hard balance → 429 insufficient_quota (no Retry-After). Never 402. - Wired into both the OpenAI proxy path (proxy_with_metrics) and the Anthropic path (estimate from the translated body). advertised_output_limit reads the loaded model's limit.output from fleet state. - Reservation prevents overshoot under concurrency: a successful reserve gates on spent+reserved+estimate ≤ cap, and settle records actual ≤ reserved, so spend can never exceed the hard cap. 4 integration tests with a hit-counting mock neuron: balance over-cap → 429 insufficient_quota (no Retry-After, not dispatched); rolling over-cap → 429 rate_limit_exceeded + Retry-After (not dispatched); within-cap served; **A0 repro** — a capped key's 20-request fan-out drains the cap, then is refused, neuron only saw the served ones, and spend never exceeds the cap. Plus 5 metering unit tests. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:35:04 +03:00
rob thijssen	4f16b8c541	feat(#47 phase 1c): per-request token metering + spend ledger ... Stage 1 accounting (#51): capture real per-request usage and feed it to the spend ledger + per-principal metrics. Establishes the reserve→settle lifecycle that budget enforcement (#52) will tighten. - cortex-gateway::metering: ReservationGuard makes reservation leaks impossible — settle() records actual spend + releases the remainder; dropping an un-settled guard releases the whole reservation, so any early return / error / dropped stream resolves it. UsageSink is the completion hook; principal_from_headers reconstructs the principal from the middleware-stamped headers (uniform across all proxy paths, no handler-signature churn); record_spend emits per-principal counters. - proxy::TokenMetrics gains an optional usage_sink, invoked exactly once in finish() with the observed (prompt, completion) — restructured so it always runs (even when no body/usage arrived → settle 0 → release), while preserving the existing per-model metric emissions unchanged. - All proxy paths metered: chat/completions/responses via proxy_with_metrics (reserve 0 → forward_request → settle in finish); Anthropic non-streaming settles from the buffered body; Anthropic streaming (anthropic_sse) now scans the upstream frames for the usage object (#48) — it captured none before — and settles at pump end. - This phase reserves 0 tokens (metering only, no enforcement); #52 flips the reserved amount to prompt+max_output and surfaces BudgetError. The settle/release plumbing is identical, so that change is localized. - New Prometheus counters: cortex_spend_tokens_total (+ prompt/completion splits), labelled by account/key. 2 integration tests: cumulative per-key spend after N requests with reservations settled to zero outstanding; anonymous requests record no spend. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:29:51 +03:00
rob thijssen	486d7e9a8f	feat(#47 phase 1b): API-key auth + principal resolution ... Stage 1 identity (#49): cortex now knows who a request is for. Identity rides standard bearer auth only (Authorization: Bearer <key>) — no custom required headers or body fields — which is what keeps every tier OpenAI-compatible by construction. - cortex-gateway::auth: `require_principal` axum middleware (from_fn_with_state), wired in build_app outer-to-inner as trace → CORS → auth → handlers (CORS outer so preflight short-circuits). It resolves the bearer key via the EntitlementProvider, inserts the typed Principal into request extensions (for metering #51 / enforcement #52), and stamps internal x-helexa-account-id / x-helexa-key-id headers so the principal reaches neuron, which trusts cortex over WireGuard (#54). - Anti-spoofing: client-supplied principal headers are stripped before the authoritative value is stamped — a client can never assert a principal it didn't authenticate as. - Rejection contract (#63): missing key under require_auth, or any present but unresolvable key, → 401 invalid_api_key in the #60 envelope. /health and / stay public. require_auth=false (default) allows anonymous through but still 401s a present-but-invalid key. - Header-name constants (HEADER_ACCOUNT_ID/KEY_ID) live in cortex-core so neuron (#54) shares them. The chat/completions/responses paths forward the stamped headers automatically via proxy::forward_request; the Anthropic streaming + non-streaming paths forward them explicitly via auth::forward_principal_headers (they build their own upstream requests). 5 integration tests: missing-key 401, invalid-key 401 (even when auth not required, not dispatched), valid key reaches neuron with principal headers + spoofed header stripped, anonymous allowed when not required, /health public. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:07:10 +03:00
rob thijssen	bc74e0e95f	feat(#47 phase 1a): EntitlementProvider trait + local/static provider ... Stage 1's build seam (#50): the interface auth, metering, and budget enforcement all hang off, with a local/static provider so the A0 amplification fix can land before any upstream clearing house exists. The future helexa-upstream client (#57) is just another impl. - cortex-core::entitlements: Principal {account_id, key_id}, CapWindow (Balance | Rolling{seconds}), Reservation handle, BudgetSnapshot, AuthError/BudgetError, and the async EntitlementProvider trait (resolve / reserve / settle / release / snapshot). BudgetError carries the window semantics so callers pick the #63 code (rate_limit_exceeded + Retry-After vs insufficient_quota) without the provider touching HTTP. - cortex-core::config: [entitlements] section on GatewayConfig (require_auth + [[entitlements.keys]] with account_id, optional key_id, hard_cap, window). Additive + serde(default) — anonymous/uncapped when omitted, so existing setups are unaffected. - cortex-gateway::entitlements_local: LocalEntitlementProvider. Budget math serialized under one Mutex so spent+reserved can never exceed a hard cap under concurrency (the #52 guarantee); rolling windows reset lazily; uncapped keys (no hard_cap) always reserve but still meter. - CortexState gains Arc<dyn EntitlementProvider> + require_auth, built in from_config. Not yet consumed by the request path — auth middleware is 1b (#49), enforcement is 1d (#52). - cortex.example.toml documents the section; test GatewayConfig literals updated for the new field. 6 provider unit tests (resolve, unknown-key, round-trip, balance/rolling over-cap codes, uncapped infra key). Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:00:05 +03:00
rob thijssen	f22d83df14	feat(#47 phase 0): centralize OpenAI error envelope + add Retry-After ... The rejection contract (#63) requires every "no" path to speak the OpenAI envelope with standard codes and, for retryable conditions, a Retry-After header. Two gaps remained despite #63 being closed: Retry-After was implemented nowhere, and the envelope was hand-built inline in four places (gateway handlers/proxy/router, neuron api) with no shared source of truth — exactly the inconsistency #63 set out to prevent, and a foundation every Stage 1-2 rejection (401/429/503) needs. - cortex-core: new `error_envelope::OpenAiError` — an axum-agnostic builder carrying status, type, code, message, param, optional retry_after, and diagnostic extras. Named constructors encode the #63 codes (invalid_api_key, rate_limit_exceeded, insufficient_quota, context_length_exceeded, service_unavailable) and which carry Retry-After. cortex-core stays a pure types crate; each HTTP crate owns a thin `envelope_response` adapter that sets the header. - cortex-gateway: route error_response, ProxyError, and RouteError through the shared builder; RouteError::retry_after_secs wires Retry-After on the transient NoHealthyNodes (5s) / ModelRecovering (2s) variants. - neuron: route inference_error_response through the shared builder; InsufficientVram (transient 503) now advertises Retry-After: 5. Behaviour for existing paths is unchanged (same status/type/code/extras); only the new Retry-After headers are added. Tests cover the builder wire shape and Retry-After presence/absence on both sides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 18:46:56 +03:00
rob thijssen	4b28a64b34	feat(#67 phase 5b): enforce the derived input as the prompt cap ... The request path now rejects prompts above the model's self-derived input budget, not the static NEURON_MAX_PROMPT_TOKENS — so a VRAM-tight host (where the VRAM ceiling binds below the static cap) rejects an over-budget prompt up front instead of accepting it and OOMing mid-prefill. - derived_input_cap: AtomicUsize on LoadedModel + TpLoadedModel; refreshed by LoadedHandle::derived_limit (runs on every /models poll). 0 = not derived yet. - effective_prompt_cap(): cached derived input when >0, else the static max_prompt_tokens() (cold-start / no-profile fallback). - validate_request takes the cap as a param; all 4 call sites (chat_completion, inference_stream, inference_tp_stream, TP chat_completion) pass the in-scope model's effective_prompt_cap(). - doc/context-limits.md: enforcement note updated from "remaining" to landed. Reads the cap lock-free from the sync validate path (no per-request VRAM query); the cap tracks live state via the poll-driven derivation. With this, advertise and enforce agree and both track the resident model. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:26:37 +03:00
rob thijssen	dd65eedb24	feat(#67 phase 5a): NEURON_MAX_PROMPT_TOKENS becomes a clamp-only backstop; docs ... Demotes the static per-host prompt cap from authority to an optional upper-bound clamp on the self-derived limit, and rewrites the context-limits doc around the computed model. - max_prompt_tokens_clamp(): reads NEURON_MAX_PROMPT_TOKENS directly so "explicitly set" is distinct from the 16384 default; returns None when unset (no clamp). Applied as derive_limit's hard_ceiling in LoadedHandle::derived_limit, so the advertised context is clamped only when an operator set a backstop — the derivation is otherwise authoritative and binds below it in practice. - doc/context-limits.md: intro + "After #62" rewritten as "After #67 — the neuron computes its own limit" (formula, live signals, config block, opencode note, NEURON_MAX_PROMPT_TOKENS demotion). Remaining (phase 5b, follow-up): enforce the *derived* input as the prompt cap (reject above computed input, not the static NEURON_MAX_PROMPT_TOKENS) so VRAM-tight hosts can't accept an OOM-inducing prompt. Needs a per-model cached cap read from the sync validate path; scoped separately. Until then the static cap remains the enforced backstop (advertised <= enforced holds when the env is set). fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:14:34 +03:00
rob thijssen	8b2e01a072	feat(#67 phase 4): advertise neuron-computed limit on /models; drop catalogue override ... The neuron now self-derives and advertises limit{context,input,output} per loaded model; cortex forwards it and stops consulting the operator-declared catalogue limit (which can't track hot-swapped models or live capacity). Operator-set `cost` still flows from the catalogue. neuron: - CandleHarness gains context_limit_cfg (from [harness.candle.context_limit]). - LoadedHandle::derived_limit(): profile + live tightest-card free VRAM (single: query_vram; TP: query_vram_tightest_free_mb) + prefill-rate EMA (bootstrap until first sample) → derive_limit. None for arches without a context profile. No operator clamp here (advertise the honest derived value; the clamp is an enforcement-side backstop). - list_models() fills ModelInfo.limit from derived_limit (was None). - derive_limit treats free_tightest_mb == 0 (unknown/CPU sentinel) as "no VRAM ceiling" instead of collapsing to zero. cortex: - ModelEntry gains `limit`, copied from ModelInfo.limit by the poller. - /v1/models: catalogue `limit` no longer flows (Pass 1 sets None); Pass 2 adopts the neuron's limit, taking the tightest across neurons via tightest_limit(). cost unchanged. - model_limits.rs rewritten: catalogue limit (999999) is ignored; the neuron's ModelEntry.limit is advertised; cost still from catalogue. - All ModelEntry literals updated with the new field. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:10:20 +03:00