7 Commits

Author	SHA1	Message	Date
rob thijssen	bc74e0e95f	feat(#47 phase 1a): EntitlementProvider trait + local/static provider ... Stage 1's build seam (#50): the interface auth, metering, and budget enforcement all hang off, with a local/static provider so the A0 amplification fix can land before any upstream clearing house exists. The future helexa-upstream client (#57) is just another impl. - cortex-core::entitlements: Principal {account_id, key_id}, CapWindow (Balance | Rolling{seconds}), Reservation handle, BudgetSnapshot, AuthError/BudgetError, and the async EntitlementProvider trait (resolve / reserve / settle / release / snapshot). BudgetError carries the window semantics so callers pick the #63 code (rate_limit_exceeded + Retry-After vs insufficient_quota) without the provider touching HTTP. - cortex-core::config: [entitlements] section on GatewayConfig (require_auth + [[entitlements.keys]] with account_id, optional key_id, hard_cap, window). Additive + serde(default) — anonymous/uncapped when omitted, so existing setups are unaffected. - cortex-gateway::entitlements_local: LocalEntitlementProvider. Budget math serialized under one Mutex so spent+reserved can never exceed a hard cap under concurrency (the #52 guarantee); rolling windows reset lazily; uncapped keys (no hard_cap) always reserve but still meter. - CortexState gains Arc<dyn EntitlementProvider> + require_auth, built in from_config. Not yet consumed by the request path — auth middleware is 1b (#49), enforcement is 1d (#52). - cortex.example.toml documents the section; test GatewayConfig literals updated for the new field. 6 provider unit tests (resolve, unknown-key, round-trip, balance/rolling over-cap codes, uncapped infra key). Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:00:05 +03:00
rob thijssen	8b2e01a072	feat(#67 phase 4): advertise neuron-computed limit on /models; drop catalogue override ... The neuron now self-derives and advertises limit{context,input,output} per loaded model; cortex forwards it and stops consulting the operator-declared catalogue limit (which can't track hot-swapped models or live capacity). Operator-set `cost` still flows from the catalogue. neuron: - CandleHarness gains context_limit_cfg (from [harness.candle.context_limit]). - LoadedHandle::derived_limit(): profile + live tightest-card free VRAM (single: query_vram; TP: query_vram_tightest_free_mb) + prefill-rate EMA (bootstrap until first sample) → derive_limit. None for arches without a context profile. No operator clamp here (advertise the honest derived value; the clamp is an enforcement-side backstop). - list_models() fills ModelInfo.limit from derived_limit (was None). - derive_limit treats free_tightest_mb == 0 (unknown/CPU sentinel) as "no VRAM ceiling" instead of collapsing to zero. cortex: - ModelEntry gains `limit`, copied from ModelInfo.limit by the poller. - /v1/models: catalogue `limit` no longer flows (Pass 1 sets None); Pass 2 adopts the neuron's limit, taking the tightest across neurons via tightest_limit(). cost unchanged. - model_limits.rs rewritten: catalogue limit (999999) is ignored; the neuron's ModelEntry.limit is advertised; cost still from catalogue. - All ModelEntry literals updated with the new field. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:10:20 +03:00
rob thijssen	8a636c687f	feat(cortex): per-model limit + cost on /v1/models; remove max_model_len ... Resolves #62. opencode's helexa provider discovers a model's serving budget from /v1/models and uses it to size context, trigger compaction, and show spend with no hand-configuration. Each model entry now carries: - limit { context, input?, output } — operator-declared in models.toml - cost { input, output, cache_read?, cache_write? } — USD per 1M tokens - tool_call / reasoning — runtime-detected by the candle harness and OR-ed in from each serving neuron Composition: the catalogue profile supplies limit/cost (Pass 1); the poller carries the neuron's detected tool_call/reasoning into ModelEntry, which the gateway unions onto the entry (Pass 2); aliases propagate every field (Pass 4). Wire types extend ModelInfo / ModelProfile / CortexModelEntry additively (serde default + skip_serializing_if), so older neurons and clients are unaffected. helexa-bench's ModelInfo constructor and the gateway test fixtures are updated for the new fields. Adds tests/model_limits.rs asserting /v1/models surfaces limit + cost (catalogue) and tool_call + reasoning (runtime), and that max_model_len is gone. Removes max_model_len. It was write-only with no consumer — opencode's source references it nowhere and it is not an OpenAI /v1/models field — and doubly misleading: vLLM's max_model_len means total sequence length, but cortex populated it from NEURON_MAX_PROMPT_TOKENS, a prompt-only cap. The limit{} contract replaces it. The neuron's max_prompt_tokens remains the enforced prompt cap (neuron-side); cortex just stops re-advertising a derived, mis-named copy. Closes #66 — its stale-max_model_len premise is moot once the field is gone. limit/cost are operator-declared (catalogue) per #62's design; auto- deriving the advertised budget from each neuron's reported cap is a tracked follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 09:26:55 +03:00
rob thijssen	04f798ec23	feat(cortex-gateway): enhance error responses with structured data ... fixes #63 Standardize error messages by adding type, code, and param fields to align with OpenAI API format. Updates include: - Structured error envelopes with broad type categorization (invalid_request_error/api_error) - Specific machine-readable codes (model_not_found/service_unavailable) - Null param field as required by OpenAI specification - Consistent error response formatting across handlers, proxy, and routing layers New tests verify correct error envelope structure for various failure scenarios. Co-Authored-By: Helexa (Qwen3.6-27B, 48k context) <noreply@helexa.ai>	2026-06-16 17:51:04 +03:00
rob thijssen	df9c490614	feat(neuron+gateway): keep auto-recovering models visible as recovering (#20) ... During the #17 auto-recovery window (unload → reload, minutes for a large TP model) the model's registry slot is absent, so it vanished from neuron's /models — and cortex, routing by /models presence, answered "model not found on any node" while a direct request to neuron would have correctly said "recovering, retry shortly". neuron: the recovery set becomes a map carrying a devices/capabilities snapshot taken at trigger time (while the registry slot still exists). list_models reports `recovering` for models in the set — both while the poisoned slot is still present and during the reload gap, where the snapshot keeps the model listed. gateway: ModelStatus grows a Recovering variant (parsed from the wire); the router holds the route — new RouteError::ModelRecovering mapped to 503 instead of 404 — and deliberately does not fall through to the catalogue cold-load, which would race a second placement against the in-flight recovery. The evictor already ignores non-Loaded entries. Tests: neuron unit test (recovering model stays listed with snapshot), gateway integration tests (poller parses `recovering`; request gets 503 retry-shortly and the model stays on /v1/models). Closes #20 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 13:42:03 +03:00
rob thijssen	e42e8ee81f	refactor: cortex talks to neurons instead of mistral.rs directly ... Replace NodeConfig (static vram_mb, pinned) with NeuronEndpoint. Hardware discovery and model pinning now come from neuron API and models.toml catalogue respectively. - config.rs: nodes -> neurons, add models_config path - catalogue.rs: ModelProfile with pinned_on, ModelCatalogue - poller.rs: poll neuron GET /models (ModelInfo format) - router.rs: resolve inference endpoint via neuron GET /models/{id}/endpoint - evictor.rs: call neuron POST /models/unload - node.rs: remove vram_mb, pinned fields (come from discovery/catalogue) - All 22 gateway tests updated to mock neuron API - Remove MistralModelsResponse, ModelLifecycleRequest (no longer needed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 14:42:52 +03:00
rob thijssen	1b339b1426	test: add Phase 1 integration tests for basic proxy ... 6 tests proving the scaffold works end-to-end: - chat completion proxied through gateway to mock backend - /health endpoint with healthy node - /v1/models returns seeded model list - 404 for unknown model - 404 when no healthy nodes available - 400 when request body missing model field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-14 19:26:12 +03:00