5 Commits

Author	SHA1	Message	Date
rob thijssen	bc74e0e95f	feat(#47 phase 1a): EntitlementProvider trait + local/static provider ... Stage 1's build seam (#50): the interface auth, metering, and budget enforcement all hang off, with a local/static provider so the A0 amplification fix can land before any upstream clearing house exists. The future helexa-upstream client (#57) is just another impl. - cortex-core::entitlements: Principal {account_id, key_id}, CapWindow (Balance | Rolling{seconds}), Reservation handle, BudgetSnapshot, AuthError/BudgetError, and the async EntitlementProvider trait (resolve / reserve / settle / release / snapshot). BudgetError carries the window semantics so callers pick the #63 code (rate_limit_exceeded + Retry-After vs insufficient_quota) without the provider touching HTTP. - cortex-core::config: [entitlements] section on GatewayConfig (require_auth + [[entitlements.keys]] with account_id, optional key_id, hard_cap, window). Additive + serde(default) — anonymous/uncapped when omitted, so existing setups are unaffected. - cortex-gateway::entitlements_local: LocalEntitlementProvider. Budget math serialized under one Mutex so spent+reserved can never exceed a hard cap under concurrency (the #52 guarantee); rolling windows reset lazily; uncapped keys (no hard_cap) always reserve but still meter. - CortexState gains Arc<dyn EntitlementProvider> + require_auth, built in from_config. Not yet consumed by the request path — auth middleware is 1b (#49), enforcement is 1d (#52). - cortex.example.toml documents the section; test GatewayConfig literals updated for the new field. 6 provider unit tests (resolve, unknown-key, round-trip, balance/rolling over-cap codes, uncapped infra key). Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:00:05 +03:00
rob thijssen	8b2e01a072	feat(#67 phase 4): advertise neuron-computed limit on /models; drop catalogue override ... The neuron now self-derives and advertises limit{context,input,output} per loaded model; cortex forwards it and stops consulting the operator-declared catalogue limit (which can't track hot-swapped models or live capacity). Operator-set `cost` still flows from the catalogue. neuron: - CandleHarness gains context_limit_cfg (from [harness.candle.context_limit]). - LoadedHandle::derived_limit(): profile + live tightest-card free VRAM (single: query_vram; TP: query_vram_tightest_free_mb) + prefill-rate EMA (bootstrap until first sample) → derive_limit. None for arches without a context profile. No operator clamp here (advertise the honest derived value; the clamp is an enforcement-side backstop). - list_models() fills ModelInfo.limit from derived_limit (was None). - derive_limit treats free_tightest_mb == 0 (unknown/CPU sentinel) as "no VRAM ceiling" instead of collapsing to zero. cortex: - ModelEntry gains `limit`, copied from ModelInfo.limit by the poller. - /v1/models: catalogue `limit` no longer flows (Pass 1 sets None); Pass 2 adopts the neuron's limit, taking the tightest across neurons via tightest_limit(). cost unchanged. - model_limits.rs rewritten: catalogue limit (999999) is ignored; the neuron's ModelEntry.limit is advertised; cost still from catalogue. - All ModelEntry literals updated with the new field. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:10:20 +03:00
rob thijssen	8a636c687f	feat(cortex): per-model limit + cost on /v1/models; remove max_model_len ... Resolves #62. opencode's helexa provider discovers a model's serving budget from /v1/models and uses it to size context, trigger compaction, and show spend with no hand-configuration. Each model entry now carries: - limit { context, input?, output } — operator-declared in models.toml - cost { input, output, cache_read?, cache_write? } — USD per 1M tokens - tool_call / reasoning — runtime-detected by the candle harness and OR-ed in from each serving neuron Composition: the catalogue profile supplies limit/cost (Pass 1); the poller carries the neuron's detected tool_call/reasoning into ModelEntry, which the gateway unions onto the entry (Pass 2); aliases propagate every field (Pass 4). Wire types extend ModelInfo / ModelProfile / CortexModelEntry additively (serde default + skip_serializing_if), so older neurons and clients are unaffected. helexa-bench's ModelInfo constructor and the gateway test fixtures are updated for the new fields. Adds tests/model_limits.rs asserting /v1/models surfaces limit + cost (catalogue) and tool_call + reasoning (runtime), and that max_model_len is gone. Removes max_model_len. It was write-only with no consumer — opencode's source references it nowhere and it is not an OpenAI /v1/models field — and doubly misleading: vLLM's max_model_len means total sequence length, but cortex populated it from NEURON_MAX_PROMPT_TOKENS, a prompt-only cap. The limit{} contract replaces it. The neuron's max_prompt_tokens remains the enforced prompt cap (neuron-side); cortex just stops re-advertising a derived, mis-named copy. Closes #66 — its stale-max_model_len premise is moot once the field is gone. limit/cost are operator-declared (catalogue) per #62's design; auto- deriving the advertised budget from each neuron's reported cap is a tracked follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 09:26:55 +03:00
rob thijssen	4972c7d1e7	feat(cortex-gateway): C3 — propagate vision capabilities through /v1/models ... ModelEntry and CortexModelEntry gain a `capabilities: Vec<String>` field (serde-default for back-compat). The poller copies it verbatim from each neuron's ModelInfo.capabilities; list_models computes the union across every node where a model is loaded so a checkpoint loaded text-only on one neuron and text+vision on another reports both to the fleet. Catalogue-only and mid-prewarm entries default to empty until the catalogue gains a capabilities declaration. Aliases inherit their target's capability union. New gateway test mocks two nodes with differing capability arrays and asserts the unioned /v1/models response. Closes part of #16 (Stage C3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 13:49:54 +03:00
rob thijssen	24e20dcb5c	feat(catalogue,gateway): model aliases (helexa/small, helexa/balanced, helexa/large) ... Operators can now define tier aliases in models.toml: [aliases] "helexa/small" = "Qwen/Qwen3-1.7B" "helexa/balanced" = "Qwen/Qwen3-8B" "helexa/large" = "Qwen/Qwen3.6-27B" A client request for `model: "helexa/small"` is resolved to the concrete model id at routing time. The gateway also rewrites the proxied body's `model` field to the concrete id so neuron sees a name that matches its loaded handle (otherwise the harness rejects the request). Motivated by the finger-in-the-wind benchmark: same "what's the capital of Georgia" probe runs in 2.5s on the 1.7B vs 6.7s on the 27B with identical correctness. Aliases let clients pick a latency tier without hardcoding model ids, and let operators swap targets without changing client code. Changes: * cortex-core: `ModelCatalogue` gains `aliases: HashMap<String, String>` + `resolve_alias(&str) -> &str`. Unit tests cover the basic resolution + TOML round-trip. * cortex-gateway: * `RouteDecision` gains `resolved_model_id: String`. `router::resolve` consumes aliases at entry and threads the concrete id through. * Handlers (chat_completions, completions, anthropic_messages streaming + non-streaming) rewrite the body's `model` field with `rewrite_model_in_body` before proxying, using the resolved id for metrics labels, LRU touch, and the body itself. * `/v1/models` (Pass 4) emits each alias as its own entry mirroring the target's `loaded` flag, feasible_on, and locations — clients browsing the endpoint see both names and can pick either. * `models.toml` declares the three tier aliases; `models.example.toml` documents the section as opt-in. * Integration tests verify: end-to-end alias→concrete request flow, alias surfacing in /v1/models, and no-op fall-through for non-alias model ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 16:10:41 +03:00