8 Commits

Author	SHA1	Message	Date
rob thijssen	87d9c291ce	fix(#68): pin the /v1/models cost wire contract — units + absent-vs-zero ... The cost code path already exists (cortex list_models populates cost: profile.cost from the catalogue; aliases inherit it), so opencode's $0.00 is a config gap (no cost in the live models.toml), not missing plumbing. What was missing is the *contract*: units pinned against a wire test, and a defined meaning for "free". - Document ModelCost as the load-bearing source of truth: USD per 1,000,000 tokens as JSON numbers (models.dev/opencode shape) — NOT per-token, NOT decimal strings (OpenRouter's pricing shape, which helexa deliberately does not emit). Define the absent-vs-zero distinction: cost omitted = "not priced / unknown"; cost present with 0.0 = "intentionally free". Note the advertised rate must equal what metering (#51) / reconciliation (#58/#59) bill against — today both read this catalogue value. - New wire test (model_cost.rs): a priced model with cache tiers flows through as per-million numbers; an explicit-0.0 free model keeps its cost block with cache tiers omitted; an unpriced model omits `cost` entirely. - models.example.toml: document cost.* in the field reference and show all three cases (priced-free explicit 0.0 vs the unpriced Qwen3-8B with no cost block). Decisions recorded on #68: source of truth = operator models.toml for now (marketplace clearing house #59 later, same value); no OpenRouter-style `pricing` (opencode/models.dev alignment is sufficient); end-to-end non-zero $ spent needs operators to populate cost in the live catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg	2026-06-20 12:02:03 +03:00
rob thijssen	8a636c687f	feat(cortex): per-model limit + cost on /v1/models; remove max_model_len ... Resolves #62. opencode's helexa provider discovers a model's serving budget from /v1/models and uses it to size context, trigger compaction, and show spend with no hand-configuration. Each model entry now carries: - limit { context, input?, output } — operator-declared in models.toml - cost { input, output, cache_read?, cache_write? } — USD per 1M tokens - tool_call / reasoning — runtime-detected by the candle harness and OR-ed in from each serving neuron Composition: the catalogue profile supplies limit/cost (Pass 1); the poller carries the neuron's detected tool_call/reasoning into ModelEntry, which the gateway unions onto the entry (Pass 2); aliases propagate every field (Pass 4). Wire types extend ModelInfo / ModelProfile / CortexModelEntry additively (serde default + skip_serializing_if), so older neurons and clients are unaffected. helexa-bench's ModelInfo constructor and the gateway test fixtures are updated for the new fields. Adds tests/model_limits.rs asserting /v1/models surfaces limit + cost (catalogue) and tool_call + reasoning (runtime), and that max_model_len is gone. Removes max_model_len. It was write-only with no consumer — opencode's source references it nowhere and it is not an OpenAI /v1/models field — and doubly misleading: vLLM's max_model_len means total sequence length, but cortex populated it from NEURON_MAX_PROMPT_TOKENS, a prompt-only cap. The limit{} contract replaces it. The neuron's max_prompt_tokens remains the enforced prompt cap (neuron-side); cortex just stops re-advertising a derived, mis-named copy. Closes #66 — its stale-max_model_len premise is moot once the field is gone. limit/cost are operator-declared (catalogue) per #62's design; auto- deriving the advertised budget from each neuron's reported cap is a tracked follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 09:26:55 +03:00
rob thijssen	d0292ed377	feat(cortex): catalogue source field + scheme-qualified /models/load ... Phase 3 of plan-source-aware-loader-preflight. Adds an optional `source` field to `ModelProfile` and threads it through the router's cold-load path so a profile pointing at the helexa registry forwards `helexa:<id>` to neuron's `/models/load` instead of leaving neuron to substitute its `default_source` (typically `huggingface`). Without this, an operator who declares `source = "helexa"` in models.toml would still see neuron fetch from HuggingFace — the catalogue → ModelSpec translation in `profile_to_spec` was dropping the scheme on the floor. What lands: - `cortex-core::catalogue::ModelProfile.source: Option<String>`. None is the default and preserves pre-Phase-3 behaviour. - `cortex-gateway::router::qualified_model_id(profile)` — small pure helper, extracted from `profile_to_spec` so it can be unit-tested. Empty-string `source` is treated as None so operators who blank out a previously-set value don't trip a scheme-with-no-scheme failure mode in neuron. - `models.example.toml` documents the new field with a commented-out helexa-scheme example pointing back at neuron.example.toml's matching sources block. Tests: - 2 new unit tests in `cortex-core::catalogue`: source-absent round-trip and source-present round-trip through TOML. - 3 new unit tests in `cortex-gateway::router`: pass-through when None, prefix when Some, pass-through on empty-string source. - ModelProfile literal in catalogue's existing test updated to carry `source: None`. CI gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (24 test groups ok, zero failures). Completes Phase 3. With Phases 1+2+3 landed: - neuron parses `scheme:org/name`, routes per-source hf-hub Api with disambiguated cache. - preflight returns structured errors before any device allocation. - cortex catalogue declares per-model source jurisdiction and forwards it to neuron. The registry itself (registry.helexa.ai service, MinIO, nginx, mirror fabric) is the next moving piece — landing under a separate project per the design discussion. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-01 14:53:58 +03:00
rob thijssen	24e20dcb5c	feat(catalogue,gateway): model aliases (helexa/small, helexa/balanced, helexa/large) ... Operators can now define tier aliases in models.toml: [aliases] "helexa/small" = "Qwen/Qwen3-1.7B" "helexa/balanced" = "Qwen/Qwen3-8B" "helexa/large" = "Qwen/Qwen3.6-27B" A client request for `model: "helexa/small"` is resolved to the concrete model id at routing time. The gateway also rewrites the proxied body's `model` field to the concrete id so neuron sees a name that matches its loaded handle (otherwise the harness rejects the request). Motivated by the finger-in-the-wind benchmark: same "what's the capital of Georgia" probe runs in 2.5s on the 1.7B vs 6.7s on the 27B with identical correctness. Aliases let clients pick a latency tier without hardcoding model ids, and let operators swap targets without changing client code. Changes: * cortex-core: `ModelCatalogue` gains `aliases: HashMap<String, String>` + `resolve_alias(&str) -> &str`. Unit tests cover the basic resolution + TOML round-trip. * cortex-gateway: * `RouteDecision` gains `resolved_model_id: String`. `router::resolve` consumes aliases at entry and threads the concrete id through. * Handlers (chat_completions, completions, anthropic_messages streaming + non-streaming) rewrite the body's `model` field with `rewrite_model_in_body` before proxying, using the resolved id for metrics labels, LRU touch, and the body itself. * `/v1/models` (Pass 4) emits each alias as its own entry mirroring the target's `loaded` flag, feasible_on, and locations — clients browsing the endpoint see both names and can pick either. * `models.toml` declares the three tier aliases; `models.example.toml` documents the section as opt-in. * Integration tests verify: end-to-end alias→concrete request flow, alias surfacing in /v1/models, and no-op fall-through for non-alias model ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 16:10:41 +03:00
rob thijssen	62ca125a68	chore: keep models.example.toml generic; deploy.sh sync's local models.toml ... Reverts the previous commit's naming of specific helexa neuron hosts in the shipped example catalogue (`models.example.toml`) — the example is supposed to be a generic starting point that any operator copies and adapts, not a record of one particular fleet's layout. - `pinned_on` in the TP example uses the placeholder `"your-multi-gpu-neuron"`. Other entries keep the model ids (since those are HuggingFace-canonical, not fleet-specific). - New `models.toml` at repo root holds the helexa-fleet catalogue (beast / benjy / quadbrat). Added to `.gitignore` alongside `cortex.toml` — both are operator-owned, gitignored, RPM-marked `%config(noreplace)`, and synced by `deploy.sh`. - `deploy.sh` now rsync's `models.toml` to `/etc/cortex/models.toml` on the gateway host on the same lifecycle as `cortex.toml`. Skips cleanly when no local file exists, so users without a catalogue aren't surprised by silent overwrites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 07:47:08 +03:00
rob thijssen	735945ee81	feat(cortex): unified /v1/models — catalogue × topology feasibility + cold-load ... Realises [project-unified-models-endpoint]: cortex now surfaces every model the operator has provisioned in the catalogue, transparently cold-loads on the first request, and routes the request once the load is done — without per-node configuration or client awareness of which neuron hosts what. cortex-core changes: - NodeState gains `discovery: Option<DiscoveryResponse>` — populated once per neuron on first successful poll, cached forever after (topology is invariant for a neuron process). - ModelProfile gains `is_feasible_on(neuron, devices)` with the pinned_on / min_devices / min_device_vram_mb logic + 5 unit tests. - CortexModelEntry expanded with OpenAI-compatible (`id`, `object`, `created`, `owned_by`) plus helexa-specific extension fields (`loaded`, `feasible_on`, `locations`). cortex-gateway changes: - poller.rs: `maybe_poll_discovery` fetches `GET /discovery` once per neuron and caches on NodeState. - handlers.rs::list_models rewritten as union of (catalogue × topology feasibility) + (currently loaded somewhere). Catalogue-defined models surface even when not yet loaded. - router.rs::resolve gains priority 3 (catalogue cold-load): 1. loaded somewhere → route there 2. unloaded somewhere → route + lazy load via neuron 3. in catalogue → pick feasible neuron, POST /models/load, wait, route. Cache the new entry locally so subsequent requests skip the poll wait. 4. else 404 - pick_feasible_neuron prefers pinned_on neurons, falls back to any feasible one (stable by name). - profile_to_spec translates ModelProfile → ModelSpec, picking devices by VRAM floor and setting tensor_parallel = min_devices for multi- device profiles. - "already loaded" responses from neuron are tolerated (two concurrent requests racing the same cold-load is a benign outcome). models.example.toml rewritten to reflect the canonical helexa fleet (beast = 2x RTX 5090, benjy = RTX 4090, quadbrat = RTX 3060) with a working TP example (Qwen3.6-27B pinned on beast) plus single-GPU profiles for the smaller models. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 07:39:04 +03:00
rob thijssen	3cccc2c56b	refactor(neuron): cut mistralrs/llamacpp, scaffold candle harness ... Stage 1 of the candle-native pivot. Replaces the external-process harness model (mistralrs over HTTP, llamacpp placeholder) with an in-process Harness trait whose sole implementation is candle. The trait keeps its shape so future engines slot in additively, but start/stop default to no-ops and HarnessConfig drops endpoint and systemd_unit since no harness needs external supervision. Behaviour is unchanged on the wire: load_model returns a "not implemented yet (Stage 2)" error and list_models is empty. The gateway-side proxy, poller, and router are untouched. CLAUDE.md Phase 11 (llama.cpp) and Phase 12 (mistral.rs COPR) are marked superseded; the staged plan lives in ~/.claude/plans/create-a-more-aggressive-calm-naur.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 15:53:04 +03:00
rob thijssen	c85d50066e	ci: add RPM packaging for cortex and neuron ... - cortex.spec: gateway binary, cortex.service systemd unit, cortex.toml + models.toml config files - neuron.spec: neuron binary, neuron.service systemd unit, neuron.toml config file - Parallel CI: srpm-cortex and srpm-neuron jobs build SRPMs concurrently, then publish to separate COPR repos (helexa/cortex and helexa/neuron) - Shared cortex user/group across both packages - Example configs: cortex.example.toml, neuron.example.toml, models.example.toml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 16:09:04 +03:00