rob thijssen
87d9c291ce
All checks were successful
CI / Format (push) Successful in 44s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m15s
CI / Test (push) Successful in 4m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The cost code path already exists (cortex list_models populates cost: profile.cost from the catalogue; aliases inherit it), so opencode's $0.00 is a config gap (no cost in the live models.toml), not missing plumbing. What was missing is the *contract*: units pinned against a wire test, and a defined meaning for "free". - Document ModelCost as the load-bearing source of truth: USD per 1,000,000 tokens as JSON numbers (models.dev/opencode shape) — NOT per-token, NOT decimal strings (OpenRouter's pricing shape, which helexa deliberately does not emit). Define the absent-vs-zero distinction: cost omitted = "not priced / unknown"; cost present with 0.0 = "intentionally free". Note the advertised rate must equal what metering (#51) / reconciliation (#58/#59) bill against — today both read this catalogue value. - New wire test (model_cost.rs): a priced model with cache tiers flows through as per-million numbers; an explicit-0.0 free model keeps its cost block with cache tiers omitted; an unpriced model omits `cost` entirely. - models.example.toml: document cost.* in the field reference and show all three cases (priced-free explicit 0.0 vs the unpriced Qwen3-8B with no cost block). Decisions recorded on #68: source of truth = operator models.toml for now (marketplace clearing house #59 later, same value); no OpenRouter-style `pricing` (opencode/models.dev alignment is sufficient); end-to-end non-zero $ spent needs operators to populate cost in the live catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5aNfNzS2fSZ5wnMeSQ9Wg
113 lines
4.9 KiB
TOML
113 lines
4.9 KiB
TOML
# models.example.toml — model catalogue
|
||
#
|
||
# Copy to /etc/cortex/models.toml and adjust for your environment.
|
||
# Describes how to serve each model. Cortex matches these profiles
|
||
# against discovered neuron topologies for placement decisions; the
|
||
# resulting `(catalogue × topology)` set is what `GET /v1/models`
|
||
# returns and what the router can cold-load on demand.
|
||
#
|
||
# Field reference:
|
||
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.6-27B").
|
||
# Exact match.
|
||
# harness - which engine handles inference (currently "candle").
|
||
# quant - GGUF quantisation tag for the file in the HF repo
|
||
# (e.g. "Q4_K_M"). Omit/empty for the dense
|
||
# safetensors path. TP requires dense.
|
||
# vram_mb - rough estimate; advisory only, not enforced.
|
||
# min_devices - GPU count this profile needs. TP profiles use
|
||
# the same value as the tensor-parallel size.
|
||
# min_device_vram_mb - each device must meet this VRAM floor for the
|
||
# neuron to be considered "feasible".
|
||
# pinned_on - optional whitelist of neuron names. Non-empty
|
||
# narrows feasibility to just those neurons and
|
||
# protects the model from LRU eviction there.
|
||
# source - optional source scheme ("huggingface", "helexa",
|
||
# operator mirror tag). When set, cortex forwards
|
||
# the load to neuron as `scheme:id` so the daemon
|
||
# fetches from the right registry. Omit to let
|
||
# neuron substitute its own `default_source`.
|
||
# cost.* - optional operator-set pricing, surfaced verbatim on
|
||
# GET /v1/models for clients (opencode) to display
|
||
# spend. USD per 1,000,000 tokens, as numbers:
|
||
# cost.input prompt tokens
|
||
# cost.output completion tokens
|
||
# cost.cache_read cache-hit tokens (optional tier)
|
||
# cost.cache_write cache-write tokens (optional tier)
|
||
# Absent vs zero is intentional (#68): OMIT the whole
|
||
# cost block to mean "price not declared / unknown";
|
||
# set cost.input/output = 0.0 to mean "intentionally
|
||
# free" (self-hosted). The advertised rate must match
|
||
# what metering bills against.
|
||
|
||
# Tensor-parallel target — needs a neuron with at least 2 large GPUs.
|
||
# The example pins to a specific neuron name; adjust or remove the
|
||
# pinned_on entry for your own fleet.
|
||
[[models]]
|
||
id = "Qwen/Qwen3.6-27B"
|
||
harness = "candle"
|
||
vram_mb = 54000
|
||
min_devices = 2
|
||
min_device_vram_mb = 24000
|
||
pinned_on = ["your-multi-gpu-neuron"]
|
||
# Token budget: context wall, compaction trigger (input headroom), max output.
|
||
limit.context = 32768
|
||
limit.input = 28672
|
||
limit.output = 4096
|
||
# Pricing in USD per 1M tokens. Explicit 0.0 = intentionally free
|
||
# (self-hosted) — distinct from omitting `cost`, which means "not priced".
|
||
cost.input = 0.0
|
||
cost.output = 0.0
|
||
# Static capability hints (unioned with runtime-detected flags).
|
||
capabilities = ["text", "reasoning"]
|
||
|
||
# Mid-size dense model — fits on any single GPU with ≥16 GB VRAM.
|
||
# No `cost` block here: this model is "not priced" — /v1/models omits the
|
||
# `cost` key for it, so opencode shows spend as unknown rather than $0.
|
||
[[models]]
|
||
id = "Qwen/Qwen3-8B"
|
||
harness = "candle"
|
||
vram_mb = 18000
|
||
min_devices = 1
|
||
min_device_vram_mb = 16000
|
||
limit.context = 16384
|
||
limit.output = 4096
|
||
|
||
# Small GGUF quantised — runs on any small GPU.
|
||
[[models]]
|
||
id = "unsloth/Qwen3-0.6B-GGUF"
|
||
harness = "candle"
|
||
quant = "Q4_K_M"
|
||
vram_mb = 500
|
||
min_devices = 1
|
||
min_device_vram_mb = 4000
|
||
limit.context = 8192
|
||
limit.output = 2048
|
||
|
||
# Helexa registry model — `source` pins this entry to the helexa
|
||
# scheme so cortex forwards `helexa:Helexa/Qwen3.6-27B-Uncensored` to
|
||
# neuron's /models/load. Requires the neuron config to declare a
|
||
# matching [harness.candle.sources.helexa] entry pointing at the
|
||
# helexa registry endpoint (see neuron.example.toml).
|
||
#
|
||
# [[models]]
|
||
# id = "Helexa/Qwen3.6-27B-Uncensored"
|
||
# harness = "candle"
|
||
# source = "helexa"
|
||
# vram_mb = 54000
|
||
# min_devices = 2
|
||
# min_device_vram_mb = 24000
|
||
|
||
# -- Tier aliases ------------------------------------------------------------
|
||
# Optional. Clients can request inference against an alias (e.g.
|
||
# `model: "helexa/small"` in /v1/chat/completions) and cortex
|
||
# transparently routes to the concrete model id below — including
|
||
# rewriting the body's model field so neuron sees a name that matches
|
||
# its loaded handle. Both the alias and the target appear in
|
||
# /v1/models so clients can discover either. Operators can swap
|
||
# targets here without changing client code.
|
||
#
|
||
# [aliases]
|
||
# "helexa/small" = "Qwen/Qwen3-1.7B"
|
||
# "helexa/balanced" = "Qwen/Qwen3-8B"
|
||
# "helexa/large" = "Qwen/Qwen3.6-27B"
|