grenade
821ca116aa
A model's `generation_config.json` is its authors' statement of intent, and honouring it (#272) is right by default. It is not always right for the workload being served: `Qwen/Qwen3.8-27B` publishes `temperature = 1.0`, which measured a 20% structural-defect rate on ~2k-token code generation against 0/60 at `<= 0.6` — Fisher exact one-sided p = 0.0031, n = 20 per arm, truncation excluded by `finish_reason` (#283). Extrapolated at constant hazard that is a ~79% chance of at least one defect in a 15k-token file, which is why the model had never once finished a task two cloud models did in ten minutes. The failures are structural, not semantic — an `else` with no `if`, a duplicate `export function` — the sampling-tail signature, not a capability gap. Adds `SamplingOverride { temperature, top_p, top_k }` to `ModelSpec` (which doubles as the `[[default_models]]` shape in neuron.toml) and to `ModelProfile` (models.toml), so the override travels with the *model* rather than the request. neuron is the only place sampling is resolved, so it has to reach there whether the load came from cortex's catalogue or the host's own config. Precedence is request > operator > model > fallback. The operator sits below the request deliberately: an explicit API parameter that is silently ignored is a contract break, and the observed failure came from callers naming nothing, so a default-tier override is sufficient. Mechanically the operator tier is folded into the model tier once at load by `with_operator_override`, so the per-request path stays a two-tier lookup and none of the five resolve sites can forget it — the same reasoning that made #272 resolve sampling once instead of at eight `LogitsProcessor` construction sites. Fields overlay independently, so correcting `temperature` does not blank `top_p`/`top_k` back to "no truncation" (the #271 failure mode). Deliberately not a heuristic. neuron will never inspect a model and guess a better temperature; a guesser is a footgun, a number in a config file is not. Two files now describe one model, which is exactly the #252 hazard, so the gate lands with the feature rather than after the next incident: - `script/check-config-consistency.py` fails when models.toml and asset/neuron/<host>.toml disagree on `quant` or `sampling` for a model they both name. It catches #252 retroactively. - CI gains a `config` job, ungated by the change classifier — a TOML-only push is neither `rust` nor `web`, and that is precisely the push that introduces drift. - deploy.yml gains a `config-check` gate ahead of both rollouts, and now actually *ships* the configs: models.toml to the gateway and asset/neuron/<host>.toml to each neuron, digest-compared, with a change restarting the service even when no new RPM landed. The sudoers grants for both rsync targets already existed and were simply unused. models.toml is untracked no longer. It carries no secrets — cortex.toml holds the API keys — and CI cannot gate a file it cannot see, which is how one half of the #252 pair stayed invisible. Refs #283, #272, #271, #252
210 lines
10 KiB
TOML
210 lines
10 KiB
TOML
# models.example.toml — model catalogue
|
||
#
|
||
# Copy to /etc/cortex/models.toml and adjust for your environment.
|
||
# Describes how to serve each model. Cortex matches these profiles
|
||
# against discovered neuron topologies for placement decisions; the
|
||
# resulting `(catalogue × topology)` set is what `GET /v1/models`
|
||
# returns and what the router can cold-load on demand.
|
||
#
|
||
# Field reference:
|
||
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.6-27B").
|
||
# Exact match.
|
||
# harness - which engine handles inference (currently "candle").
|
||
# quant - quantisation to serve this model with. Either a
|
||
# GGUF tag naming a file in the repo (e.g.
|
||
# "Q4_K_M"), or an in-situ tag applied to dense
|
||
# safetensors at load (e.g. "q6k", "q4k", "q8_0")
|
||
# — the latter is what TP models use, since TP
|
||
# requires dense weights.
|
||
#
|
||
# SET THIS whenever the model is served
|
||
# quantised, even if a neuron's `default_models`
|
||
# already says so. The two are separate paths to
|
||
# the same model: activation reads neuron.toml,
|
||
# cold-load reads this file. Omitting it here
|
||
# means a cold-load silently loads full-precision
|
||
# weights, which for a large model can leave too
|
||
# little VRAM to prefill — it reports `loaded`
|
||
# and then fails every request (#252).
|
||
#
|
||
# Omit only when full precision is genuinely what
|
||
# you want served.
|
||
# sampling - operator sampling override (#283). Optional
|
||
# sub-table `[models.sampling]` with any of
|
||
# `temperature`, `top_p`, `top_k`.
|
||
#
|
||
# Precedence is request > operator > model's
|
||
# generation_config.json > built-in fallback: this
|
||
# replaces what the model published, but a caller
|
||
# that names a value still wins.
|
||
#
|
||
# Use when a model's own published sampling is
|
||
# wrong for the workload you serve. Qwen3.8-27B
|
||
# publishes temperature = 1.0, which measured a
|
||
# 20% structural-defect rate on ~2k-token code
|
||
# generation against 0/60 at <= 0.6 (Fisher exact
|
||
# one-sided p = 0.0031, #283).
|
||
#
|
||
# Like `quant`, this must MATCH the same model's
|
||
# `[default_models.sampling]` in the host's
|
||
# neuron.toml — the same two-paths-to-one-model
|
||
# hazard that made #252 possible.
|
||
# `script/check-config-consistency.py` gates it.
|
||
# vram_mb - rough estimate; advisory only, not enforced.
|
||
# min_devices - GPU count this profile needs. TP profiles use
|
||
# the same value as the tensor-parallel size.
|
||
# min_device_vram_mb - each device must meet this VRAM floor for the
|
||
# neuron to be considered "feasible".
|
||
# pinned_on - optional whitelist of neuron names. Non-empty
|
||
# narrows feasibility to just those neurons.
|
||
# Affinity only: it says where the model may run,
|
||
# not whether it may be evicted once there.
|
||
# residency_priority - which residency CLASS this model belongs to
|
||
# when a node runs out of VRAM. A model may
|
||
# displace residents of its own class or below,
|
||
# and none above it. Default 100.
|
||
#
|
||
# Equal rank permits displacement, in BOTH
|
||
# directions — that is what a cold-swap is. Two
|
||
# models that share a node and take turns on it
|
||
# must each be able to evict the other, so give
|
||
# peers the SAME number. Giving them different
|
||
# numbers makes the swap one-directional: the
|
||
# higher takes the node and the lower never gets
|
||
# it back, silently.
|
||
#
|
||
# Protect a model by ranking it ABOVE whatever
|
||
# must not evict it — not by giving it a number
|
||
# nobody else has.
|
||
#
|
||
# This is what lets a fleet say "the image model
|
||
# may take the mid tier's node, but never the
|
||
# flagship's, while the frontier tier may take
|
||
# the flagship's" — three rules that a single
|
||
# pinned/unpinned flag cannot express, because it
|
||
# can only protect a model from everything or
|
||
# from nothing.
|
||
#
|
||
# Priority decides who MAY be displaced, never
|
||
# whether a displacement is NEEDED: a node with
|
||
# room for both evicts nothing, however the two
|
||
# rank.
|
||
#
|
||
# A profile with pinned_on and no explicit
|
||
# priority defaults to 1000, preserving the
|
||
# immunity pinned_on used to imply on its own.
|
||
#
|
||
# See https://helexa.ai/docs/operating/placement for
|
||
# worked examples.
|
||
# source - optional source scheme ("huggingface", "helexa",
|
||
# operator mirror tag). When set, cortex forwards
|
||
# the load to neuron as `scheme:id` so the daemon
|
||
# fetches from the right registry. Omit to let
|
||
# neuron substitute its own `default_source`.
|
||
# cost.* - optional operator-set pricing, surfaced verbatim on
|
||
# GET /v1/models for clients (opencode) to display
|
||
# spend. USD per 1,000,000 tokens, as numbers:
|
||
# cost.input prompt tokens
|
||
# cost.output completion tokens
|
||
# cost.cache_read cache-hit tokens (optional tier)
|
||
# cost.cache_write cache-write tokens (optional tier)
|
||
# Absent vs zero is intentional (#68): OMIT the whole
|
||
# cost block to mean "price not declared / unknown";
|
||
# set cost.input/output = 0.0 to mean "intentionally
|
||
# free" (self-hosted). The advertised rate must match
|
||
# what metering bills against.
|
||
|
||
# Tensor-parallel target — needs a neuron with at least 2 large GPUs.
|
||
# The example pins to a specific neuron name; adjust or remove the
|
||
# pinned_on entry for your own fleet.
|
||
[[models]]
|
||
id = "Qwen/Qwen3.6-27B"
|
||
harness = "candle"
|
||
vram_mb = 54000
|
||
min_devices = 2
|
||
min_device_vram_mb = 24000
|
||
pinned_on = ["your-multi-gpu-neuron"]
|
||
# Above the everyday models, so none of them can cold-swap the flagship
|
||
# out. Give anything that SHOULD be able to take this neuron — a second
|
||
# large model you alternate with — the same 300, not a higher number.
|
||
residency_priority = 300
|
||
# Token budget: context wall, compaction trigger (input headroom), max output.
|
||
limit.context = 32768
|
||
limit.input = 28672
|
||
limit.output = 4096
|
||
# Pricing in USD per 1M tokens. Explicit 0.0 = intentionally free
|
||
# (self-hosted) — distinct from omitting `cost`, which means "not priced".
|
||
cost.input = 0.0
|
||
cost.output = 0.0
|
||
# Optional, and usually unnecessary (#241). Neurons derive a model's
|
||
# modalities from their local cache — `model_index.json` at a repo root
|
||
# means image generation, a `vision_config` in `config.json` means
|
||
# vision — so `/v1/models` reports them whether or not anything is
|
||
# loaded. Discovery wins where the two disagree, because it is read from
|
||
# the weights that would actually be served; this field is a fallback
|
||
# for a model no node has downloaded yet.
|
||
capabilities = ["text", "reasoning"]
|
||
|
||
# Mid-size dense model — fits on any single GPU with ≥16 GB VRAM.
|
||
# No `cost` block here: this model is "not priced" — /v1/models omits the
|
||
# `cost` key for it, so opencode shows spend as unknown rather than $0.
|
||
[[models]]
|
||
id = "Qwen/Qwen3-8B"
|
||
harness = "candle"
|
||
vram_mb = 18000
|
||
min_devices = 1
|
||
min_device_vram_mb = 16000
|
||
limit.context = 16384
|
||
limit.output = 4096
|
||
|
||
# Small GGUF quantised — runs on any small GPU.
|
||
[[models]]
|
||
id = "unsloth/Qwen3-0.6B-GGUF"
|
||
harness = "candle"
|
||
quant = "Q4_K_M"
|
||
vram_mb = 500
|
||
min_devices = 1
|
||
min_device_vram_mb = 4000
|
||
limit.context = 8192
|
||
limit.output = 2048
|
||
|
||
# Helexa registry model — `source` pins this entry to the helexa
|
||
# scheme so cortex forwards `helexa:Helexa/Qwen3.6-27B-Uncensored` to
|
||
# neuron's /models/load. Requires the neuron config to declare a
|
||
# matching [harness.candle.sources.helexa] entry pointing at the
|
||
# helexa registry endpoint (see neuron.example.toml).
|
||
#
|
||
# [[models]]
|
||
# id = "Helexa/Qwen3.6-27B-Uncensored"
|
||
# harness = "candle"
|
||
# source = "helexa"
|
||
# vram_mb = 54000
|
||
# min_devices = 2
|
||
# min_device_vram_mb = 24000
|
||
|
||
# -- Tier aliases ------------------------------------------------------------
|
||
# Optional. Clients can request inference against an alias (e.g.
|
||
# `model: "helexa/small"` in /v1/chat/completions) and cortex
|
||
# transparently routes to the concrete model id below — including
|
||
# rewriting the body's model field so neuron sees a name that matches
|
||
# its loaded handle. Both the alias and the target appear in
|
||
# /v1/models so clients can discover either. Operators can swap
|
||
# targets here without changing client code.
|
||
#
|
||
# [aliases]
|
||
# "helexa/small" = "Qwen/Qwen3-1.7B"
|
||
# "helexa/balanced" = "Qwen/Qwen3-8B"
|
||
# "helexa/large" = "Qwen/Qwen3.6-27B"
|
||
|
||
# Text-to-image (#197): Z-Image-Turbo (Apache 2.0). 6B S3-DiT + Qwen3-4B
|
||
# text encoder (host CPU by default; see [harness.candle.image]) + VAE.
|
||
# BF16 DiT+VAE resident ~= 14.6 GB -> any 20 GB+ card. Cold-loads on the
|
||
# first /v1/images/generations request; metered in megapixel-steps, not
|
||
# tokens.
|
||
[[models]]
|
||
id = "Tongyi-MAI/Z-Image-Turbo"
|
||
harness = "candle"
|
||
vram_mb = 15000
|
||
min_devices = 1
|
||
min_device_vram_mb = 20000
|