CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m7s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m39s
CI / CUDA type-check (push) Successful in 19m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for ~50 minutes on 2026-08-27 with 32 GB held on one card, the other empty, `models: []`, and `activation.state: "ready"`. The driver was the router's alias table. The web chat's authenticated default is `helexa/balanced` (Chat.tsx:35), and on gallumbits that alias still resolved to Qwen/Qwen3.6-27B while beast's resident model is the 3.8. The two sat at equal `residency_priority` deliberately, so they displaced each other in both directions — meaning every authenticated web-chat turn evicted the flagship and cold-loaded the 3.6 (~70 s of TP layer loading), and the next flagship request swapped it straight back. At 21:27–21:33 four of those overlapped and left the leader holding accumulated shards with nothing registered. Mutual displacement is not the bug; #99 wants it and the equal ranks were right for the A/B they were set up for (#251). The bug is that the A/B was over and the loser stayed catalogued, so a stale alias pointed real user traffic at a model whose only remaining job was to evict the one people were actually being served. Retires the 3.6 from models.toml and repoints every operator-facing mention at the 3.8: the docs under helexa.ai/content/docs/2-operating/, the example catalogues and router/acp configs, and the helexa-acp README (which told readers to set a default_model that would now 404). Deliberately NOT renamed: doc/benchmarks.md and crates/neuron/tests/fixtures/numerical/ record measurements and a parity snapshot actually taken against the 3.6 — rewriting those would falsify the record. Test fixtures elsewhere keep the old string too; it is an arbitrary model id in those, and the churn would bury the real diff. The live alias on gallumbits was corrected and the router restarted ahead of this commit, since it was the active cause; helexa-router.toml is gitignored, so that file is still hand-synced (#287).
210 lines
10 KiB
TOML
210 lines
10 KiB
TOML
# models.example.toml — model catalogue
|
||
#
|
||
# Copy to /etc/cortex/models.toml and adjust for your environment.
|
||
# Describes how to serve each model. Cortex matches these profiles
|
||
# against discovered neuron topologies for placement decisions; the
|
||
# resulting `(catalogue × topology)` set is what `GET /v1/models`
|
||
# returns and what the router can cold-load on demand.
|
||
#
|
||
# Field reference:
|
||
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.8-27B").
|
||
# Exact match.
|
||
# harness - which engine handles inference (currently "candle").
|
||
# quant - quantisation to serve this model with. Either a
|
||
# GGUF tag naming a file in the repo (e.g.
|
||
# "Q4_K_M"), or an in-situ tag applied to dense
|
||
# safetensors at load (e.g. "q6k", "q4k", "q8_0")
|
||
# — the latter is what TP models use, since TP
|
||
# requires dense weights.
|
||
#
|
||
# SET THIS whenever the model is served
|
||
# quantised, even if a neuron's `default_models`
|
||
# already says so. The two are separate paths to
|
||
# the same model: activation reads neuron.toml,
|
||
# cold-load reads this file. Omitting it here
|
||
# means a cold-load silently loads full-precision
|
||
# weights, which for a large model can leave too
|
||
# little VRAM to prefill — it reports `loaded`
|
||
# and then fails every request (#252).
|
||
#
|
||
# Omit only when full precision is genuinely what
|
||
# you want served.
|
||
# sampling - operator sampling override (#283). Optional
|
||
# sub-table `[models.sampling]` with any of
|
||
# `temperature`, `top_p`, `top_k`.
|
||
#
|
||
# Precedence is request > operator > model's
|
||
# generation_config.json > built-in fallback: this
|
||
# replaces what the model published, but a caller
|
||
# that names a value still wins.
|
||
#
|
||
# Use when a model's own published sampling is
|
||
# wrong for the workload you serve. Qwen3.8-27B
|
||
# publishes temperature = 1.0, which measured a
|
||
# 20% structural-defect rate on ~2k-token code
|
||
# generation against 0/60 at <= 0.6 (Fisher exact
|
||
# one-sided p = 0.0031, #283).
|
||
#
|
||
# Like `quant`, this must MATCH the same model's
|
||
# `[default_models.sampling]` in the host's
|
||
# neuron.toml — the same two-paths-to-one-model
|
||
# hazard that made #252 possible.
|
||
# `script/check-config-consistency.py` gates it.
|
||
# vram_mb - rough estimate; advisory only, not enforced.
|
||
# min_devices - GPU count this profile needs. TP profiles use
|
||
# the same value as the tensor-parallel size.
|
||
# min_device_vram_mb - each device must meet this VRAM floor for the
|
||
# neuron to be considered "feasible".
|
||
# pinned_on - optional whitelist of neuron names. Non-empty
|
||
# narrows feasibility to just those neurons.
|
||
# Affinity only: it says where the model may run,
|
||
# not whether it may be evicted once there.
|
||
# residency_priority - which residency CLASS this model belongs to
|
||
# when a node runs out of VRAM. A model may
|
||
# displace residents of its own class or below,
|
||
# and none above it. Default 100.
|
||
#
|
||
# Equal rank permits displacement, in BOTH
|
||
# directions — that is what a cold-swap is. Two
|
||
# models that share a node and take turns on it
|
||
# must each be able to evict the other, so give
|
||
# peers the SAME number. Giving them different
|
||
# numbers makes the swap one-directional: the
|
||
# higher takes the node and the lower never gets
|
||
# it back, silently.
|
||
#
|
||
# Protect a model by ranking it ABOVE whatever
|
||
# must not evict it — not by giving it a number
|
||
# nobody else has.
|
||
#
|
||
# This is what lets a fleet say "the image model
|
||
# may take the mid tier's node, but never the
|
||
# flagship's, while the frontier tier may take
|
||
# the flagship's" — three rules that a single
|
||
# pinned/unpinned flag cannot express, because it
|
||
# can only protect a model from everything or
|
||
# from nothing.
|
||
#
|
||
# Priority decides who MAY be displaced, never
|
||
# whether a displacement is NEEDED: a node with
|
||
# room for both evicts nothing, however the two
|
||
# rank.
|
||
#
|
||
# A profile with pinned_on and no explicit
|
||
# priority defaults to 1000, preserving the
|
||
# immunity pinned_on used to imply on its own.
|
||
#
|
||
# See https://helexa.ai/docs/operating/placement for
|
||
# worked examples.
|
||
# source - optional source scheme ("huggingface", "helexa",
|
||
# operator mirror tag). When set, cortex forwards
|
||
# the load to neuron as `scheme:id` so the daemon
|
||
# fetches from the right registry. Omit to let
|
||
# neuron substitute its own `default_source`.
|
||
# cost.* - optional operator-set pricing, surfaced verbatim on
|
||
# GET /v1/models for clients (opencode) to display
|
||
# spend. USD per 1,000,000 tokens, as numbers:
|
||
# cost.input prompt tokens
|
||
# cost.output completion tokens
|
||
# cost.cache_read cache-hit tokens (optional tier)
|
||
# cost.cache_write cache-write tokens (optional tier)
|
||
# Absent vs zero is intentional (#68): OMIT the whole
|
||
# cost block to mean "price not declared / unknown";
|
||
# set cost.input/output = 0.0 to mean "intentionally
|
||
# free" (self-hosted). The advertised rate must match
|
||
# what metering bills against.
|
||
|
||
# Tensor-parallel target — needs a neuron with at least 2 large GPUs.
|
||
# The example pins to a specific neuron name; adjust or remove the
|
||
# pinned_on entry for your own fleet.
|
||
[[models]]
|
||
id = "Qwen/Qwen3.8-27B"
|
||
harness = "candle"
|
||
vram_mb = 54000
|
||
min_devices = 2
|
||
min_device_vram_mb = 24000
|
||
pinned_on = ["your-multi-gpu-neuron"]
|
||
# Above the everyday models, so none of them can cold-swap the flagship
|
||
# out. Give anything that SHOULD be able to take this neuron — a second
|
||
# large model you alternate with — the same 300, not a higher number.
|
||
residency_priority = 300
|
||
# Token budget: context wall, compaction trigger (input headroom), max output.
|
||
limit.context = 32768
|
||
limit.input = 28672
|
||
limit.output = 4096
|
||
# Pricing in USD per 1M tokens. Explicit 0.0 = intentionally free
|
||
# (self-hosted) — distinct from omitting `cost`, which means "not priced".
|
||
cost.input = 0.0
|
||
cost.output = 0.0
|
||
# Optional, and usually unnecessary (#241). Neurons derive a model's
|
||
# modalities from their local cache — `model_index.json` at a repo root
|
||
# means image generation, a `vision_config` in `config.json` means
|
||
# vision — so `/v1/models` reports them whether or not anything is
|
||
# loaded. Discovery wins where the two disagree, because it is read from
|
||
# the weights that would actually be served; this field is a fallback
|
||
# for a model no node has downloaded yet.
|
||
capabilities = ["text", "reasoning"]
|
||
|
||
# Mid-size dense model — fits on any single GPU with ≥16 GB VRAM.
|
||
# No `cost` block here: this model is "not priced" — /v1/models omits the
|
||
# `cost` key for it, so opencode shows spend as unknown rather than $0.
|
||
[[models]]
|
||
id = "Qwen/Qwen3-8B"
|
||
harness = "candle"
|
||
vram_mb = 18000
|
||
min_devices = 1
|
||
min_device_vram_mb = 16000
|
||
limit.context = 16384
|
||
limit.output = 4096
|
||
|
||
# Small GGUF quantised — runs on any small GPU.
|
||
[[models]]
|
||
id = "unsloth/Qwen3-0.6B-GGUF"
|
||
harness = "candle"
|
||
quant = "Q4_K_M"
|
||
vram_mb = 500
|
||
min_devices = 1
|
||
min_device_vram_mb = 4000
|
||
limit.context = 8192
|
||
limit.output = 2048
|
||
|
||
# Helexa registry model — `source` pins this entry to the helexa
|
||
# scheme so cortex forwards `helexa:Helexa/Qwen3.8-27B-Uncensored` to
|
||
# neuron's /models/load. Requires the neuron config to declare a
|
||
# matching [harness.candle.sources.helexa] entry pointing at the
|
||
# helexa registry endpoint (see neuron.example.toml).
|
||
#
|
||
# [[models]]
|
||
# id = "Helexa/Qwen3.8-27B-Uncensored"
|
||
# harness = "candle"
|
||
# source = "helexa"
|
||
# vram_mb = 54000
|
||
# min_devices = 2
|
||
# min_device_vram_mb = 24000
|
||
|
||
# -- Tier aliases ------------------------------------------------------------
|
||
# Optional. Clients can request inference against an alias (e.g.
|
||
# `model: "helexa/small"` in /v1/chat/completions) and cortex
|
||
# transparently routes to the concrete model id below — including
|
||
# rewriting the body's model field so neuron sees a name that matches
|
||
# its loaded handle. Both the alias and the target appear in
|
||
# /v1/models so clients can discover either. Operators can swap
|
||
# targets here without changing client code.
|
||
#
|
||
# [aliases]
|
||
# "helexa/small" = "Qwen/Qwen3-1.7B"
|
||
# "helexa/balanced" = "Qwen/Qwen3-8B"
|
||
# "helexa/large" = "Qwen/Qwen3.8-27B"
|
||
|
||
# Text-to-image (#197): Z-Image-Turbo (Apache 2.0). 6B S3-DiT + Qwen3-4B
|
||
# text encoder (host CPU by default; see [harness.candle.image]) + VAE.
|
||
# BF16 DiT+VAE resident ~= 14.6 GB -> any 20 GB+ card. Cold-loads on the
|
||
# first /v1/images/generations request; metered in megapixel-steps, not
|
||
# tokens.
|
||
[[models]]
|
||
id = "Tongyi-MAI/Z-Image-Turbo"
|
||
harness = "candle"
|
||
vram_mb = 15000
|
||
min_devices = 1
|
||
min_device_vram_mb = 20000
|