A model's `generation_config.json` is its authors' statement of intent, and honouring it (#272) is right by default. It is not always right for the workload being served: `Qwen/Qwen3.8-27B` publishes `temperature = 1.0`, which measured a 20% structural-defect rate on ~2k-token code generation against 0/60 at `<= 0.6` — Fisher exact one-sided p = 0.0031, n = 20 per arm, truncation excluded by `finish_reason` (#283). Extrapolated at constant hazard that is a ~79% chance of at least one defect in a 15k-token file, which is why the model had never once finished a task two cloud models did in ten minutes. The failures are structural, not semantic — an `else` with no `if`, a duplicate `export function` — the sampling-tail signature, not a capability gap. Adds `SamplingOverride { temperature, top_p, top_k }` to `ModelSpec` (which doubles as the `[[default_models]]` shape in neuron.toml) and to `ModelProfile` (models.toml), so the override travels with the *model* rather than the request. neuron is the only place sampling is resolved, so it has to reach there whether the load came from cortex's catalogue or the host's own config. Precedence is request > operator > model > fallback. The operator sits below the request deliberately: an explicit API parameter that is silently ignored is a contract break, and the observed failure came from callers naming nothing, so a default-tier override is sufficient. Mechanically the operator tier is folded into the model tier once at load by `with_operator_override`, so the per-request path stays a two-tier lookup and none of the five resolve sites can forget it — the same reasoning that made #272 resolve sampling once instead of at eight `LogitsProcessor` construction sites. Fields overlay independently, so correcting `temperature` does not blank `top_p`/`top_k` back to "no truncation" (the #271 failure mode). Deliberately not a heuristic. neuron will never inspect a model and guess a better temperature; a guesser is a footgun, a number in a config file is not. Two files now describe one model, which is exactly the #252 hazard, so the gate lands with the feature rather than after the next incident: - `script/check-config-consistency.py` fails when models.toml and asset/neuron/<host>.toml disagree on `quant` or `sampling` for a model they both name. It catches #252 retroactively. - CI gains a `config` job, ungated by the change classifier — a TOML-only push is neither `rust` nor `web`, and that is precisely the push that introduces drift. - deploy.yml gains a `config-check` gate ahead of both rollouts, and now actually *ships* the configs: models.toml to the gateway and asset/neuron/<host>.toml to each neuron, digest-compared, with a change restarting the service even when no new RPM landed. The sudoers grants for both rsync targets already existed and were simply unused. models.toml is untracked no longer. It carries no secrets — cortex.toml holds the API keys — and CI cannot gate a file it cannot see, which is how one half of the #252 pair stayed invisible. Refs #283, #272, #271, #252
154 lines
6.7 KiB
TOML
154 lines
6.7 KiB
TOML
# neuron.example.toml — example configuration
|
|
#
|
|
# Copy to /etc/neuron/neuron.toml and adjust for your environment.
|
|
#
|
|
# Environment variable overrides use NEURON_ prefix with __ separators:
|
|
# NEURON_PORT=13131
|
|
|
|
port = 13131
|
|
|
|
# -- Harnesses ---------------------------------------------------------------
|
|
# Each [[harnesses]] entry enables an inference engine. Currently only
|
|
# "candle" is supported — it runs in-process and uses huggingface/candle
|
|
# for inference on local CUDA devices (or CPU when CUDA is unavailable).
|
|
|
|
[[harnesses]]
|
|
name = "candle"
|
|
|
|
# -- Candle harness settings -------------------------------------------------
|
|
# Optional tuning for the candle harness.
|
|
|
|
[harness.candle]
|
|
# HuggingFace cache directory for model weights.
|
|
#
|
|
# Resolution order (first hit wins):
|
|
# 1. `hf_cache` here in this file (applies to the synth `huggingface`
|
|
# source only — see [harness.candle.sources.*] below for explicit
|
|
# per-source paths).
|
|
# 2. `HF_HUB_CACHE` env var — same convention as the Python
|
|
# `huggingface_hub` library, so an existing cache directory shared
|
|
# with other tooling can be reused without per-tool config.
|
|
# 3. `HF_HOME` env var (cache appended as `$HF_HOME/hub`).
|
|
# 4. hf-hub's default (`~/.cache/huggingface/hub`).
|
|
#
|
|
# For per-host overrides (e.g. one neuron has an SSD with prefetched
|
|
# weights), prefer a systemd drop-in over editing this file:
|
|
# /etc/systemd/system/neuron.service.d/local.conf:
|
|
# [Service]
|
|
# Environment=HF_HUB_CACHE=/archive/hf-cache
|
|
# hf_cache = "/var/lib/neuron/hf-cache"
|
|
|
|
# Default scheme applied to bare `org/name` model ids (those without a
|
|
# `scheme:` prefix). Defaults to "huggingface" when unset. Set to
|
|
# "helexa" to make `default_models = [{ model_id = "Helexa/Foo" }]`
|
|
# resolve via the helexa registry without prefixing every entry.
|
|
# default_source = "huggingface"
|
|
|
|
# Per-scheme source endpoints. Each scheme maps to an HF-compatible
|
|
# registry. The `huggingface` source is auto-synthesised pointing at
|
|
# `https://huggingface.co` when omitted; declare it explicitly here to
|
|
# override the endpoint, auth env, or cache dir.
|
|
#
|
|
# [harness.candle.sources.huggingface]
|
|
# endpoint = "https://huggingface.co"
|
|
# auth_env = "HF_TOKEN" # optional bearer token via env var
|
|
# cache_dir = "/archive3/llm-cache/huggingface"
|
|
#
|
|
# Add helexa (or any operator-run mirror speaking the HF-compatible
|
|
# wire format) by adding another sources entry. Caches are
|
|
# disambiguated per scheme so a mirror serving the same `org/name` as
|
|
# HF cannot collide on disk.
|
|
#
|
|
# [harness.candle.sources.helexa]
|
|
# endpoint = "https://registry.helexa.ai"
|
|
# auth_env = "HELEXA_TOKEN"
|
|
# cache_dir = "/archive3/llm-cache/helexa"
|
|
|
|
# -- Prefix KV cache ----------------------------------------------------------
|
|
# Reuse cache state across requests when a new prompt starts with the
|
|
# exact token sequence of a previous one (chat/agent workloads), so
|
|
# prefill only runs on the new suffix. Applies per loaded model, on
|
|
# architectures that expose their cache state (qwen3_5). Snapshots
|
|
# live in device memory: budget_mb is per loaded model and comes out
|
|
# of the same VRAM that serves inference.
|
|
#
|
|
# [harness.candle.prefix_cache]
|
|
# enabled = true
|
|
# budget_mb = 1024
|
|
# max_entries = 8
|
|
|
|
# -- Self-derived context limits (#67) ---------------------------------------
|
|
# neuron computes the most-efficient limit{context,input,output} that still
|
|
# allows coherent agentic performance, from model architecture + live free
|
|
# VRAM + a self-measured prefill throughput ceiling, advertises it on
|
|
# /models, and enforces it. The advertised limit tracks the resident model
|
|
# and rises automatically as efficiency work (e.g. prefix caching, #11)
|
|
# frees headroom or speeds prefill — no operator action. These defaults
|
|
# rarely need changing; raise target_prefill_latency_secs once prefix
|
|
# caching makes long-context re-prefill cheap.
|
|
#
|
|
# [harness.candle.context_limit]
|
|
# enabled = true
|
|
# target_prefill_latency_secs = 120.0 # coherence wall (longest prefill/turn)
|
|
# bootstrap_prefill_tok_per_sec = 800.0 # cold-start estimate until measured
|
|
# activation_headroom_mb = 2048 # per-card prefill activation reserve
|
|
# min_free_floor_mb = 1500 # per-card free-VRAM floor to keep
|
|
# output_reserve_tokens = 8192 # generation reserve below the wall
|
|
|
|
# -- Default models ----------------------------------------------------------
|
|
# Models listed here are loaded automatically when the neuron service
|
|
# activates. Loading is sequential — a slow or failing entry doesn't
|
|
# block the rest of the fleet, but it does push out the time before
|
|
# neuron starts serving HTTP, so keep the list short. Operators can
|
|
# load additional models on demand via POST /models/load.
|
|
#
|
|
# Make sure data/neuron.service's TimeoutStartSec is generous enough to
|
|
# cover the slowest entry's first-time download + materialisation.
|
|
|
|
# [[default_models]]
|
|
# model_id = "Qwen/Qwen3-0.6B-GGUF"
|
|
# harness = "candle"
|
|
# quant = "Q4_K_M"
|
|
# devices = [0]
|
|
#
|
|
# Operator sampling override (#283) — optional, per model.
|
|
#
|
|
# By default neuron honours what the model published in its
|
|
# `generation_config.json`; that is the model authors' stated intent and
|
|
# is right most of the time. It is not always right for the workload you
|
|
# serve: Qwen3.8-27B publishes `temperature = 1.0`, which measured a 20%
|
|
# structural-defect rate on ~2k-token code generation against 0/60 at
|
|
# <= 0.6 (Fisher exact one-sided p = 0.0031).
|
|
#
|
|
# Precedence is request > operator > model > built-in fallback, so a
|
|
# caller that names a temperature still gets what it asked for; this only
|
|
# changes what a caller who names nothing receives. Each field overlays
|
|
# independently — setting `temperature` alone leaves the model's `top_p`
|
|
# and `top_k` in force.
|
|
#
|
|
# Deliberately explicit. neuron will never inspect a model and guess a
|
|
# "better" temperature: a guesser is a footgun, a number you wrote is not.
|
|
#
|
|
# If cortex can also cold-load this model, the same values must appear in
|
|
# its `models.toml` entry — `script/check-config-consistency.py` gates on
|
|
# that, because two files describing one model is how #252 happened.
|
|
#
|
|
# [default_models.sampling]
|
|
# temperature = 0.6
|
|
# top_p = 0.95
|
|
# top_k = 20
|
|
|
|
# Text-to-image serving (#198). Applies to diffusers-format image
|
|
# models (Z-Image family). All fields optional.
|
|
#[harness.candle.image]
|
|
# Where the ~8 GB Qwen3-4B text encoder runs. "cpu" (default) keeps it
|
|
# off the GPU entirely — required on 24 GB cards; "cuda" trades VRAM
|
|
# for faster prompt encoding on cards with headroom.
|
|
#te_device = "cpu"
|
|
# Pin the text encoder between generations. Defaults follow te_device:
|
|
# resident on cpu (system RAM), rebuilt per request on cuda (VRAM).
|
|
#te_resident = true
|
|
# Resolution ceiling per side; oversized requests are rejected before
|
|
# admission. Dimensions must be multiples of 16.
|
|
#max_dim = 2048
|