Files
helexa/neuron.example.toml
T
grenade 821ca116aa feat(sampling): let the operator correct a model's published temperature
A model's `generation_config.json` is its authors' statement of intent,
and honouring it (#272) is right by default. It is not always right for
the workload being served: `Qwen/Qwen3.8-27B` publishes
`temperature = 1.0`, which measured a 20% structural-defect rate on
~2k-token code generation against 0/60 at `<= 0.6` — Fisher exact
one-sided p = 0.0031, n = 20 per arm, truncation excluded by
`finish_reason` (#283).

Extrapolated at constant hazard that is a ~79% chance of at least one
defect in a 15k-token file, which is why the model had never once
finished a task two cloud models did in ten minutes. The failures are
structural, not semantic — an `else` with no `if`, a duplicate `export
function` — the sampling-tail signature, not a capability gap.

Adds `SamplingOverride { temperature, top_p, top_k }` to `ModelSpec`
(which doubles as the `[[default_models]]` shape in neuron.toml) and to
`ModelProfile` (models.toml), so the override travels with the *model*
rather than the request. neuron is the only place sampling is resolved,
so it has to reach there whether the load came from cortex's catalogue
or the host's own config.

Precedence is request > operator > model > fallback. The operator sits
below the request deliberately: an explicit API parameter that is
silently ignored is a contract break, and the observed failure came from
callers naming nothing, so a default-tier override is sufficient.

Mechanically the operator tier is folded into the model tier once at
load by `with_operator_override`, so the per-request path stays a
two-tier lookup and none of the five resolve sites can forget it — the
same reasoning that made #272 resolve sampling once instead of at eight
`LogitsProcessor` construction sites. Fields overlay independently, so
correcting `temperature` does not blank `top_p`/`top_k` back to "no
truncation" (the #271 failure mode).

Deliberately not a heuristic. neuron will never inspect a model and
guess a better temperature; a guesser is a footgun, a number in a config
file is not.

Two files now describe one model, which is exactly the #252 hazard, so
the gate lands with the feature rather than after the next incident:

- `script/check-config-consistency.py` fails when models.toml and
  asset/neuron/<host>.toml disagree on `quant` or `sampling` for a model
  they both name. It catches #252 retroactively.
- CI gains a `config` job, ungated by the change classifier — a
  TOML-only push is neither `rust` nor `web`, and that is precisely the
  push that introduces drift.
- deploy.yml gains a `config-check` gate ahead of both rollouts, and now
  actually *ships* the configs: models.toml to the gateway and
  asset/neuron/<host>.toml to each neuron, digest-compared, with a
  change restarting the service even when no new RPM landed. The sudoers
  grants for both rsync targets already existed and were simply unused.

models.toml is untracked no longer. It carries no secrets — cortex.toml
holds the API keys — and CI cannot gate a file it cannot see, which is
how one half of the #252 pair stayed invisible.

Refs #283, #272, #271, #252
2026-08-25 12:47:56 +03:00

154 lines
6.7 KiB
TOML

# neuron.example.toml — example configuration
#
# Copy to /etc/neuron/neuron.toml and adjust for your environment.
#
# Environment variable overrides use NEURON_ prefix with __ separators:
# NEURON_PORT=13131
port = 13131
# -- Harnesses ---------------------------------------------------------------
# Each [[harnesses]] entry enables an inference engine. Currently only
# "candle" is supported — it runs in-process and uses huggingface/candle
# for inference on local CUDA devices (or CPU when CUDA is unavailable).
[[harnesses]]
name = "candle"
# -- Candle harness settings -------------------------------------------------
# Optional tuning for the candle harness.
[harness.candle]
# HuggingFace cache directory for model weights.
#
# Resolution order (first hit wins):
# 1. `hf_cache` here in this file (applies to the synth `huggingface`
# source only — see [harness.candle.sources.*] below for explicit
# per-source paths).
# 2. `HF_HUB_CACHE` env var — same convention as the Python
# `huggingface_hub` library, so an existing cache directory shared
# with other tooling can be reused without per-tool config.
# 3. `HF_HOME` env var (cache appended as `$HF_HOME/hub`).
# 4. hf-hub's default (`~/.cache/huggingface/hub`).
#
# For per-host overrides (e.g. one neuron has an SSD with prefetched
# weights), prefer a systemd drop-in over editing this file:
# /etc/systemd/system/neuron.service.d/local.conf:
# [Service]
# Environment=HF_HUB_CACHE=/archive/hf-cache
# hf_cache = "/var/lib/neuron/hf-cache"
# Default scheme applied to bare `org/name` model ids (those without a
# `scheme:` prefix). Defaults to "huggingface" when unset. Set to
# "helexa" to make `default_models = [{ model_id = "Helexa/Foo" }]`
# resolve via the helexa registry without prefixing every entry.
# default_source = "huggingface"
# Per-scheme source endpoints. Each scheme maps to an HF-compatible
# registry. The `huggingface` source is auto-synthesised pointing at
# `https://huggingface.co` when omitted; declare it explicitly here to
# override the endpoint, auth env, or cache dir.
#
# [harness.candle.sources.huggingface]
# endpoint = "https://huggingface.co"
# auth_env = "HF_TOKEN" # optional bearer token via env var
# cache_dir = "/archive3/llm-cache/huggingface"
#
# Add helexa (or any operator-run mirror speaking the HF-compatible
# wire format) by adding another sources entry. Caches are
# disambiguated per scheme so a mirror serving the same `org/name` as
# HF cannot collide on disk.
#
# [harness.candle.sources.helexa]
# endpoint = "https://registry.helexa.ai"
# auth_env = "HELEXA_TOKEN"
# cache_dir = "/archive3/llm-cache/helexa"
# -- Prefix KV cache ----------------------------------------------------------
# Reuse cache state across requests when a new prompt starts with the
# exact token sequence of a previous one (chat/agent workloads), so
# prefill only runs on the new suffix. Applies per loaded model, on
# architectures that expose their cache state (qwen3_5). Snapshots
# live in device memory: budget_mb is per loaded model and comes out
# of the same VRAM that serves inference.
#
# [harness.candle.prefix_cache]
# enabled = true
# budget_mb = 1024
# max_entries = 8
# -- Self-derived context limits (#67) ---------------------------------------
# neuron computes the most-efficient limit{context,input,output} that still
# allows coherent agentic performance, from model architecture + live free
# VRAM + a self-measured prefill throughput ceiling, advertises it on
# /models, and enforces it. The advertised limit tracks the resident model
# and rises automatically as efficiency work (e.g. prefix caching, #11)
# frees headroom or speeds prefill — no operator action. These defaults
# rarely need changing; raise target_prefill_latency_secs once prefix
# caching makes long-context re-prefill cheap.
#
# [harness.candle.context_limit]
# enabled = true
# target_prefill_latency_secs = 120.0 # coherence wall (longest prefill/turn)
# bootstrap_prefill_tok_per_sec = 800.0 # cold-start estimate until measured
# activation_headroom_mb = 2048 # per-card prefill activation reserve
# min_free_floor_mb = 1500 # per-card free-VRAM floor to keep
# output_reserve_tokens = 8192 # generation reserve below the wall
# -- Default models ----------------------------------------------------------
# Models listed here are loaded automatically when the neuron service
# activates. Loading is sequential — a slow or failing entry doesn't
# block the rest of the fleet, but it does push out the time before
# neuron starts serving HTTP, so keep the list short. Operators can
# load additional models on demand via POST /models/load.
#
# Make sure data/neuron.service's TimeoutStartSec is generous enough to
# cover the slowest entry's first-time download + materialisation.
# [[default_models]]
# model_id = "Qwen/Qwen3-0.6B-GGUF"
# harness = "candle"
# quant = "Q4_K_M"
# devices = [0]
#
# Operator sampling override (#283) — optional, per model.
#
# By default neuron honours what the model published in its
# `generation_config.json`; that is the model authors' stated intent and
# is right most of the time. It is not always right for the workload you
# serve: Qwen3.8-27B publishes `temperature = 1.0`, which measured a 20%
# structural-defect rate on ~2k-token code generation against 0/60 at
# <= 0.6 (Fisher exact one-sided p = 0.0031).
#
# Precedence is request > operator > model > built-in fallback, so a
# caller that names a temperature still gets what it asked for; this only
# changes what a caller who names nothing receives. Each field overlays
# independently — setting `temperature` alone leaves the model's `top_p`
# and `top_k` in force.
#
# Deliberately explicit. neuron will never inspect a model and guess a
# "better" temperature: a guesser is a footgun, a number you wrote is not.
#
# If cortex can also cold-load this model, the same values must appear in
# its `models.toml` entry — `script/check-config-consistency.py` gates on
# that, because two files describing one model is how #252 happened.
#
# [default_models.sampling]
# temperature = 0.6
# top_p = 0.95
# top_k = 20
# Text-to-image serving (#198). Applies to diffusers-format image
# models (Z-Image family). All fields optional.
#[harness.candle.image]
# Where the ~8 GB Qwen3-4B text encoder runs. "cpu" (default) keeps it
# off the GPU entirely — required on 24 GB cards; "cuda" trades VRAM
# for faster prompt encoding on cards with headroom.
#te_device = "cpu"
# Pin the text encoder between generations. Defaults follow te_device:
# resident on cpu (system RAM), rebuilt per request on cuda (VRAM).
#te_resident = true
# Resolution ceiling per side; oversized requests are rejected before
# admission. Dimensions must be multiples of 16.
#max_dim = 2048