feat(sampling): resolve from request, model config, then fallback

A caller that names no sampling parameters — the common case, and what
the DeepSeek Harness does — was generated at a hardcoded
`temperature = 0.7` with `top_p = None`. `None` selects `Sampling::All`,
so that is untruncated sampling across the full ~250k vocabulary, on a
model whose authors published `top_k = 20, top_p = 0.95,
temperature = 1.0` in a `generation_config.json` we never opened.

Choosing "no truncation" for an absent field is not a neutral default.
It is the widest possible one. The neutral default is what the model
shipped with.

Sampling now resolves once from three sources in priority order —
request, then the model's `generation_config.json`, then a built-in
fallback — into a `SamplingParams` threaded whole.

New knobs, first-class on chat completions and responses rather than
falling into `extra` where they were accepted and discarded:

- `top_k` — required by models that publish it; candle's `Sampling`
  already had `TopK` and `TopKThenTopP`, so the combination Qwen3 asks
  for was expressible all along and simply unreachable.
- `seed` — every text path called `unix_subsec_nanos()` unconditionally,
  so a caller asking for reproducible output got fresh randomness and a
  200. The image path already honoured it; text now matches.
- `repetition_penalty` / `repeat_last_n` — previously constants. 1.1
  over 64 tokens is reasonable for chat and poorly suited to a
  24k-token reasoning block, and there was no way to say so.

Anthropic's Messages API carries `top_k` natively, so it is modelled and
mapped through translation instead of being dropped at the hop.

Consolidated while touching it: 8 `LogitsProcessor` construction sites
collapse to one `to_sampling()`, 4 request-extraction sites to one
`requested_sampling()`, and 5 signatures shed a `temperature`/`top_p`/
`seed` trio. Eight independent copies of a default is how two paths drift
apart — the same shape as #252, where two files described one model and
disagreed. It also makes #273 a one-place change rather than an
eight-place one.

Defaults are unchanged for a model without a `generation_config.json`
and for a request that specifies everything, so no existing client's
behaviour moves. A missing or malformed config falls back and logs
rather than failing the load.

Ten unit tests cover the resolution order, each knob selecting its
strategy, greedy beating truncation, seed pinning, and the real
Qwen3.8-27B config parsing with its unmodelled fields.

This does not yet prove the repetition hypothesis in #271 — the A/B was
inconclusive because the pathology did not reproduce on a task small
enough to measure. It makes the hypothesis testable, which it was not
before: there was no way to tune our way out of a sampling problem.

Closes #272. Refs #271.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGhUi88ANnSfvk86DtYc9g

crates/cortex-core/src/anthropic.rs

@@ -20,6 +20,11 @@ pub struct MessagesRequest {
     pub temperature: Option<f64>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub top_p: Option<f64>,
+    /// Anthropic's Messages API carries `top_k` natively, so it maps
+    /// straight onto the OpenAI-side field rather than being dropped in
+    /// translation (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub top_k: Option<usize>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub stream: Option<bool>,
     #[serde(flatten)]

crates/cortex-core/src/openai.rs

@@ -18,6 +18,31 @@ pub struct ChatCompletionRequest {
     pub temperature: Option<f64>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub top_p: Option<f64>,
+    /// Truncate sampling to the `k` most likely tokens (#272).
+    ///
+    /// Not in the OpenAI schema but universal among serving stacks
+    /// (vLLM, SGLang, llama.cpp, Ollama), and required by models that
+    /// publish it — Qwen3 specifies `top_k = 20`. A named field rather
+    /// than `extra`, because a sampling parameter that lands in `extra`
+    /// is accepted and silently discarded.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub top_k: Option<usize>,
+    /// Pin the sampler's RNG for reproducible output (OpenAI core).
+    ///
+    /// Previously ignored on every text path, which called
+    /// `unix_subsec_nanos()` regardless — a caller asking for
+    /// determinism got fresh randomness and a `200`.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub seed: Option<u64>,
+    /// Penalty applied to recently-generated tokens; `1.0` disables.
+    /// Defaults to 1.1 when absent (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub repetition_penalty: Option<f32>,
+    /// How many recent tokens `repetition_penalty` considers. Defaults
+    /// to 64 — fine for chat, short for a long reasoning block, which is
+    /// why it is a knob rather than a constant (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub repeat_last_n: Option<usize>,
     /// The deprecated spelling of the output cap. Still what most
     /// clients send; see [`ChatCompletionRequest::effective_max_tokens`].
     #[serde(skip_serializing_if = "Option::is_none")]

crates/cortex-core/src/responses.rs

@@ -55,6 +55,21 @@ pub struct ResponsesRequest {
     pub temperature: Option<f64>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub top_p: Option<f64>,
+    /// Truncate sampling to the `k` most likely tokens (#272). Not in
+    /// the Responses schema, but a sampling parameter that lands in
+    /// `extra` is accepted and silently discarded, which is the defect
+    /// this closes.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub top_k: Option<usize>,
+    /// Pin the sampler's RNG for reproducible output (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub seed: Option<u64>,
+    /// Penalty on recently-generated tokens; `1.0` disables (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub repetition_penalty: Option<f32>,
+    /// Window `repetition_penalty` considers (#272).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub repeat_last_n: Option<usize>,
     /// Chained-conversation identifier. We don't store responses
     /// server-side yet; if this is `Some`, the handler returns 400.
     #[serde(default, skip_serializing_if = "Option::is_none")]

crates/cortex-core/src/translate.rs

@@ -101,6 +101,13 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
         messages,
         temperature: req.temperature,
         top_p: req.top_p,
+        // Anthropic models top_k too, so it survives the hop (#272).
+        top_k: req.top_k,
+        // Anthropic's Messages API has no seed or penalty knobs; absent
+        // means the model's own generation_config decides.
+        seed: None,
+        repetition_penalty: None,
+        repeat_last_n: None,
         max_tokens: Some(req.max_tokens),
         // Anthropic has one spelling; it maps onto the legacy field.
         max_completion_tokens: None,

crates/neuron/src/harness/candle.rs

@@ -13,7 +13,7 @@ use async_trait::async_trait;
 use candle_core::quantized::gguf_file;
 use candle_core::{DType, Device, Tensor};
 use candle_nn::VarBuilder;
-use candle_transformers::generation::{LogitsProcessor, Sampling};
+use candle_transformers::generation::LogitsProcessor;
 use candle_transformers::models::llama as llama_dense;
 use candle_transformers::models::quantized_llama::ModelWeights as QuantizedLlamaWeights;
 use candle_transformers::models::quantized_qwen3::ModelWeights as QuantizedQwen3Weights;
@@ -568,8 +568,30 @@ impl LoadedImageModel {
 /// A loaded model with its tokenizer, device placement, and architecture-
 /// specific weights. The `arch` is `Arc<Mutex<>>` so the lock guard can be
 /// moved into `spawn_blocking` for synchronous candle forward passes.
+/// Collect the sampling knobs a request expressed (#272).
+///
+/// Everything optional — absent means "no opinion", which defers to the
+/// model's `generation_config.json` and then to the built-in fallback.
+/// Kept in one place so a new knob is added once rather than at each of
+/// the four request-handling paths.
+fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::RequestedSampling {
+    super::sampling::RequestedSampling {
+        temperature: request.temperature,
+        top_p: request.top_p,
+        top_k: request.top_k,
+        seed: request.seed,
+        repeat_penalty: request.repetition_penalty,
+        repeat_last_n: request.repeat_last_n,
+    }
+}
+
 pub struct LoadedModel {
     pub model_id: String,
+    /// What the model published in `generation_config.json` (#272).
+    /// Defaults for any sampling field the request leaves unset — so a
+    /// caller that says nothing gets what the model's authors intended
+    /// rather than an untruncated distribution.
+    pub generation_defaults: super::sampling::ModelGenerationDefaults,
     /// Local (async-side) handle to the model architecture. `Some`
     /// only when the model loaded onto the CPU device (no CUDA
     /// available); the inference path then takes this mutex via
@@ -742,6 +764,8 @@ impl LoadedModel {
 #[cfg(feature = "cuda")]
 pub struct TpLoadedModel {
     pub model_id: String,
+    /// What the model published in `generation_config.json` (#272).
+    pub generation_defaults: super::sampling::ModelGenerationDefaults,
     pub tokenizer: Tokenizer,
     pub devices: Vec<u32>,
     /// One end-to-end gate: the pool's RPC stream to the subprocess
@@ -2815,10 +2839,14 @@ impl CandleHarness {
             // inference closure takes ownership of `prompt_tokens`.
             let prompt_opened_reasoning =
                 prompt_opens_reasoning(&prompt_tokens, loaded.reasoning_tokens.as_ref());
-            let temperature = request.temperature.unwrap_or(0.7);
-            let top_p = request.top_p;
+            // Resolve sampling once: request → the model's
+            // generation_config.json → built-in fallback (#272).
+            let sampling = super::sampling::SamplingParams::resolve(
+                &requested_sampling(&request),
+                &loaded.generation_defaults,
+                unix_subsec_nanos(),
+            );
             let max_new = request.max_tokens.unwrap_or(8192) as usize;
-            let seed = unix_subsec_nanos();
 
             let eos_id = loaded
                 .tokenizer
@@ -2829,8 +2857,10 @@ impl CandleHarness {
             tracing::info!(
                 prompt_len,
                 max_new,
-                temperature,
-                ?top_p,
+                temperature = sampling.temperature,
+                top_p = ?sampling.top_p,
+                top_k = ?sampling.top_k,
+                seed = sampling.seed,
                 ?eos_id,
                 vram_free_mb,
                 vram_total_mb,
@@ -2874,9 +2904,7 @@ impl CandleHarness {
                                 images.clone(),
                                 *image_token_id,
                                 max_new,
-                                temperature,
-                                top_p,
-                                seed,
+                                sampling.clone(),
                                 eos_id,
                             )
                             .await
@@ -2889,9 +2917,7 @@ impl CandleHarness {
                                 loaded.prefix_cache.as_deref(),
                                 loaded.tokenizer.token_to_id("<|im_start|>"),
                                 max_new,
-                                temperature,
-                                top_p,
-                                seed,
+                                sampling.clone(),
                                 eos_id,
                             )
                             .await
@@ -2941,9 +2967,7 @@ impl CandleHarness {
                             loaded_for_cache.prefix_cache.as_deref(),
                             im_start_id,
                             max_new,
-                            temperature,
-                            top_p,
-                            seed,
+                            sampling.clone(),
                             eos_id,
                         )
                     })
@@ -3273,10 +3297,14 @@ impl CandleHarness {
                 None
             };
 
-        let temperature = request.temperature.unwrap_or(0.7);
-        let top_p = request.top_p;
+        // Resolve sampling once: request → the model's
+        // generation_config.json → built-in fallback (#272).
+        let sampling = super::sampling::SamplingParams::resolve(
+            &requested_sampling(&request),
+            &loaded.generation_defaults,
+            unix_subsec_nanos(),
+        );
         let max_new = request.max_tokens.unwrap_or(8192) as usize;
-        let seed = unix_subsec_nanos();
 
         let eos_id = loaded
             .tokenizer
@@ -3333,8 +3361,10 @@ impl CandleHarness {
             tracing::info!(
                 prompt_len,
                 max_new,
-                temperature,
-                ?top_p,
+                temperature = sampling.temperature,
+                top_p = ?sampling.top_p,
+                top_k = ?sampling.top_k,
+                seed = sampling.seed,
                 ?eos_id,
                 vram_free_mb,
                 vram_total_mb,
@@ -3392,9 +3422,7 @@ impl CandleHarness {
                 .submit(super::engine::EngineRequest {
                     prompt_tokens,
                     max_new,
-                    temperature,
-                    top_p,
-                    seed,
+                    sampling: sampling.clone(),
                     eos_id,
                     tool_schemas,
                     tx,
@@ -3423,9 +3451,7 @@ impl CandleHarness {
                             loaded_for_task.prefix_cache.as_deref(),
                             &loaded_for_task.prefill_rate,
                             max_new,
-                            temperature,
-                            top_p,
-                            seed,
+                            sampling.clone(),
                             eos_id,
                             reasoning_tokens_inner,
                             tool_call_tokens_inner,
@@ -3487,9 +3513,7 @@ impl CandleHarness {
                     &prompt_tokens,
                     loaded_for_task.prefix_cache.as_deref(),
                     max_new,
-                    temperature,
-                    top_p,
-                    seed,
+                    sampling.clone(),
                     eos_id,
                     reasoning_tokens_inner.as_ref(),
                     tool_call_tokens_inner.as_ref(),
@@ -4047,8 +4071,17 @@ impl Harness for CandleHarness {
             }
             _ => None,
         };
+        // The model's published sampling, read from the snapshot dir
+        // beside the tokenizer (#272). Absent is fine — it falls back to
+        // the built-in defaults, which is what every model did before.
+        let generation_defaults = super::sampling::ModelGenerationDefaults::load_from_dir(
+            tokenizer_path
+                .parent()
+                .unwrap_or_else(|| std::path::Path::new(".")),
+        );
         let loaded = Arc::new(LoadedModel {
             model_id: spec.model_id.clone(),
+            generation_defaults,
             arch: arch_local,
             tokenizer,
             device,
@@ -4680,8 +4713,16 @@ impl CandleHarness {
             );
         }
 
+        // Same source as the single-GPU path (#272): the model's own
+        // generation_config.json, read from the snapshot dir.
+        let generation_defaults = super::sampling::ModelGenerationDefaults::load_from_dir(
+            tokenizer_path
+                .parent()
+                .unwrap_or_else(|| std::path::Path::new(".")),
+        );
         let tp_loaded = StdArc::new(TpLoadedModel {
             model_id: spec.model_id.clone(),
+            generation_defaults,
             tokenizer,
             devices: devices.clone(),
             pool: StdArc::new(TMutex::new(pool)),
@@ -4991,10 +5032,14 @@ impl CandleHarness {
 
         let prompt_len = prompt_tokens.len();
 
-        let temperature = request.temperature.unwrap_or(0.7);
-        let top_p = request.top_p;
+        // Resolve sampling once: request → the model's
+        // generation_config.json → built-in fallback (#272).
+        let sampling = super::sampling::SamplingParams::resolve(
+            &requested_sampling(&request),
+            &tp.generation_defaults,
+            unix_subsec_nanos(),
+        );
         let max_new = request.max_tokens.unwrap_or(8192) as usize;
-        let seed = unix_subsec_nanos();
 
         let eos_id = tp
             .tokenizer
@@ -5047,8 +5092,10 @@ impl CandleHarness {
             parent: &span,
             prompt_len,
             max_new,
-            temperature,
-            ?top_p,
+            temperature = sampling.temperature,
+            top_p = ?sampling.top_p,
+            top_k = ?sampling.top_k,
+            seed = sampling.seed,
             ?eos_id,
             vram_free_mb,
             vram_total_mb,
@@ -5094,9 +5141,7 @@ impl CandleHarness {
                 .submit(super::engine::EngineRequest {
                     prompt_tokens,
                     max_new,
-                    temperature,
-                    top_p,
-                    seed,
+                    sampling: sampling.clone(),
                     eos_id,
                     tool_schemas,
                     tx,
@@ -5188,15 +5233,7 @@ impl CandleHarness {
                     reused_for_task = reused as u32;
 
                     let mut logits_processor = {
-                        let sampling = if temperature <= 0.0 {
-                            Sampling::ArgMax
-                        } else {
-                            match top_p {
-                                Some(p) => Sampling::TopP { p, temperature },
-                                None => Sampling::All { temperature },
-                            }
-                        };
-                        LogitsProcessor::from_sampling(seed, sampling)
+                        LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling())
                     };
 
                     // Chunked prefill — see `chunked_prefill_tp`. Each
@@ -5732,10 +5769,14 @@ async fn chat_completion_tp_inner(
 
     let prompt_len = prompt_tokens.len();
 
-    let temperature = request.temperature.unwrap_or(0.7);
-    let top_p = request.top_p;
+    // Resolve sampling once: request → the model's
+    // generation_config.json → built-in fallback (#272).
+    let sampling = super::sampling::SamplingParams::resolve(
+        &requested_sampling(&request),
+        &tp.generation_defaults,
+        unix_subsec_nanos(),
+    );
     let max_new = request.max_tokens.unwrap_or(8192) as usize;
-    let seed = unix_subsec_nanos();
 
     let eos_id = tp
         .tokenizer
@@ -5747,8 +5788,10 @@ async fn chat_completion_tp_inner(
         model = %model_id,
         prompt_len,
         max_new,
-        temperature,
-        ?top_p,
+        temperature = sampling.temperature,
+        top_p = ?sampling.top_p,
+        top_k = ?sampling.top_k,
+        seed = sampling.seed,
         ?eos_id,
         vram_free_mb,
         vram_total_mb,
@@ -5808,17 +5851,8 @@ async fn chat_completion_tp_inner(
         "TP chat_completion: kv cache ready"
     );
 
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut generated: Vec<u32> = Vec::new();
     let mut finish_reason = "length".to_string();
@@ -6881,22 +6915,11 @@ async fn run_inference_with_images_via_worker(
     images: Vec<super::device_worker::jobs::ImageInput>,
     image_token_id: u32,
     max_new: usize,
-    temperature: f64,
-    top_p: Option<f64>,
-    seed: u64,
+    sampling: super::sampling::SamplingParams,
     eos_id: Option<u32>,
 ) -> Result<(Vec<u32>, String)> {
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut generated: Vec<u32> = Vec::new();
     let prompt_len = prompt_tokens.len();
@@ -7245,22 +7268,11 @@ async fn run_inference_via_worker(
     prefix_cache: Option<&ModelPrefixCache>,
     im_start_id: Option<u32>,
     max_new: usize,
-    temperature: f64,
-    top_p: Option<f64>,
-    seed: u64,
+    sampling: super::sampling::SamplingParams,
     eos_id: Option<u32>,
 ) -> Result<(Vec<u32>, String)> {
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut generated: Vec<u32> = Vec::new();
     let prompt_len = prompt_tokens.len();
@@ -7372,9 +7384,7 @@ async fn stream_inference_via_worker(
     prefix_cache: Option<&ModelPrefixCache>,
     prefill_rate: &super::context_limit::PrefillRateEma,
     max_new: usize,
-    temperature: f64,
-    top_p: Option<f64>,
-    seed: u64,
+    sampling: super::sampling::SamplingParams,
     eos_id: Option<u32>,
     reasoning_tokens: Option<ReasoningTokenPair>,
     tool_call_tokens: Option<ToolCallTokenPair>,
@@ -7385,17 +7395,8 @@ async fn stream_inference_via_worker(
     // identity would be unsound for vision requests — they bypass the
     // prefix cache entirely (no restore, no snapshot).
     let prefix_cache = if images.is_some() { None } else { prefix_cache };
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut all_tokens: Vec<u32> = Vec::new();
     // Incremental detokenizer. Replaces the old "decode cumulative
@@ -7676,22 +7677,11 @@ fn run_inference(
     prefix_cache: Option<&ModelPrefixCache>,
     im_start_id: Option<u32>,
     max_new: usize,
-    temperature: f64,
-    top_p: Option<f64>,
-    seed: u64,
+    sampling: super::sampling::SamplingParams,
     eos_id: Option<u32>,
 ) -> Result<(Vec<u32>, String)> {
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut generated: Vec<u32> = Vec::new();
 
@@ -7747,26 +7737,15 @@ fn run_inference_streaming(
     prompt_tokens: &[u32],
     prefix_cache: Option<&ModelPrefixCache>,
     max_new: usize,
-    temperature: f64,
-    top_p: Option<f64>,
-    seed: u64,
+    sampling: super::sampling::SamplingParams,
     eos_id: Option<u32>,
     reasoning_tokens: Option<&ReasoningTokenPair>,
     tool_call_tokens: Option<&ToolCallTokenPair>,
     tool_schemas: ToolSchemas,
     tx: &mpsc::Sender<InferenceEvent>,
 ) -> Result<()> {
-    let mut logits_processor = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut logits_processor =
+        { LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling()) };
 
     let mut all_tokens: Vec<u32> = Vec::new();
     // Incremental detokenizer. See `stream_inference_via_worker` for

crates/neuron/src/harness/engine.rs

@@ -406,9 +406,10 @@ async fn op_prefill(
 pub struct EngineRequest {
     pub prompt_tokens: Vec<u32>,
     pub max_new: usize,
-    pub temperature: f64,
-    pub top_p: Option<f64>,
-    pub seed: u64,
+    /// Sampling resolved once from request → model `generation_config`
+    /// → fallback (#272), rather than loose scalars each caller could
+    /// default differently.
+    pub sampling: crate::harness::sampling::SamplingParams,
     pub eos_id: Option<u32>,
     pub tool_schemas: ToolSchemas,
     pub tx: mpsc::Sender<InferenceEvent>,
@@ -774,14 +775,10 @@ async fn prefill_join(
     session: &mut ActiveSession,
     req: EngineRequest,
 ) -> Result<Option<(Slot, u64)>> {
-    use candle_transformers::generation::Sampling;
-
     let EngineRequest {
         prompt_tokens,
         max_new,
-        temperature,
-        top_p,
-        seed,
+        sampling,
         eos_id,
         tool_schemas,
         tx,
@@ -789,17 +786,7 @@ async fn prefill_join(
         span,
     } = req;
 
-    let mut lp = {
-        let sampling = if temperature <= 0.0 {
-            Sampling::ArgMax
-        } else {
-            match top_p {
-                Some(p) => Sampling::TopP { p, temperature },
-                None => Sampling::All { temperature },
-            }
-        };
-        LogitsProcessor::from_sampling(seed, sampling)
-    };
+    let mut lp = LogitsProcessor::from_sampling(sampling.seed, sampling.to_sampling());
 
     let prompt_len = prompt_tokens.len();
     let prefill_start = std::time::Instant::now();
@@ -1188,9 +1175,17 @@ mod tests {
             .submit(EngineRequest {
                 prompt_tokens: prompt,
                 max_new,
-                temperature: 0.0, // greedy — deterministic
-                top_p: None,
-                seed: 0,
+                // Greedy, deterministic: the fixture asserts exact
+                // token sequences.
+                sampling: crate::harness::sampling::SamplingParams::resolve(
+                    &crate::harness::sampling::RequestedSampling {
+                        temperature: Some(0.0),
+                        seed: Some(0),
+                        ..Default::default()
+                    },
+                    &crate::harness::sampling::ModelGenerationDefaults::default(),
+                    0,
+                ),
                 eos_id: None,
                 tool_schemas: ToolSchemas::new(),
                 tx,

crates/neuron/src/harness/mod.rs

@@ -12,6 +12,7 @@ pub mod image;
 pub mod prefix_cache;
 pub mod preflight;
 pub mod preprocess;
+pub mod sampling;
 pub mod speculative;
 pub mod tp;
 

crates/neuron/src/harness/sampling.rs

@@ -0,0 +1,368 @@
+//! Sampling parameter resolution (#272).
+//!
+//! Three sources, in priority order:
+//!
+//! 1. what the **request** asked for,
+//! 2. what the **model** published in `generation_config.json`,
+//! 3. a built-in fallback.
+//!
+//! Before this, (2) did not exist and (3) was a hardcoded
+//! `temperature = 0.7` with `top_p = None` — and `None` selects
+//! `Sampling::All`, meaning no truncation at all. So a caller that
+//! omitted sampling fields (the common case: the DeepSeek Harness sends
+//! only `temperature`) got untruncated sampling across the full ~250k
+//! vocabulary on a model whose authors published `top_k = 20,
+//! top_p = 0.95, temperature = 1.0`.
+//!
+//! Picking "no truncation" for an absent field is not a neutral default,
+//! it is the widest possible one. The neutral default is what the model
+//! shipped with.
+//!
+//! Resolved **once** per request and threaded whole. The eight
+//! `LogitsProcessor` construction sites this replaces could each have
+//! disagreed about a default — the same shape of bug as #252, where two
+//! config files described one model and drifted.
+
+use serde::{Deserialize, Serialize};
+use std::path::Path;
+
+/// Fallback temperature when neither the request nor the model says.
+/// Retained from the previous hardcoded value so a model without a
+/// `generation_config.json` behaves exactly as it did before.
+const FALLBACK_TEMPERATURE: f64 = 0.7;
+
+/// Repetition penalty applied to recently-generated tokens before
+/// sampling. 1.0 disables it; >1.0 makes recently-emitted tokens less
+/// likely. mistral.rs and llama.cpp default to 1.1, which is enough to
+/// stop small quantized models from degenerating into "Wait, no, no..."
+/// loops without distorting normal output.
+pub const DEFAULT_REPEAT_PENALTY: f32 = 1.1;
+
+/// Number of recently-generated tokens fed into the repetition penalty.
+/// Matches the candle quantized-qwen3 example default.
+///
+/// Defensible for chat and poorly suited to a 24k-token reasoning block,
+/// where a derivation repeated 500 tokens later is well outside the
+/// window — which is why it is now a per-request knob rather than a
+/// constant.
+pub const DEFAULT_REPEAT_LAST_N: usize = 64;
+
+/// What a model published in its `generation_config.json`.
+///
+/// Every field optional: the file is a HuggingFace convention, not a
+/// schema, and models populate different subsets of it.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct ModelGenerationDefaults {
+    pub temperature: Option<f64>,
+    pub top_p: Option<f64>,
+    pub top_k: Option<usize>,
+}
+
+impl ModelGenerationDefaults {
+    /// Read `generation_config.json` from a model snapshot directory.
+    ///
+    /// Absent or unparseable is not an error — we fall back exactly as
+    /// before — but it is logged, because silently guessing the model's
+    /// sampling is how the untruncated default went unnoticed.
+    pub fn load_from_dir(dir: &Path) -> Self {
+        let path = dir.join("generation_config.json");
+        let text = match std::fs::read_to_string(&path) {
+            Ok(t) => t,
+            Err(e) => {
+                tracing::debug!(
+                    path = %path.display(),
+                    error = %e,
+                    "sampling: no generation_config.json; using built-in fallbacks"
+                );
+                return Self::default();
+            }
+        };
+        match serde_json::from_str::<Self>(&text) {
+            Ok(d) => {
+                tracing::info!(
+                    path = %path.display(),
+                    temperature = ?d.temperature,
+                    top_p = ?d.top_p,
+                    top_k = ?d.top_k,
+                    "sampling: model generation defaults loaded"
+                );
+                d
+            }
+            Err(e) => {
+                tracing::warn!(
+                    path = %path.display(),
+                    error = %e,
+                    "sampling: generation_config.json did not parse; using built-in fallbacks"
+                );
+                Self::default()
+            }
+        }
+    }
+}
+
+/// What the caller asked for. All optional — absent means "no opinion",
+/// which defers to the model's published value.
+#[derive(Debug, Clone, Default)]
+pub struct RequestedSampling {
+    pub temperature: Option<f64>,
+    pub top_p: Option<f64>,
+    pub top_k: Option<usize>,
+    pub seed: Option<u64>,
+    pub repeat_penalty: Option<f32>,
+    pub repeat_last_n: Option<usize>,
+}
+
+/// Fully-resolved sampling for one request.
+#[derive(Debug, Clone, PartialEq)]
+pub struct SamplingParams {
+    pub temperature: f64,
+    pub top_p: Option<f64>,
+    pub top_k: Option<usize>,
+    pub seed: u64,
+    pub repeat_penalty: f32,
+    pub repeat_last_n: usize,
+}
+
+impl SamplingParams {
+    /// Resolve request over model defaults over fallback.
+    ///
+    /// `random_seed` is the per-request seed used when the caller did
+    /// not pin one; it is passed in rather than generated here so this
+    /// stays a pure function and the "same seed reproduces" property is
+    /// testable without a clock.
+    pub fn resolve(
+        requested: &RequestedSampling,
+        model: &ModelGenerationDefaults,
+        random_seed: u64,
+    ) -> Self {
+        Self {
+            temperature: requested
+                .temperature
+                .or(model.temperature)
+                .unwrap_or(FALLBACK_TEMPERATURE),
+            // No `unwrap_or` here on purpose: `None` still means "do not
+            // truncate", but it is now only reachable when neither the
+            // caller nor the model expressed a preference, rather than
+            // whenever the caller stayed quiet.
+            top_p: requested.top_p.or(model.top_p),
+            top_k: requested.top_k.or(model.top_k),
+            seed: requested.seed.unwrap_or(random_seed),
+            repeat_penalty: requested.repeat_penalty.unwrap_or(DEFAULT_REPEAT_PENALTY),
+            repeat_last_n: requested.repeat_last_n.unwrap_or(DEFAULT_REPEAT_LAST_N),
+        }
+    }
+
+    /// The candle sampling strategy these parameters select.
+    ///
+    /// `temperature <= 0` is greedy regardless of the rest — a caller
+    /// asking for determinism that way should not have it silently
+    /// softened by a `top_k` the model happened to publish.
+    pub fn to_sampling(&self) -> candle_transformers::generation::Sampling {
+        use candle_transformers::generation::Sampling;
+        if self.temperature <= 0.0 {
+            return Sampling::ArgMax;
+        }
+        let temperature = self.temperature;
+        match (self.top_k, self.top_p) {
+            (Some(k), Some(p)) => Sampling::TopKThenTopP { k, p, temperature },
+            (Some(k), None) => Sampling::TopK { k, temperature },
+            (None, Some(p)) => Sampling::TopP { p, temperature },
+            (None, None) => Sampling::All { temperature },
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use candle_transformers::generation::Sampling;
+
+    fn qwen3_defaults() -> ModelGenerationDefaults {
+        ModelGenerationDefaults {
+            temperature: Some(1.0),
+            top_p: Some(0.95),
+            top_k: Some(20),
+        }
+    }
+
+    /// The case that motivated #271: a caller that names no sampling
+    /// parameters must get the model's published ones, not untruncated
+    /// sampling at a hardcoded 0.7.
+    #[test]
+    fn a_silent_request_inherits_the_models_published_sampling() {
+        let p = SamplingParams::resolve(&RequestedSampling::default(), &qwen3_defaults(), 42);
+        assert_eq!(p.temperature, 1.0);
+        assert_eq!(p.top_p, Some(0.95));
+        assert_eq!(p.top_k, Some(20));
+        assert_eq!(
+            p.to_sampling(),
+            Sampling::TopKThenTopP {
+                k: 20,
+                p: 0.95,
+                temperature: 1.0
+            },
+            "the combination Qwen3 asks for must be expressible"
+        );
+    }
+
+    /// The caller still wins. A harness that deliberately sets a low
+    /// temperature must not have it overridden by the model's file.
+    #[test]
+    fn the_request_overrides_the_model() {
+        let requested = RequestedSampling {
+            temperature: Some(0.2),
+            top_k: Some(5),
+            ..Default::default()
+        };
+        let p = SamplingParams::resolve(&requested, &qwen3_defaults(), 42);
+        assert_eq!(p.temperature, 0.2);
+        assert_eq!(p.top_k, Some(5));
+        // Untouched fields still fall through to the model.
+        assert_eq!(p.top_p, Some(0.95));
+    }
+
+    /// A model with no `generation_config.json` must behave exactly as
+    /// it did before this landed, so nothing regresses on the models
+    /// that do not ship one.
+    #[test]
+    fn without_model_defaults_the_old_behaviour_is_preserved() {
+        let p = SamplingParams::resolve(
+            &RequestedSampling::default(),
+            &ModelGenerationDefaults::default(),
+            42,
+        );
+        assert_eq!(p.temperature, FALLBACK_TEMPERATURE);
+        assert_eq!(p.top_p, None);
+        assert_eq!(p.top_k, None);
+        assert_eq!(p.to_sampling(), Sampling::All { temperature: 0.7 });
+    }
+
+    /// A pinned seed is honoured; an absent one takes the per-request
+    /// random value. Reproducibility is the whole point — the text paths
+    /// previously called `unix_subsec_nanos()` unconditionally, so
+    /// `seed` was accepted and discarded.
+    #[test]
+    fn seed_is_pinned_when_given_and_random_otherwise() {
+        let pinned = SamplingParams::resolve(
+            &RequestedSampling {
+                seed: Some(7),
+                ..Default::default()
+            },
+            &ModelGenerationDefaults::default(),
+            42,
+        );
+        assert_eq!(pinned.seed, 7);
+
+        let unpinned = SamplingParams::resolve(
+            &RequestedSampling::default(),
+            &ModelGenerationDefaults::default(),
+            42,
+        );
+        assert_eq!(unpinned.seed, 42);
+    }
+
+    /// Greedy beats every truncation setting: a caller asking for
+    /// `temperature = 0` wants determinism, not a softened version of it.
+    #[test]
+    fn zero_temperature_is_greedy_even_with_model_top_k() {
+        let requested = RequestedSampling {
+            temperature: Some(0.0),
+            ..Default::default()
+        };
+        let p = SamplingParams::resolve(&requested, &qwen3_defaults(), 42);
+        assert_eq!(p.to_sampling(), Sampling::ArgMax);
+    }
+
+    /// Each truncation knob alone selects the matching strategy.
+    #[test]
+    fn single_knobs_select_their_own_strategy() {
+        let only_k = SamplingParams::resolve(
+            &RequestedSampling {
+                top_k: Some(20),
+                ..Default::default()
+            },
+            &ModelGenerationDefaults::default(),
+            1,
+        );
+        assert_eq!(
+            only_k.to_sampling(),
+            Sampling::TopK {
+                k: 20,
+                temperature: 0.7
+            }
+        );
+
+        let only_p = SamplingParams::resolve(
+            &RequestedSampling {
+                top_p: Some(0.9),
+                ..Default::default()
+            },
+            &ModelGenerationDefaults::default(),
+            1,
+        );
+        assert_eq!(
+            only_p.to_sampling(),
+            Sampling::TopP {
+                p: 0.9,
+                temperature: 0.7
+            }
+        );
+    }
+
+    /// Repetition penalty and its window are now per-request, defaulting
+    /// to the values that were previously constants.
+    #[test]
+    fn repetition_knobs_default_to_the_former_constants() {
+        let d = SamplingParams::resolve(
+            &RequestedSampling::default(),
+            &ModelGenerationDefaults::default(),
+            1,
+        );
+        assert_eq!(d.repeat_penalty, DEFAULT_REPEAT_PENALTY);
+        assert_eq!(d.repeat_last_n, DEFAULT_REPEAT_LAST_N);
+
+        let tuned = SamplingParams::resolve(
+            &RequestedSampling {
+                repeat_penalty: Some(1.05),
+                repeat_last_n: Some(512),
+                ..Default::default()
+            },
+            &ModelGenerationDefaults::default(),
+            1,
+        );
+        assert_eq!(tuned.repeat_penalty, 1.05);
+        assert_eq!(tuned.repeat_last_n, 512);
+    }
+
+    #[test]
+    fn absent_generation_config_is_not_an_error() {
+        let dir = tempfile::tempdir().expect("tempdir");
+        let d = ModelGenerationDefaults::load_from_dir(dir.path());
+        assert!(d.temperature.is_none() && d.top_p.is_none() && d.top_k.is_none());
+    }
+
+    #[test]
+    fn a_real_generation_config_parses() {
+        let dir = tempfile::tempdir().expect("tempdir");
+        // Verbatim from Qwen/Qwen3.8-27B, extra fields included — the
+        // reader must tolerate the ones it does not model.
+        std::fs::write(
+            dir.path().join("generation_config.json"),
+            r#"{"bos_token_id":248044,"do_sample":true,"eos_token_id":[248046,248044],
+                "pad_token_id":248044,"temperature":1.0,"top_k":20,"top_p":0.95}"#,
+        )
+        .expect("write");
+        let d = ModelGenerationDefaults::load_from_dir(dir.path());
+        assert_eq!(d.temperature, Some(1.0));
+        assert_eq!(d.top_p, Some(0.95));
+        assert_eq!(d.top_k, Some(20));
+    }
+
+    #[test]
+    fn malformed_generation_config_falls_back_rather_than_failing() {
+        let dir = tempfile::tempdir().expect("tempdir");
+        std::fs::write(dir.path().join("generation_config.json"), "{not json").expect("write");
+        let d = ModelGenerationDefaults::load_from_dir(dir.path());
+        assert!(d.temperature.is_none());
+    }
+}

crates/neuron/src/wire/openai_responses.rs

@@ -156,6 +156,10 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
         messages,
         temperature: req.temperature,
         top_p: req.top_p,
+        top_k: req.top_k,
+        seed: req.seed,
+        repetition_penalty: req.repetition_penalty,
+        repeat_last_n: req.repeat_last_n,
         max_tokens: req.max_output_tokens,
         // Responses spells the cap `max_output_tokens`, already
         // resolved onto the legacy field above.
@@ -1008,6 +1012,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1030,6 +1038,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1053,6 +1065,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: Some("resp_prev".into()),
             extra: Value::Object(Default::default()),
         };
@@ -1085,6 +1101,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1115,6 +1135,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1160,6 +1184,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1197,6 +1225,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };
@@ -1223,6 +1255,10 @@ mod tests {
             max_output_tokens: None,
             temperature: None,
             top_p: None,
+            top_k: None,
+            seed: None,
+            repetition_penalty: None,
+            repeat_last_n: None,
             previous_response_id: None,
             extra: Value::Object(Default::default()),
         };