feat(neuron): strip reasoning from chat completions by default
Some checks failed
CI / CUDA type-check (push) Failing after 18s
build-prerelease / Resolve version stamps (push) Successful in 32s
CI / Format (push) Successful in 32s
CI / Clippy (push) Successful in 2m36s
build-prerelease / Build cortex binary (push) Successful in 4m29s
CI / Test (push) Successful in 5m19s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 5m56s
build-prerelease / Package cortex RPM (push) Successful in 1m21s
build-prerelease / Build neuron-ampere (push) Successful in 7m45s
build-prerelease / Build neuron-ada (push) Successful in 5m24s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 2m53s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 3m0s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m43s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 1m2s
Some checks failed
CI / CUDA type-check (push) Failing after 18s
build-prerelease / Resolve version stamps (push) Successful in 32s
CI / Format (push) Successful in 32s
CI / Clippy (push) Successful in 2m36s
build-prerelease / Build cortex binary (push) Successful in 4m29s
CI / Test (push) Successful in 5m19s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 5m56s
build-prerelease / Package cortex RPM (push) Successful in 1m21s
build-prerelease / Build neuron-ampere (push) Successful in 7m45s
build-prerelease / Build neuron-ada (push) Successful in 5m24s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 2m53s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 3m0s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m43s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 1m2s
Closes #8. Reasoning-capable models (Qwen3, DeepSeek-R1, gpt-oss, Mistral Magistral, …) emit `<think>...</think>` blocks inline in their content stream. The chat-completions wire format has no slot for reasoning, so until this change every consumer either parsed the markers themselves (helexa-acp) or wrote the raw scratchpad content into their UI (Zed's commit-message generator — visible as the leaked reasoning block on every generated commit message against benjy's Qwen3-8B). ## Implementation, model-agnostic by design The neuron side now does token-level routing without any hardcoded model knowledge: 1. **At load time** (`detect_reasoning_token_pair` in `wire::event`), probe the tokenizer's vocabulary for a known reasoning-marker pair: `<think>` / `</think>` (Qwen3, DeepSeek-R1, gpt-oss), `[THINK]` / `[/THINK]` (Mistral Magistral), and a couple of derivatives. Each marker must resolve to a single token id; if both open and close resolve, stash on `LoadedModel.reasoning_tokens` (similarly `TpLoadedModel`). Non-reasoning models get `None` and pass through unchanged. 2. **At inference time**, the three streaming paths (`run_inference_streaming` CPU, `stream_inference_via_worker` CUDA single-GPU, `chat_completion_tp_stream` CUDA TP) now check each sampled token against the pair via the new `handle_reasoning_marker` helper before feeding it to the detokeniser. Open marker → set `in_reasoning = true`, drop the marker. Close marker → unset, drop. Other tokens go through `emit_delta(_blocking)` which now picks `ReasoningDelta` or `TextDelta` based on state. Markers never appear in the streamed output. 3. **In `wire::openai_chat`**, the projector splits into: - `project_chat_stream` (unchanged signature; default behaviour — drops `ReasoningDelta`) - `project_chat_stream_with(rx, …, ChatProjectionConfig)` — when `include_thinking: true` and `reasoning_markers: Some(_)`, re-wraps reasoning content with the literal open/close marker text and emits as content deltas. Preserves the on-the-wire shape that helexa-acp's `ThinkParser` expects. 4. **HTTP handler** reads `x-include-thinking: true` (case- insensitive `1`/`true`/`yes`) from the request headers and threads it into the projection config. cortex-gateway already forwards arbitrary headers verbatim, so the opt-in works end-to-end without gateway changes. 5. **helexa-acp's `openai_chat` provider** sets `x-include-thinking: true` on every request so its existing `ThinkParser` keeps receiving the marked content stream. `ThinkParser` itself is unchanged — needed for endpoints that aren't reasoning-aware (OpenRouter, OpenAI directly, etc.). ## Acceptance - Zed's commit-message generator (vanilla chat-completions client, no `x-include-thinking`) gets clean commit messages with no `<think>` block. - helexa-acp sessions continue to render thinking in Zed's thought UI via the opt-in path. - Models without reasoning tokens declared in their tokenizer pass through unchanged. - Implementation contains zero references to "qwen3" or any specific model — entirely driven by tokenizer metadata. ## Tests 9 new tests in `wire::event` (token-pair detection across 4 marker conventions, edge cases) and `wire::openai_chat` (default drop, opt-in re-wrap with multi-chunk reasoning, close-marker on Finish, fallback when markers absent, off-switch with markers present). All 213 workspace tests pass; fmt + clippy clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -97,3 +97,134 @@ impl FinishReason {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Open/close token IDs for the reasoning marker a loaded model uses
|
||||
/// (or `None` for non-reasoning models). The harness reads this once
|
||||
/// at load time from the tokenizer's added-tokens table, then the
|
||||
/// inference loop checks `next_token` against the pair to flip
|
||||
/// between [`InferenceEvent::TextDelta`] and
|
||||
/// [`InferenceEvent::ReasoningDelta`].
|
||||
///
|
||||
/// `open` and `close` text are kept alongside the IDs so wire
|
||||
/// projectors that want to re-emit the literal markers (the
|
||||
/// opt-in `include_thinking` path on chat completions) don't have
|
||||
/// to reach back into the tokenizer for the strings.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ReasoningTokenPair {
|
||||
pub open_id: u32,
|
||||
pub close_id: u32,
|
||||
pub open_text: String,
|
||||
pub close_text: String,
|
||||
}
|
||||
|
||||
/// Known reasoning-marker conventions. Each is a `(open, close)`
|
||||
/// pair of literal token strings. Each modern reasoning model
|
||||
/// declares its markers in the tokenizer's `added_tokens` table;
|
||||
/// at load time we probe for whichever pair the loaded tokenizer
|
||||
/// has and stash both IDs.
|
||||
///
|
||||
/// Ordering matters only for tie-breaking when a model declares
|
||||
/// multiple pairs (shouldn't happen in practice); the first hit
|
||||
/// wins.
|
||||
const KNOWN_REASONING_MARKERS: &[(&str, &str)] = &[
|
||||
// Qwen3, DeepSeek-R1, gpt-oss, and most other open-weight
|
||||
// reasoning models.
|
||||
("<think>", "</think>"),
|
||||
// Mistral Magistral.
|
||||
("[THINK]", "[/THINK]"),
|
||||
// Some older derivatives; harmless to probe.
|
||||
("<thought>", "</thought>"),
|
||||
("<reasoning>", "</reasoning>"),
|
||||
];
|
||||
|
||||
/// Inspect a tokenizer for known reasoning-marker pairs and return
|
||||
/// the first match. The tokenizer types this trait is defined over
|
||||
/// just need to expose `token_to_id(&str) -> Option<u32>` so this
|
||||
/// stays decoupled from the candle crate — the production caller
|
||||
/// passes a `tokenizers::Tokenizer`, but tests can fake one.
|
||||
///
|
||||
/// Returns `None` when no known marker pair is fully declared
|
||||
/// (both open AND close token ids must resolve). That's the
|
||||
/// pass-through case — non-reasoning models, or reasoning models
|
||||
/// whose tokenizer split the markers across multiple tokens (rare
|
||||
/// in practice; modern reasoning tokenizers list them as
|
||||
/// `added_tokens`).
|
||||
pub fn detect_reasoning_token_pair<F>(token_to_id: F) -> Option<ReasoningTokenPair>
|
||||
where
|
||||
F: Fn(&str) -> Option<u32>,
|
||||
{
|
||||
for (open_text, close_text) in KNOWN_REASONING_MARKERS {
|
||||
let open_id = token_to_id(open_text);
|
||||
let close_id = token_to_id(close_text);
|
||||
if let (Some(open_id), Some(close_id)) = (open_id, close_id) {
|
||||
return Some(ReasoningTokenPair {
|
||||
open_id,
|
||||
close_id,
|
||||
open_text: (*open_text).into(),
|
||||
close_text: (*close_text).into(),
|
||||
});
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::collections::HashMap;
|
||||
|
||||
fn lookup<'a>(map: &'a HashMap<&'static str, u32>) -> impl Fn(&str) -> Option<u32> + 'a {
|
||||
|s| map.get(s).copied()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detects_qwen3_style_think_markers() {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("<think>", 151648);
|
||||
m.insert("</think>", 151649);
|
||||
let pair = detect_reasoning_token_pair(lookup(&m)).expect("pair detected");
|
||||
assert_eq!(pair.open_id, 151648);
|
||||
assert_eq!(pair.close_id, 151649);
|
||||
assert_eq!(pair.open_text, "<think>");
|
||||
assert_eq!(pair.close_text, "</think>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detects_mistral_magistral_markers() {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("[THINK]", 100);
|
||||
m.insert("[/THINK]", 101);
|
||||
let pair = detect_reasoning_token_pair(lookup(&m)).expect("pair detected");
|
||||
assert_eq!(pair.open_text, "[THINK]");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn returns_none_when_only_open_marker_present() {
|
||||
// A pathological tokenizer that has `<think>` but not
|
||||
// `</think>` shouldn't half-detect. Pass-through.
|
||||
let mut m = HashMap::new();
|
||||
m.insert("<think>", 1);
|
||||
assert!(detect_reasoning_token_pair(lookup(&m)).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn returns_none_for_non_reasoning_tokenizer() {
|
||||
let m: HashMap<&'static str, u32> = HashMap::new();
|
||||
assert!(detect_reasoning_token_pair(lookup(&m)).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn first_match_wins_when_multiple_pairs_declared() {
|
||||
// Hypothetical tokenizer with both Qwen-style AND Mistral-style
|
||||
// markers — the `<think>` pair is earlier in the convention
|
||||
// table so it wins.
|
||||
let mut m = HashMap::new();
|
||||
m.insert("<think>", 1);
|
||||
m.insert("</think>", 2);
|
||||
m.insert("[THINK]", 3);
|
||||
m.insert("[/THINK]", 4);
|
||||
let pair = detect_reasoning_token_pair(lookup(&m)).unwrap();
|
||||
assert_eq!(pair.open_id, 1);
|
||||
assert_eq!(pair.close_id, 2);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user