feat(neuron): extract <tool_call> blocks to structured tool_calls deltas
Some checks failed
build-prerelease / Build cortex binary (push) Blocked by required conditions
CI / Clippy (push) Waiting to run
CI / Test (push) Waiting to run
CI / CUDA type-check (push) Failing after 17s
build-prerelease / Resolve version stamps (push) Successful in 32s
CI / Format (push) Successful in 32s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
CI / Build cortex SRPM (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Some checks failed
build-prerelease / Build cortex binary (push) Blocked by required conditions
CI / Clippy (push) Waiting to run
CI / Test (push) Waiting to run
CI / CUDA type-check (push) Failing after 17s
build-prerelease / Resolve version stamps (push) Successful in 32s
CI / Format (push) Successful in 32s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
CI / Build cortex SRPM (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Closes #6. Same model-agnostic seam as #8 but for tool-call markers (`<tool_call>` / `</tool_call>` on Qwen3-Coder, Hermes-format, DeepSeek-Coder, gpt-oss, …). Lets Zed's tool-use feature and any other vanilla OpenAI chat client get structured `tool_calls` deltas out of cortex without having to parse markers themselves. ## Implementation 1. **Tokenizer probe at load time** (`detect_tool_call_token_pair` in `wire::event`) — same shape as the reasoning-marker probe from #8. Both open AND close must resolve to single token ids; non-tool-use models get `None` and pass through unchanged. Stored on `LoadedModel.tool_call_tokens` and the TP analogue. 2. **New `InferenceEvent::ToolCall` variant** — carries `index` (call slot, per-turn counter), generated `id` (`call_<hex>_<idx>`), `name`, and the complete `arguments` JSON string. One event per parsed call. 3. **Token-level state machine** in all three streaming paths (CPU `run_inference_streaming`, CUDA single-GPU `stream_inference_via_worker`, CUDA TP `chat_completion_tp_stream`) layered on top of #8's reasoning routing: - `<tool_call>` token → enter buffering state, clear buffer. - Tokens while buffering → accumulate into `tool_call_buf` via the decoder (so multi-byte UTF-8 still buffers correctly) without emitting anything visible. - `</tool_call>` token → take the buffer, parse with `parse_tool_call_body` (extract `name` + `arguments`), emit a structured `ToolCall` event with a fresh `call_<hex>` id and the parsed fields. - On parse failure → fall back to re-emitting the original `<tool_call>{buf}</tool_call>` block as plain text content so helexa-acp's existing `ToolCallParser` repair passes still have a chance to recover the call. 4. **OpenAI chat projector** emits the OpenAI streaming `tool_calls` delta shape on `InferenceEvent::ToolCall` — `{tool_calls: [{index, id, type:"function", function:{name, arguments}}]}`. One chunk per call slot. 5. **OpenAI Responses projector** drops `ToolCall` events for now (Responses-side function_call event family routing tracked under #7); the chat path is what unblocks Zed's tool use today. ## Acceptance - Vanilla OpenAI chat clients (Zed's tool-use feature, any other OpenAI-compatible tool-call consumer) get structured tool_calls deltas against cortex+neuron without having to parse `<tool_call>` markers in content. - helexa-acp continues to work — when neuron parses cleanly, it consumes the structured deltas through its existing decoder. When the model emits malformed JSON, neuron falls back to text pass-through and helexa-acp's `ToolCallParser` recovers via the same path it always did. - Models without tool-call markers in their tokenizer pass through unchanged. - No hardcoded model knowledge — entirely driven by tokenizer metadata. ## Tests 2 new detection tests in `wire::event` (Qwen3-style marker detection, no-marker case). The streaming paths themselves stay covered by the existing chat-completions integration tests; full end-to-end exercise of the new path requires GPU-loaded models and lives outside the CI test surface. 215 workspace tests pass; clippy + fmt clean across the workspace. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -44,16 +44,29 @@ pub enum InferenceEvent {
|
||||
/// concatenate into the complete reply.
|
||||
TextDelta(String),
|
||||
/// Reasoning / scratchpad text the model emitted inside a
|
||||
/// `<think>` block (or equivalent). Producers that don't
|
||||
/// surface reasoning separately use [`TextDelta`] for
|
||||
/// everything; future split lives here.
|
||||
///
|
||||
/// Not yet emitted by the candle harness — present so future
|
||||
/// stages (qwen3 `<think>` routing, OpenAI o-series reasoning)
|
||||
/// have a typed home without breaking the existing
|
||||
/// projections.
|
||||
#[allow(dead_code)]
|
||||
/// `<think>` block (or equivalent). The harness routes
|
||||
/// content between marker tokens here so wire projectors can
|
||||
/// decide what to do with it (chat completions drops by
|
||||
/// default; Responses API has a dedicated event family).
|
||||
ReasoningDelta(String),
|
||||
/// A tool call has been parsed out of a `<tool_call>{json}</tool_call>`
|
||||
/// block. Carries the parsed name + arguments JSON string
|
||||
/// (Anthropic / OpenAI projectors emit their own wire shape
|
||||
/// from this).
|
||||
///
|
||||
/// `index` is the call slot — incremented per tool call in a
|
||||
/// turn so wire formats that order calls by index
|
||||
/// (OpenAI chat completions) can correlate.
|
||||
ToolCall {
|
||||
index: usize,
|
||||
id: String,
|
||||
name: String,
|
||||
/// Complete JSON arguments string. The model could in
|
||||
/// principle stream these token-by-token, but our
|
||||
/// extraction buffers the whole block until `</tool_call>`
|
||||
/// arrives and emits exactly one event per call.
|
||||
arguments: String,
|
||||
},
|
||||
/// The stream is complete. Carries the reason so wire formats
|
||||
/// that use it (OpenAI's `finish_reason`, Anthropic's
|
||||
/// `stop_reason`) can render it without re-parsing.
|
||||
@@ -137,6 +150,51 @@ const KNOWN_REASONING_MARKERS: &[(&str, &str)] = &[
|
||||
("<reasoning>", "</reasoning>"),
|
||||
];
|
||||
|
||||
/// Open/close token IDs for the model's tool-call marker
|
||||
/// convention (or `None` for models that don't emit structured
|
||||
/// tool calls). Same shape as [`ReasoningTokenPair`]: probed once
|
||||
/// at load time, consumed by the inference loop to switch between
|
||||
/// "emit visible deltas" and "buffer JSON for the next tool
|
||||
/// call".
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ToolCallTokenPair {
|
||||
pub open_id: u32,
|
||||
pub close_id: u32,
|
||||
pub open_text: String,
|
||||
pub close_text: String,
|
||||
}
|
||||
|
||||
/// Tool-call marker conventions. Open-weight tool-use models
|
||||
/// converged on `<tool_call>` / `</tool_call>` (Qwen3-Coder /
|
||||
/// -Instruct, the Hermes function-call format, DeepSeek-Coder,
|
||||
/// gpt-oss). The pair lives alongside the reasoning markers in
|
||||
/// the same `added_tokens` table.
|
||||
const KNOWN_TOOL_CALL_MARKERS: &[(&str, &str)] = &[("<tool_call>", "</tool_call>")];
|
||||
|
||||
/// Probe a tokenizer for known tool-call marker pairs. Mirrors
|
||||
/// [`detect_reasoning_token_pair`] — both open AND close must
|
||||
/// resolve for the pair to be returned. `None` means the model
|
||||
/// doesn't emit structured tool calls (or its tokenizer split
|
||||
/// the markers across tokens).
|
||||
pub fn detect_tool_call_token_pair<F>(token_to_id: F) -> Option<ToolCallTokenPair>
|
||||
where
|
||||
F: Fn(&str) -> Option<u32>,
|
||||
{
|
||||
for (open_text, close_text) in KNOWN_TOOL_CALL_MARKERS {
|
||||
let open_id = token_to_id(open_text);
|
||||
let close_id = token_to_id(close_text);
|
||||
if let (Some(open_id), Some(close_id)) = (open_id, close_id) {
|
||||
return Some(ToolCallTokenPair {
|
||||
open_id,
|
||||
close_id,
|
||||
open_text: (*open_text).into(),
|
||||
close_text: (*close_text).into(),
|
||||
});
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Inspect a tokenizer for known reasoning-marker pairs and return
|
||||
/// the first match. The tokenizer types this trait is defined over
|
||||
/// just need to expose `token_to_id(&str) -> Option<u32>` so this
|
||||
@@ -213,6 +271,24 @@ mod tests {
|
||||
assert!(detect_reasoning_token_pair(lookup(&m)).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detects_tool_call_markers() {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("<tool_call>", 151657);
|
||||
m.insert("</tool_call>", 151658);
|
||||
let pair = detect_tool_call_token_pair(lookup(&m)).expect("pair detected");
|
||||
assert_eq!(pair.open_id, 151657);
|
||||
assert_eq!(pair.close_id, 151658);
|
||||
assert_eq!(pair.open_text, "<tool_call>");
|
||||
assert_eq!(pair.close_text, "</tool_call>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn returns_none_for_non_tool_use_tokenizer() {
|
||||
let m: HashMap<&'static str, u32> = HashMap::new();
|
||||
assert!(detect_tool_call_token_pair(lookup(&m)).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn first_match_wins_when_multiple_pairs_declared() {
|
||||
// Hypothetical tokenizer with both Qwen-style AND Mistral-style
|
||||
|
||||
Reference in New Issue
Block a user