rob thijssen
fc9a8c42a3
Some checks failed
build-prerelease / Build cortex binary (push) Blocked by required conditions
CI / Clippy (push) Waiting to run
CI / Test (push) Waiting to run
CI / CUDA type-check (push) Failing after 17s
build-prerelease / Resolve version stamps (push) Successful in 32s
CI / Format (push) Successful in 32s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
CI / Build cortex SRPM (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Closes #6. Same model-agnostic seam as #8 but for tool-call markers (`<tool_call>` / `</tool_call>` on Qwen3-Coder, Hermes-format, DeepSeek-Coder, gpt-oss, …). Lets Zed's tool-use feature and any other vanilla OpenAI chat client get structured `tool_calls` deltas out of cortex without having to parse markers themselves. ## Implementation 1. **Tokenizer probe at load time** (`detect_tool_call_token_pair` in `wire::event`) — same shape as the reasoning-marker probe from #8. Both open AND close must resolve to single token ids; non-tool-use models get `None` and pass through unchanged. Stored on `LoadedModel.tool_call_tokens` and the TP analogue. 2. **New `InferenceEvent::ToolCall` variant** — carries `index` (call slot, per-turn counter), generated `id` (`call_<hex>_<idx>`), `name`, and the complete `arguments` JSON string. One event per parsed call. 3. **Token-level state machine** in all three streaming paths (CPU `run_inference_streaming`, CUDA single-GPU `stream_inference_via_worker`, CUDA TP `chat_completion_tp_stream`) layered on top of #8's reasoning routing: - `<tool_call>` token → enter buffering state, clear buffer. - Tokens while buffering → accumulate into `tool_call_buf` via the decoder (so multi-byte UTF-8 still buffers correctly) without emitting anything visible. - `</tool_call>` token → take the buffer, parse with `parse_tool_call_body` (extract `name` + `arguments`), emit a structured `ToolCall` event with a fresh `call_<hex>` id and the parsed fields. - On parse failure → fall back to re-emitting the original `<tool_call>{buf}</tool_call>` block as plain text content so helexa-acp's existing `ToolCallParser` repair passes still have a chance to recover the call. 4. **OpenAI chat projector** emits the OpenAI streaming `tool_calls` delta shape on `InferenceEvent::ToolCall` — `{tool_calls: [{index, id, type:"function", function:{name, arguments}}]}`. One chunk per call slot. 5. **OpenAI Responses projector** drops `ToolCall` events for now (Responses-side function_call event family routing tracked under #7); the chat path is what unblocks Zed's tool use today. ## Acceptance - Vanilla OpenAI chat clients (Zed's tool-use feature, any other OpenAI-compatible tool-call consumer) get structured tool_calls deltas against cortex+neuron without having to parse `<tool_call>` markers in content. - helexa-acp continues to work — when neuron parses cleanly, it consumes the structured deltas through its existing decoder. When the model emits malformed JSON, neuron falls back to text pass-through and helexa-acp's `ToolCallParser` recovers via the same path it always did. - Models without tool-call markers in their tokenizer pass through unchanged. - No hardcoded model knowledge — entirely driven by tokenizer metadata. ## Tests 2 new detection tests in `wire::event` (Qwen3-style marker detection, no-marker case). The streaming paths themselves stay covered by the existing chat-completions integration tests; full end-to-end exercise of the new path requires GPU-loaded models and lives outside the CI test surface. 215 workspace tests pass; clippy + fmt clean across the workspace. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 lines
1.1 KiB
Rust
28 lines
1.1 KiB
Rust
//! Wire-format projection layer.
|
|
//!
|
|
//! The candle harness produces a single, format-agnostic stream of
|
|
//! [`InferenceEvent`]s. Each wire format (OpenAI chat completions,
|
|
//! OpenAI Responses, Anthropic messages, …) lives in its own module
|
|
//! under `wire::` and projects that event stream into the chunks /
|
|
//! events its HTTP clients expect.
|
|
//!
|
|
//! The benefit over translating *between* wire shapes (OpenAI chat
|
|
//! → Anthropic, etc.) is that we never have to reason about a
|
|
//! wire-N → wire-M conversion: every translation is wire-N ↔ the
|
|
//! internal event currency, and the projections are independent. A
|
|
//! new wire format adds a new file under `wire::`; nothing else
|
|
//! needs to know about it.
|
|
//!
|
|
//! Today: [`openai_chat`]. Stage 2 adds `openai_responses`. Stage 3
|
|
//! could add a native Anthropic projection that replaces the
|
|
//! gateway-side translation.
|
|
|
|
pub mod event;
|
|
pub mod openai_chat;
|
|
pub mod openai_responses;
|
|
|
|
pub use event::{
|
|
FinishReason, InferenceEvent, ReasoningTokenPair, ToolCallTokenPair,
|
|
detect_reasoning_token_pair, detect_tool_call_token_pair,
|
|
};
|