helexa/helexa
1
0
Fork 0
Code Issues 135 Pull Requests Actions Packages Projects Releases Wiki Activity

Three of five API surfaces silently discard the model's reasoning #300

New Issue
Closed
opened 2026-08-27 05:37:00 +00:00 by grenade · 1 comment
grenade commented 2026-08-27 05:37:00 +00:00
Owner
Copy Link
Copy Source

Reasoning is generated, counted in usage, and then thrown away on
three of our five response paths. The client never sees it, so it can
never replay it, so on the next turn the model has no record of its own
thinking — only bare assertions it cannot account for.

The surface

path reasoning returned
/v1/chat/completions streaming yes — choice.delta.reasoning_content
/v1/chat/completions non-streaming discarded
/v1/responses streaming yes — response.reasoning_summary_* family
/v1/responses non-streaming discarded
/v1/messages (Anthropic) discarded

/v1/chat/completions, non-streaming

wire/openai_chat.rs, in collect_chat_completion:

InferenceEvent::ReasoningDelta(_) => {}

Explicitly dropped on the floor.

/v1/responses, non-streaming

wire/openai_responses.rs, build_response emits a single Message item:

output: vec![ResponsesOutputItem::Message { … content: vec![OutputText { text: full_text }] … }],

No reasoning output item is ever constructed, though the streaming
projector builds the full item lifecycle.

Observed live: a request returning
{"output_tokens": 4000, "output_tokens_details": {"reasoning_tokens": 4000}}
with an empty output — 4,000 tokens generated, billed, and invisible.

/v1/messages (Anthropic) — the worst of the three

cortex-core/src/translate.rs:97-113 deliberately accepts the
extended-thinking control:

Anthropic's extended-thinking budget is the one quantitative reasoning
control any of our wire formats offers (#223). Normalised onto
reasoning.max_tokens …

and openai_to_anthropic returns:

content: vec![ContentBlock { block_type: "text".into(), data: json!({ "text": content_text }) }],

A bare text block. Anthropic's own shape is a {"type":"thinking", "thinking": …} block alongside the text. So a client asks for extended
thinking, we take the parameter, spend the tokens, and hand back
nothing — every turn, silently.

No thinking_delta appears anywhere in the gateway, so this is not
streaming-only.

Why this matters beyond the wire

The visible symptom is a model that cannot account for its own prior
work: it asserts things it "decided" with no trace of the deciding, and
when asked to verify, has nothing to check against. On these surfaces
that is not a memory failure — the record was deleted before it ever
reached the client.

It also silently inflates cost: reasoning_tokens are metered and
unusable.

Not the cause of #296

pi streams /v1/responses, so it does receive reasoning. This issue is
a separate, wider defect found while investigating that one, and should
not be conflated with it.

Fix

  1. collect_chat_completion accumulates reasoning into
    message.reasoning_content, matching what the streaming path emits.
  2. build_response constructs a reasoning output item when reasoning
    was produced, mirroring the streaming projector's item lifecycle.
  3. openai_to_anthropic emits a thinking content block. Needs care:
    Anthropic's shape carries a signature, and a block we cannot sign
    may be rejected on replay — worth checking what a real client does
    with an unsigned thinking block before choosing between emitting one
    and documenting the gap.
  4. A test per surface asserting reasoning survives, so this cannot
    silently reopen. The absence of exactly these tests is why three
    paths diverged.

Related

#293 (a client-visible field we omitted, same shape of defect), #294
(schema conformance would catch the Responses case), #296, #223.

Reasoning is generated, counted in `usage`, and then thrown away on three of our five response paths. The client never sees it, so it can never replay it, so on the next turn the model has no record of its own thinking — only bare assertions it cannot account for. ## The surface | path | reasoning returned | |---|---| | `/v1/chat/completions` streaming | yes — `choice.delta.reasoning_content` | | `/v1/chat/completions` non-streaming | **discarded** | | `/v1/responses` streaming | yes — `response.reasoning_summary_*` family | | `/v1/responses` non-streaming | **discarded** | | `/v1/messages` (Anthropic) | **discarded** | ### `/v1/chat/completions`, non-streaming `wire/openai_chat.rs`, in `collect_chat_completion`: ```rust InferenceEvent::ReasoningDelta(_) => {} ``` Explicitly dropped on the floor. ### `/v1/responses`, non-streaming `wire/openai_responses.rs`, `build_response` emits a single Message item: ```rust output: vec![ResponsesOutputItem::Message { … content: vec![OutputText { text: full_text }] … }], ``` No `reasoning` output item is ever constructed, though the streaming projector builds the full item lifecycle. Observed live: a request returning `{"output_tokens": 4000, "output_tokens_details": {"reasoning_tokens": 4000}}` with an empty `output` — 4,000 tokens generated, billed, and invisible. ### `/v1/messages` (Anthropic) — the worst of the three `cortex-core/src/translate.rs:97-113` deliberately *accepts* the extended-thinking control: > Anthropic's extended-thinking budget is the one quantitative reasoning > control any of our wire formats offers (#223). Normalised onto > `reasoning.max_tokens` … and `openai_to_anthropic` returns: ```rust content: vec![ContentBlock { block_type: "text".into(), data: json!({ "text": content_text }) }], ``` A bare text block. Anthropic's own shape is a `{"type":"thinking", "thinking": …}` block alongside the text. So a client asks for extended thinking, we take the parameter, spend the tokens, and hand back nothing — every turn, silently. No `thinking_delta` appears anywhere in the gateway, so this is not streaming-only. ## Why this matters beyond the wire The visible symptom is a model that cannot account for its own prior work: it asserts things it "decided" with no trace of the deciding, and when asked to verify, has nothing to check against. On these surfaces that is not a memory failure — the record was deleted before it ever reached the client. It also silently inflates cost: `reasoning_tokens` are metered and unusable. ## Not the cause of #296 pi streams `/v1/responses`, so it does receive reasoning. This issue is a separate, wider defect found while investigating that one, and should not be conflated with it. ## Fix 1. `collect_chat_completion` accumulates reasoning into `message.reasoning_content`, matching what the streaming path emits. 2. `build_response` constructs a `reasoning` output item when reasoning was produced, mirroring the streaming projector's item lifecycle. 3. `openai_to_anthropic` emits a `thinking` content block. Needs care: Anthropic's shape carries a `signature`, and a block we cannot sign may be rejected on replay — worth checking what a real client does with an unsigned thinking block before choosing between emitting one and documenting the gap. 4. A test per surface asserting reasoning survives, so this cannot silently reopen. The absence of exactly these tests is why three paths diverged. ## Related #293 (a client-visible field we omitted, same shape of defect), #294 (schema conformance would catch the Responses case), #296, #223.
grenade commented 2026-08-27 09:21:15 +00:00
Author
Owner
Copy Link
Copy Source

Fixed across all five surfaces — 039a353a and eb783c34.

path before now
/v1/chat/completions streaming returned unchanged
/v1/chat/completions non-streaming discarded message.reasoning_content
/v1/responses streaming returned unchanged
/v1/responses non-streaming discarded a reasoning output item
/v1/messages streaming discarded thinking_delta events
/v1/messages non-streaming discarded a thinking block

Six tests, one pair per surface: reasoning survives, and its absence
leaves the previous message shape untouched (absent, not empty).

The near-miss worth recording

The first fix (039a353a) covered three paths and I called the issue
done. It had missed AnthropicStreamTranslator — which reads
delta.content and delta.tool_calls and nothing else, and which is
the path Claude Code actually uses. anthropic_sse.rs contained
zero occurrences of "thinking".

So the fix would have merged, passed CI, closed the issue, and changed
nothing for the client that needed it most. It surfaced only because
someone asked what a different client had been displaying.

That is the same pattern this issue documents — parallel paths diverging
with nothing comparing them — reproduced inside the fix for it. The
streaming and non-streaming halves of one surface are exactly the kind
of pair no test was checking, which is why three paths had drifted in
the first place.

Carried forward

The Anthropic thinking block is emitted without a signature.
Anthropic signs theirs; we cannot mint a valid one and inventing a value
would be worse than omitting it. If a real client rejects an unsigned
block the remedy is a flag, not a return to deleting the model's work —
recorded in both code paths so it does not need rediscovering. Worth a
deliberate short Claude Code session after this deploys rather than
finding out mid-task: the failure would appear on the second turn of a
session, when the block is replayed, which is an awkward thing to
diagnose cold.

Fixed across all five surfaces — `039a353a` and `eb783c34`. | path | before | now | |---|---|---| | `/v1/chat/completions` streaming | returned | unchanged | | `/v1/chat/completions` non-streaming | discarded | `message.reasoning_content` | | `/v1/responses` streaming | returned | unchanged | | `/v1/responses` non-streaming | discarded | a `reasoning` output item | | `/v1/messages` streaming | discarded | `thinking_delta` events | | `/v1/messages` non-streaming | discarded | a `thinking` block | Six tests, one pair per surface: reasoning survives, and its absence leaves the previous message shape untouched (absent, not empty). ## The near-miss worth recording The first fix (`039a353a`) covered three paths and I called the issue done. It had missed `AnthropicStreamTranslator` — which reads `delta.content` and `delta.tool_calls` and nothing else, and which is **the path Claude Code actually uses**. `anthropic_sse.rs` contained zero occurrences of "thinking". So the fix would have merged, passed CI, closed the issue, and changed nothing for the client that needed it most. It surfaced only because someone asked what a *different* client had been displaying. That is the same pattern this issue documents — parallel paths diverging with nothing comparing them — reproduced inside the fix for it. The streaming and non-streaming halves of one surface are exactly the kind of pair no test was checking, which is why three paths had drifted in the first place. ## Carried forward The Anthropic `thinking` block is emitted **without a `signature`**. Anthropic signs theirs; we cannot mint a valid one and inventing a value would be worse than omitting it. If a real client rejects an unsigned block the remedy is a flag, not a return to deleting the model's work — recorded in both code paths so it does not need rediscovering. Worth a deliberate short Claude Code session after this deploys rather than finding out mid-task: the failure would appear on the *second* turn of a session, when the block is replayed, which is an awkward thing to diagnose cold.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#300