## What changed
- Pass only compaction message and window metadata into
`Session::replace_compacted_history`.
- Build the persisted `CompactedItem` there after assigning any missing response
item IDs, so its replacement history matches the live history without callers
cloning or assembling it themselves.
GitOrigin-RevId: a6106c33b6d3eef39ac0b563a50e17f6782970dc
## What changed
- Add an `ExtensionData` store to each `StepContext`.
- Pass the step store to context, world-state, turn-input, and tool contributors so extensions can use capabilities bound to the current sampling step.
- Preserve the captured step context when rebuilding initial context during compaction.
GitOrigin-RevId: 51f3105d3a793f60534f122f9be8247c131c2ca5
## Why
Remote compaction can process large histories. Repeatedly estimating and replacing the full history, and cloning it when tracing is disabled, adds avoidable CPU and memory overhead.
## What changed
- Estimate each history item's token count once, update the total as trailing tool outputs are rewritten, and replace history only after all rewrites are selected.
- Preserve unclamped token totals while calculating removed tokens so saturated estimates do not hide overflow.
- Snapshot compaction input history only when rollout tracing is enabled.
- Reuse the v2 request input instead of cloning it before adding the compaction trigger.
## Testing
- Cover enabled and disabled compaction trace contexts through `is_enabled()` assertions.
GitOrigin-RevId: 8de9c9704ba29532fba05430eb204bbae2e8bf83
## What changed
- Add remote and local audio variants to core and app-server user inputs, including generated JSON and TypeScript schemas.
- Preserve audio fields when converting app-server inputs and response content into thread items.
- Keep unsupported audio out of model requests, TUI rendering, text extraction, and truncated tool output, using explicit text placeholders where audio is submitted as a model input.
## Testing
- Cover app-server media conversion, unsupported-input placeholders, TUI omission, and tool-output truncation.
GitOrigin-RevId: 16152bf3af674575ce569764a8b472e032e7569d
## What changed
- Parse `cache_write_tokens` from response input token details and carry it through token usage aggregation.
- Expose `cache_write_input_tokens` in protocol, app-server, exec, and TypeScript SDK usage events, defaulting it to zero for compatibility with older payloads.
- Report cache-write usage in analytics, metrics, traces, and rollout data alongside existing cache-read usage.
## Testing
- Add coverage for response parsing, usage event output, aggregation, analytics, metrics, and telemetry fields.
GitOrigin-RevId: 634e4fbe3086a8923c0ee13b622573e7d9d221bd
## Why
Clients that account for individual upstream Responses API calls need the exact
usage for each completion rather than accumulated or estimated thread usage.
## What changed
- Add the opt-in `rawResponse/completed` app-server notification with the
`threadId`, `turnId`, `responseId`, and upstream `usage` payload.
- Emit the transient event for normal turns and compaction requests when
`experimentalRawEvents` is enabled. The event is not persisted or replayed,
and `usage` is `null` when the upstream response omits it.
- Export the notification in the app-server schemas and document its behavior.
## Testing
- Cover exact token breakdowns for turn and compaction completions.
GitOrigin-RevId: 49040a060bf1784c1ff1dd62112efb4a778428c4
## Why
`TurnContext` stored model and reasoning settings both as resolved turn fields and inside `CollaborationMode`, requiring the copies to stay synchronized when the model changed.
## What changed
- Store the mode and collaboration-mode developer instructions directly on `TurnContext`.
- Reconstruct `CollaborationMode` from those values plus the turn's current model and reasoning effort when callers need it.
- Read the mode directly for plan-mode checks, lifecycle events, and telemetry.
GitOrigin-RevId: b8fccafdd5d5cbccc84bc70d542588087ddf0093
## Why
Compacting a resumed conversation can fail when its previous model is no longer
available. Some model-specific failures, such as a model-not-found response, are
not represented as invalid requests and therefore did not trigger the existing
fallback to the currently selected model.
## What changed
- Share a retry predicate between both remote compaction implementations.
- Fall back to the current model for invalid requests, unexpected statuses,
context-window and usage-limit errors, server failures, and exhausted retries.
## Testing
Add an integration test that resumes a conversation whose previous model returns
model-not-found and verifies that compaction and the next turn use the renamed
model.
GitOrigin-RevId: c3857beae2ce79b50589deb13c2e59710ce14725
## Why
Pre-sampling compaction intentionally uses the previous turn's model
when the compaction compatibility hash changes or when switching to a
model with a smaller context window. This keeps compaction aligned with
the settings that produced the history, but it can block the next turn
when a resumed ChatGPT thread still references a model slug that has
since been retired. The Codex backend rejects that compaction request
before the user's currently selected model gets a chance to sample.
This PR lets those threads recover without changing previous-model
compaction behavior for API-key authentication or custom providers. It
is stacked on #31316, which is a behavior-preserving extraction of the
individual remote compaction attempts; this PR contains the fallback
behavior.
## What changed
- For automatic previous-model compaction, capture the selected model's
request context when using ChatGPT authentication with the OpenAI
provider and the selected model differs from the previous model.
- If the previous-model attempt returns an `InvalidRequest`, retry
compaction once with the selected model for both `/responses/compact`
and Responses Compaction V2.
- Complete history processing, lifecycle events, and token accounting
with the context of the model that successfully compacted the thread.
- If the fallback also fails, return the original previous-model error
so the retry does not change the user-visible failure.
- Record fallback attempts with reason, implementation, and outcome
telemetry.
- Leave API-key authentication, custom providers, same-model turns, and
non-`InvalidRequest` failures on their existing paths.
## Testing
- `just test -p codex-core -E 'test(pre_sampling_compact) |
test(model_unavailable_error)'` (10 tests)
- Added integration coverage for a resumed thread whose model was
renamed, a model downshift using Responses Compaction V2, and API-key
authentication with a custom provider.
## Why
This PR is a behavior-preserving refactor only. It does not add a
fallback, change which model is used for compaction, or otherwise change
compaction behavior. The behavioral change is implemented in the stacked
follow-up, #30319.
Pre-sampling compaction deliberately uses the previous turn's context
when the compaction compatibility hash changes or when switching to a
model with a smaller context window. That preserves the model settings
that produced the history being compacted, but the previous context is
not always usable. For example, a resumed thread can still reference a
model slug that has since been retired, causing compaction to fail
before the currently selected model can sample.
#30319 addresses that failure mode by retrying compaction with the
current turn's selected model when the backend rejects the
previous-model attempt. This PR performs only that preparatory refactor.
## What changed
- Extracted one legacy `/responses/compact` request attempt into
`compact_remote_request.rs`.
- Extracted one Responses-based remote compaction request attempt into
`compact_remote_v2_attempt.rs`.
- Kept hooks, lifecycle events, analytics, window advancement, history
processing and installation, and error behavior unchanged in the
existing orchestration paths.
- Preserved standalone Responses-based compaction's owned client-session
lifetime through lifecycle completion.
## Testing
- `just test -p codex-core -E 'test(remote_compact)'` (22 tests)
## Why
With deferred executors, an environment can become ready between two
sampling requests in the same turn. The model-visible environment
update, advertised tools, and eventual tool execution must all describe
the same request-time view.
Otherwise, a request built while only environment B is ready can
advertise a tool without an `environment_id`; if higher-priority
environment A becomes ready before execution, that call could silently
run in A instead.
This PR is stacked on #29527.
## Design
`run_turn` captures one `Arc<StepContext>` at each sampling-request
boundary. That step owns the request's `TurnContext` and environment
snapshot.
- World-state environment updates and tool planning borrow that same
step.
- `ToolCallRuntime` retains the `Arc` while asynchronous tool calls
execute.
- `ToolInvocation` carries the step to handlers; its temporary `turn`
compatibility field is derived from the same object.
- `ToolRouter` does not retain `StepContext`; it only uses it while
constructing the request's tool set.
- With `DeferredExecutor` disabled, step capture keeps using the
environments frozen at turn start.
Simply: every sampling request gets one consistent picture of its
environments, from what the model sees through where its tool calls run.
## What changed
- Build environment-dependent tool specs from the current request's
`StepContext`.
- Use that same step for unified exec, legacy shell, `apply_patch`,
`view_image`, and `request_permissions` execution.
- Hide environment-backed tools, including `request_permissions`, while
no environment is attached.
- Resolve legacy shell paths and metadata from the selected step
environment instead of the stale turn-start environment.
- Capture explicit steps at non-turn-loop boundaries such as compaction,
prompt debug, and startup prewarm.
- Reconcile prompt-debug history from the same step used to build its
tools.
## Follow-up
- Bind yielded code-mode cells to the tool runtime that created them, so
nested calls made after yielding continue to use the originating
request's `StepContext`.
## Test plan
- `just test -p codex-core
deferred_executor_updates_context_and_tools_after_startup`
- `just test -p codex-core
environment_count_controls_environment_backed_tools`
- `just test -p codex-core
build_prompt_input_includes_context_and_user_message`
## Why
Follow-up to #29249 and its [compaction review
thread](https://github.com/openai/codex/pull/29249#discussion_r3455055101).
During a turn, environment readiness can change between sampling
requests. Inline compaction must render the same model-visible
`WorldState` used by the request it follows. Rebuilding that state
during compaction can observe a newer environment, make replacement
history disagree with what the model saw, and suppress the next
environment update.
## What changed
- Make `run_turn` own the current `Arc<WorldState>` and replace it only
between sampling requests.
- Build each state from an explicitly chosen environment snapshot, diff
deferred-executor steps against the turn-owned state, and retain the
latest state in `ContextManager` only for cross-turn and resume
tracking.
- Pass the exact turn-owned state into inline compaction and explicit
new-context-window replacement.
- Carry that state with
`InitialContextInjection::BeforeLastUserMessage`, so replacement context
and its stored baseline cannot come from different snapshots.
- Remove obsolete state-recapture helpers and ambiguous TurnContext-only
WorldState builders.
- Add an integration test that moves an environment from starting to
ready during a paused turn, triggers compaction, and verifies the next
request receives the readiness update exactly once.
## Test plan
- `just test -p codex-core
deferred_executor_compaction_preserves_then_updates_environment_once`
- `just test -p codex-core process_compacted_history`
- `just test -p codex-core mid_turn_continuation_compaction`
- `just test -p codex-core build_initial_context`
- `just test -p codex-core
ignores_session_prefix_messages_when_truncating`
## Summary
- express remote compaction result handling as an exhaustive match
- preserve the special `TurnAborted` path without emitting a generic
compaction error
- rely on the standard `test_codex` provider setup in the compaction
budget test
Follow-up to review feedback on #28707.
## Testing
- `just test -p codex-core
compaction_budget_exhaustion_aborts_without_error_or_retry`
- `just fmt`
## Description
This PR is a followup to https://github.com/openai/codex/pull/28355 and
starts assigning `internal_chat_message_metadata_passthrough.turn_id` to
durable Responses API items created during a turn.
The goal is that those items keep the `turn_id` that introduced them
when Codex resends stateless HTTP context, reconstructs history for
resume/fork paths, or reuses websocket response state.
## What changed
- Set `internal_chat_message_metadata_passthrough.turn_id` when missing
as response items enter durable history, initial/replacement history,
inter-agent communication history, and local compaction summaries.
- Preserve existing item turn IDs instead of overwriting them during
persistence, resume reconstruction, compaction, forked history, and
websocket incremental reuse.
- Keep `compaction_trigger` fieldless because it is a request control,
not a durable response item.
- Update focused history/request assertions and fixtures for stateless
requests, websocket incrementals, compaction, thread injection, prompt
debug, and related CI coverage.
## Description
This PR cuts Codex over from generic `ResponseItem.metadata` (introduced
here: https://github.com/openai/codex/pull/28355) to
`ResponseItem.internal_chat_message_metadata_passthrough`, which is the
blessed path and has strongly-typed keys.
For now we have to drop this MAv2 usage of `metadata`:
https://github.com/openai/codex/pull/28561 until we figure out where
that should live.
## Why
The rendered `<token_budget>` fragment identifies the thread and current
context window, but it does not expose enough lineage to identify the
first window in the thread or the immediately preceding window. Those
IDs also need to remain stable across compaction, resume, and rollback.
## What changed
- Track first, previous, and current UUIDv7 context-window IDs in
auto-compaction state.
- Render `thread_id`, `first_window_id`, `previous_window_id`, and the
current window ID in the full `<token_budget>` fragment.
- Persist the first and previous window IDs in compacted rollout
checkpoints and restore them during rollout reconstruction.
- Preserve compatibility with older compacted records that do not
contain the new optional fields.
- Update focused state, rendering, reconstruction, rollback, and
serialization coverage.
## Validation
- `just test -p codex-core token_budget`
- `just test -p codex-protocol compacted_item::tests`
- `just test -p codex-core tracks_prefill_and_window_boundaries`
- `just test -p codex-core
reconstruct_history_uses_replacement_history_verbatim`
- `just test -p codex-core
thread_rollback_restores_cleared_reference_context_item_after_compaction`
## Stack
Depends on #28494.
## Description
This PR propagates shared rollout-budget exhaustion through the existing
`CodexErr::TurnAborted` task result.
Each thread records its model usage against the same ledger. Once the
ledger is exhausted, that usage update and all later usage updates
return `TurnAborted`. The task wrapper emits the normal aborted-turn
event and lifecycle instead of completing the turn.
This is intentionally a soft boundary: there is no cross-thread
`Op::Interrupt` fanout. An in-flight thread can finish its current
response before it observes the exhausted ledger, but every thread
aborts at its next usage-accounting boundary.
## Tests
The integration coverage verifies that:
- the response that exhausts the budget aborts its turn;
- a later response also aborts because the shared ledger remains
exhausted; and
- sub-agent usage draws from the same shared ledger; and
- local and remote-v2 compaction abort without retrying or emitting a
generic error.
Local checks:
- `just test -p codex-core
exhausted_budget_aborts_current_and_later_turns`
- `just test -p codex-core subagent_usage_draws_from_the_shared_budget`
- `just test -p codex-core
abort_regular_task_emits_marker_before_turn_aborted`
- `just test -p codex-core
compaction_budget_exhaustion_aborts_without_error_or_retry`
- `just fix -p codex-core`
- `just fmt`
- `git diff --check`
The full workspace test suite was not run locally.
## Why
Remote v2 compaction can return replacement-history items without IDs.
Because replacement history is installed directly, those items bypass
normal history preparation and remain ID-less in later Responses
requests even when the `item_ids` feature is enabled.
## What changed
- Pass the active `TurnContext` into `replace_compacted_history`.
- When `item_ids` is enabled, assign missing IDs before installing and
persisting replacement history.
- Rebuild `CompactedItem` from the prepared history so live and
persisted replacement histories match.
- Add integration coverage requiring IDs on every ID-capable input item
in the initial, remote v2 compaction, and post-compaction requests.
## Test plan
- `just test -p codex-core response_item_ids`
- `just test -p codex-core websocket_v2_test_codex_shell_chain`
- `just test -p codex-core remote_compaction_parity_pre_turn_auto`
- `just test -p codex-app-server
thread_inject_items_adds_raw_response_items_to_thread_history`
## Why
Client-created response items enter history without IDs, so their
identity is lost across rollout persistence and resume. IDs should be
assigned once at the history-recording boundary, while IDs returned by
the server must remain unchanged.
The Responses API validates item IDs using type-specific prefixes.
Locally generated IDs therefore use the matching prefix plus a
hyphenated UUIDv7, keeping them valid while distinguishable from
server-generated IDs. Because this changes persisted history and
provider request shapes, the behavior is opt-in behind the
under-development `item_ids` feature. Compaction triggers remain request
controls whose API shape does not accept an ID.
## What changed
- Register the disabled-by-default `item_ids` feature and expose it in
`config.schema.json`.
- Make supported optional `ResponseItem` IDs serializable and expose
them in the generated app-server schemas.
- When `item_ids` is enabled, assign an ID during conversation-history
preparation if an item has no ID.
- Generate type-prefixed, hyphenated UUIDv7 IDs using the Responses API
item conventions.
- Preserve existing server IDs without rewriting them.
- Persist assigned IDs in rollouts and include them in subsequent
Responses requests.
- Remove the unsupported ID field from `CompactionTrigger` and document
why it has no ID.
- Add integration coverage for enabled ID persistence, preservation of
server IDs, and omission of generated IDs while the feature is disabled.
`prepare_conversation_items_for_history` is the single response-item ID
allocation boundary.
## Test plan
- `just test -p codex-features`
- `just test -p codex-core
response_item_ids_persist_across_resume_and_preserve_server_ids`
- `just test -p codex-core
non_openai_responses_requests_omit_item_turn_metadata`
- `just test -p codex-core
resize_all_images_prepares_failures_before_history_insertion`
- `just test -p codex-protocol`
- `just test -p codex-app-server-protocol`
- `just test -p codex-api azure_default_store_attaches_ids_and_headers`
## Why
The token-budget context currently identifies a context window by its
thread-local sequence number. A UUIDv7 gives the model a stable opaque
identity that remains fixed for a window and rotates when compaction or
`new_context` starts the next one.
## What changed
- Preserve the existing monotonic value as `window_number` and add a
UUIDv7 `window_id` to `CompactedItem`.
- Generate and rotate the UUID with auto-compaction window state,
persist it alongside the number, and reconstruct it on resume and
rollback.
- Accept legacy compacted rollout records where the numeric `window_id`
represented the window number.
- Use the UUID only in token-budget context; existing request headers
and metadata continue using `thread_id:window_number`.
## Testing
- `just test -p codex-protocol compacted_item::tests`
- `just test -p codex-core token_budget`
## Stack
Depends on #28746. This PR implements shared rollout-budget accounting
and model-visible reminders using the configuration defined in #28746.
# Description / Main changes to Core:
`AgentControl` will now be the area where "rollout level" features &
accounting will have to live. It is incorrectly named for this
responsibility, but I think it can hold all the necessary shared state &
features (rollout token budget, mutliple thread interruption
responsibilitym etc)
In this PR, we have one "token ledger" that each thread will subtract
from when sampling. The "charge" will occur when response.completed() is
done and the calculation will be done on the responses api usage
carrier. The calculation will weigh sampling and pre-fill tokens as
specified.
Every time the budget crosses the configured reminder threshold, a
developer message is appended before the thread's next request
This remaining budget will _always_ be restated/reminded after a
compaction event.
Expiration and fan-out interruption will be in the stacked follow-up
(and also live in Agent Control).
## Reminders
"You have weighted {session_tokens_left} tokens left in the shared
session token budget."
The first request in each thread context receives the current remainder.
Later reminders are emitted after aggregate weighted usage crosses a
configured interval. If several intervals are crossed before a thread
sends another request, Core inserts one reminder with the latest
remainder.
Compaction response usage is charged before the next context starts. The
next reminder is appended after the compaction summary, leaving the
initial context content stable.
## Tests
Integration coverage verifies:
- weighted output and non-cached input accounting
- initial and periodic reminders
- shared accounting between a root and sub-agent
- post-compaction remainder and message placement
Local checks:
- `just fmt`
- `just test -p codex-core rollout_budget`
- `git diff --check`
The full workspace test suite was not run locally.
## Why
`ResponseItem` variants do not have a consistent internal ID shape: some
variants carry required IDs, some carry optional IDs, and some cannot
represent an ID at all. The existing fields also use inconsistent serde,
TypeScript, and JSON-schema annotations. A single enum-level access path
is needed before history recording can assign and retain IDs.
This PR establishes that internal model only. It intentionally does not
generate or serialize IDs; allocation and wire persistence are isolated
in the stacked follow-up.
## What changed
- Give every concrete `ResponseItem` variant an `Option<String>` ID
field.
- Apply the same internal-only annotations to every ID field:
`#[serde(default, skip_serializing)]`, `#[ts(skip)]`, and
`#[schemars(skip)]`.
- Add `ResponseItem::id()` and `ResponseItem::set_id()` as the shared
accessors.
- Preserve IDs when history items are rewritten for truncation.
- Adapt consumers that previously assumed reasoning and image-generation
IDs were required.
- Regenerate app-server schemas so the hidden fields are represented
consistently.
The serde catch-all `ResponseItem::Other` remains ID-less because it
must remain a unit variant.
## Test plan
- `cargo check --tests -p codex-core -p codex-api -p codex-rollout-trace
-p codex-image-generation-extension`
- `just test -p codex-protocol`
- `just test -p codex-app-server-protocol`
- `just test -p codex-api -p codex-rollout-trace -p
codex-image-generation-extension`
- `just test -p codex-core event_mapping`
## Why
`TurnContext` had accumulated dead fields and cached projections of
values already owned by its per-turn `Config` or `ModelInfo`. Keeping
both copies made ownership unclear and allowed artificial split-brain
states, such as a compatibility hash differing from the model metadata
it came from.
`Prompt` similarly carried a write-only personality after personality
selection had already been materialized into its base instructions.
This makes the canonical owner explicit: configuration-backed values
come from `config`, model-derived values come from `model_info`, and
prompts contain only data consumed by request construction.
## What changed
- Remove the unused `ghost_snapshot`, `codex_self_exe`, and
`thread_source` fields.
- Remove duplicate `comp_hash`, `truncation_policy`, `features`,
`shell_environment_policy`, `codex_linux_sandbox_exe`, `compact_prompt`,
and `tool_mode` fields.
- Read those values directly from `TurnContext::config` or
`TurnContext::model_info` at their consumers.
- Remove the write-only `Prompt::personality` field and its constructor
assignments.
- Preserve review-turn inheritance of the parent turn's shell policy,
Linux sandbox executable, and compact prompt through the review config.
## Testing
- `cargo check -p codex-core --tests`
## Description
This PR adds an optional `metadata` field to `ResponseItem` for
Responses API calls. Only mechanical plumbing, no actual values
populated and sent yet. Turns out just adding a new field to
`ResponseItem` has quite a large blast radius already.
This change is backwards compatible because `metadata` is optional and
omitted when absent, so existing response items and rollout history
without it still deserialize and requests that do not set it keep the
same wire shape. For provider compatibility, we strip out `metadata`
before non-OpenAI Responses requests so Azure and AWS Bedrock never see
this field.
My followup PR here will actually make use of it to start storing and
passing along `turn_id`: https://github.com/openai/codex/pull/28360
## What changed
- Added `ResponseItemMetadata` with optional `turn_id`, plus optional
`metadata` on Responses API item variants and inter-agent communication.
- Preserved item metadata through response-item rewrites such as
truncation, missing tool-output synthesis, compaction history
rebuilding, visible-history conversion, rollout/resume, and generated
app-server schemas/types.
- Strip item metadata from non-OpenAI Responses requests while
preserving it for OpenAI-shaped requests.
- Updated the mechanical fixture/test construction churn required by the
new optional field.
## What
Introduce a `CodexResponsesMetadata` struct that defines all the core
metadata we send to Responses API. Example fields are `thread_id`,
`turn_id`, `window_id`, etc.
Going forward, `client_metadata["x-codex-turn-metadata"]` will be the
canonical way Codex sends metadata to Responses API across both HTTP and
websocket transports.
For now, we continue to emit the existing top-level HTTP headers and
top-level `client_metadata` fields from the same
`CodexResponsesMetadata` struct for compatibility reasons.
Also, app-server clients who specify additional
`responsesapi_client_metadata` via `turn/start` and `turn/steer` will
have those fields merged into
`client_metadata["x-codex-turn-metadata"]`, but cannot override the
reserved fields that core uses (i.e. the fields in
`CodexResponsesMetadata`).
## Why
Responses API request instrumentation is the source of truth for
downstream Codex analytics that join requests by Codex IDs such as
session, thread, turn, and context window. Before this change, those
values were assembled through several request-specific paths: HTTP
request bodies, websocket handshake headers, websocket `response.create`
payloads, compaction requests, and the rich `x-codex-turn-metadata`
envelope all had their own wiring.
That made metadata propagation easy to drift across API-key/direct
Responses API requests, ChatGPT-auth/proxied requests, websocket
requests, and compaction requests. It also made additions like
`window_id` error-prone because a field could be added to one transport
projection but missed in another.
## What changed
- Added `CodexResponsesMetadata` as the core-owned snapshot for Codex
metadata sent to ResponsesAPI.
- Render `client_metadata["x-codex-turn-metadata"]`, flat
`client_metadata` projections, and direct compatibility headers from
that same snapshot.
- Include the known Codex-owned fields in the turn metadata blob,
including installation/session/thread/turn/window IDs, request kind,
lineage, sandbox/workspace metadata, timing, and compaction details.
- Treat app-server `responsesapi_client_metadata` as enrichment for the
Codex turn metadata blob while preventing those extras from overriding
Codex-owned fields.
- Use the same metadata path for normal turns, websocket prewarm, local
compaction, remote v1 compaction, and remote v2 compaction.
- Keep websocket connection-only preconnect metadata separate so
handshakes carry compatibility identity headers without inventing a fake
turn metadata blob.
## Verification
- `cargo check -p codex-core`
- `just fix -p codex-core`
## Summary
- replace raw compaction `error` analytics with `codex_error_kind` and
`codex_error_http_status_code`
- derive compaction error telemetry from `CodexErr` using the same
`CodexErrKind` mapping and HTTP status helper used by turn events
- remove the pre-compact hook stop reason from the internal compaction
outcome now that it is no longer emitted as raw analytics text
## Why
Compaction `error` was a raw `CodexErr::to_string()` value, which can
carry free-form provider or user-derived text. Structured Codex error
fields preserve useful low-cardinality telemetry without sending the raw
string.
## Validation
- `just fmt`
- `just test -p codex-analytics`
- `just test -p codex-core
compact::tests::build_token_limited_compacted_history_appends_summary_message`
Attempted `just test -p codex-core`; the changed crate compiled, but the
full target failed in unrelated environment-dependent tests such as
missing helper binaries and shell snapshot timeouts.
## Summary
- add nullable `cached_input_tokens` to the compaction analytics event
- populate it from response usage for compaction v2
- leave it `null` for other compaction implementations
This adds visibility into prompt-cache usage for v2 compaction without
changing compaction behavior.
## Testing
- `just test -p codex-analytics`
- `just test -p codex-core
collect_compaction_output_accepts_additional_output_items`
## Why
The model should be able to see bounded context-window budget metadata
when the `token_budget` feature is enabled. The full-window message is
only injected with full context, while normal turns get a smaller
follow-up only when reported usage first crosses a budget threshold.
## What changed
- Added the `TokenBudget` feature flag.
- Added `<token_budget>` developer fragments for full context-window
metadata and current-window remaining tokens.
- Inserted the threshold message during normal turn handling by
comparing token usage before and after sampling, avoiding persistent
threshold bookkeeping.
- Added core integration coverage for full-context-only metadata and
25/50/75 percent threshold messages.
## Verification
- `just test -p codex-core token_budget`
- `git diff --check`
## Why
Compaction window identity is part of session history, not model-client
transport state. Persisting it with the compacted rollout item lets
resumed threads continue from the reconstructed window without keeping
mutable window state on `ModelClient`.
## What changed
- Added `window_id` to `CompactedItem` and stamp it when
`replace_compacted_history` installs compacted history.
- Moved auto-compact window id ownership into `AutoCompactWindow` /
`SessionState`; `ModelClient` now receives the request window id from
callers instead of storing it.
- Returned `window_id` from rollout reconstruction for resume.
Reconstruction uses the newest surviving compacted item's stored
`window_id` when present, and falls back to the legacy compacted-item
count when it is absent.
- Kept fork startup at the fresh default window id and updated direct
model-client tests to pass explicit test window ids.
## Validation
- `cargo check -p codex-core --tests`
## Why
`log_remote_compact_failure` was the only consumer of the
compact-request logging payload and most of the token-usage breakdown
fields. Once that failure log is removed, keeping the surrounding
carrier types leaves dead plumbing in the compaction path and context
manager.
## What changed
- Remove `log_remote_compact_failure`, `CompactRequestLogData`, and the
v2 wrapper that only fed that log.
- Let both remote compaction implementations return the original
compaction error directly.
- Replace `TotalTokenUsageBreakdown` with a narrow helper that returns
only the remaining value needed by compaction analytics.
- Keep `estimate_response_item_model_visible_bytes` private to the
context manager implementation.
## Validation
- `cargo check -p codex-core`
## Why
Compaction analytics adds retained image count and compaction summary
output tokens for v1.5 specifically.
## What changed
- Add nullable `retained_image_count` and `compaction_summary_tokens`
fields to `codex_compaction_event`.
- Populate them only for `responses_compaction_v2`: retained images come
from the retained v2 compacted history, and summary tokens come from
`response.completed.token_usage.output_tokens`.
- Leave local and legacy remote compaction events as `null` for these
detail fields.
## Verification
- `just fmt`
- `just fix -p codex-core`
- `just test -p codex-core
build_v2_compacted_history_counts_retained_input_images`
- `git diff --check`
## Summary
- accept non-empty model-defined reasoning effort values while
preserving built-in effort behavior
- propagate the non-Copy effort type through core, app-server, TUI,
telemetry, and persistence call sites
- preserve string wire encoding and expose an open-string schema for
clients
- update model selection and shortcut behavior for model-advertised
effort values
## Root cause
`ReasoningEffort` gained a string-backed custom variant, so it could no
longer implement `Copy` or rely on derived closed-enum serialization.
Existing consumers still moved effort values from shared references and
assumed a fixed built-in value set.
## Validation
- `just fmt`
- Local tests and compilation were not run per request; relying on CI.
## Why
The Responses websocket client no longer needs to send a follow-up
`response.processed` request after a turn response has already been
recorded. Keeping that extra acknowledgement path adds feature-gated
control flow and a second websocket request shape that no longer carries
useful behavior.
## What Changed
- Removed the `response.processed` websocket request type and sender.
- Removed the `responses_websocket_response_processed` feature flag and
schema entry.
- Removed turn and remote-compaction plumbing that only tracked response
IDs to send the acknowledgement.
- Removed tests that existed solely to cover the deleted feature path.
## Validation
- `just fix -p codex-core -p codex-api -p codex-features`
## Why
Compaction analytics need token counts that better represent the request
being compacted. The existing session snapshot can diverge from the
actual remote compaction request after output rewriting, and remote v2
can use server-side Responses usage when available.
## What changed
- Add an optional `active_context_tokens_before` override to
`CompactionAnalyticsAttempt::track(...)` for remote compaction when it
has a better before-token value than the begin-time session snapshot.
The local `/compact` path passes no override.
- For remote v1 `responses_compact`, subtract the estimated token delta
from pre-compaction output rewriting from the session snapshot, capped
by locally-added tokens since the last successful API response.
- For remote v2 `responses_compaction_v2`, use the same bounded
output-rewrite fallback as remote v1, then overwrite
`active_context_tokens_before` with server `token_usage.input_tokens`
from the `response.completed` event when present.
- Keep the existing v2 compaction-output validation while carrying the
completed response token usage through `collect_compaction_output`.
## Verification
- `just fmt`
- `just test -p codex-core
collect_compaction_output_accepts_additional_output_items`
- `git diff --check`
## Why
When trying to fit history under compaction limit rewrite output items
instead of removing them entirely. Otherwise we're breaking
incrementality in relation to the previous response.
## Summary
- add analytics-only `CodexErr` telemetry to `codex_turn_event` while
leaving existing `turn_error` unchanged
- record terminal `CodexErr` facts from core immediately before the
existing turn error event is sent
- emit source-truth `codex_error_*` fields for downstream analytics,
including the raw `CodexErr::InvalidRequest(String)` message as
`codex_error_subreason`
## Validation
- `just test -p codex-analytics`
- attempted `just test -p codex-core`, but the local run timed out
across unrelated integration suites in this environment and is not being
used as validation
## Summary
- Add `request_kind` values for foreground turn, startup prewarm,
compaction, and detached memory model requests.
- Attach compaction dispatch metadata to local Responses, legacy
`/v1/responses/compact`, and remote v2 compact requests.
- Add the existing logical context-window identifier as `window_id` on
turn-owned model request metadata.
- Keep identity fields optional for detached memory requests, while
still emitting `request_kind="memory"` in non-git/no-sandbox workspaces.
## Root Cause
`x-codex-turn-metadata` has more than one producer. Foreground turns and
compaction requests own a real turn and should carry that turn identity.
Detached memory stage-one requests do not own a foreground turn, so
absent identity fields are valid rather than missing data. Startup
websocket prewarm is also a model request, but it has `generate=false`
and must not be counted as a foreground turn.
`thread_source` or session source identifies where a thread came from
(for example review, guardian, or another subagent). `request_kind`
identifies what the current outbound model request is doing (`turn`,
`prewarm`, `compaction`, or `memory`). A review or guardian thread can
issue either a normal turn request or a compaction request, so source
cannot replace request kind.
## Behavior / Impact
- Ordinary foreground requests send `request_kind="turn"`, their real
identity fields, and `window_id="<thread_id>:<window_generation>"`.
- Startup websocket warmup requests send `request_kind="prewarm"` so
they are not counted as foreground turns.
- Compaction requests send `request_kind="compaction"`, their real
owning turn identity, the existing `window_id`, and
`compaction.{trigger,reason,implementation,phase,strategy}`.
- Detached memory stage-one requests send `request_kind="memory"`
without `session_id`, `thread_id`, `turn_id`, or `window_id`; when no
workspace metadata exists, the kind-only header is still emitted.
- `session_id`, `thread_id`, `turn_id`, and `window_id` remain optional
in the header schema because detached memory requests do not own a
foreground turn or context window.
- `window_id` is not a new ID system: it is copied from the already-sent
`x-codex-window-id` / WS client metadata value at model-request dispatch
time.
- Existing `x-codex-window-id` HTTP/WS emission, value format,
generation advancement, resume behavior, and fork reset behavior are
unchanged.
- `request_kind`, `window_id`, and upstream turn-owned identity fields
remain schema-owned; input `responsesapi_client_metadata` cannot replace
their canonical values.
- No table, DAG, export, app-server API, or MCP `_meta` schema changes
are included.
A compaction attempt stopped by a pre-compact hook issues no model
request and therefore has no request header; its outcome remains in
analytics events. Status, error, duration, and token deltas also remain
analytics fields rather than request-header fields.
Future detached-memory attribution using a real initiating turn ID as
`trigger_turn_id` is intentionally not part of this PR.
## Sync With Main
- Final pushed head `716342e79` is rebased onto `origin/main@0d37db4b2`.
- The metadata conflict came from upstream `#24160`, which added
`forked_from_thread_id` on the same `turn_metadata` surface. Resolution
preserves that field and its protection from client metadata override
alongside this PR's request-kind, compaction, and window-id fields.
- While resolving the overlapping commits, I removed an accidental
recursive model-request overlay and a duplicate detached-memory header
builder before completing the rebase.
## Latency / User Experience Boundary
- Foreground turns perform no new filesystem, git, or network work. New
fields are inserted into metadata already serialized for outgoing
requests.
- Compaction issues the same model/HTTP requests with the same prompt,
model, service tier, and sampling settings; only metadata bytes change.
- Startup prewarm already sent metadata; it is now correctly classified
as `prewarm`.
- Non-git detached memory now sends a small kind-only metadata header
rather than no header.
- This client diff adds no user-visible latency mechanism beyond
negligible serialization and header bytes on already-existing requests.
## Validation
On conflict-resolved head `1d35c2cfb` based on `origin/main@487521733`:
- `just fmt` (passed)
- `just fix -p codex-core` (passed)
- `git diff --check origin/main...HEAD` (passed)
- `just test -p codex-core -E 'test(turn_metadata) |
test(websocket_first_turn_uses_startup_prewarm_and_create) |
test(responses_stream_includes_turn_metadata_header_for_git_workspace_e2e)
|
test(responses_websocket_forwards_turn_metadata_on_initial_and_incremental_create)
| test(remote_compact_v2_retries_failures_with_stream_retry_budget) |
test(window_id_advances_after_compact_persists_on_resume_and_resets_on_fork)'`
(`23 passed`; `bench-smoke` passed)
- `just test -p codex-app-server -E
'test(turn_start_forwards_client_metadata_to_responses_request_v2) |
test(turn_start_forwards_client_metadata_to_responses_websocket_request_body_v2)
| test(auto_compaction_remote_emits_started_and_completed_items)'` (`3
passed`; `bench-smoke` passed)
- `just test -p codex-memories-write` (`29 passed`; `bench-smoke`
passed)
## Why
#23951 added remote compaction v2 retries, but it left the retry and WS
-> HTTPS fallback behavior duplicated between normal Responses turns and
compaction. This follow-up centralizes the common retry handling so
future changes to fallback, retry delay, retry notifications, and retry
sleep do not have to be kept in sync across both callsites.
## What changed
- Added `core/src/responses_retry.rs` with a shared handler for
retryable Responses stream errors.
- Reused that handler from normal turn sampling and remote compaction
v2.
- Kept each callsite responsible for its retry budget: normal turns
still use `stream_max_retries`, while compaction v2 still uses
`min(stream_max_retries, 2)`.
- Preserved caller-specific behavior around non-retryable errors,
context-window errors, usage-limit errors, and compact-specific final
failure logging.
The shared handler now owns:
- WS -> HTTPS fallback warning emission
- retry delay selection, including server-requested stream retry delay
- retry logging
- first-WebSocket-retry notification suppression
- `Reconnecting... n/max` stream-error notification
- sleeping before the next retry attempt
## Verification
- `cargo test -p codex-core remote_compact_v2`
- `cargo test -p codex-core websocket_fallback`
- `just fix -p codex-core`
Did not run the full workspace test suite.
---------
Co-authored-by: jif-oai <jif@openai.com>
## What changed
- Add a distinct `responses_compaction_v2` value for
`CodexCompactionEvent.implementation`.
- Emit that value from the remote compaction v2 path.
- Keep local compaction as `responses` and legacy `/responses/compact`
as `responses_compact`.
## Why
Remote compaction v2 and local prompt-based compaction were both
reported as `responses`, which made the analytics table collapse two
different compaction mechanisms into one implementation bucket.
## Validation
- `just fmt`
- `just test -p codex-analytics`
`just test -p codex-core` was started locally, but this PR is
intentionally being pushed for CI to finish the remaining validation.
## Why
[Recent PR](https://github.com/openai/codex/pull/22709) removed
`trace_id` from `TurnContextItem`.
## What changed
- Add to `TurnStartedEvent` so rollout consumers can correlate turns
with telemetry traces.
- Note that the branch name is out of date because I originally re-added
to `TurnContextItem`, but we decided to move it to `TurnStartedEvent`.
## Verification
- `cargo test -p codex-protocol`
- `cargo test -p codex-core --lib
regular_turn_emits_turn_started_without_waiting_for_startup_prewarm`
- `cargo test -p codex-core --test all
emits_warning_when_resumed_model_differs`
- `cargo test -p codex-rollout`
- `cargo test -p codex-state`
## Why
Remote compaction v2 sends a normal `/responses` request with a
compaction trigger. It should follow the retry semantics used by normal
Responses streaming calls for transient stream/request failures, while
keeping a smaller per-transport retry budget because compact attempts
can run much longer than normal turns.
## What changed
- Add a v2 compaction retry loop that uses `stream_max_retries`,
matching normal Responses turn retry mechanics.
- Cap the compact v2 retry budget at 2 retries per transport with
`min(stream_max_retries, 2)`.
- Retry retryable request-open and post-open stream collection failures
through the same loop.
- Use the existing 200ms exponential backoff and requested retry delay
handling used by normal turn retries.
- Emit the same `Reconnecting... n/max` stream-error notification
pattern.
- Fall back from WebSockets to HTTPS after the compact v2 stream retry
budget is exhausted, then reset the retry counter for HTTPS.
- Keep final remote-compaction failure logging after retries/fallback
are exhausted.
- Treat compact stream EOF before `response.completed` as a retryable
stream failure.
- Add compact v2 regression coverage with `request_max_retries = 0` and
`stream_max_retries = 2`, covering both request-open failure and
opened-stream EOF in one end-to-end test.
## Tests
- `just fmt`
- `cargo test -p codex-core remote_compact_v2`
- `just fix -p codex-core`
## Why
Remote compaction now has two implementations: the existing
server-rebuilt v1 path and the newer client-rebuilt v2 path behind
`remote_compaction_v2`. The v1 path bounds retained
user/developer/system history before installing the compaction item,
while v2 was previously carrying the full retained history forward. That
made the two paths diverge for large pre-compaction transcripts even
though they are meant to preserve the same compaction contract.
This aligns v2 with the retained-history budget expected from v1 so
switching the feature flag does not materially change which
pre-compaction messages survive into the rebuilt history.
## What changed
- Apply a retained-message character budget while rebuilding v2
compacted history in `core/src/compact_remote_v2.rs`.
- Keep newest retained messages first, truncate the boundary message
with the shared `truncate_text(...)` helper, and drop older retained
messages once the budget is exhausted.
- Preserve non-text retained message content such as images while
truncating text content.
- Use the current `64_000` token retained-message default translated to
the existing `4x` character budget.
## Testing
- `cargo test -p codex-core compact_remote_v2::tests::`
- Added focused coverage for newest-first retention and truncating
multipart retained messages without dropping images.
## Why
Remote compaction v2 is the `/responses` implementation of
session-history compaction, but it still needs to preserve the
observable contract of the legacy `/responses/compact` path. In
particular, users and integrations that rely on `PreCompact` and
`PostCompact` hooks should not see different behavior when
`remote_compaction_v2` is enabled.
## What Changed
- Runs `PreCompact` before issuing the remote compaction v2 request,
including `Interrupted` analytics when a pre-hook stops execution.
- Runs `PostCompact` after a successful v2 compaction and aborts the
turn if the post-hook stops execution.
- Adds `compact_remote_parity` coverage that compares legacy and v2
compaction across manual transcript shapes, automatic pre-turn
compaction, automatic mid-turn compaction, hook payloads, replacement
history, follow-up request payloads, and API-key `service_tier=fast`
behavior.
- Registers the new parity suite under `core/tests/suite`.
Relevant code:
-
[`compact_remote_v2.rs`](https://github.com/openai/codex/blob/af63745cb502183a6fc447d0240f8150934d70b7/codex-rs/core/src/compact_remote_v2.rs)
-
[`compact_remote_parity.rs`](https://github.com/openai/codex/blob/af63745cb502183a6fc447d0240f8150934d70b7/codex-rs/core/tests/suite/compact_remote_parity.rs)
## Verification
- Added `core/tests/suite/compact_remote_parity.rs` to assert parity
between legacy remote compaction and remote compaction v2 for the
affected request, hook, rollout-history, and follow-up paths.
- Existing `compact_remote_v2` unit coverage still exercises v2
replacement-history retention and compaction-output collection.
## Why
Remote compaction v2 was still using `context_compaction` as both the
request trigger and the compacted output shape. The Responses API now
has the landed contract for this flow: Codex sends a dedicated `{
"type": "compaction_trigger" }` input item, and the backend returns the
standard `compaction` output item with encrypted content.
This aligns the v2 path with that wire contract while preserving the
existing local compacted-history post-processing behavior.
## What changed
- Add `ResponseItem::CompactionTrigger` and regenerate the app-server
protocol schema fixtures.
- Send `compaction_trigger` from `remote_compaction_v2` instead of a
payload-less `context_compaction`.
- Collect exactly one backend `compaction` output item, then reuse the
existing compacted-history rebuilding path.
- Treat the trigger item as a transient request marker rather than model
output or persisted rollout/memory content.
## Verification
- `cargo test -p codex-protocol compaction_trigger`
- `cargo test -p codex-core remote_compact_v2`
- `cargo test -p codex-core compact_remote_v2`
- `cargo test -p codex-core
responses_websocket_sends_response_processed_after_remote_compaction_v2`
- `just write-app-server-schema`
- `cargo test -p codex-app-server-protocol schema_fixtures`
## Why
Recent session history showed no active use of the raw `shell`,
`local_shell`, or `container.exec` execution surfaces. Keeping those
handlers/specs wired into core leaves duplicate shell execution paths
alongside the supported `shell_command` and unified exec tools.
## What changed
- Removed the raw `shell` handler/spec and its `ShellToolCallParams`
protocol helper.
- Removed the legacy `local_shell` and `container.exec` handler/spec
plumbing while preserving persisted-history compatibility for old
response items.
- Normalized model/config `default` and `local` shell selections to
`shell_command`.
- Pruned tests that exercised removed raw-shell/local-shell/apply-patch
variants and kept coverage on `shell_command`, unified exec, and
freeform `apply_patch`.
## Verification
- `git diff --check`
- `cargo test -p codex-protocol`
- `cargo test -p codex-tools`
- `cargo test -p codex-core tools::handlers::shell`
- `cargo test -p codex-core tools::spec`
- `cargo test -p codex-core tools::router`
- `cargo test -p codex-core
active_call_preserves_triggering_command_context`
- `cargo test -p codex-core guardian_tests`
- `cargo test -p codex-core --test all shell_serialization`
- `cargo test -p codex-core --test all apply_patch_cli`
- `cargo test -p codex-core --test all shell_command_`
- `cargo test -p codex-core --test all local_shell`
- `cargo test -p codex-core --test all otel::`
- `cargo test -p codex-core --test all hooks::`
- `just fix -p codex-core`
- `just fix -p codex-tools`
## Why
Remote compaction v2 consumes a normal Responses stream, but that
compaction-specific stream consumer dropped the `response.completed` id.
As a result, the `responses_websocket_response_processed` lifecycle
notification was emitted for normal turn sampling but not after a v2
remote compaction response was fully processed.
## What changed
- Return the completed response id alongside the v2 `context_compaction`
output item.
- After v2 compacted history is installed, send `response.processed`
through the same websocket session when the feature is enabled.
- Add websocket regression coverage for a remote compaction v2 request
followed by `response.processed`.
## Verification
- `cargo test -p codex-core --test all
responses_websocket_sends_response_processed_after_remote_compaction_v2
-- --nocapture`
- `cargo test -p codex-core
collect_context_compaction_output_accepts_additional_output_items --
--nocapture`
## Why
This adds the `remote_compaction_v2` client path so remote compaction
can run through the normal Responses stream and install a
`context_compaction` item that trigger a compaction.
The goal is to migrate some of the compaction logic on the client side
We keeps the v2 transport behind a feature flag while letting follow-up
requests reuse the compacted context instead of falling back to the
legacy compaction item shape.
## What changed
- add `ResponseItem::ContextCompaction` and refresh the generated
app-server / schema / TypeScript fixtures that expose response items on
the wire
- add `core/src/compact_remote_v2.rs` to send compaction through the
standard streamed Responses client, require exactly one
`context_compaction` output item, and install that item into compacted
history
- route manual compact and auto-compaction through the v2 path when
`remote_compaction_v2` is enabled, while keeping the existing remote
compaction path as the fallback
- preserve the new item type across history retention, follow-up request
construction, telemetry, rollout persistence, and rollout-trace
normalization
- add targeted coverage for the feature flag, `context_compaction`
serialization, rollout-trace normalization, and remote-compaction
follow-up behavior
## Verification
- added protocol tests for `context_compaction`
serialization/deserialization in `protocol/src/models.rs`
- added rollout-trace coverage for `context_compaction` normalization in
`rollout-trace/src/reducer/conversation_tests.rs`
- added remote compaction integration coverage for v2 follow-up reuse
and mixed compaction output streams in
`core/tests/suite/compact_remote.rs`
---------
Co-authored-by: Codex <noreply@openai.com>
Charlie Marsh