## Why
Enabled rollout tracing could silently fail to initialize, and a terminal task status did not establish whether all recording had finished. Trace consumers need to distinguish a complete recording from a partial bundle, including when a task is interrupted.
## What changed
- Fail thread startup or resume if an enabled recorder cannot initialize, while cleaning up session resources.
- Stop recording after the first write or serialization failure and preserve that failure across shared producers.
- Append a closing record when the last writer handle is released without recording errors. Expose `recording_complete` in `state.json` independently of task success, and mark submission-channel closure as an aborted rollout.
- Reject replay logs with missing or reordered events, mismatched rollout IDs, events after completion, or missing or invalid JSON payloads. Valid prefixes remain replayable with `recording_complete` set to `false`.
## Testing
Add coverage for startup failure and recovery, shared-writer failures, finalization after child handles close, incomplete logs, and missing or malformed payloads. Extend session and CLI tests to check recording completion after shutdown, interruption, stream failure, and submission-channel closure.
GitOrigin-RevId: 30d62fa95912fb0442ad1199776d110884066c30
## Why
Repeated image outputs in different request payloads could fail replay with a call ID reuse error even when their content was unchanged.
## What changed
- Hash canonical JSON for non-text content parts and compare their `content_sha256` values independently of payload location.
- Preserve empty content arrays as empty bodies so they also match across snapshots.
## Testing
Add a regression test verifying that repeated image and empty tool outputs reuse conversation items, while changed image content with the same call and item IDs still fails replay.
GitOrigin-RevId: 07699312675764079f679fd2c12ba6679332d709
## Why
Model output alone does not establish that an item entered conversation history. Treating every response item as accepted can include discarded attempts, while relying on a later request misses accepted final output when a turn ends or is interrupted.
## What changed
- Record `HistoryItemAccepted` events and retain Responses item IDs, generating missing IDs consistently across item events.
- Build thread conversation history from request inputs, compaction checkpoints, and explicit acceptance. Keep unaccepted output accessible through its inference call.
- Preserve inference provenance across retries and late response records without overwriting newer request or compaction snapshots.
- Mark turns with errors as failed. Bump trace schemas to version 2 and reject incompatible bundles and event schemas during replay.
## Testing
Add reducer coverage for discarded attempts with identical text, late responses, compaction, inherited context, and legacy bundle rejection. Add CLI integration tests that run `debug trace-reduce` after success, stream failure, and interruption followed by shutdown, including responses without provider item IDs.
GitOrigin-RevId: 8e387a91d9b3f704df82847d630e6f0b08f998b3
## Why
Runtime process handles are recyclable and thread-local. Using them as terminal session identities can merge separate commands or associate `write_stdin` operations with the wrong exec.
## What changed
- Give each exec a distinct terminal session and track the latest session for each thread-local process handle.
- Record the exec selected by `write_stdin` at runtime and use it to correct session membership during replay, including when dispatch is delayed until a handle is reused.
- Fall back to the latest session for the thread and handle when replaying older traces without runtime target events.
## Testing
Extend reducer coverage for process ID reuse across threads and successive execs, stdin session membership, and delayed runtime target resolution.
GitOrigin-RevId: 330c6fd26a4eebdef03ae600a030dc76920e0dd9
## Why
Reducing code cell lifecycle events discarded their response payload references and lost the initial response status when the runtime status changed.
## What changed
Store the initial response status and separate raw payload IDs for the initial response and final result on `CodeCell`. Carry payload references through queued lifecycle events so they survive replay when the source item arrives later.
## Testing
Extend lifecycle tests to verify distinct initial and final payload IDs and the preserved initial status for yielded cells that complete and failed cells whose lifecycle events precede their source item.
GitOrigin-RevId: 8f49a3174ea487a5d91ebc2a6c97133aacaabfd2
## Why
MCP callable namespaces and tool names can differ from their backend identities or match built-in tool names. Classifying calls by name can misidentify MCP calls and incorrectly parse namespaced `write_stdin` calls as terminal operations.
## What changed
Resolve `ToolCallKind::Mcp` from the runtime invocation's server and tool as soon as the runtime starts, before MCP correlation assignment. Preserve the caller-facing summary and raw invocation, and skip built-in runtime handling for MCP calls.
Only create dispatch-based `write_stdin` terminal operations for calls without a namespace.
## Testing
Add replay regression coverage for prefixed and unprefixed MCP aliases, names that match built-in tools, and `write_stdin` arguments with and without a session ID. Verify backend classification, preserved callable identity and dispatch lifecycle, and the absence of terminal operations and sessions.
GitOrigin-RevId: 71f31335c1eb36b6b7e1a34383d10fc500bd2c4f
## What changed
- Parse `cache_write_tokens` from response input token details and carry it through token usage aggregation.
- Expose `cache_write_input_tokens` in protocol, app-server, exec, and TypeScript SDK usage events, defaulting it to zero for compatibility with older payloads.
- Report cache-write usage in analytics, metrics, traces, and rollout data alongside existing cache-read usage.
## Testing
- Add coverage for response parsing, usage event output, aggregation, analytics, metrics, and telemetry fields.
GitOrigin-RevId: 634e4fbe3086a8923c0ee13b622573e7d9d221bd
## Why
Multi-agent v2 `send_message` deliveries already reach the receiving
model as typed `agent_message` items with encrypted content.
Child-completion notifications are generated by Codex itself, so their
content is plaintext and previously fell back to a serialized JSON
envelope inside an assistant message.
With plaintext `input_text` supported for `agent_message`, both delivery
paths can use the same model-visible type while preserving explicit
author and recipient metadata.
## What changed
- add plaintext `input_text` support to `AgentMessageInputContent` and
regenerate the affected app-server schemas
- preserve `InterAgentCommunication` as structured mailbox input instead
of converting it to assistant text
- record delivered communications as typed `agent_message` history items
- persist a dedicated rollout item so local delivery metadata such as
`trigger_turn` remains available without leaking into the Responses
request
- reconstruct typed agent messages on resume and preserve fork-turn
truncation behavior
- remove request-time assistant-content parsing
- preserve plaintext and encrypted inter-agent deliveries in stage-one
memory inputs
- normalize and link plaintext and encrypted agent messages in rollout
traces without treating inbound messages as child results
- cover the real MultiAgent V2 child-completion path end to end with
deterministic mailbox synchronization
## Verification
- `just test -p codex-core
plaintext_multi_agent_v2_completion_sends_agent_message`
- `just test -p codex-core input_queue_drains_mailbox_in_delivery_order
record_initial_history_reconstructs_typed_inter_agent_message
fork_turn_positions_use_inter_agent_delivery_metadata`
- `just test -p codex-memories-write
serializes_inter_agent_communications_for_memory`
- `just test -p codex-rollout-trace
agent_messages_preserve_routing_and_content
sub_agent_started_activity_creates_spawn_edge`
- `just test -p codex-rollout-trace
agent_result_edge_falls_back_to_child_thread_without_result_message`
- `just test -p codex-protocol -p codex-rollout -p
codex-app-server-protocol`
## Why
Rollout traces need an identifier that can be used to correlate a Codex
inference with upstream Responses API, proxy, and engine logs. The
reduced trace model already exposed `upstream_request_id`, but it was
being populated from the Responses API `response.id`. That value is
useful for `previous_response_id` chaining, but it is not the transport
request id that upstream systems key on.
This PR separates those concepts so trace consumers can reliably answer
both questions:
- which Responses API response did this inference produce?
- which upstream request handled it?
## Structure
The change keeps the upstream request id at the same lifecycle level as
the provider stream:
- `codex-api` captures the `x-request-id` HTTP response header when the
SSE stream is created and exposes it on `ResponseStream`. Fixture and
websocket streams set the field to `None` because they do not have that
HTTP response header.
- `codex-core` carries that stream-level id into `InferenceTraceAttempt`
when recording terminal stream outcomes. Completed, failed, cancelled,
dropped-stream, and pre-response error paths all record the id when it
is available.
- `rollout-trace` now records both identifiers in raw terminal inference
events and response payloads: `response_id` for the Responses API
`response.id`, and `upstream_request_id` for `x-request-id`.
- The reducer stores both fields on `InferenceCall`. It also uses
`response_id` for `previous_response_id` conversation linking, which
removes the old accidental dependency on the misnamed
`upstream_request_id` field.
- Terminal inference reduction now consumes the full terminal payload
(`InferenceCompleted`, `InferenceFailed`, or `InferenceCancelled`) in
one place. That keeps status, partial payloads, response ids, and
upstream request ids consistent across success, failure, cancellation,
and late stream-mapper events.
## Why This Shape
`x-request-id` is a property of the HTTP/provider response envelope, not
an SSE event. Capturing it once in `codex-api` and plumbing it through
terminal trace recording avoids trying to infer the value from stream
contents, and it preserves the id even when the stream fails or is
cancelled after only partial output.
Keeping `response_id` separate from `upstream_request_id` also makes the
reduced trace model less surprising: `response_id` remains the
conversation-continuation id, while `upstream_request_id` is the
operational correlation id for upstream debugging.
## Validation
The PR updates trace and reducer coverage for:
- reading `x-request-id` from SSE response headers;
- storing the true upstream request id on completed inference calls;
- preserving upstream request ids for cancelled and late-cancelled
inference streams;
- keeping `previous_response_id` reconstruction tied to `response_id`
rather than transport request ids.
## Summary
Adds the standalone `codex-rollout-trace` crate, which defines the raw
trace event format, replay/reduction model, writer, and reducer logic
for reconstructing model-visible conversation/runtime state from
recorded rollout data.
The crate-level design is documented in
[`codex-rs/rollout-trace/README.md`](https://github.com/openai/codex/blob/codex/rollout-trace-crate/codex-rs/rollout-trace/README.md).
## Stack
This is PR 1/5 in the rollout trace stack.
- [#18876](https://github.com/openai/codex/pull/18876): Add rollout
trace crate
- [#18877](https://github.com/openai/codex/pull/18877): Record core
session rollout traces
- [#18878](https://github.com/openai/codex/pull/18878): Trace tool and
code-mode boundaries
- [#18879](https://github.com/openai/codex/pull/18879): Trace sessions
and multi-agent edges
- [#18880](https://github.com/openai/codex/pull/18880): Add debug trace
reduction command
## Review Notes
This PR intentionally does not wire tracing into live Codex execution.
It establishes the data model and reducer contract first, with
crate-local tests covering conversation reconstruction, compaction
boundaries, tool/session edges, and code-cell lifecycle reduction. Later
PRs emit into this model.
The README is the best entry point for reviewing the intended trace
format and reduction semantics before diving into the reducer modules.
cassirer-openai