Files
codex/codex-rs/rollout-trace
cassirer-openai ef6ce8100a Bound rollout trace capture with configurable storage limits (#61896)
## What changed

- Add `rollout_trace.max_payload_bytes` and `rollout_trace.max_bytes`, defaulting to 64 MiB per body or separated value and 256 MiB per bundle. Root and child threads share the bundle budget.
- Record explicit truncations and omissions when limits are reached, reserve space for later events, and stop at the last complete event if the event log fills.
- Reuse identical prepared-prompt items and store inline images separately from message structure. Preserve exact JSON bytes for complete composite payloads and surviving content and links in partial captures.
- Let reducers handle partial payloads while retaining lifecycle information and reporting `closed_with_partial_payloads` for closed bundles with incomplete captures.

## Testing

Add coverage for payload reuse and byte-exact reconstruction, truncation, compressed bundle budgets, later events after omissions, readable prefixes after event-space exhaustion, and CLI capture-limit configuration and startup failures.

GitOrigin-RevId: be99a68da427414a7a12bf8209f232c1a0bbe545
2026-11-16 22:26:31 +00:00
..

Rollout Trace

Privacy: Rollout tracing is not telemetry. Codex does not upload or report these traces; it writes local bundles only when CODEX_ROLLOUT_TRACE_ROOT is set. Those local bundles can contain prompts, responses, tool inputs/outputs, terminal output, and paths, so treat them as sensitive.

Rollout tracing is an opt-in diagnostic path for understanding what happened during a Codex session. It records raw runtime evidence into a local bundle on disk, then replays that bundle into a semantic graph that a debugger or UI can inspect.

Same-binary replay only. The exact Codex executable that recorded a bundle must reduce it. Matching release or schema versions do not make a different binary compatible. The raw event log and payload formats are temporary implementation details and can change with the writer and reducer.

Viewers and other consumers read state.json, the reduced trace API. They can use manifest.json for bundle discovery and identity, such as selecting a trace by root_thread_id. Payload references in the reduced state let consumers inspect the original evidence. Every change to the state.json schema bumps REDUCED_TRACE_SCHEMA_VERSION. Consider the impact on existing consumers. Breaking changes are allowed.

The key design choice is: observe first, interpret later.

Hot-path Codex code does not try to build the final graph while the session is running. It writes ordered raw events and payload references. The offline reducer then decides which events became model-visible conversation, which events were runtime work, and how information moved between threads, tools, code cells, and terminal sessions.

What This Gives Us

Rollout traces make failures debuggable when the normal transcript is not enough. They preserve enough evidence to answer questions like:

  • Which model request produced this tool call?
  • Did this output come from the model-visible transcript, a code-mode runtime value, a terminal operation, or an agent notification?
  • Which code-mode exec cell issued a nested tool call?
  • Which terminal operation created or reused a running process?
  • Which multi-agent v2 tool call spawned, messaged, received from, or closed a child thread?

The reduced state.json is intentionally not just a transcript. It is a graph of model-visible conversation plus the runtime objects that explain how Codex got there.

System Shape

flowchart TD
    subgraph Runtime["codex-core runtime"]
        Protocol["protocol lifecycle\nthread start/end, turn start/end"]
        Inference["inference + compaction\nrequests, responses, checkpoints"]
        Tools["tool dispatch\ndirect model tools + code-mode nested tools"]
        CodeMode["code-mode runtime\nexec cells, yields, waits, termination"]
        Terminal["terminal runtime\nexec_command / write_stdin operations"]
        Agents["multi_agent_v2\nspawn, task delivery, result, close"]
    end

    Context["ThreadTraceContext\nroot/child no-op-capable producer"]
    Writer["TraceWriter\nassigns seq and writes payloads before events"]

    subgraph Bundle["trace bundle"]
        Manifest["manifest.json\ntrace_id, rollout_id, root_thread_id"]
        Events["trace.jsonl\nordered raw event spine"]
        Payloads["payloads/*.zst\nlarge raw evidence"]
    end

    Reducer["replay_bundle\ndeterministic offline reducer"]

    subgraph State["state.json"]
        Threads["threads + turns"]
        Conversation["conversation_items\nwhat the model saw"]
        RuntimeObjects["inference_calls, tool_calls,\ncode_cells, terminals, compactions"]
        Edges["interaction_edges\nspawn, task, result, close"]
        RawRefs["raw_payload refs"]
    end

    Protocol --> Context
    Inference --> Context
    Tools --> Context
    CodeMode --> Context
    Terminal --> Context
    Agents --> Context

    Context --> Writer
    Writer --> Manifest
    Writer --> Payloads
    Writer --> Events

    Manifest --> Reducer
    Events --> Reducer
    Payloads --> Reducer

    Reducer --> Threads
    Reducer --> Conversation
    Reducer --> RuntimeObjects
    Reducer --> Edges
    Reducer --> RawRefs

The thread context is deliberately small and no-op capable. A root session starts one from CODEX_ROLLOUT_TRACE_ROOT; fresh spawned child threads derive their own context from the parent's context so the whole rollout tree shares one writer. Disabled contexts accept the same calls and record nothing.

When tracing is enabled, a thread can start or resume only if its recorder initializes successfully. A later write failure stops recording. Partial bundles remain available for replay. The reduced recording_status is unclosed until the closing record arrives. It then becomes complete or closed_with_partial_payloads according to the referenced captures.

Bundle Layout

A trace bundle contains:

  • manifest.json: trace identity and bundle metadata.
  • trace.jsonl: append-only raw events ordered by writer-assigned seq.
  • payloads/*.zst: raw requests, responses, tool inputs/results, runtime events, terminal output, compaction data, and protocol snapshots.
  • state.json: reduced trace for consumers, written by codex debug trace-reduce.

Schema v3 indexes numbered Zstandard shards in state.json under raw_payloads. Prepared prompts share exact input items. Inline images have independent references and shards, so losing an attachment preserves its position, sibling text and tool links. A composite record stores the literal JSON fragments around its ordered parts. Interleaving both reconstructs the original bytes, including whitespace and escapes. Hashes cover each entire value before truncation. PayloadReader::read returns the original bytes for complete bodies. A truncated body records its original and retained lengths and can be inspected with read_prefix.

Configure the capture limits in config.toml. Both values are in bytes.

[rollout_trace]
max_payload_bytes = 67108864  # Default: 64 MiB per new body or separated value
max_bytes = 268435456         # Default: 256 MiB per bundle

The bundle budget covers the manifest, event log, layouts, and compressed payloads. Space is reserved for later events carrying explicit omissions. If even an event cannot fit, the log ends at the preceding complete event. The optional reduced-state cache is generated separately. Reduce a bundle with its recording binary.

For an inference call, raw_request_payload_id resolves to the outgoing HTTP body or WebSocket message, including its previous_response_id and incremental input. prepared_prompt_payload_id resolves to Core's input to formatting, including base instructions and their provenance, ordered history, tools and output settings. For a continuation from an unrecorded WebSocket warmup, the start event also links the full formatted input so the reducer can resolve the missing history.

One worker per bundle records events in submission order. For inference starts, the transport shares its encoded request and Core shares the prepared prompt. The worker serializes the prompt, compresses both bodies, and writes them before publishing the start event. Remote compaction uses the same worker to snapshot the model, instructions, input history, and parallel tool-call setting. Up to eight jobs can wait for the worker. Use TraceWriter::flush to read a live snapshot. Shutdown waits for the worker to finish accepted jobs, seal the shard, and write the closing marker. A recording failure prevents the marker.

Each shard is one Zstandard frame (level 3, 8 MiB window, checksum enabled). Each record contains a little-endian 16-bit ID length, a little-endian 64-bit retained body length, the UTF-8 ID, then the retained bytes. The writer flushes a record before publishing its reference, and rotates before the next record would exceed 8 MiB uncompressed. An oversized record occupies one shard. The final frame is sealed before the writer publishes RecordingCompleted. A live bundle can be reduced through its published records with the last frame still open.

trace_id identifies this diagnostic artifact. rollout_id identifies the Codex rollout/session being observed. Keeping those separate lets us reason about the stored trace without confusing it with the product-level session identity.

Run the recording executable to reduce its bundle:

codex debug trace-reduce <trace-bundle>

By default this writes <trace-bundle>/state.json. Within the recording binary, codex_rollout_trace::replay_bundle returns the same reduced graph in memory.

Raw Evidence vs Reduced Graph

flowchart LR
    Model["model-visible payloads\nrequests and response output items"]
    Runtime["runtime observations\ntool dispatch, terminal output, code-mode JSON"]
    RawPayloads["payloads/*.zst\nexact evidence"]
    Reducer["reducer"]
    Conversation["ConversationItem\nwhat the model saw"]
    ToolCall["ToolCall\nruntime tool boundary"]
    CodeCell["CodeCell\nmodel-authored exec cell"]
    TerminalOperation["TerminalOperation\ncommand/write/poll"]
    InteractionEdge["InteractionEdge\ninformation flow"]

    Model --> RawPayloads
    Runtime --> RawPayloads
    RawPayloads --> Reducer

    Reducer --> Conversation
    Reducer --> ToolCall
    Reducer --> CodeCell
    Reducer --> TerminalOperation
    Reducer --> InteractionEdge

    CodeCell --> ToolCall
    ToolCall --> TerminalOperation
    ToolCall --> InteractionEdge
    Conversation --> InteractionEdge

This distinction is the reason the model has both raw payload references and semantic objects. A code-mode nested tool call, for example, has JSON input and output at the JavaScript runtime boundary, but the model-visible transcript only contains the surrounding exec custom tool call and its eventual output.

The reducer keeps those facts separate:

  • ConversationItem records what appeared in model-facing requests/responses.
  • ToolCall, CodeCell, TerminalOperation, InferenceCall, and Compaction record runtime/debug boundaries.
  • InteractionEdge records information flow between objects, such as a spawn_agent tool call delivering a task into a child thread.
  • RawPayloadRef points back to exact evidence when a viewer needs more detail than the reduced graph stores inline.

Multi-Agent v2

Multi-agent v2 child threads share the root trace writer. That means one root bundle reduces into one graph containing the parent thread, child threads, and the edges between them.

flowchart LR
    RootTool["root ToolCall\nspawn_agent / followup_task / send_message"]
    ChildInput["child ConversationItem\ninjected task/message"]
    ChildThread["child AgentThread"]
    ChildResult["child assistant ConversationItem\nresult message"]
    RootNotice["root ConversationItem\nsubagent notification"]
    CloseTool["root ToolCall\nclose_agent"]
    TargetThread["target AgentThread"]

    RootTool -- "spawn/task edge" --> ChildInput
    ChildInput --> ChildThread
    ChildThread --> ChildResult
    ChildResult -- "agent_result edge" --> RootNotice
    CloseTool -- "close_agent edge" --> TargetThread

Top-level independent threads still get independent bundles. Spawned child threads are different: they are part of the same rollout tree, so they belong in the same raw event log, payload directory, and reduced state.json.

Reducer Invariants

The reducer is strict where the raw evidence should be self-consistent:

  • raw events are replayed in seq order;
  • payload records must be readable before events refer to them;
  • reduced object IDs are stable within one replay;
  • runtime events may be queued until the model-visible source or delivery target has been observed;
  • model-visible conversation is derived from model-facing payloads, not from runtime convenience output;
  • runtime payloads are evidence, not proof that the model saw the same bytes.

Those invariants let the reduced graph stay small while preserving a path back to the original evidence whenever a debugger needs to explain why an object or edge exists.