mirror of
https://github.com/openai/codex.git
synced 2026-11-19 22:30:42 +00:00
## Why Repeated tool output, such as the same image returned in several results, can consume the trace bundle budget. When output bytes are unavailable, the surrounding result can still carry terminal metadata and conversation links. ## What changed - Store JSON string values of at least 64 KiB, measured after encoding, separately from their tool result or runtime event. Identical encoded values share one record per bundle, while each occurrence retains its position and complete captures reconstruct the original JSON bytes. - Keep separate compression shards open for content and surrounding records. Budget for the serialized event and fall back to a single capped or omitted payload when the split layout or its references exceed the applicable limit. - Track missing values separately from literal `null` so reduction can preserve surviving terminal metadata and keep tool output IDs attached to their original items. ## Testing Add regression tests for shared image and text payloads, byte-for-byte reconstruction, recording completion under tight bundle limits, and reduction with missing output. GitOrigin-RevId: a003197ea998639b6c2ae27f408990c0186406bd
296 lines
13 KiB
Markdown
296 lines
13 KiB
Markdown
# Rollout Trace
|
|
|
|
> **Privacy:** Rollout tracing is not telemetry. Codex does **not** upload or
|
|
> report these traces; it writes local bundles only when
|
|
> `CODEX_ROLLOUT_TRACE_ROOT` is set. Those local bundles can contain prompts,
|
|
> responses, tool inputs/outputs, terminal output, and paths, so treat them as
|
|
> sensitive.
|
|
|
|
Rollout tracing is an opt-in diagnostic path for understanding what happened
|
|
during a Codex session. It records raw runtime evidence into a local bundle on
|
|
disk, then replays that bundle into a semantic graph that a debugger or UI can
|
|
inspect.
|
|
|
|
**Same-binary replay only.** The exact Codex executable that recorded a bundle
|
|
must reduce it. Matching release or schema versions do not make a different
|
|
binary compatible. The raw event log and payload formats are temporary
|
|
implementation details and can change with the writer and reducer.
|
|
|
|
Viewers and other consumers read `state.json`, the reduced trace API. They can
|
|
use `manifest.json` for bundle discovery and identity, such as selecting a trace
|
|
by `root_thread_id`. Payload references in the reduced state let consumers
|
|
inspect the original evidence. Every change to the `state.json` schema bumps
|
|
`REDUCED_TRACE_SCHEMA_VERSION`. Consider the impact on existing consumers.
|
|
Breaking changes are allowed.
|
|
|
|
The key design choice is: **observe first, interpret later**.
|
|
|
|
Hot-path Codex code does not try to build the final graph while the session is
|
|
running. It writes ordered raw events and payload references. The offline reducer
|
|
then decides which events became model-visible conversation, which events were
|
|
runtime work, and how information moved between threads, tools, code cells, and
|
|
terminal sessions.
|
|
|
|
## What This Gives Us
|
|
|
|
Rollout traces make failures debuggable when the normal transcript is not enough.
|
|
They preserve enough evidence to answer questions like:
|
|
|
|
- Which model request produced this tool call?
|
|
- Did this output come from the model-visible transcript, a code-mode runtime
|
|
value, a terminal operation, or an agent notification?
|
|
- Which code-mode `exec` cell issued a nested tool call?
|
|
- Which terminal operation created or reused a running process?
|
|
- Which multi-agent v2 tool call spawned, messaged, received from, or closed a
|
|
child thread?
|
|
|
|
The reduced `state.json` is intentionally not just a transcript. It is a graph of
|
|
model-visible conversation plus the runtime objects that explain how Codex got
|
|
there.
|
|
|
|
## System Shape
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph Runtime["codex-core runtime"]
|
|
Protocol["protocol lifecycle\nthread start/end, turn start/end"]
|
|
Inference["inference + compaction\nrequests, responses, checkpoints"]
|
|
Tools["tool dispatch\ndirect model tools + code-mode nested tools"]
|
|
CodeMode["code-mode runtime\nexec cells, yields, waits, termination"]
|
|
Terminal["terminal runtime\nexec_command / write_stdin operations"]
|
|
Agents["multi_agent_v2\nspawn, task delivery, result, close"]
|
|
end
|
|
|
|
Context["ThreadTraceContext\nroot/child no-op-capable producer"]
|
|
Writer["TraceWriter\nassigns seq and writes payloads before events"]
|
|
|
|
subgraph Bundle["trace bundle"]
|
|
Manifest["manifest.json\ntrace_id, rollout_id, root_thread_id"]
|
|
Events["trace.jsonl\nordered raw event spine"]
|
|
Payloads["payloads/*.zst\nlarge raw evidence"]
|
|
end
|
|
|
|
Reducer["replay_bundle\ndeterministic offline reducer"]
|
|
|
|
subgraph State["state.json"]
|
|
Threads["threads + turns"]
|
|
Conversation["conversation_items\nwhat the model saw"]
|
|
RuntimeObjects["inference_calls, tool_calls,\ncode_cells, terminals, compactions"]
|
|
Edges["interaction_edges\nspawn, task, result, close"]
|
|
RawRefs["raw_payload refs"]
|
|
end
|
|
|
|
Protocol --> Context
|
|
Inference --> Context
|
|
Tools --> Context
|
|
CodeMode --> Context
|
|
Terminal --> Context
|
|
Agents --> Context
|
|
|
|
Context --> Writer
|
|
Writer --> Manifest
|
|
Writer --> Payloads
|
|
Writer --> Events
|
|
|
|
Manifest --> Reducer
|
|
Events --> Reducer
|
|
Payloads --> Reducer
|
|
|
|
Reducer --> Threads
|
|
Reducer --> Conversation
|
|
Reducer --> RuntimeObjects
|
|
Reducer --> Edges
|
|
Reducer --> RawRefs
|
|
```
|
|
|
|
The thread context is deliberately small and no-op capable. A root session starts
|
|
one from `CODEX_ROLLOUT_TRACE_ROOT`; fresh spawned child threads derive their
|
|
own context from the parent's context so the whole rollout tree shares one
|
|
writer. Disabled contexts accept the same calls and record nothing.
|
|
|
|
When tracing is enabled, a thread can start or resume only if its recorder
|
|
initializes successfully. A later write failure stops recording. Partial
|
|
bundles remain available for replay. The reduced `recording_status` is `unclosed` until the closing record arrives. It then becomes
|
|
`complete` or `closed_with_partial_payloads` according to the referenced captures.
|
|
|
|
## Bundle Layout
|
|
|
|
A trace bundle contains:
|
|
|
|
- `manifest.json`: trace identity and bundle metadata.
|
|
- `trace.jsonl`: append-only raw events ordered by writer-assigned `seq`.
|
|
- `payloads/*.zst`: raw requests, responses, tool inputs/results, runtime
|
|
events, terminal output, compaction data, and protocol snapshots.
|
|
- `state.json`: reduced trace for consumers, written by `codex debug trace-reduce`.
|
|
|
|
Schema v3 indexes numbered Zstandard shards in `state.json` under `raw_payloads`.
|
|
Prepared prompts share exact input items. In tool results and tool runtime events, the writer
|
|
can share any string value whose JSON encoding is at least 64 KiB, including quotes and escapes.
|
|
Each occurrence keeps its own parent record. Within one bundle, identical encoded values reuse
|
|
one child reference. The writer hashes the complete value before applying capture limits.
|
|
|
|
For example, two results might carry the same large string under unrelated keys:
|
|
|
|
```text
|
|
result A: {"preview": <large string>, "source": "raw_payload:7"}
|
|
result B: {"attachments": [<same large string>]}
|
|
|
|
A's literal fragments {"preview": | , "source": "raw_payload:7"}
|
|
A's parts [child #7]
|
|
B's literal fragments {"attachments": [ | ]}
|
|
B's parts [child #7]
|
|
```
|
|
|
|
Only entries in the recorder's ordered `parts` are references. The reader inserts each child's
|
|
bytes between adjacent fragments. Once it reaches a record with no parts, it returns its bytes
|
|
verbatim. The literal `"raw_payload:7"` survives even when it names a real child in that bundle.
|
|
|
|
`PayloadReader::read` reconstructs the original bytes, including whitespace and escapes, or
|
|
reports an incomplete capture. The reducer can also read surviving JSON with a separate map of
|
|
missing slots. This preserves the difference between a literal `null` and a value omitted by
|
|
the recorder. A truncated record has original and retained lengths and can be inspected with
|
|
`read_prefix`.
|
|
|
|
Configure the capture limits in `config.toml`. Both values are in bytes.
|
|
|
|
```toml
|
|
[rollout_trace]
|
|
max_payload_bytes = 67108864 # Default: 64 MiB per new body or separated value
|
|
max_bytes = 268435456 # Default: 256 MiB per bundle
|
|
```
|
|
|
|
The bundle budget covers the manifest, event log, layouts, and compressed payloads.
|
|
Space is reserved for later events carrying explicit omissions.
|
|
If even an event cannot fit, the log ends at the preceding complete event.
|
|
The optional reduced-state cache is generated separately. Reduce a bundle with its recording binary.
|
|
|
|
For an inference call, `raw_request_payload_id` resolves to the outgoing HTTP
|
|
body or WebSocket message, including its `previous_response_id` and incremental
|
|
`input`. `prepared_prompt_payload_id` resolves to Core's input to formatting,
|
|
including base instructions and their provenance, ordered history, tools and
|
|
output settings. For a continuation from an unrecorded WebSocket warmup, the
|
|
start event also links the full formatted input so the reducer can resolve the
|
|
missing history.
|
|
|
|
One worker per bundle records events in submission order. For inference starts,
|
|
the transport shares its encoded request and Core shares the prepared prompt.
|
|
The worker serializes the prompt, compresses both bodies, and writes them before
|
|
publishing the start event. Remote compaction uses the same worker to snapshot
|
|
the model, instructions, input history, and parallel tool-call setting. Up to
|
|
eight jobs can wait for the worker. Use `TraceWriter::flush` to read a live
|
|
snapshot. Shutdown waits for the worker to finish accepted jobs, seal the shards,
|
|
and write the closing marker. A recording failure prevents the marker.
|
|
|
|
The writer keeps one active shard for separated content values and one for all other
|
|
records. Alternating between a value and its parent layout keeps both compression windows open.
|
|
Each shard is one Zstandard frame (level 3, 8 MiB window, checksum enabled).
|
|
Each record contains a little-endian 16-bit ID length, a little-endian 64-bit
|
|
retained body length, the UTF-8 ID, then the retained bytes. The writer flushes a record
|
|
before publishing its reference. Each stream rotates before its next record would
|
|
exceed 8 MiB uncompressed. An oversized record occupies one shard. Near the
|
|
bundle byte limit, a record can be admitted as a separately compressed frame.
|
|
A live bundle can be reduced with both current frames open. The writer seals
|
|
both before publishing `RecordingCompleted`.
|
|
|
|
`trace_id` identifies this diagnostic artifact. `rollout_id` identifies the
|
|
Codex rollout/session being observed. Keeping those separate lets us reason about
|
|
the stored trace without confusing it with the product-level session identity.
|
|
|
|
Run the recording executable to reduce its bundle:
|
|
|
|
```bash
|
|
codex debug trace-reduce <trace-bundle>
|
|
```
|
|
|
|
By default this writes `<trace-bundle>/state.json`. Within the recording binary,
|
|
`codex_rollout_trace::replay_bundle` returns the same reduced graph in memory.
|
|
|
|
## Raw Evidence vs Reduced Graph
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Model["model-visible payloads\nrequests and response output items"]
|
|
Runtime["runtime observations\ntool dispatch, terminal output, code-mode JSON"]
|
|
RawPayloads["payloads/*.zst\nexact evidence"]
|
|
Reducer["reducer"]
|
|
Conversation["ConversationItem\nwhat the model saw"]
|
|
ToolCall["ToolCall\nruntime tool boundary"]
|
|
CodeCell["CodeCell\nmodel-authored exec cell"]
|
|
TerminalOperation["TerminalOperation\ncommand/write/poll"]
|
|
InteractionEdge["InteractionEdge\ninformation flow"]
|
|
|
|
Model --> RawPayloads
|
|
Runtime --> RawPayloads
|
|
RawPayloads --> Reducer
|
|
|
|
Reducer --> Conversation
|
|
Reducer --> ToolCall
|
|
Reducer --> CodeCell
|
|
Reducer --> TerminalOperation
|
|
Reducer --> InteractionEdge
|
|
|
|
CodeCell --> ToolCall
|
|
ToolCall --> TerminalOperation
|
|
ToolCall --> InteractionEdge
|
|
Conversation --> InteractionEdge
|
|
```
|
|
|
|
This distinction is the reason the model has both raw payload references and
|
|
semantic objects. A code-mode nested tool call, for example, has JSON input and
|
|
output at the JavaScript runtime boundary, but the model-visible transcript only
|
|
contains the surrounding `exec` custom tool call and its eventual output.
|
|
|
|
The reducer keeps those facts separate:
|
|
|
|
- `ConversationItem` records what appeared in model-facing requests/responses.
|
|
- `ToolCall`, `CodeCell`, `TerminalOperation`, `InferenceCall`, and
|
|
`Compaction` record runtime/debug boundaries.
|
|
- `InteractionEdge` records information flow between objects, such as a
|
|
`spawn_agent` tool call delivering a task into a child thread.
|
|
- `RawPayloadRef` points back to exact evidence when a viewer needs more detail
|
|
than the reduced graph stores inline.
|
|
|
|
## Multi-Agent v2
|
|
|
|
Multi-agent v2 child threads share the root trace writer. That means one root
|
|
bundle reduces into one graph containing the parent thread, child threads, and
|
|
the edges between them.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
RootTool["root ToolCall\nspawn_agent / followup_task / send_message"]
|
|
ChildInput["child ConversationItem\ninjected task/message"]
|
|
ChildThread["child AgentThread"]
|
|
ChildResult["child assistant ConversationItem\nresult message"]
|
|
RootNotice["root ConversationItem\nsubagent notification"]
|
|
CloseTool["root ToolCall\nclose_agent"]
|
|
TargetThread["target AgentThread"]
|
|
|
|
RootTool -- "spawn/task edge" --> ChildInput
|
|
ChildInput --> ChildThread
|
|
ChildThread --> ChildResult
|
|
ChildResult -- "agent_result edge" --> RootNotice
|
|
CloseTool -- "close_agent edge" --> TargetThread
|
|
```
|
|
|
|
Top-level independent threads still get independent bundles. Spawned child
|
|
threads are different: they are part of the same rollout tree, so they belong in
|
|
the same raw event log, payload directory, and reduced `state.json`.
|
|
|
|
## Reducer Invariants
|
|
|
|
The reducer is strict where the raw evidence should be self-consistent:
|
|
|
|
- raw events are replayed in `seq` order;
|
|
- payload records must be readable before events refer to them;
|
|
- reduced object IDs are stable within one replay;
|
|
- runtime events may be queued until the model-visible source or delivery target
|
|
has been observed;
|
|
- model-visible conversation is derived from model-facing payloads, not from
|
|
runtime convenience output;
|
|
- runtime payloads are evidence, not proof that the model saw the same bytes.
|
|
|
|
Those invariants let the reduced graph stay small while preserving a path back
|
|
to the original evidence whenever a debugger needs to explain why an object or
|
|
edge exists.
|