Commit Graph
9 Commits
Author SHA1 Message Date
Max Johnson 9ac8c3f8d7 Close abandoned tool dispatch traces as aborted (#61434)
## Why

Interrupting a turn while a tool awaits approval can leave its rollout trace execution marked as running without an end timestamp.

## What changed

Record an `Aborted` end event when a tool dispatch trace is dropped before producing a result. Make completion and failure methods consume the trace handle so each dispatch is closed once.

## Testing

Add coverage for dropping a pending dispatch and for interrupting a turn with a pending command approval, verifying that replay reports an aborted execution with an end timestamp.

GitOrigin-RevId: 63cf68312fcbff8c6b85a01a7713cf4460c9d795
2026-11-14 01:32:55 +00:00
tongzhou wang 9d12d007b2 Record host-confirmed Code Mode wait targets in rollout traces (#61338)
## Why

Inferring wait targets from request arguments can abort trace replay on malformed arguments and link waits to cells the host never resolved.

## What changed

Record an explicit `CodeCellWaitTargetResolved` event when the host confirms a live cell, and expose its runtime handle as `resolved_wait_runtime_cell_id` on the tool call. Build cell wait links from these observations, preserving links while cell source is pending and preventing late confirmations from linking to recycled handles after thread resume.

Older traces no longer infer wait links from request arguments, so an empty `wait_tool_call_ids` list does not establish that nobody waited on a cell.

## Testing

Extend dispatch and replay tests to cover live, missing, and invalid wait targets, pending cell source, legacy traces, and runtime handle reuse across threads and resume.

GitOrigin-RevId: 0f13b291c197c5487258b6ce36276cf1f19e2b6d
2026-11-13 18:43:21 +00:00
cassirer-openai e73744b3f4 Store rollout trace payloads in compressed shards (#61222)
## What changed

- Replace individual JSON payload files with numbered Zstandard shards in trace schema v3. Add `PayloadReader` to resolve payload references, cache decoded shards, and validate payloads during replay.
- Capture exact outgoing HTTP bodies and WebSocket messages alongside prepared prompts. Preserve full formatted context for continuations from unrecorded WebSocket warmups while retaining the actual incremental request.
- Flush payload records before publishing references and seal the final shard before recording completion. Replace `recording_complete` with `recording_status` to distinguish unclosed recordings, closed recordings with partial payloads, and complete recordings.

Reduce trace bundles with the binary that recorded them.

## Testing

Add coverage for shard rotation, oversized records, live reads, cache eviction, and malformed or corrupt payloads. Extend HTTP and WebSocket integration assertions to verify captured requests and prepared prompts, including warmup continuations.

GitOrigin-RevId: 40ce3c11383f48e08f52e7120b706c61a449df9c
2026-11-13 01:22:19 +00:00
tongzhou wang 6103cf6c78 Expose acceptance metadata and code-cell links in rollout traces (#59778)
## What changed

- Add `accepted_at_unix_ms` and `accepted_seq` to conversation items, recording the first local `HistoryItemAccepted` event in trace order. Inherited context alone does not count as local acceptance.
- Add `started_code_cell_id` to model-requested `exec` tool calls when a matching code cell exists in the same thread and, when known, turn. Resolve links after replay so deferred cell starts can be matched.
- Default the new optional fields to absent when deserializing older traces and omit them from serialized output when unset.

## Testing

Extend reducer tests to cover duplicate acceptance events with out-of-order clock readings, inherited items without local acceptance, and `exec` calls with and without matching code cells.

GitOrigin-RevId: dfb594630f77d7c899398aa181c88a681587b1df
2026-11-06 01:20:54 +00:00
cassirer-openai 68b67f0ff4 Preserve rollout tracing across sub-agent reloads (#57615)
## Why

Reloading an evicted sub-agent disabled its rollout tracing, leaving later turns out of the trace. Replacement Code Mode sessions could also reuse cell IDs from earlier sessions.

## What changed

- Reattach reloaded children to the root rollout trace and emit `ThreadResumed` without duplicating `ThreadStarted`.
- Reopen existing threads during replay, preserving their identity and history while clearing stale runtime cell and terminal process mappings.
- Allocate Code Mode cell generations across sessions for both gRPC and stdio hosts to prevent ID reuse after reload.
- Record an aborted trace when session startup fails before the session loop begins.

## Testing

Add regression coverage for child eviction and reload, including a failed reload, continued inference tracing, distinct cell IDs, and preserved mail and follow-up context. Extend replay tests for history preservation and reused runtime handles, and host tests for cell IDs across session replacement and reconnection.

GitOrigin-RevId: 6c146c04b5cda9295afb7d973c0d5474b363a822
2026-10-28 01:29:34 +00:00
cassirer-openai e977cb394d Preserve tool output versions and history order in rollout traces (#57311)
## Why

Tool calls can emit multiple outputs, and compaction can trim output content while retaining the same call and item IDs. The trace reducer rejected these versions as conflicting content. Late tool and code-cell events could also link outputs in map-key order instead of conversation order.

## What changed

- Preserve distinct tool output versions while retaining consistency checks for tool invocations.
- Link tool and code-cell outputs in thread history order, and resolve pending code cells after the entire event has been reduced.
- Prevent snapshot reconciliation from reusing the same conversation item twice when identical items move within history.

## Testing

Extend reducer regression tests for trimmed outputs, changed images and metadata, reordered identical messages, and late tool and code-cell events spanning the item-ID ordering boundary.

GitOrigin-RevId: eb4242d6a2d32d61c3caab9d3cc3cf6b374ce799
2026-10-26 23:55:28 +00:00
cassirer-openai d571eaf1f6 Preserve code cell response metadata in rollout traces (#55725)
## Why

Reducing code cell lifecycle events discarded their response payload references and lost the initial response status when the runtime status changed.

## What changed

Store the initial response status and separate raw payload IDs for the initial response and final result on `CodeCell`. Carry payload references through queued lifecycle events so they survive replay when the source item arrives later.

## Testing

Extend lifecycle tests to verify distinct initial and final payload IDs and the preserved initial status for yielded cells that complete and failed cells whose lifecycle events precede their source item.

GitOrigin-RevId: 8f49a3174ea487a5d91ebc2a6c97133aacaabfd2
2026-10-21 02:02:49 +00:00
cassirer-openai 89698ad1c3 [rollout-trace] Include x-request-id in rollout trace. (#20066)
## Why

Rollout traces need an identifier that can be used to correlate a Codex
inference with upstream Responses API, proxy, and engine logs. The
reduced trace model already exposed `upstream_request_id`, but it was
being populated from the Responses API `response.id`. That value is
useful for `previous_response_id` chaining, but it is not the transport
request id that upstream systems key on.

This PR separates those concepts so trace consumers can reliably answer
both questions:

- which Responses API response did this inference produce?
- which upstream request handled it?

## Structure

The change keeps the upstream request id at the same lifecycle level as
the provider stream:

- `codex-api` captures the `x-request-id` HTTP response header when the
SSE stream is created and exposes it on `ResponseStream`. Fixture and
websocket streams set the field to `None` because they do not have that
HTTP response header.
- `codex-core` carries that stream-level id into `InferenceTraceAttempt`
when recording terminal stream outcomes. Completed, failed, cancelled,
dropped-stream, and pre-response error paths all record the id when it
is available.
- `rollout-trace` now records both identifiers in raw terminal inference
events and response payloads: `response_id` for the Responses API
`response.id`, and `upstream_request_id` for `x-request-id`.
- The reducer stores both fields on `InferenceCall`. It also uses
`response_id` for `previous_response_id` conversation linking, which
removes the old accidental dependency on the misnamed
`upstream_request_id` field.
- Terminal inference reduction now consumes the full terminal payload
(`InferenceCompleted`, `InferenceFailed`, or `InferenceCancelled`) in
one place. That keeps status, partial payloads, response ids, and
upstream request ids consistent across success, failure, cancellation,
and late stream-mapper events.

## Why This Shape

`x-request-id` is a property of the HTTP/provider response envelope, not
an SSE event. Capturing it once in `codex-api` and plumbing it through
terminal trace recording avoids trying to infer the value from stream
contents, and it preserves the id even when the stream fails or is
cancelled after only partial output.

Keeping `response_id` separate from `upstream_request_id` also makes the
reduced trace model less surprising: `response_id` remains the
conversation-continuation id, while `upstream_request_id` is the
operational correlation id for upstream debugging.

## Validation

The PR updates trace and reducer coverage for:

- reading `x-request-id` from SSE response headers;
- storing the true upstream request id on completed inference calls;
- preserving upstream request ids for cancelled and late-cancelled
inference streams;
- keeping `previous_response_id` reconstruction tied to `response_id`
rather than transport request ids.
2026-04-28 21:11:17 +00:00
cassirer-openai 27d9673273 [rollout_trace] Add rollout trace crate (#18876)
## Summary

Adds the standalone `codex-rollout-trace` crate, which defines the raw
trace event format, replay/reduction model, writer, and reducer logic
for reconstructing model-visible conversation/runtime state from
recorded rollout data.

The crate-level design is documented in
[`codex-rs/rollout-trace/README.md`](https://github.com/openai/codex/blob/codex/rollout-trace-crate/codex-rs/rollout-trace/README.md).

## Stack

This is PR 1/5 in the rollout trace stack.

- [#18876](https://github.com/openai/codex/pull/18876): Add rollout
trace crate
- [#18877](https://github.com/openai/codex/pull/18877): Record core
session rollout traces
- [#18878](https://github.com/openai/codex/pull/18878): Trace tool and
code-mode boundaries
- [#18879](https://github.com/openai/codex/pull/18879): Trace sessions
and multi-agent edges
- [#18880](https://github.com/openai/codex/pull/18880): Add debug trace
reduction command

## Review Notes

This PR intentionally does not wire tracing into live Codex execution.
It establishes the data model and reducer contract first, with
crate-local tests covering conversation reconstruction, compaction
boundaries, tool/session edges, and code-cell lifecycle reduction. Later
PRs emit into this model.

The README is the best entry point for reviewing the intended trace
format and reduction semantics before diving into the reducer modules.
2026-04-21 21:54:05 +00:00