Deduplicate large tool output strings in rollout traces (#62407)

## Why

Repeated tool output, such as the same image returned in several results,
can consume the trace bundle budget. When output bytes are unavailable,
the surrounding result can still carry terminal metadata and conversation
links.

## What changed

- Store JSON string values of at least 64 KiB, measured after encoding,
  separately from their tool result or runtime event. Identical encoded
  values share one record per bundle, while each occurrence retains its
  position and complete captures reconstruct the original JSON bytes.
- Keep separate compression shards open for content and surrounding
  records. Budget for the serialized event and fall back to a single
  capped or omitted payload when the split layout or its references
  exceed the applicable limit.
- Track missing values separately from literal `null` so reduction can
  preserve surviving terminal metadata and keep tool output IDs attached
  to their original items.

## Testing

Add regression tests for shared image and text payloads, byte-for-byte
reconstruction, recording completion under tight bundle limits, and
reduction with missing output.

GitOrigin-RevId: a003197ea998639b6c2ae27f408990c0186406bd
This commit is contained in:
cassirer-openai
2026-11-19 02:41:02 +00:00
committed by copyberry
parent 9274511443
commit 508a429666
18 changed files with 1072 additions and 259 deletions
+34 -11
View File
@@ -124,12 +124,32 @@ A trace bundle contains:
- `state.json`: reduced trace for consumers, written by `codex debug trace-reduce`.
Schema v3 indexes numbered Zstandard shards in `state.json` under `raw_payloads`.
Prepared prompts share exact input items. Inline images have independent references and shards,
so losing an attachment preserves its position, sibling text and tool links. A composite record
stores the literal JSON fragments around its ordered `parts`. Interleaving both reconstructs the
original bytes, including whitespace and escapes. Hashes cover each entire value before truncation.
`PayloadReader::read` returns the original bytes for complete bodies. A truncated body
records its original and retained lengths and can be inspected with `read_prefix`.
Prepared prompts share exact input items. In tool results and tool runtime events, the writer
can share any string value whose JSON encoding is at least 64 KiB, including quotes and escapes.
Each occurrence keeps its own parent record. Within one bundle, identical encoded values reuse
one child reference. The writer hashes the complete value before applying capture limits.
For example, two results might carry the same large string under unrelated keys:
```text
result A: {"preview": <large string>, "source": "raw_payload:7"}
result B: {"attachments": [<same large string>]}
A's literal fragments {"preview": | , "source": "raw_payload:7"}
A's parts [child #7]
B's literal fragments {"attachments": [ | ]}
B's parts [child #7]
```
Only entries in the recorder's ordered `parts` are references. The reader inserts each child's
bytes between adjacent fragments. Once it reaches a record with no parts, it returns its bytes
verbatim. The literal `"raw_payload:7"` survives even when it names a real child in that bundle.
`PayloadReader::read` reconstructs the original bytes, including whitespace and escapes, or
reports an incomplete capture. The reducer can also read surviving JSON with a separate map of
missing slots. This preserves the difference between a literal `null` and a value omitted by
the recorder. A truncated record has original and retained lengths and can be inspected with
`read_prefix`.
Configure the capture limits in `config.toml`. Both values are in bytes.
@@ -158,16 +178,19 @@ The worker serializes the prompt, compresses both bodies, and writes them before
publishing the start event. Remote compaction uses the same worker to snapshot
the model, instructions, input history, and parallel tool-call setting. Up to
eight jobs can wait for the worker. Use `TraceWriter::flush` to read a live
snapshot. Shutdown waits for the worker to finish accepted jobs, seal the shard,
snapshot. Shutdown waits for the worker to finish accepted jobs, seal the shards,
and write the closing marker. A recording failure prevents the marker.
The writer keeps one active shard for separated content values and one for all other
records. Alternating between a value and its parent layout keeps both compression windows open.
Each shard is one Zstandard frame (level 3, 8 MiB window, checksum enabled).
Each record contains a little-endian 16-bit ID length, a little-endian 64-bit
retained body length, the UTF-8 ID, then the retained bytes. The writer flushes a record
before publishing its reference, and rotates before the next record would
exceed 8 MiB uncompressed. An oversized record occupies one shard. The final
frame is sealed before the writer publishes `RecordingCompleted`. A live bundle
can be reduced through its published records with the last frame still open.
before publishing its reference. Each stream rotates before its next record would
exceed 8 MiB uncompressed. An oversized record occupies one shard. Near the
bundle byte limit, a record can be admitted as a separately compressed frame.
A live bundle can be reduced with both current frames open. The writer seals
both before publishing `RecordingCompleted`.
`trace_id` identifies this diagnostic artifact. `rollout_id` identifies the
Codex rollout/session being observed. Keeping those separate lets us reason about
+219
View File
@@ -67,6 +67,43 @@ fn parts_reuse_the_complete_identity_and_round_trip_in_order() -> Result<()> {
Ok(())
}
#[test]
fn separate_tool_results_reuse_open_content_and_structure_shards() -> Result<()> {
let (root, writer) = limited_trace(/*body*/ 1 << 20, /*bundle*/ 4 << 20)?;
let prefix = "x".repeat(96 << 10);
let mut reader = PayloadReader::new(root.path());
let mut shard_numbers = None;
for line in 0..16 {
let body = json!({"output": format!("{prefix}\n{line}")});
let parent = writer.write_json_payload(RawPayloadKind::ToolResult, &body)?;
let [child] = parent.parts.as_slice() else {
panic!("expected a separate output value");
};
let pair = (child.shard.unwrap(), parent.shard.unwrap());
assert_ne!(pair.0, pair.1);
assert_eq!(pair, *shard_numbers.get_or_insert(pair));
writer.append(RawTraceEventPayload::ProtocolEventObserved {
event_type: "tool_result".into(),
event_payload: parent.clone(),
})?;
assert_eq!(reader.read_json(&parent)?, body);
}
assert_eq!(
replay_bundle(root.path())?.recording_status,
crate::RecordingStatus::Unclosed
);
drop(writer);
assert_eq!(
replay_bundle(root.path())?.recording_status,
crate::RecordingStatus::Complete
);
assert_eq!(
std::fs::read_dir(root.path().join(PAYLOADS_DIR_NAME))?.count(),
2
);
Ok(())
}
#[test]
fn compressed_budget_admits_small_later_events_after_an_omission() -> Result<()> {
let limit = 4096;
@@ -138,3 +175,185 @@ fn exhausting_event_space_leaves_a_readable_prefix() -> Result<()> {
assert_eq!(state.raw_payloads.len(), 1);
Ok(())
}
#[test]
fn repeated_mcp_images_leave_room_for_their_event_and_recording_completion() -> Result<()> {
let image = json!({"type":"image","mimeType":"image/png","data":"A".repeat(64 << 10)});
let result = json!({"result":{"Ok":{"content":vec![image; 512]}}});
let cap = 48 << 10;
let original_bytes = serde_json::to_vec(&result)?.len();
for (bundle_bytes, turn, expected_capture) in [
(
64 << 10,
"turn".to_string(),
PayloadCapture::Truncated {
original_bytes,
retained_bytes: cap,
},
),
(128 << 10, "turn".to_string(), PayloadCapture::Complete),
// The refs fit by themselves, but the serialized event envelope consumes the difference.
(
128 << 10,
"t".repeat(120 << 10),
PayloadCapture::Omitted { original_bytes },
),
] {
let (root, writer) = limited_trace(cap, bundle_bytes)?;
writer.event_with_payload(
RawTraceEventContext::turn(ROOT_THREAD_ID, &turn),
RawPayloadKind::ToolRuntimeEvent,
&result,
|runtime_payload| RawTraceEventPayload::ToolCallRuntimeEnded {
tool_call_id: "images".into(),
status: crate::ExecutionStatus::Completed,
runtime_payload,
},
)?;
let next = RawTraceEventPayload::CodexTurnStarted {
codex_turn_id: "next".into(),
thread_id: ROOT_THREAD_ID.into(),
};
writer.append(next.clone())?;
drop(writer);
let log = std::fs::read_to_string(root.path().join(RAW_EVENT_LOG_FILE_NAME))?;
let events = log
.lines()
.map(serde_json::from_str)
.collect::<serde_json::Result<Vec<RawTraceEvent>>>()?;
let [tool, later, closing] = events.as_slice() else {
panic!("expected tool result, next turn and recording completion");
};
let RawTraceEventPayload::ToolCallRuntimeEnded {
runtime_payload, ..
} = &tool.payload
else {
panic!("missing MCP result");
};
assert_eq!(later.payload, next);
assert_eq!(closing.payload, RawTraceEventPayload::RecordingCompleted);
assert_eq!(runtime_payload.capture, expected_capture);
if expected_capture.is_complete() {
assert_eq!(runtime_payload.parts.len(), 512);
let first = &runtime_payload.parts[0];
assert!(runtime_payload.parts.iter().all(|part| part == first));
assert_eq!(
PayloadReader::new(root.path()).read_prefix(first)?.len(),
cap
);
} else {
// No child was written before selecting the compact representation.
assert!(runtime_payload.parts.is_empty());
assert_eq!(runtime_payload.raw_payload_id, "raw_payload:1");
}
let mut total =
log.len() as u64 + std::fs::metadata(root.path().join(MANIFEST_FILE_NAME))?.len();
for shard in std::fs::read_dir(root.path().join(PAYLOADS_DIR_NAME))? {
total += shard?.metadata()?.len();
}
assert!(total <= bundle_bytes);
}
Ok(())
}
#[test]
fn tool_and_cell_results_keep_distinct_parents_and_share_the_same_image() -> Result<()> {
let image = format!("data:image/png;base64,{}", "A".repeat(96 << 10));
let block = json!({"type":"input_image", "image_url":image});
let cell = json!({"response":{"Result":{"content_items":[block],"error_text":null}}});
let inner =
json!({"type":"code_mode_response","value":{"image_url":image,"detail":"original"}});
let outer = json!({"type":"direct_response","response_item":{"type":"custom_tool_call_output","output":[block]}});
for cap in [4096, 128 << 10] {
let (bundle, writer) = limited_trace(cap, /*bundle*/ 1 << 20)?;
let mut reader = PayloadReader::new(bundle.path());
let mut parents = std::collections::HashSet::new();
let mut children = std::collections::HashSet::new();
for body in [&inner, &cell, &cell, &outer] {
let reference = writer.write_json_payload(RawPayloadKind::ToolResult, body)?;
let [child] = reference.parts.as_slice() else {
panic!("expected one image reference")
};
parents.insert(reference.raw_payload_id.clone());
children.insert(child.raw_payload_id.clone());
if image.len() < cap {
assert_eq!(reader.read(&reference)?, serde_json::to_vec(body)?);
assert_eq!(reader.read(child)?, serde_json::to_vec(&image)?);
} else {
assert!(reference.capture.is_complete());
assert_eq!(
reader.read_prefix(child)?,
&serde_json::to_vec(&image)?[..cap]
);
assert!(reader.read(&reference).is_err());
}
}
assert_eq!((parents.len(), children.len()), (4, 1));
}
Ok(())
}
#[test]
fn arbitrary_tool_strings_share_bytes_and_gaps_have_exact_positions() -> Result<()> {
let (root, writer) = limited_trace(/*body*/ 4096, /*bundle*/ 1 << 20)?;
let data = "x".repeat(96 << 10);
let literal = json!({"$trace_unavailable":"raw_payload:1"});
let body = json!({
"structuredContent": {"unknown/field": [literal, null, data]},
"another": data,
"path": "payloads/000001.zst",
"id": "raw_payload:1"
});
let reference = writer.write_json_payload(RawPayloadKind::ToolRuntimeEvent, &body)?;
let [first, second] = reference.parts.as_slice() else {
panic!("expected two slots")
};
assert_eq!(first, second);
let mut reader = PayloadReader::new(root.path());
assert!(reader.read(&reference).is_err());
let observed = reader.observed_json(&reference)?;
assert_eq!(
observed.value,
json!({
"structuredContent": {"unknown/field": [literal, null, null]},
"another": null, "path": "payloads/000001.zst", "id": "raw_payload:1"
})
);
assert_eq!(
observed.missing,
json!({
"structuredContent": {"unknown/field": [null, null, first.raw_payload_id]},
"another": first.raw_payload_id
})
);
Ok(())
}
#[test]
fn tool_string_selection_preserves_json_tokens_and_respects_the_body_limit() -> Result<()> {
let (root, writer) = limited_trace(/*body*/ 256 << 10, /*bundle*/ 1 << 20)?;
let text = "x".repeat((64 << 10) - 2);
let short = &text[1..];
// Duplicate and escaped keys are legal in raw JSON. Only value tokens are shared.
let bytes = format!(
r#"{{"{text}" : "{text}", "a\u002fb":[ "{short}", "{text}"], "a\u002fb": "before\"after"}}"#
);
let raw = RawValue::from_string(bytes.clone())?;
let reference = writer.write_json_payload(RawPayloadKind::ToolResult, &raw)?;
assert_eq!(reference.parts.len(), 2);
assert_eq!(reference.parts[0], reference.parts[1]);
assert_eq!(
PayloadReader::new(root.path()).read(&reference)?,
bytes.as_bytes()
);
let (root, writer) = limited_trace(/*body*/ 4096, /*bundle*/ 1 << 20)?;
let fallback = writer.write_json_payload(RawPayloadKind::ToolResult, &raw)?;
assert!(fallback.parts.is_empty()); // The key alone exceeds the cap.
assert_eq!(fallback.raw_payload_id, "raw_payload:1");
assert_eq!(
PayloadReader::new(root.path()).read_prefix(&fallback)?,
&bytes.as_bytes()[..4096]
);
Ok(())
}
+1 -1
View File
@@ -386,7 +386,7 @@ fn record_partial_response(
attempt: &EnabledInferenceTraceAttempt,
upstream_request_id: Option<&str>,
output_items: &[ResponseItem],
event: impl FnOnce(Option<crate::RawPayloadRef>) -> RawTraceEventPayload + Send + 'static,
event: impl FnOnce(Option<crate::RawPayloadRef>) -> RawTraceEventPayload + Clone + Send + 'static,
) {
let ctx = &attempt.context;
let envelope = RawTraceEventContext::turn(&ctx.thread_id, &ctx.codex_turn_id);
+1
View File
@@ -19,6 +19,7 @@ mod raw_event;
mod reducer;
mod shard;
mod thread;
mod tool_content;
mod tool_dispatch;
mod writer;
+2 -3
View File
@@ -71,7 +71,7 @@ pub enum RawPayloadKind {
PreparedPrompt,
/// One exact item in a prepared prompt or oversized provider input/output array.
ModelItem,
/// An inline image or oversized text part. Losing its bytes preserves its message and position.
/// A shared JSON value, such as inline media or a large tool-result string.
ContentValue,
/// Provider-formatted context for replay of a continuation from an untraced warmup.
WarmupContext,
@@ -94,8 +94,7 @@ pub enum RawPayloadKind {
AgentResult,
}
/// Select whole JSON values whose bytes can be stored independently. Only model-facing
/// payloads use this layout. Other consumers can continue reading their single raw record.
/// Select input items and inline content from model-facing payloads.
pub(crate) fn capture_parts(
kind: RawPayloadKind,
body: &[u8],
@@ -327,6 +327,40 @@ fn accepted_tool_results_survive_without_inference_and_reuse_their_ids() -> anyh
Ok(())
}
#[test]
fn capped_tool_output_keeps_the_next_stripped_id_in_its_original_slot() -> anyhow::Result<()> {
let (temp, writer) =
crate::writer::capture_tests::limited_trace(/*body*/ 512, /*bundle*/ 64 << 10)?;
start_thread(&writer, "thread-root", "/root")?;
start_turn(&writer, "turn-1")?;
let request = json!({"input": [
{"type": "function_call_output", "output": "x".repeat(1024)},
{"type": "function_call_output", "output": "retained"}
]});
writer.append(RawTraceEventPayload::InferenceStarted {
inference_call_id: "sample".into(),
thread_id: "thread-root".into(),
codex_turn_id: "turn-1".into(),
model: "model".into(),
provider_name: "test".into(),
request_payload: writer.write_json_payload(RawPayloadKind::InferenceRequest, &request)?,
prepared_prompt_payload: writer
.write_json_payload(RawPayloadKind::PreparedPrompt, &request)?,
warmup_context_payload: None,
stripped_tool_output_ids: vec![(0, "large".into()), (1, "small".into())],
})?;
let state = replay_bundle(temp.path())?;
let ids = &state.inference_calls["sample"].request_item_ids;
assert_eq!(ids.len(), 1);
assert_eq!(
state.conversation_items[&ids[0]]
.responses_item_id
.as_deref(),
Some("small")
);
Ok(())
}
#[test]
fn injected_outputs_keep_their_request_positions_and_native_ids() -> anyhow::Result<()> {
for accept_first in [true, false] {
@@ -23,6 +23,7 @@ use crate::model::ConversationRole;
use crate::model::InferenceCallId;
use crate::model::ProducerRef;
use crate::payload::RawPayloadRef;
use crate::shard::ObservedJson;
mod normalize;
@@ -40,8 +41,8 @@ impl TraceReducer {
if self.receivable_ids.contains_key(&key) {
return Ok(());
}
let item =
normalize::normalize_model_item(&self.payload_reader.observed_json(payload)?, payload)?;
let ObservedJson { mut value, missing } = self.payload_reader.observed_json(payload)?;
let item = normalize::normalize_model_item(&mut value, payload, &missing)?;
if !item.is_delivery_or_tool_output() || item.responses_item_id.as_deref() != Some(wire_id)
{
bail!("accepted item payload does not match {wire_id}");
@@ -81,7 +82,10 @@ impl TraceReducer {
if !request_payload.capture.is_complete() {
return Ok(Vec::new());
}
let mut payload = self.payload_reader.observed_json(request_payload)?;
let ObservedJson {
value: mut payload,
missing,
} = self.payload_reader.observed_json(request_payload)?;
let Some(input) = payload.get_mut("input") else {
bail!(
"inference request payload {} did not contain input",
@@ -101,6 +105,9 @@ impl TraceReducer {
let item = request_items
.get_mut(index)
.context("stripped tool output index outside request")?;
if missing["input"][index].is_string() {
continue; // The captured request has no item to attach the history ID to.
}
if item.get("id").is_some()
|| !matches!(
item["type"].as_str(),
@@ -113,7 +120,8 @@ impl TraceReducer {
}
item["id"] = Value::String(history_id);
}
let items = normalize::normalize_model_items(request_items, request_payload)?;
let items =
normalize::normalize_model_items(request_items, request_payload, &missing["input"])?;
let previous_response_id = payload.get("previous_response_id").and_then(Value::as_str);
// After compaction, the next full request is compared against the installed replacement
@@ -203,7 +211,10 @@ impl TraceReducer {
if !response_payload.capture.is_complete() {
return Ok(Vec::new());
}
let payload = self.payload_reader.observed_json(response_payload)?;
let ObservedJson {
value: payload,
missing,
} = self.payload_reader.observed_json(response_payload)?;
let Some(output_items) = payload.get("output_items").and_then(Value::as_array) else {
bail!(
"inference response payload {} did not contain output_items",
@@ -220,7 +231,11 @@ impl TraceReducer {
bail!("inference response referenced unknown call {inference_call_id}");
};
let items = normalize::normalize_model_items(output_items, response_payload)?;
let items = normalize::normalize_model_items(
output_items,
response_payload,
&missing["output_items"],
)?;
// A later request or compaction can be recorded before this response.
// Only match items with source IDs, so retries with identical text stay distinct.
let candidates = self.rollout.threads[&thread_id]
@@ -380,12 +395,23 @@ impl TraceReducer {
checkpoint_payload: &RawPayloadRef,
) -> Result<ReducedCompactionCheckpoint> {
let (input_items, replacement_items) = if checkpoint_payload.capture.is_complete() {
let payload = self.payload_reader.observed_json(checkpoint_payload)?;
let ObservedJson {
value: payload,
missing,
} = self.payload_reader.observed_json(checkpoint_payload)?;
let input = required_array(&payload, "input_history", checkpoint_payload)?;
let replacement = required_array(&payload, "replacement_history", checkpoint_payload)?;
(
normalize::normalize_model_items(input, checkpoint_payload)?,
normalize::normalize_model_items(replacement, checkpoint_payload)?,
normalize::normalize_model_items(
input,
checkpoint_payload,
&missing["input_history"],
)?,
normalize::normalize_model_items(
replacement,
checkpoint_payload,
&missing["replacement_history"],
)?,
)
} else {
// The install event still establishes a compaction boundary.
@@ -414,6 +440,7 @@ impl TraceReducer {
Some(codex_turn_id.to_string()),
wall_time_unix_ms,
NormalizedConversationItem {
structured_output_hash: None,
responses_item_id: None,
role: ConversationRole::Assistant,
channel: None,
@@ -517,6 +544,9 @@ impl TraceReducer {
}
}
let item_id = self.next_conversation_item_id();
if let Some(hash) = item.structured_output_hash {
self.structured_output_hashes.insert(item_id.clone(), hash);
}
self.rollout.conversation_items.insert(
item_id.clone(),
ConversationItem {
@@ -566,6 +596,12 @@ impl TraceReducer {
if accepted_id.as_deref() == Some(item_id) {
*accepted_id = None;
item.body.clone_from(&normalized.body);
if let Some(hash) = normalized.structured_output_hash {
self.structured_output_hashes
.insert(item_id.to_owned(), hash);
} else {
self.structured_output_hashes.remove(item_id);
}
if normalized.agent_message.is_none() {
self.snapshot_output_anchors.insert(item_id.to_owned());
}
@@ -664,6 +700,10 @@ impl TraceReducer {
{
return false;
}
let existing_hash = self.structured_output_hashes.get(item_id);
if existing_hash.is_some() || normalized.structured_output_hash.is_some() {
return existing_hash == normalized.structured_output_hash.as_ref();
}
conversation_item_matches(item, normalized)
}
@@ -8,6 +8,7 @@ use serde::Deserialize;
use serde_json::Value;
use sha2::Digest;
use sha2::Sha256;
use std::collections::BTreeMap;
use crate::model::AgentMessageMetadata;
use crate::model::ConversationBody;
@@ -31,6 +32,7 @@ pub(super) struct NormalizedConversationItem {
pub(super) kind: ConversationItemKind,
pub(super) agent_message: Option<AgentMessageMetadata>,
pub(super) body: ConversationBody,
pub(super) structured_output_hash: Option<[u8; 32]>,
pub(super) call_id: Option<String>,
}
@@ -48,14 +50,15 @@ impl NormalizedConversationItem {
pub(super) fn normalize_model_items(
items: &[Value],
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<Vec<NormalizedConversationItem>> {
let mut normalized_items = Vec::new();
for item in items {
for (index, item) in items.iter().enumerate() {
// Tools and reasoning settings configure inference. Their raw payload retains the control.
if matches!(
item.get("type").and_then(Value::as_str),
Some("additional_tools" | "configuration_update")
) || unavailable_ref(item, raw_payload).is_some()
) || missing[index].is_string()
{
continue;
}
@@ -64,7 +67,11 @@ pub(super) fn normalize_model_items(
if let Some(object) = model_visible_item.as_object_mut() {
object.remove("internal_chat_message_metadata_passthrough");
}
normalized_items.push(normalize_model_item(&model_visible_item, raw_payload)?);
normalized_items.push(normalize_model_item(
&mut model_visible_item,
raw_payload,
&missing[index],
)?);
}
Ok(normalized_items)
}
@@ -80,8 +87,9 @@ pub(super) fn token_usage_from_value(value: &Value) -> Option<TokenUsage> {
}
pub(super) fn normalize_model_item(
item: &Value,
item: &mut Value,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<NormalizedConversationItem> {
let Some(item_type) = item.get("type").and_then(Value::as_str) else {
bail!(
@@ -90,11 +98,12 @@ pub(super) fn normalize_model_item(
);
};
let mut normalized = match item_type {
"message" => normalize_message_item(item, raw_payload),
"agent_message" => normalize_agent_message_item(item, raw_payload),
"reasoning" => normalize_reasoning_item(item, raw_payload),
"message" => normalize_message_item(item, raw_payload, missing),
"agent_message" => normalize_agent_message_item(item, raw_payload, missing),
"reasoning" => normalize_reasoning_item(item, raw_payload, missing),
"function_call" => Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::FunctionCall,
@@ -107,11 +116,12 @@ pub(super) fn normalize_model_item(
}),
"function_call_output" => Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Tool,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::FunctionCallOutput,
agent_message: None,
body: tool_output_body(item.get("output"), raw_payload)?,
body: tool_output_body(item.get("output"), raw_payload, &missing["output"])?,
call_id: item
.get("call_id")
.and_then(Value::as_str)
@@ -119,6 +129,7 @@ pub(super) fn normalize_model_item(
}),
"custom_tool_call" => Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::CustomToolCall,
@@ -131,11 +142,12 @@ pub(super) fn normalize_model_item(
}),
"custom_tool_call_output" => Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Tool,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::CustomToolCallOutput,
agent_message: None,
body: tool_output_body(item.get("output"), raw_payload)?,
body: tool_output_body(item.get("output"), raw_payload, &missing["output"])?,
call_id: item
.get("call_id")
.and_then(Value::as_str)
@@ -144,6 +156,7 @@ pub(super) fn normalize_model_item(
"tool_search_call" | "web_search_call" | "image_generation_call" | "local_shell_call" => {
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::FunctionCall,
@@ -155,21 +168,37 @@ pub(super) fn normalize_model_item(
.map(ToString::to_string),
})
}
"tool_search_output" | "mcp_tool_call_output" => Ok(NormalizedConversationItem {
responses_item_id: None,
role: ConversationRole::Tool,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::FunctionCallOutput,
agent_message: None,
body: json_body(item, raw_payload),
call_id: item
.get("call_id")
.and_then(Value::as_str)
.map(ToString::to_string),
}),
"tool_search_output" | "mcp_tool_call_output" => {
// A preview can truncate a difference or show null for two different missing
// values. Hash the full visible JSON and the missing child IDs at their paths.
let body = json_body(item, raw_payload);
item.sort_all_objects();
let gaps: BTreeMap<_, _> = missing
.as_object()
.into_iter()
.flatten()
.filter(|(key, _)| item.get(key).is_some())
.collect();
let mut hash = Sha256::new();
serde_json::to_writer(&mut hash, &(&*item, gaps))?;
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: Some(hash.finalize().into()),
role: ConversationRole::Tool,
channel: Some(ConversationChannel::Commentary),
kind: ConversationItemKind::FunctionCallOutput,
agent_message: None,
body,
call_id: item
.get("call_id")
.and_then(Value::as_str)
.map(ToString::to_string),
})
}
"compaction" | "compaction_summary" | "context_compaction" => {
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Summary),
kind: ConversationItemKind::Message,
@@ -190,6 +219,7 @@ pub(super) fn normalize_model_item(
fn normalize_message_item(
item: &Value,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<NormalizedConversationItem> {
let Some(role) = item.get("role").and_then(Value::as_str) else {
bail!(
@@ -205,6 +235,7 @@ fn normalize_message_item(
};
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role,
channel: item
.get("phase")
@@ -213,7 +244,7 @@ fn normalize_message_item(
kind: ConversationItemKind::Message,
agent_message: None,
body: ConversationBody {
parts: content_parts(item.get("content"), raw_payload)?,
parts: content_parts(item.get("content"), raw_payload, &missing["content"])?,
},
call_id: None,
})
@@ -222,17 +253,20 @@ fn normalize_message_item(
fn normalize_agent_message_item(
item: &Value,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<NormalizedConversationItem> {
let raw_payload_id = &raw_payload.raw_payload_id;
let routing = AgentMessageMetadata::deserialize(item).with_context(|| {
format!("failed to parse agent_message routing in payload {raw_payload_id}")
})?;
let mut parts = Vec::new();
for part in item["content"]
for (index, part) in item["content"]
.as_array()
.context("expected agent_message content array")?
.iter()
.enumerate()
{
if let Some(missing) = unavailable_ref(&part["text"], raw_payload) {
if let Some(missing) = missing["content"][index]["text"].as_str() {
parts.push(payload_ref_part("text", missing));
continue;
}
@@ -252,6 +286,7 @@ fn normalize_agent_message_item(
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Analysis),
kind: ConversationItemKind::Message,
@@ -264,6 +299,7 @@ fn normalize_agent_message_item(
fn normalize_reasoning_item(
item: &Value,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<NormalizedConversationItem> {
let mut parts = Vec::new();
append_reasoning_parts(
@@ -271,6 +307,7 @@ fn normalize_reasoning_item(
"content",
ReasoningPartKind::Content,
raw_payload,
missing,
&mut parts,
)?;
append_reasoning_parts(
@@ -278,6 +315,7 @@ fn normalize_reasoning_item(
"summary",
ReasoningPartKind::Summary,
raw_payload,
missing,
&mut parts,
)?;
@@ -309,6 +347,7 @@ fn normalize_reasoning_item(
Ok(NormalizedConversationItem {
responses_item_id: None,
structured_output_hash: None,
role: ConversationRole::Assistant,
channel: Some(ConversationChannel::Analysis),
kind: ConversationItemKind::Reasoning,
@@ -329,6 +368,7 @@ fn append_reasoning_parts(
key: &str,
kind: ReasoningPartKind,
raw_payload: &RawPayloadRef,
missing: &Value,
parts: &mut Vec<ConversationPart>,
) -> Result<()> {
let Some(items) = item.get(key) else {
@@ -344,7 +384,7 @@ fn append_reasoning_parts(
);
};
for content_item in items {
for (index, content_item) in items.iter().enumerate() {
let Some(item_type) = content_item.get("type").and_then(Value::as_str) else {
bail!(
"reasoning item in payload {} had {key} entry without string type",
@@ -372,7 +412,7 @@ fn append_reasoning_parts(
}
};
if let Some(missing) = unavailable_ref(&content_item["text"], raw_payload) {
if let Some(missing) = missing[key][index]["text"].as_str() {
parts.push(payload_ref_part(expected_type, missing));
continue;
}
@@ -417,31 +457,27 @@ fn channel_from_phase(phase: &str) -> Option<ConversationChannel> {
}
}
/// Only interpret recorder-produced gaps backed by a partial reference in this payload.
fn unavailable_ref<'a>(value: &Value, root: &'a RawPayloadRef) -> Option<&'a RawPayloadRef> {
let marker = value.as_object().filter(|obj| obj.len() == 1)?;
let id = marker.get("$trace_unavailable")?.as_str()?;
root.references()
.find(|r| r.raw_payload_id == id && !r.capture.is_complete())
}
fn content_parts(
content: Option<&Value>,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<Vec<ConversationPart>> {
let Some(content) = content.and_then(Value::as_array) else {
return Ok(vec![payload_ref_part("content", raw_payload)]);
return Ok(vec![payload_ref_part(
"content",
&raw_payload.raw_payload_id,
)]);
};
let mut parts = Vec::new();
for part in content {
for (index, part) in content.iter().enumerate() {
match part.get("type").and_then(Value::as_str) {
Some("input_text" | "output_text" | "text") => {
if let Some(text) = part.get("text").and_then(Value::as_str) {
parts.push(ConversationPart::Text {
text: text.to_string(),
});
} else if let Some(missing) = unavailable_ref(&part["text"], raw_payload) {
} else if let Some(missing) = missing[index]["text"].as_str() {
parts.push(payload_ref_part("text", missing));
}
}
@@ -452,12 +488,16 @@ fn content_parts(
canonical.sort_all_objects();
let mut digest = Sha256::new();
serde_json::to_writer(&mut digest, &canonical)?;
let missing_image = missing[index]["image_url"].as_str();
if let Some(id) = missing_image {
digest.update([0]); // Include the shared child's identity in a partial observation.
digest.update(id.as_bytes());
}
parts.push(ConversationPart::PayloadRef {
label: other.unwrap_or("content").to_string(),
raw_payload_id: unavailable_ref(&part["image_url"], raw_payload)
.unwrap_or(raw_payload)
.raw_payload_id
.clone(),
raw_payload_id: missing_image
.unwrap_or(&raw_payload.raw_payload_id)
.to_owned(),
content_sha256: Some(format!("{:x}", digest.finalize())),
});
}
@@ -500,7 +540,7 @@ fn raw_text_or_json_body(value: Option<&Value>, raw_payload: &RawPayloadRef) ->
}
Some(value) => json_body(value, raw_payload),
None => ConversationBody {
parts: vec![payload_ref_part("payload", raw_payload)],
parts: vec![payload_ref_part("payload", &raw_payload.raw_payload_id)],
},
}
}
@@ -508,17 +548,18 @@ fn raw_text_or_json_body(value: Option<&Value>, raw_payload: &RawPayloadRef) ->
fn tool_output_body(
output: Option<&Value>,
raw_payload: &RawPayloadRef,
missing: &Value,
) -> Result<ConversationBody> {
Ok(match output {
Some(Value::String(text)) => ConversationBody {
parts: vec![ConversationPart::Text { text: text.clone() }],
},
Some(Value::Array(_)) => ConversationBody {
parts: content_parts(output, raw_payload)?,
parts: content_parts(output, raw_payload, missing)?,
},
Some(value) => json_body(value, raw_payload),
None => ConversationBody {
parts: vec![payload_ref_part("tool_output", raw_payload)],
parts: vec![payload_ref_part("tool_output", &raw_payload.raw_payload_id)],
},
})
}
@@ -550,10 +591,10 @@ fn json_body(value: &Value, raw_payload: &RawPayloadRef) -> ConversationBody {
}
}
fn payload_ref_part(label: &str, raw_payload: &RawPayloadRef) -> ConversationPart {
fn payload_ref_part(label: &str, id: &str) -> ConversationPart {
ConversationPart::PayloadRef {
label: label.to_string(),
raw_payload_id: raw_payload.raw_payload_id.clone(),
raw_payload_id: id.to_string(),
content_sha256: None,
}
}
@@ -555,6 +555,62 @@ fn request_reuses_prior_tool_outputs_with_internal_metadata() -> anyhow::Result<
Ok(())
}
#[test]
fn tool_search_outputs_keep_the_evidence_for_each_version() -> anyhow::Result<()> {
let long_text = "prefix".repeat(340);
let a = json!(format!("{long_text}A"));
let b = json!(format!("{long_text}B"));
let null = json!(null);
for body_cap in [512, 8192] {
let (root, writer) = crate::writer::capture_tests::limited_trace(body_cap, 128 << 10)?;
start_thread(&writer, ROOT_THREAD_ID, "/root")?;
let mut requests = Vec::new();
for (index, (first, second)) in [
(&a, &null),
(&b, &null),
(&b, &null),
(&null, &b),
(&null, &null),
]
.into_iter()
.enumerate()
{
let turn = format!("turn-{index}");
start_turn(&writer, &turn)?;
let mut output = json!({"type": "tool_search_output", "call_id": "search", "tools": [
{"name": "one", "text": first}, {"name": "two", "text": second}
]});
if index == 2 {
output["internal_chat_message_metadata_passthrough"] = json!({"text": a});
}
let request = writer
.write_json_payload(RawPayloadKind::InferenceRequest, &json!({"input":[output]}))?;
assert_eq!(
request.is_complete(),
body_cap > long_text.len() || index == 4
);
requests.push(request.raw_payload_id.clone());
append_inference_start(&writer, &format!("inference-{index}"), &turn, request)?;
}
drop(writer);
let rollout = replay_bundle(root.path())?;
// A changed value or a moved missing slot gets its own evidence. An exact repeat
// reuses the first sighting, and genuine JSON nulls form a distinct observation.
for (index, source) in [0, 1, 1, 3, 4].into_iter().enumerate() {
let item_id =
&rollout.inference_calls[&format!("inference-{index}")].request_item_ids[0];
let [ConversationPart::Json { raw_payload_id, .. }] =
rollout.conversation_items[item_id].body.parts.as_slice()
else {
panic!("expected JSON tool-search output");
};
assert_eq!(raw_payload_id, &requests[source]);
}
assert_eq!(rollout.conversation_items.len(), 4);
}
Ok(())
}
#[test]
fn tool_output_versions_preserve_different_nested_metadata() -> anyhow::Result<()> {
let temp = TempDir::new()?;
@@ -73,6 +73,7 @@ pub fn replay_bundle(bundle_dir: impl AsRef<Path>) -> Result<RolloutTrace> {
terminal_ids_by_process: BTreeMap::new(),
process_ids_by_terminal: BTreeMap::new(),
thread_conversation_snapshots: BTreeMap::new(),
structured_output_hashes: BTreeMap::new(),
pending_compaction_replacement_item_ids: BTreeMap::new(),
code_cell_ids_by_runtime: BTreeMap::new(),
last_thread_resume_seq: BTreeMap::new(),
@@ -220,6 +221,8 @@ struct TraceReducer {
process_ids_by_terminal: BTreeMap<String, String>,
/// Latest request and its response items, used to reconcile the next request.
thread_conversation_snapshots: BTreeMap<String, Vec<String>>,
/// Full identities for JSON tool outputs whose reduced bodies contain only a preview.
structured_output_hashes: BTreeMap<String, [u8; 32]>,
/// Replacement snapshot installed by compaction but not yet seen in a sampling request.
///
/// The first full request after compaction should compare against the installed replacement
@@ -72,7 +72,7 @@ impl TraceReducer {
tool_call_id: &str,
runtime_payload: &RawPayloadRef,
) -> Result<()> {
if !runtime_payload.capture.is_complete() {
if !runtime_payload.is_complete() {
return Ok(());
}
let kind = self
@@ -135,7 +135,7 @@ impl TraceReducer {
) -> Result<()> {
let kind = self.rollout.tool_calls[tool_call_id].kind.clone();
// MCP results are opaque evidence; they cannot describe agent interactions.
if !runtime_payload.capture.is_complete() || matches!(kind, ToolCallKind::Mcp { .. }) {
if !runtime_payload.is_complete() || matches!(kind, ToolCallKind::Mcp { .. }) {
return Ok(());
}
let runtime_payload_json = self.payload_reader.read_json(runtime_payload)?;
@@ -28,6 +28,7 @@ use crate::model::TerminalRequest;
use crate::model::TerminalResult;
use crate::model::TerminalSession;
use crate::model::ToolCallKind;
use crate::payload::RawPayloadKind;
use crate::payload::RawPayloadRef;
use crate::raw_event::RawEventSeq;
use crate::reducer::TraceReducer;
@@ -93,7 +94,7 @@ impl TraceReducer {
return Ok(None);
};
if !runtime_payload.capture.is_complete() {
if !runtime_payload.is_complete() {
return Ok(None);
}
let payload = self.payload_reader.read_json(runtime_payload)?;
@@ -175,13 +176,44 @@ impl TraceReducer {
else {
bail!("terminal end referenced unknown operation {operation_id}");
};
let response = response_payload
.filter(|p| p.capture.is_complete())
.map(|payload| {
let value = self.payload_reader.read_json(payload)?;
parse_terminal_response_payload(value, &operation_kind, &payload.raw_payload_id)
})
.transpose()?;
let response = match response_payload {
Some(payload)
if payload.capture.is_complete()
&& (payload.kind == RawPayloadKind::ToolRuntimeEvent
|| payload.is_complete()) =>
{
// A retained exec-end parent still carries the exit code and process ID.
// observed_json maps an unavailable output child to null.
let value = self.payload_reader.observed_json(payload)?.value;
Some(parse_terminal_response_payload(
value,
&operation_kind,
&payload.raw_payload_id,
)?)
}
Some(payload)
if payload.capture.is_complete()
&& operation_kind == TerminalOperationKind::WriteStdin =>
{
let observed = self.payload_reader.observed_json(payload)?;
// Missing stdout has a child ID at the same path. A literal null does not.
let has_exec_output = observed.value["value"]["output"].is_string()
|| observed.missing["value"]["output"].is_string();
if has_exec_output
&& let DispatchedToolTraceResponsePayload::CodeModeResponse { value } =
serde_json::from_value(observed.value)?
{
let result: CodeModeExecResult = serde_json::from_value(value)?;
Some(ParsedTerminalResponse {
process_id: None,
result: result.into(),
})
} else {
None
}
}
_ => None,
};
let process_id = {
let Some(operation) = self.rollout.terminal_operations.get_mut(operation_id) else {
@@ -507,7 +539,7 @@ fn parse_protocol_terminal_response(payload: ExecCommandEndPayload) -> ParsedTer
process_id: payload.process_id,
result: TerminalResult {
exit_code: Some(payload.exit_code),
stdout: payload.aggregated_output,
stdout: payload.aggregated_output.unwrap_or_default(),
stderr: String::new(),
formatted_output: None,
original_token_count: None,
@@ -557,15 +589,8 @@ fn parse_dispatch_terminal_response(value: JsonValue) -> Result<ParsedTerminalRe
fn parse_code_mode_exec_result(value: JsonValue) -> TerminalResult {
match serde_json::from_value::<CodeModeExecResult>(value.clone()) {
Ok(result) => TerminalResult {
exit_code: result.exit_code,
stdout: result.output.clone(),
stderr: String::new(),
formatted_output: Some(result.output),
original_token_count: result.original_token_count,
chunk_id: result.chunk_id,
},
Err(_) => {
Ok(result) if result.output.is_some() => result.into(),
_ => {
let output = json_text_content(&value).unwrap_or_else(|| value.to_string());
TerminalResult {
exit_code: None,
@@ -614,8 +639,7 @@ struct ExecCommandBeginPayload {
#[derive(Deserialize)]
struct ExecCommandEndPayload {
process_id: Option<String>,
#[serde(default)]
aggregated_output: String,
aggregated_output: Option<String>,
exit_code: i32,
}
@@ -655,7 +679,20 @@ struct CodeModeExecResult {
chunk_id: Option<String>,
exit_code: Option<i32>,
original_token_count: Option<usize>,
output: String,
output: Option<String>,
}
impl From<CodeModeExecResult> for TerminalResult {
fn from(result: CodeModeExecResult) -> Self {
Self {
exit_code: result.exit_code,
stdout: result.output.clone().unwrap_or_default(),
stderr: String::new(),
formatted_output: result.output,
original_token_count: result.original_token_count,
chunk_id: result.chunk_id,
}
}
}
#[cfg(test)]
@@ -19,6 +19,7 @@ use crate::model::TerminalResult;
use crate::model::TerminalSession;
use crate::model::ToolCallKind;
use crate::model::ToolCallSummary;
use crate::payload::PayloadCapture;
use crate::payload::RawPayloadKind;
use crate::raw_event::RawTraceEventPayload;
use crate::reducer::test_support::ROOT_THREAD_ID;
@@ -37,8 +38,30 @@ use crate::writer::TraceWriter;
#[test]
fn exec_tool_reduces_to_terminal_operation_and_session() -> anyhow::Result<()> {
let temp = TempDir::new()?;
let writer = create_started_writer(&temp)?;
check_exec_terminal_result("ok\n", /*exit_code*/ 0, Some("ok\n"))
}
#[test]
fn exec_end_retains_exit_code_and_session_when_output_is_omitted() -> anyhow::Result<()> {
// Numbered lines stay larger than the bundle budget even after compression.
let output = (0..32_768)
.map(|line| format!("{line}\n"))
.collect::<String>();
check_exec_terminal_result(
&output, /*exit_code*/ 17, /*expected_stdout*/ None,
)
}
fn check_exec_terminal_result(
output: &str,
exit_code: i32,
expected_stdout: Option<&str>,
) -> anyhow::Result<()> {
let (temp, writer) = crate::writer::capture_tests::limited_trace(
/*body*/ 256 << 10,
/*bundle*/ 24 << 10,
)?;
start_thread(&writer, ROOT_THREAD_ID, "/root")?;
start_turn(&writer, "turn-1")?;
append_inference_with_tool_call(&writer)?;
@@ -94,11 +117,16 @@ fn exec_tool_reduces_to_terminal_operation_and_session() -> anyhow::Result<()> {
"turn_id": "turn-1",
"command": ["cargo", "test"],
"cwd": "/repo",
"aggregated_output": "ok\n",
"exit_code": 0,
"aggregated_output": output,
"exit_code": exit_code,
"status": "completed"
}),
)?;
if expected_stdout.is_none() {
assert!(runtime_end_payload.capture.is_complete());
assert!(matches!(runtime_end_payload.parts.as_slice(), [child]
if matches!(child.capture, PayloadCapture::Omitted { .. })));
}
let runtime_end_payload_id = runtime_end_payload.raw_payload_id.clone();
let runtime_end = writer.append_with_context(
trace_context("turn-1"),
@@ -188,8 +216,8 @@ fn exec_tool_reduces_to_terminal_operation_and_session() -> anyhow::Result<()> {
max_output_tokens: None,
},
result: Some(TerminalResult {
exit_code: Some(0),
stdout: "ok\n".to_string(),
exit_code: Some(exit_code),
stdout: expected_stdout.unwrap_or_default().to_string(),
stderr: String::new(),
formatted_output: None,
original_token_count: None,
@@ -570,8 +598,91 @@ fn invalid_write_stdin_arguments_remain_failed_tool_calls() -> anyhow::Result<()
#[test]
fn code_mode_write_stdin_result_projects_structured_exec_fields() -> anyhow::Result<()> {
let temp = TempDir::new()?;
let writer = create_started_writer(&temp)?;
check_code_mode_write_stdin_result(/*body_limit*/ 128 << 10)
}
#[test]
fn capped_code_mode_write_stdin_retains_structured_exec_fields() -> anyhow::Result<()> {
check_code_mode_write_stdin_result(/*body_limit*/ 4096)
}
fn check_code_mode_write_stdin_result(body_limit: usize) -> anyhow::Result<()> {
let output = "x".repeat(80 << 10);
let response = json!({
"type": "code_mode_response",
"value": {
"chunk_id": "abc123",
"wall_time_seconds": 1.25,
"exit_code": 0,
"original_token_count": 3,
"output": output
}
});
let retained_output = (body_limit > output.len()).then_some(output);
assert_eq!(
replay_write_stdin_result(body_limit, response, ExecutionStatus::Completed)?,
Some(TerminalResult {
exit_code: Some(0),
stdout: retained_output.clone().unwrap_or_default(),
stderr: String::new(),
formatted_output: retained_output,
original_token_count: Some(3),
chunk_id: Some("abc123".to_string()),
}),
);
Ok(())
}
#[test]
fn capped_write_stdin_only_recovers_structured_output() -> anyhow::Result<()> {
let body_limit = 4 << 10;
let text = "x".repeat(80 << 10);
for (response, status) in [
(
json!({"type": "direct_response", "response_item": {"type": "function_call_output", "output": text}}),
ExecutionStatus::Completed,
),
(
json!({"type": "error", "error": text}),
ExecutionStatus::Failed,
),
// A literal null and a gap elsewhere do not identify a missing terminal output.
(
json!({"type": "code_mode_response", "value": {"output": null, "text": text}}),
ExecutionStatus::Completed,
),
] {
assert_eq!(
replay_write_stdin_result(body_limit, response, status)?,
None
);
}
let response = json!({
"type": "code_mode_response",
"value": {"output": "done", "exit_code": 17, "text": text},
});
assert_eq!(
replay_write_stdin_result(body_limit, response, ExecutionStatus::Completed)?,
Some(TerminalResult {
exit_code: Some(17),
stdout: "done".into(),
formatted_output: Some("done".into()),
stderr: String::new(),
original_token_count: None,
chunk_id: None,
})
);
Ok(())
}
fn replay_write_stdin_result(
body_limit: usize,
response: serde_json::Value,
status: ExecutionStatus,
) -> anyhow::Result<Option<TerminalResult>> {
let (temp, writer) =
crate::writer::capture_tests::limited_trace(body_limit, /*bundle*/ 1 << 20)?;
start_thread(&writer, ROOT_THREAD_ID, "/root")?;
start_turn(&writer, "turn-1")?;
let request_payload = writer.write_json_payload(
@@ -590,19 +701,6 @@ fn code_mode_write_stdin_result_projects_structured_exec_fields() -> anyhow::Res
}
}),
)?;
let response_payload = writer.write_json_payload(
RawPayloadKind::ToolResult,
&json!({
"type": "code_mode_response",
"value": {
"chunk_id": "abc123",
"wall_time_seconds": 1.25,
"exit_code": 0,
"original_token_count": 3,
"output": "done\n"
}
}),
)?;
writer.append_with_context(
trace_context("turn-1"),
RawTraceEventPayload::CodeCellStarted {
@@ -625,29 +723,22 @@ fn code_mode_write_stdin_result_projects_structured_exec_fields() -> anyhow::Res
invocation_payload: Some(request_payload),
},
)?;
writer.append_with_context(
writer.event_with_payload(
trace_context("turn-1"),
RawTraceEventPayload::ToolCallEnded {
RawPayloadKind::ToolResult,
&response,
move |response_payload| RawTraceEventPayload::ToolCallEnded {
tool_call_id: "tool-stdin".to_string(),
status: ExecutionStatus::Completed,
status,
result_payload: Some(response_payload),
},
)?;
writer.flush()?;
let rollout = replay_bundle(temp.path())?;
assert_eq!(
rollout.terminal_operations["terminal_operation:1"].result,
Some(TerminalResult {
exit_code: Some(0),
stdout: "done\n".to_string(),
stderr: String::new(),
formatted_output: Some("done\n".to_string()),
original_token_count: Some(3),
chunk_id: Some("abc123".to_string()),
}),
);
Ok(())
Ok(rollout.terminal_operations["terminal_operation:1"]
.result
.clone())
}
fn append_inference_with_tool_call(writer: &TraceWriter) -> anyhow::Result<()> {
+134 -51
View File
@@ -17,11 +17,14 @@ use anyhow::Context;
use anyhow::Result;
use anyhow::ensure;
use serde::de::IgnoredAny;
use serde_json::Value;
use serde_json::value::RawValue;
use zstd::stream::write::Encoder;
use crate::bundle::PAYLOADS_DIR_NAME;
use crate::payload::PayloadCapture;
use crate::payload::RawPayloadId;
use crate::payload::RawPayloadKind;
use crate::payload::RawPayloadRef;
const WINDOW_LOG: u32 = 23;
@@ -35,92 +38,124 @@ fn shard_path(directory: &Path, number: u64) -> PathBuf {
pub(crate) struct ShardWriter {
directory: PathBuf,
active: Option<Encoder<'static, BufWriter<File>>>,
number: u64,
bytes: usize,
active: [Option<ActiveShard>; 2],
last_number: u64,
closed_bytes: u64,
active_bytes: u64,
}
struct ActiveShard {
number: u64,
encoder: Encoder<'static, BufWriter<File>>,
uncompressed_bytes: usize,
compressed_bytes: u64,
}
// Bridge can omit large content shards while retaining their parent layouts.
// Each stream keeps its compression window across writes to the other stream.
fn stream_index(kind: RawPayloadKind) -> usize {
usize::from(kind == RawPayloadKind::ContentValue)
}
impl ShardWriter {
pub(crate) fn new(directory: PathBuf) -> Self {
Self {
directory,
active: None,
number: 0,
bytes: 0,
active: [None, None],
last_number: 0,
closed_bytes: 0,
active_bytes: 0,
}
}
/// Reserve the final empty block and checksum while a flushed frame is still live.
/// Reserve the final empty block and checksum for each flushed, live frame.
pub(crate) fn charged_bytes(&self) -> u64 {
self.closed_bytes + self.active_bytes + if self.active.is_some() { 7 } else { 0 }
self.closed_bytes
+ self
.active
.iter()
.flatten()
.map(|shard| shard.compressed_bytes + 7)
.sum::<u64>()
}
/// Near the budget boundary, trial-compress one independent frame. A body that
/// does not fit then leaves both the published prefix and future small writes intact.
pub(crate) fn append_with_limit(
&mut self,
kind: RawPayloadKind,
id: &str,
body: &[u8],
limit: u64,
) -> Result<Option<u64>> {
let size = HEADER_BYTES + id.len() + body.len();
let bound = zstd::zstd_safe::compress_bound(size) as u64 + 128;
if self.charged_bytes().saturating_add(bound) <= limit {
return self.append(id, body).map(Some);
let remaining = limit.saturating_sub(self.charged_bytes());
if bound <= remaining {
return self.append(kind, id, body).map(Some);
}
self.finish()?;
if self.closed_bytes >= limit {
if remaining == 0 {
return Ok(None);
}
let mut encoder = new_encoder(Vec::new())?;
write_record(&mut encoder, id, body)?;
let frame = encoder.finish()?;
if frame.len() as u64 > limit - self.closed_bytes {
if frame.len() as u64 > remaining {
return Ok(None);
}
let number = self.number + 1;
self.finish_stream(stream_index(kind))?;
let number = self.last_number + 1;
let mut file = File::create_new(shard_path(&self.directory, number))?;
file.write_all(&frame)?;
self.closed_bytes += frame.len() as u64;
self.number = number;
self.last_number = number;
Ok(Some(number))
}
/// Returns the shard containing this whole record once it is readable.
pub(crate) fn append(&mut self, id: &str, body: &[u8]) -> Result<u64> {
fn append(&mut self, kind: RawPayloadKind, id: &str, body: &[u8]) -> Result<u64> {
let stream = stream_index(kind);
let size = HEADER_BYTES + id.len() + body.len();
if self.bytes.saturating_add(size) > TARGET_BYTES {
self.finish()?;
if self.active[stream]
.as_ref()
.is_some_and(|shard| shard.uncompressed_bytes.saturating_add(size) > TARGET_BYTES)
{
self.finish_stream(stream)?;
}
let encoder = match self.active.as_mut() {
let shard = match self.active[stream].as_mut() {
Some(active) => active,
None => {
self.number += 1;
self.bytes = 0;
let path = shard_path(&self.directory, self.number);
let number = self.last_number + 1;
let path = shard_path(&self.directory, number);
let file = File::create_new(&path)
.with_context(|| format!("create shard {}", path.display()))?;
self.active.insert(new_encoder(BufWriter::new(file))?)
self.last_number = number;
self.active[stream].insert(ActiveShard {
number,
encoder: new_encoder(BufWriter::new(file))?,
uncompressed_bytes: 0,
compressed_bytes: 0,
})
}
};
write_record(encoder, id, body)?;
encoder.flush()?;
self.active_bytes = encoder.get_ref().get_ref().metadata()?.len();
self.bytes += size;
Ok(self.number)
write_record(&mut shard.encoder, id, body)?;
shard.encoder.flush()?;
shard.compressed_bytes = shard.encoder.get_ref().get_ref().metadata()?.len();
shard.uncompressed_bytes += size;
Ok(shard.number)
}
/// Flushes the frame footer before the event log can publish RecordingCompleted.
pub(crate) fn finish(&mut self) -> Result<()> {
if let Some(active) = self.active.take() {
let mut file = active.finish()?;
fn finish_stream(&mut self, stream: usize) -> Result<()> {
if let Some(shard) = self.active[stream].take() {
let mut file = shard.encoder.finish()?;
file.flush()?;
self.closed_bytes += file.get_ref().metadata()?.len();
self.active_bytes = 0;
}
Ok(())
}
/// Publish all frame footers before the event log publishes RecordingCompleted.
pub(crate) fn finish(&mut self) -> Result<()> {
for stream in 0..self.active.len() {
self.finish_stream(stream)?;
}
Ok(())
}
@@ -200,6 +235,13 @@ impl DecodedShard {
}
}
/// Partial JSON for semantic reduction. The parallel tree contains a payload ID at each
/// missing slot. Other positions have no ID, even if the literal value looks like one.
pub(crate) struct ObservedJson {
pub(crate) value: Value,
pub(crate) missing: Value,
}
/// Keeps the two most recently read shards so replay can cross a shard boundary
/// in either direction. A shard larger than the target occupies the cache alone.
pub struct PayloadReader {
@@ -230,28 +272,37 @@ impl PayloadReader {
return self.read_inner(payload, /*require_sealed*/ false);
}
let mut bytes = Vec::new();
self.assemble(payload, &mut bytes)?;
self.assemble(payload, &mut bytes, &mut BTreeMap::new())?;
self.assembled = bytes;
Ok(&self.assembled)
}
/// Read the surviving structure for reduction. Known gaps occupy their original slots.
/// Exact replay uses `read`, which rejects a body with any missing descendant.
pub(crate) fn observed_json(&mut self, payload: &RawPayloadRef) -> Result<serde_json::Value> {
/// Read what survived capture, keeping gap metadata separate from the observed value.
pub(crate) fn observed_json(&mut self, payload: &RawPayloadRef) -> Result<ObservedJson> {
if payload.is_complete() {
return self.read_json(payload);
return Ok(ObservedJson {
value: self.read_json(payload)?,
missing: Value::Null,
});
}
let mut bytes = Vec::new();
self.assemble(payload, &mut bytes)?;
Ok(serde_json::from_slice(&bytes)?)
let mut gaps = BTreeMap::new();
self.assemble(payload, &mut bytes, &mut gaps)?;
let value = serde_json::from_slice(&bytes)?;
let raw = serde_json::from_slice(&bytes)?;
let missing = missing_slots(raw, bytes.as_ptr() as usize, &gaps)?;
Ok(ObservedJson { value, missing })
}
fn assemble(&mut self, payload: &RawPayloadRef, bytes: &mut Vec<u8>) -> Result<()> {
fn assemble<'a>(
&mut self,
payload: &'a RawPayloadRef,
bytes: &mut Vec<u8>,
gaps: &mut BTreeMap<usize, &'a str>,
) -> Result<()> {
if !payload.capture.is_complete() {
serde_json::to_writer(
bytes,
&serde_json::json!({"$trace_unavailable": payload.raw_payload_id}),
)?;
gaps.insert(bytes.len(), &payload.raw_payload_id);
bytes.extend_from_slice(b"null");
} else if payload.parts.is_empty() {
bytes.extend_from_slice(self.read_inner(payload, /*require_sealed*/ false)?);
} else {
@@ -261,7 +312,7 @@ impl PayloadReader {
)?;
bytes.extend_from_slice(fragments[0].as_bytes());
for (part, suffix) in payload.parts.iter().zip(&fragments[1..]) {
self.assemble(part, bytes)?;
self.assemble(part, bytes, gaps)?;
bytes.extend_from_slice(suffix.as_bytes());
}
}
@@ -290,16 +341,20 @@ impl PayloadReader {
recording_closed: bool,
) -> Result<()> {
// Group reads to decompress each shard just once during validation.
// Only the last shard can legitimately lack its footer in a live recording.
let mut refs: Vec<_> = index.values().collect();
refs.sort_unstable_by_key(|p| p.shard);
let last = refs.last().and_then(|p| p.shard);
// Either stream can rotate while the other stays open at an older shard number.
let mut last_shard = [None; 2];
for payload in &refs {
last_shard[stream_index(payload.kind)] = payload.shard;
}
for payload in refs {
if matches!(payload.capture, PayloadCapture::Omitted { .. }) {
ensure!(payload.shard.is_none(), "omitted payload has a shard");
continue;
}
let sealed = recording_closed || payload.shard != last;
let sealed =
recording_closed || payload.shard != last_shard[stream_index(payload.kind)];
let bytes = self.read_inner(payload, sealed)?;
if let PayloadCapture::Truncated {
original_bytes,
@@ -363,6 +418,34 @@ impl PayloadReader {
}
}
/// Project missing byte offsets onto JSON positions. Only descend into containers with gaps.
fn missing_slots(raw: &RawValue, base: usize, gaps: &BTreeMap<usize, &str>) -> Result<Value> {
let token = raw.get();
let start = token.as_ptr() as usize - base;
if let Some(id) = gaps.get(&start) {
return Ok(Value::String((*id).to_owned()));
}
if gaps.range(start..start + token.len()).next().is_none() {
return Ok(Value::Null);
}
if token.starts_with('[') {
let items: Vec<&RawValue> = serde_json::from_str(token)?;
return items
.into_iter()
.map(|item| missing_slots(item, base, gaps))
.collect();
}
let fields: BTreeMap<String, &RawValue> = serde_json::from_str(token)?;
let mut missing = serde_json::Map::new();
for (key, child) in fields {
let slot = missing_slots(child, base, gaps)?;
if !slot.is_null() {
missing.insert(key, slot);
}
}
Ok(Value::Object(missing))
}
fn decode_fragments(bytes: &[u8], parts: usize) -> Result<Vec<String>> {
let fragments: Vec<String> = serde_json::from_slice(bytes)?;
ensure!(
+91 -17
View File
@@ -1,7 +1,6 @@
#![cfg(test)]
use super::*;
use crate::RawPayloadKind;
use pretty_assertions::assert_eq;
use tempfile::TempDir;
@@ -10,23 +9,25 @@ fn published_records_survive_rotation_and_remain_readable_while_open() -> Result
let bundle = TempDir::new()?;
let directory = bundle.path().join(PAYLOADS_DIR_NAME);
std::fs::create_dir(&directory)?;
let mut writer = ShardWriter::new(directory);
let mut writer = ShardWriter::new(directory.clone());
let mut reader = PayloadReader::new(bundle.path());
let mut append = |id: &str, body: &[u8]| -> Result<RawPayloadRef> {
let shard = writer.append(id, body)?;
let mut append = |kind, id: &str, body: &[u8]| -> Result<RawPayloadRef> {
let shard = writer.append(kind, id, body)?;
Ok(RawPayloadRef {
raw_payload_id: id.to_owned(),
kind: RawPayloadKind::InferenceRequest,
kind,
shard: Some(shard),
capture: Default::default(),
parts: Vec::new(),
})
};
let first = append("first", b"{}")?;
let content = append(RawPayloadKind::ContentValue, "content", b"true")?;
let kind = RawPayloadKind::InferenceRequest;
let first = append(kind, "first", b"{}")?;
assert_eq!(reader.read(&first)?, b"{}");
let body = br#"{"text":"second\nline"}"#;
let second = append("second", body)?;
let second = append(kind, "second", body)?;
assert_eq!(second.shard, first.shard);
assert_eq!(reader.read(&second)?, body);
@@ -34,17 +35,83 @@ fn published_records_survive_rotation_and_remain_readable_while_open() -> Result
let mut large = vec![b'x'; TARGET_BYTES + 1];
large[0] = b'"';
*large.last_mut().unwrap() = b'"';
let oversized = append("large", &large)?;
let last = append("last", b"null")?;
let oversized = append(kind, "large", &large)?;
let last = append(kind, "last", b"null")?;
assert_ne!(first.shard, oversized.shard);
assert_ne!(oversized.shard, last.shard);
assert_eq!(reader.read(&oversized)?, large.as_slice());
assert_eq!(reader.shards.len(), 1);
assert_eq!(reader.read(&first)?, b"{}");
assert_eq!(reader.read(&last)?, b"null");
// Content is still live even though the structure stream has rotated twice.
let index = [
content.clone(),
first.clone(),
second,
oversized,
last.clone(),
]
.into_iter()
.map(|payload| (payload.raw_payload_id.clone(), payload))
.collect();
reader.validate(&index, /*recording_closed*/ false)?;
assert!(
reader
.read_inner(&content, /*require_sealed*/ true)
.is_err()
);
assert!(reader.read_inner(&last, /*require_sealed*/ true).is_err());
writer.finish()?;
assert_eq!(reader.read_inner(&last, /*require_sealed*/ true)?, b"null");
reader.validate(&index, /*recording_closed*/ true)?;
// A superseded shard must retain its footer even when validating a live snapshot.
let file = File::options()
.write(true)
.open(shard_path(&directory, first.shard.unwrap()))?;
file.set_len(file.metadata()?.len() - 1)?;
assert!(
PayloadReader::new(bundle.path())
.validate(&index, /*recording_closed*/ false)
.is_err_and(|error| error.to_string().contains("missing footer"))
);
Ok(())
}
#[test]
fn trial_frames_share_the_budget_with_both_open_shards() -> Result<()> {
let directory = TempDir::new()?;
let mut writer = ShardWriter::new(directory.path().to_owned());
let content = RawPayloadKind::ContentValue;
let structure = RawPayloadKind::ToolResult;
let child = writer.append(content, "child", &(0..=255).collect::<Vec<u8>>())?;
let parent = writer.append(structure, "parent", b"{}")?;
let charged = writer.charged_bytes();
let child_bytes = shard_path(directory.path(), child).metadata()?.len();
let parent_bytes = shard_path(directory.path(), parent).metadata()?.len();
assert_eq!(charged, child_bytes + parent_bytes + 2 * 7);
let body = [b'x'; 512];
assert_eq!(
writer.append_with_limit(structure, "trial", &body, charged + 1)?,
None
);
assert!(!DecodedShard::open(directory.path(), parent)?.sealed);
let limit = charged + 64;
let trial = writer
.append_with_limit(structure, "trial", &body, limit)?
.unwrap();
assert!(DecodedShard::open(directory.path(), trial)?.sealed);
assert!(DecodedShard::open(directory.path(), parent)?.sealed);
assert!(!DecodedShard::open(directory.path(), child)?.sealed);
let before_close = writer.charged_bytes();
writer.finish()?;
let mut total = 0;
for entry in directory.path().read_dir()? {
total += entry?.metadata()?.len();
}
assert_eq!(total, before_close);
assert!(total <= limit);
Ok(())
}
@@ -52,13 +119,15 @@ fn published_records_survive_rotation_and_remain_readable_while_open() -> Result
fn rotation_counts_the_record_header_and_id() -> Result<()> {
let directory = TempDir::new()?;
let mut writer = ShardWriter::new(directory.path().to_owned());
let first = writer.append("a", b"x")?;
let kind = RawPayloadKind::InferenceRequest;
let first = writer.append(kind, "a", b"x")?;
let remaining = TARGET_BYTES - (HEADER_BYTES + 1 + 1);
let exact_fit = writer.append(
kind,
"longer",
&vec![0; remaining - HEADER_BYTES - "longer".len()],
)?;
let next = writer.append("c", b"x")?;
let next = writer.append(kind, "c", b"x")?;
assert_eq!([first, exact_fit, next], [1, 1, 2]);
Ok(())
}
@@ -67,8 +136,12 @@ fn rotation_counts_the_record_header_and_id() -> Result<()> {
fn live_prefix_survives_a_partial_record_but_a_sealed_frame_rejects_it() -> Result<()> {
let directory = TempDir::new()?;
let mut writer = ShardWriter::new(directory.path().to_owned());
let shard = writer.append("published", b"true")?;
let encoder = writer.active.as_mut().expect("open shard");
let kind = RawPayloadKind::InferenceRequest;
let shard = writer.append(kind, "published", b"true")?;
let encoder = &mut writer.active[stream_index(kind)]
.as_mut()
.expect("open shard")
.encoder;
// A valid header and ID promise four body bytes, but the writer stops after two.
encoder.write_all(&1u16.to_le_bytes())?;
encoder.write_all(&4u64.to_le_bytes())?;
@@ -90,8 +163,9 @@ fn live_prefix_survives_a_partial_record_but_a_sealed_frame_rejects_it() -> Resu
fn duplicate_ids_cannot_resolve_to_different_bodies() -> Result<()> {
let directory = TempDir::new()?;
let mut writer = ShardWriter::new(directory.path().to_owned());
let shard = writer.append("same", b"true")?;
writer.append("same", b"false")?;
let kind = RawPayloadKind::InferenceRequest;
let shard = writer.append(kind, "same", b"true")?;
writer.append(kind, "same", b"false")?;
writer.finish()?;
assert!(
DecodedShard::open(directory.path(), shard)
@@ -109,7 +183,7 @@ fn reads_across_a_boundary_keep_the_two_most_recent_shards() -> Result<()> {
let mut refs = Vec::new();
for shard in 1..=3 {
let id = format!("id-{shard}");
let shard = writer.append(&id, b"null")?;
let shard = writer.append(RawPayloadKind::InferenceRequest, &id, b"null")?;
writer.finish()?;
refs.push(RawPayloadRef {
raw_payload_id: id,
@@ -0,0 +1,36 @@
//! Locate large string values in a serialized tool observation without decoding or copying them.
use crate::RawPayloadKind;
use std::ops::Range;
pub(crate) fn large_string_values(body: &[u8]) -> Vec<(Range<usize>, RawPayloadKind)> {
const MIN_BYTES: usize = 64 << 10;
let mut parts = Vec::new();
if body.len() < MIN_BYTES {
return parts;
}
// The writer supplies valid JSON. A quoted token followed by a colon is an object key.
// Take slices of the original encoding so escapes, duplicate keys and order survive.
let mut bytes = body.iter().copied().enumerate();
while let Some((start, _)) = bytes.find(|(_, byte)| *byte == b'"') {
while let Some((index, byte)) = bytes.next() {
match byte {
b'\\' => {
bytes.next();
}
b'"' => {
let end = index + 1;
if end - start >= MIN_BYTES
&& body[end..].iter().find(|b| !b.is_ascii_whitespace()) != Some(&b':')
{
parts.push((start..end, RawPayloadKind::ContentValue));
}
break;
}
_ => {}
}
}
}
parts
}
+3 -28
View File
@@ -103,21 +103,6 @@ struct DispatchedToolTraceRequest<'a> {
payload: &'a JsonValue,
}
/// Raw response payload for dispatch-level tool trace events.
#[derive(Serialize)]
#[serde(rename_all = "snake_case", tag = "type")]
enum DispatchedToolTraceResponse<'a> {
DirectResponse {
response_item: &'a ResponseInputItem,
},
CodeModeResponse {
value: &'a JsonValue,
},
Error {
error: String,
},
}
impl ToolDispatchTraceContext {
/// Builds a context that accepts trace calls and records nothing.
pub(crate) fn disabled() -> Self {
@@ -153,15 +138,7 @@ impl ToolDispatchTraceContext {
let Some(context) = self.active.take() else {
return;
};
let response = match &result {
ToolDispatchResult::DirectResponse { response_item } => {
DispatchedToolTraceResponse::DirectResponse { response_item }
}
ToolDispatchResult::CodeModeResponse { value } => {
DispatchedToolTraceResponse::CodeModeResponse { value }
}
};
append_tool_call_ended(&context, status, &response);
append_tool_call_ended(&context, status, &result);
}
/// Records a dispatch failure before the tool produced a normal result payload.
@@ -172,9 +149,7 @@ impl ToolDispatchTraceContext {
append_tool_call_ended(
&context,
ExecutionStatus::Failed,
&DispatchedToolTraceResponse::Error {
error: error.to_string(),
},
&json!({"type": "error", "error": error.to_string()}),
);
}
}
@@ -344,7 +319,7 @@ fn truncate_preview(value: &str) -> String {
fn append_tool_call_ended(
context: &EnabledToolDispatchTraceContext,
status: ExecutionStatus,
response: &DispatchedToolTraceResponse<'_>,
response: &impl Serialize,
) {
let tool_call_id = context.tool_call_id.clone();
let _ = context.writer.event_with_payload(
+134 -33
View File
@@ -79,7 +79,6 @@ pub(crate) struct BundleWriter {
metadata_bytes: u64,
event_log_full: bool,
items: HashMap<(RawPayloadKind, [u8; 32]), RawPayloadRef>,
writing_content: bool,
next_seq: u64,
next_payload_ordinal: u64,
// Keep the first error so explicit reads and shutdown can detect an incomplete recording.
@@ -107,31 +106,119 @@ impl BundleWriter {
kind: RawPayloadKind,
body: &[u8],
) -> Result<RawPayloadRef> {
let parts = capture_parts(kind, body, self.limits.max_payload_bytes)?;
self.write_body_for_event(kind, body, |_, reference| {
Ok(serde_json::to_vec(&reference)?.len() as u64)
})
}
fn write_body_for_event(
&mut self,
kind: RawPayloadKind,
body: &[u8],
event_bytes: impl Fn(&Self, RawPayloadRef) -> Result<u64>,
) -> Result<RawPayloadRef> {
let tool = matches!(
kind,
RawPayloadKind::ToolResult | RawPayloadKind::ToolRuntimeEvent
);
let parts = if tool {
crate::tool_content::large_string_values(body)
} else {
capture_parts(kind, body, self.limits.max_payload_bytes)?
};
// Deduplication determines how many IDs we assign, so budget for the last possible ID.
// Shard admission determines capture status. Truncation carries two byte counts.
// Within the body cap, omission costs more than a complete ref even with a 20-digit shard.
let last_ordinal = self.next_payload_ordinal.saturating_add(parts.len() as u64);
let ref_bound = |kind, bytes| RawPayloadRef {
raw_payload_id: format!("raw_payload:{last_ordinal}"),
kind,
shard: (bytes > self.limits.max_payload_bytes).then_some(last_ordinal),
capture: if bytes > self.limits.max_payload_bytes {
PayloadCapture::Truncated {
original_bytes: bytes,
retained_bytes: self.limits.max_payload_bytes,
}
} else {
PayloadCapture::Omitted {
original_bytes: bytes,
}
},
parts: Vec::new(),
};
let compact_reserve = if tool {
event_bytes(self, ref_bound(kind, body.len()))?
} else {
0
};
let unsplit = |writer: &mut Self| {
writer.store_body(kind, body, body.len(), Vec::new(), compact_reserve)
};
if parts.is_empty() {
return self.store_body(kind, body, body.len(), Vec::new());
return unsplit(self);
}
// Literal fragments preserve whitespace, key order and escapes without reserving a
// sentinel in user JSON. A missing child occupies exactly one original value slot.
let mut fragments = Vec::with_capacity(parts.len() + 1);
let mut children = Vec::with_capacity(parts.len());
let mut end = 0;
for (range, child_kind) in parts {
for (range, _) in &parts {
fragments.push(std::str::from_utf8(&body[end..range.start])?);
end = range.end;
}
fragments.push(std::str::from_utf8(&body[end..])?);
// The literal structure can itself exceed the cap. Fall back before any child is written.
if tool
&& fragments
.iter()
.map(|fragment| fragment.len())
.sum::<usize>()
> self.limits.max_payload_bytes
{
return unsplit(self);
}
let layout = serde_json::to_vec(&fragments)?;
if tool && layout.len() > self.limits.max_payload_bytes {
return unsplit(self);
}
let mut reserved = 0;
if tool {
// The split stores a compressed layout and shares each distinct value across
// occurrences. The uncompressed event log still lists a ref for every occurrence.
// Budget the whole event before writing children. If it would crowd out later
// lifecycle events, capture the original as one capped or omitted body.
let mut parent = ref_bound(kind, body.len());
parent.parts = parts
.iter()
.map(|(range, kind)| ref_bound(*kind, range.len()))
.collect();
reserved = event_bytes(self, parent)?;
let used = self.metadata_bytes + self.shards.charged_bytes() + self.lifecycle_reserve();
if reserved > self.limits.max_bytes.saturating_sub(used) {
return unsplit(self);
}
}
let mut children = Vec::with_capacity(parts.len());
for (range, child_kind) in parts {
let bytes = &body[range];
let key = (child_kind, Sha256::digest(bytes).into());
let child = if let Some(child) = self.items.get(&key) {
child.clone()
} else {
let child = self.write_body(child_kind, bytes)?;
let child = if child_kind == RawPayloadKind::ContentValue {
self.store_body(child_kind, bytes, bytes.len(), Vec::new(), reserved)?
} else {
self.write_body(child_kind, bytes)?
};
self.items.insert(key, child.clone());
child
};
children.push(child);
}
fragments.push(std::str::from_utf8(&body[end..])?);
self.store_body(kind, &serde_json::to_vec(&fragments)?, body.len(), children)
self.store_body(kind, &layout, body.len(), children, reserved)
}
fn lifecycle_reserve(&self) -> u64 {
(self.limits.max_bytes / 8).min(1 << 20)
}
fn store_body(
@@ -140,14 +227,8 @@ impl BundleWriter {
body: &[u8],
original_bytes: usize,
parts: Vec<RawPayloadRef>,
event_reserve: u64,
) -> Result<RawPayloadRef> {
// Bridge admits whole shards. Keep attachment bytes out of shards that carry
// message structure, so omitting a media shard cannot hide a tool or its links.
let content = kind == RawPayloadKind::ContentValue;
if self.writing_content != content {
self.shards.finish()?;
self.writing_content = content;
}
let ordinal = self.next_payload_ordinal;
let raw_payload_id = format!("raw_payload:{ordinal}");
// Layout is metadata. Applying the individual body cap to it would hide surviving siblings.
@@ -156,15 +237,18 @@ impl BundleWriter {
} else {
body.len()
};
// Leave room for lifecycle events and their explicit omitted-body references.
let reserve = (self.limits.max_bytes / 8).min(1 << 20);
// Admit each shard write by its compressed size. Hold back space for this tool
// event's references until append publishes it, plus the usual lifecycle reserve.
let reserve = self.lifecycle_reserve();
let content = kind == RawPayloadKind::ContentValue;
let shard_limit = self
.limits
.max_bytes
.saturating_sub(event_reserve)
.saturating_sub(self.metadata_bytes + reserve + if content { reserve } else { 0 });
let shard =
self.shards
.append_with_limit(&raw_payload_id, &body[..retained], shard_limit)?;
.append_with_limit(kind, &raw_payload_id, &body[..retained], shard_limit)?;
let capture = if shard.is_none() {
PayloadCapture::Omitted { original_bytes }
} else if retained != body.len() {
@@ -196,15 +280,7 @@ impl BundleWriter {
// Closure is observed here, after draining earlier jobs.
wall_time_unix_ms = unix_time_ms();
}
let event = RawTraceEvent {
schema_version: RAW_TRACE_EVENT_SCHEMA_VERSION,
seq: self.next_seq,
wall_time_unix_ms,
rollout_id: self.rollout_id.clone(),
thread_id: context.thread_id,
codex_turn_id: context.codex_turn_id,
payload,
};
let event = self.raw_event(context, payload, wall_time_unix_ms);
let mut bytes = serde_json::to_vec(&event)?;
bytes.push(b'\n');
if self.metadata_bytes + self.shards.charged_bytes() + bytes.len() as u64
@@ -219,6 +295,23 @@ impl BundleWriter {
self.next_seq += 1;
Ok(event)
}
fn raw_event(
&self,
context: RawTraceEventContext,
payload: RawTraceEventPayload,
wall_time_unix_ms: i64,
) -> RawTraceEvent {
RawTraceEvent {
schema_version: RAW_TRACE_EVENT_SCHEMA_VERSION,
seq: self.next_seq,
wall_time_unix_ms,
rollout_id: self.rollout_id.clone(),
thread_id: context.thread_id,
codex_turn_id: context.codex_turn_id,
payload,
}
}
}
impl TraceWriter {
@@ -274,7 +367,6 @@ impl TraceWriter {
metadata_bytes,
event_log_full: false,
items: HashMap::new(),
writing_content: false,
next_seq: 1,
next_payload_ordinal: 1,
failure: Arc::clone(&failure),
@@ -367,18 +459,27 @@ impl TraceWriter {
/// Encodes a borrowed observation before returning. The worker handles its
/// bytes or serialization error in order, preserving previously accepted evidence.
/// The event constructor also supplies the envelope when budgeting tool references.
pub(crate) fn event_with_payload(
&self,
context: RawTraceEventContext,
kind: RawPayloadKind,
value: &impl Serialize,
event: impl FnOnce(RawPayloadRef) -> RawTraceEventPayload + Send + 'static,
event: impl FnOnce(RawPayloadRef) -> RawTraceEventPayload + Clone + Send + 'static,
) -> Result<()> {
check_recording(&self.failure)?;
let body = serde_json::to_vec(value);
self.defer_event(context, move |bundle| {
let reference = bundle.write_body(kind, &body?)?;
Ok(event(reference))
let now = unix_time_ms();
self.submit(move |bundle| {
let _ = bundle.record(|bundle| {
let reference =
bundle.write_body_for_event(kind, &body?, |writer, reference| {
let candidate =
writer.raw_event(context.clone(), event.clone()(reference), now);
Ok(serde_json::to_vec(&candidate)?.len() as u64 + 1)
})?;
bundle.append(context, event(reference), now)
});
})
}
@@ -390,7 +491,7 @@ impl TraceWriter {
impl Drop for TraceWriter {
fn drop(&mut self) {
// The closing event seals the shard after earlier jobs. Joining makes it visible on return.
// The closing event seals all shards after earlier jobs. Joining makes it visible on return.
if !thread::panicking() {
let _ = self.enqueue_event(
RawTraceEventContext::default(),