Move rollout trace recording to a background worker (#61253)

## Why

Recording inference starts synchronously serializes prompts and writes trace payloads before model work can proceed. Defer that work so inference dispatch can continue while the trace is recorded.

## What changed

- Use one worker per trace bundle to write events in submission order, with up to eight pending jobs.
- Share prepared prompts and encoded requests with the worker, and build remote compaction snapshots there. Write payloads before publishing events that reference them.
- Add `TraceWriter::flush` and `ThreadTraceContext::flush` for live reads. Drain accepted jobs and seal the bundle on shutdown; suppress the closing marker after a recording failure.

## Testing

Add tests for deferred prompt serialization, ordered start and completion events, shutdown draining, and serialization failures that preserve earlier evidence without publishing a closing marker. Verify compaction snapshots are built on the worker and skipped when tracing is disabled.

GitOrigin-RevId: 32fd787bb1d281f1814df2f08c73b667628f8add
This commit is contained in:
cassirer-openai
2026-11-13 04:36:03 +00:00
committed by copyberry
parent aa5f492031
commit b8dde1eb40
26 changed files with 599 additions and 442 deletions
+1 -1
View File
@@ -441,7 +441,7 @@ async fn schedule_startup_prewarm_inner(
// Prewarm establishes the request baseline before the first turn can change effort.
client_session
.prewarm_websocket(
&startup_prompt,
&Arc::new(startup_prompt),
&step_context.settings.model_info,
&step_context.session_telemetry,
session
+2 -1
View File
@@ -3000,7 +3000,8 @@ async fn try_run_sampling_request(
None
};
let mut resampling = crate::guardian::ResponseResampling::new(&sess);
let mut stream_request = async |prompt: &Prompt| {
let prompt = Arc::new(prompt);
let mut stream_request = async |prompt: &Arc<Prompt>| {
client_session
.stream(
prompt,