Add Frameless Bidi support for realtime conversations (#33261)

## What changed

- Add realtime conversation version `v3`, which preserves the V1 Codex Voice behavior while using Frameless Bidi `delegation.*` events.
- Translate audio, transcripts, handoffs, session context, and lifecycle events between the app server and the Frameless Bidi wire protocol.
- Support `v3` over WebSocket and WebRTC, including the Frameless `/live` endpoint, session configuration, headers, and default model selection.
- Update the app-server protocol schemas and documentation for the new version.

## Testing

- Add unit coverage for Frameless event parsing, outbound messages, context chunking, URL construction, and call creation.
- Add app-server end-to-end coverage for WebSocket delegation and WebRTC session startup.

GitOrigin-RevId: 79a3307bc209e1a54582ebd2febc07c909fca016
This commit is contained in:
jiayuhuang-openai
2026-07-15 05:59:37 +00:00
committed by copyberry
parent 768330dd6c
commit 2e1607ee2f
24 changed files with 1117 additions and 147 deletions
@@ -3304,7 +3304,7 @@ mod tests {
prompt: Some(Some("You are on a call".to_string())),
realtime_session_id: Some("sess_456".to_string()),
transport: None,
version: Some(RealtimeConversationVersion::V1),
version: Some(RealtimeConversationVersion::V3),
voice: Some(RealtimeVoice::Marin),
},
};
@@ -3325,7 +3325,7 @@ mod tests {
"prompt": "You are on a call",
"realtimeSessionId": "sess_456",
"transport": null,
"version": "v1",
"version": "v3",
"voice": "marin"
}
}),
@@ -80,9 +80,9 @@ pub struct ThreadRealtimeStartParams {
/// Optional prefix added to automatic Codex response items when `codexResponsesAsItems` is true.
#[ts(optional = nullable)]
pub codex_response_item_prefix: Option<String>,
/// Optional prefix added to automatic V1 Codex commentary sent with
/// `conversation.handoff.append` when `codexResponsesAsItems` is not true. Final answers are
/// sent without the prefix.
/// Optional prefix added to automatic V1 or V3 Codex commentary sent through the selected
/// Bidi handoff wire event when `codexResponsesAsItems` is not true. Final answers are sent
/// without the prefix.
#[ts(optional = nullable)]
pub codex_response_handoff_prefix: Option<String>,
/// Overrides the configured realtime model for this session only.