mirror of
https://github.com/openai/codex.git
synced 2026-10-22 17:58:46 +00:00
Add realtime speech append control (#27917)
## Why Realtime voice harness tuning needs app-side control over what backend Codex text is spoken. Backend orchestrator text is written for a reading UI, so automatically speaking every preamble, progress update, or final assistant message can make the realtime voice model too chatty. For experimentation, clients need two simple controls: keep app/client text-item injection on the existing item-create path, and add an explicit speakable path that app code can call only when it wants realtime to speak. Automatic Codex output also needs an opt-in way to switch from the protocol's default speakable path to regular realtime items, with a caller-provided prefix so prompt wording can be tuned outside core. The default remains unchanged: if a client omits the new start fields and never calls `appendSpeech`, automatic backend output continues down the existing speakable path for the selected realtime protocol. ## What Changed - Adds experimental `thread/realtime/appendSpeech` for app-provided speakable text. - Keeps existing `thread/realtime/appendText` as the item-create API for app-provided realtime text items. - Adds `codexResponsesAsItems` / `codex_responses_as_items` on `thread/realtime/start` to send automatic Codex responses with `conversation.item.create` instead of the protocol's default speakable output path. - Adds `codexResponseItemPrefix` / `codex_response_item_prefix` so clients can prepend experiment instructions to those automatic Codex response items. - Keeps literal `conversation.handoff.append` routing scoped to the v1 speakable path; v2 default speech uses its item/function-output plus `response.create` behavior. - Removes the earlier public silent-context API and hardcoded silent-context prefix. - Updates realtime tests to cover default automatic speakable behavior, opt-in automatic item-create behavior, and explicit `appendSpeech` behavior. ## Validation - `cargo check -p codex-core -p codex-app-server -p codex-api` - `just test -p codex-app-server realtime_conversation` - `just test -p codex-core realtime_conversation` (50/51 passed in the filtered parallel run; the lone failure passed when rerun in isolation) - `just test -p codex-core conversation_mirrors_assistant_message_text_to_realtime_handoff` - `just test -p codex-api e2e_connect_and_exchange_events_against_mock_ws_server` - `just fix -p codex-core` - `just fix -p codex-app-server` - `cargo build -p codex-cli`
This commit is contained in:
@@ -70,6 +70,12 @@ pub struct ThreadRealtimeStartParams {
|
||||
/// Overrides the configured realtime architecture for this session only.
|
||||
#[ts(optional = nullable)]
|
||||
pub architecture: Option<RealtimeConversationArchitecture>,
|
||||
/// Sends automatic Codex responses as realtime conversation items instead of handoff appends.
|
||||
#[ts(optional = nullable)]
|
||||
pub codex_responses_as_items: Option<bool>,
|
||||
/// Optional prefix added to automatic Codex response items when `codexResponsesAsItems` is true.
|
||||
#[ts(optional = nullable)]
|
||||
pub codex_response_item_prefix: Option<String>,
|
||||
/// Overrides the configured realtime model for this session only.
|
||||
#[ts(optional = nullable)]
|
||||
pub model: Option<String>,
|
||||
@@ -146,6 +152,21 @@ pub struct ThreadRealtimeAppendTextParams {
|
||||
#[ts(export_to = "v2/")]
|
||||
pub struct ThreadRealtimeAppendTextResponse {}
|
||||
|
||||
/// EXPERIMENTAL - append speakable text to thread realtime.
|
||||
#[derive(Serialize, Deserialize, Debug, Default, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[ts(export_to = "v2/")]
|
||||
pub struct ThreadRealtimeAppendSpeechParams {
|
||||
pub thread_id: String,
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
/// EXPERIMENTAL - response for appending realtime speech.
|
||||
#[derive(Serialize, Deserialize, Debug, Default, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[ts(export_to = "v2/")]
|
||||
pub struct ThreadRealtimeAppendSpeechResponse {}
|
||||
|
||||
/// EXPERIMENTAL - stop thread realtime.
|
||||
#[derive(Serialize, Deserialize, Debug, Default, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
|
||||
Reference in New Issue
Block a user