Commit Graph
21 Commits
Author SHA1 Message Date
cooper-oai 44f1b0e867 Apply runtime authentication to realtime connections (#60003)
## Why

Realtime connections bypassed runtime authentication and reused captured authentication headers for sideband handshakes. They need to honor the selected credential source and its transport policy across call creation, retries, and reconnects.

## What changed

- Use runtime authentication for realtime WebSocket connections, WebRTC call creation, and sideband attachment when enabled for supported OpenAI destinations.
- Resolve authentication on every WebSocket handshake, preserving credential ownership and request scope while generating fresh assertions. Reject authentication failures without falling back to bearer authentication.
- Keep call creation and sideband attachment bound to the selected credentials and account network policy, including when inference uses a different provider.
- Avoid runtime initialization for local marketplace and MCP operations, plugin commands that do not use the remote catalog, and model debugging with a configured catalog.

## Testing

Add coverage for fresh handshake credentials on retries and reconnects, credential and account changes, runtime authentication modes, destination validation, and registration failures.

GitOrigin-RevId: 57e53122912fa78a524427b0f51013f76f9f7681
2026-11-06 22:27:23 +00:00
guinness-oai c9750d7f50 Preserve realtime V3 transcripts in conversation history (#58221)
## Why

Long realtime conversations could lose speech when active transcripts and handoff transcript fields were truncated.

## What changed

- Record completed V3 transcript chunks in model and client history once the active transcript exceeds 4 KiB, without triggering inference.
- Keep unfinished utterances pending to preserve conversation order, and persist completed chunks before subsequent handoffs.
- Preserve full V3 transcript text at handoff and session-end tail flush, and avoid adding handoff input already recorded in transcript history.

## Testing

Add integration coverage for long transcripts, XML escaping, recording before handoff without inference, repeated handoff input, and both session-end tail flush settings.

GitOrigin-RevId: 62218b811809c4cf5b16fb0247a335eb474f9ab9
2026-10-30 02:43:02 +00:00
lsingh-oai 3d1001e865 Route FedRAMP API keys to government endpoints by default (#57178)
## Why

API keys with FedRAMP issuance markers previously used commercial endpoint defaults without a FedRAMP request header.

## What changed

- Recognize `sk-proj-fedramp-` and `sk-svcacct-fedramp-` prefixes as routing hints and send `X-OpenAI-Fedramp: true`, preserving the complete key for authentication.
- Select `https://gov.api.openai.com/v1` for default OpenAI API routing and `https://gov.chatgpt.com/backend-api/` for default ChatGPT backend requests, including model discovery and analytics.
- Preserve explicit endpoint overrides, including an explicitly configured commercial default URL.
- Resolve provider routing and authentication from the same credential snapshot, and apply key-dependent routing to realtime WebSocket startup.

## Testing

Add regression coverage for prefix recognition, unchanged key storage and authorization, provider credential precedence, endpoint overrides, credential rotation, model discovery, analytics routing, and realtime request preparation.

GitOrigin-RevId: a62576ce205c34e067532b8e6c0e431f87a157e2
2026-10-26 18:20:30 +00:00
Chris Dong 6786dccbe7 Forward client voice session IDs in realtime requests (#53784)
## What changed

Add optional `voiceSessionId` to realtime start parameters and forward it as the `voice-session-id` header for WebSocket, WebRTC, and existing-call transports. Keep this client-generated voice session identifier distinct from `realtimeSessionId`, which uses `x-session-id`.

## Testing

Extend serialization tests and add coverage for distinct session ID headers, WebRTC request forwarding, and existing-call handshake forwarding.

GitOrigin-RevId: cb558dbb9d438f2d34e281199a7e81fe5d33b879
2026-10-13 22:21:31 +00:00
Zahan Malkani 4aaee872e3 Add completion-aware realtime attachment and session-scoped detach (#51539)
## Why

A delayed detach or cleanup from an older realtime conversation must not close its replacement or end the replacement's history segment. Realtime start parameters can also contain credentials in text and transport endpoints, so diagnostics should avoid exposing them.

## What changed

- Add `RealtimeConversationAttach` to report the connection attempt's result through a reply channel, using the sequential start path.
- Add `RealtimeConversationDetach` to close only the matching `realtime_session_id` after earlier starts finish. Detaching an older session leaves the replacement connected.
- Emit the previous conversation's close event before starting its replacement, and prevent stale fanout cleanup from closing or clearing the replacement's context.
- Limit `ConversationStartParams` debug output to output modality, initial item count, and protocol version.

## Testing

Add coverage for attachment completion after the handshake, stale detaches, failed replacement and retry, natural-close ordering, handoff history after replacement, and debug redaction for existing-call and WebRTC transports.

GitOrigin-RevId: 23aee6f70522f0b17e6aa9349dbbcc2ea985f233
2026-10-07 01:28:59 +00:00
rb-oai 822e58cc3d Handle partial answers in realtime routing and thread search (#51260)
## Why

Messages with the `partial_answer` phase contain answer text without declaring a terminal answer. They should remain searchable and speakable without being treated as final messages.

## What changed

- Preserve explicit `partial_answer` phases in streamed and completed realtime handoffs, bypassing legacy channel-prefix parsing and routing them to the `speakable` channel.
- Treat partial answers as nonterminal in V1 handoffs and omit the final-message prefix.
- Include partial answers in thread occurrence searches and normalize them to the final conversation channel in rollout traces.

## Testing

Extend coverage for realtime phase routing, streamed and completed handoffs, partial-answer search results, and rollout normalization. Verify that turn summaries still exclude nonterminal messages.

GitOrigin-RevId: aa183da783f6e7e0b78ddd51073cba533297e121
2026-10-06 03:58:55 +00:00
guinness-oai 90e653014a Add opt-in reasoning status for realtime V3 delegations (#47377)
## What changed

Add `backend_reasoning_status`, disabled by default, to relay public reasoning summaries during active, server-managed realtime V3 delegations.

Use the last non-empty summary from each completed reasoning item, truncate it to 256 bytes at a character boundary, and append it as `[STATUS]` context on the `commentary` channel. Discard queued updates if their delegation is no longer active.

## Testing

Add an integration test covering enabled and disabled behavior, selection of the latest non-empty summary, exclusion of raw reasoning and partial summaries, and delivery without triggering `response.create` or changing the model's reasoning-summary setting.

GitOrigin-RevId: 75ce87dbbbab15a6d9d0ad0b23737738837192fd
2026-09-22 22:20:00 +00:00
Peter Bakkum ecb8013dfa Reconnect WebRTC Realtime sideband transports (#39257)
## What changed

- Reconnect frameless bidirectional WebRTC sideband sockets after unexpected transport loss with capped exponential backoff.
- Preserve pending text or handoff output and bounded transcript state across reconnects.
- Treat `404 Not Found` and `410 Gone` handshake responses as terminal session completion, and preserve HTTP status information from failed WebSocket handshakes.
- Bound Realtime delegation input and transcript fields while retaining the most relevant portion of each.

## Testing

- Add unit coverage for reconnect backoff, terminal status handling, transport error classification, transcript continuity and truncation, and pending outbound messages.
- Add an end-to-end WebRTC conversation test covering an unclean disconnect, reconnect, transcript reconciliation, outbound progress during sustained inbound traffic, and terminal session completion.

GitOrigin-RevId: 544fd7e542c3434d0bb42cfbcc7143604419a037
2026-08-18 18:41:40 +00:00
jiayuhuang-openai 4ebd976312 Support configurable realtime BEM channel prefixes (#34816)
## What changed

- Route realtime V3 `bemTags` handoffs using `[ANALYSIS]`, `[COMMENTARY]`, and `[FINAL]` prefixes by default.
- Add `codexResponseHandoffChannelPrefixes` to realtime start requests so clients can replace the accepted prefixes for individual channels. Each channel may accept multiple prefixes, and omitted channels retain their defaults.
- Apply the configured prefixes to both complete and streamed handoff messages, without treating empty prefixes as matches.

## Testing

- Cover request deserialization, default and custom routing, partial streamed prefixes, empty-prefix handling, and end-to-end V3 handoffs.

GitOrigin-RevId: d0aef50aa71362a5beef2c6f6d28b8633da4e529
2026-07-22 19:43:35 +00:00
jiayuhuang-openai 025db22058 Route realtime V3 handoffs by response channel (#33903)
## What changed

- Add `codexResponseHandoffMode` to `thread/realtime/start` for V3 sessions, with `thinking` as the default and `commentary` and `bemTags` routing modes.
- Map BEM `analysis` and `commentary` output to the commentary channel and `final` output to the speakable channel, while preserving the original BEM envelope. Unrecognized output falls back to speakable.
- Mark explicit speech appends as speakable and stop adding the legacy `"Agent Final Message"` prefix to V3 handoffs. V1 and V2 continue to ignore the new setting.

## Testing

- Cover channel selection for every routing mode, streamed BEM header parsing, fallback behavior, explicit speech, and V1 compatibility.

GitOrigin-RevId: 1974578257488f2914b252c9c1990ba38831c96b
2026-07-17 21:12:40 +00:00
jiayuhuang-openai b03545b5b8 Stream realtime V3 Codex handoff output (#33856)
## What changed

- Stream V3 agent message deltas to `delegation.context.append` at 200 ms intervals instead of waiting for each completed message.
- Send commentary without a prefix and label final or phase-less V1/V3 output with `"Agent Final Message":`.
- Bound streamed output while preserving both its beginning and final tail when truncation is required.
- Remove `codexResponseHandoffPrefix` from `thread/realtime/start`; unknown fields from older clients remain ignored.

## Testing

- Cover incremental V3 commentary and final output, Unicode-safe truncation, V1/V3 phase labeling, and legacy request compatibility.

GitOrigin-RevId: 633c18b5eb9a4f5756a264b63b7bfc185b073951
2026-07-17 15:54:33 +00:00
guinness-oai 22f413bdbf Tag realtime transcript tail flush delegations (#33855)
## What changed

- Add `<source>transcript_tail_flush</source>` to realtime delegation payloads
  emitted when a session flushes its remaining transcript tail.
- Keep explicit realtime handoff payloads unchanged.
- Render both delegation variants through a shared contextual user fragment.

## Testing

- Update realtime conversation tests to cover tagged transcript-tail flushes and
  preserve the existing handoff format and XML escaping behavior.

GitOrigin-RevId: 33f70d16cf45b3af8068565c4cf2b1785be3f1d9
2026-07-17 15:50:13 +00:00
jiayuhuang-openai 2e1607ee2f Add Frameless Bidi support for realtime conversations (#33261)
## What changed

- Add realtime conversation version `v3`, which preserves the V1 Codex Voice behavior while using Frameless Bidi `delegation.*` events.
- Translate audio, transcripts, handoffs, session context, and lifecycle events between the app server and the Frameless Bidi wire protocol.
- Support `v3` over WebSocket and WebRTC, including the Frameless `/live` endpoint, session configuration, headers, and default model selection.
- Update the app-server protocol schemas and documentation for the new version.

## Testing

- Add unit coverage for Frameless event parsing, outbound messages, context chunking, URL construction, and call creation.
- Add app-server end-to-end coverage for WebSocket delegation and WebRTC session startup.

GitOrigin-RevId: 79a3307bc209e1a54582ebd2febc07c909fca016
2026-07-15 05:59:37 +00:00
alexsong-oai 1acb722e8a Support thread-level originator overrides (#29477)
## Why

Work(TPP) threads can be launched from the Desktop app, but if they all
keep the Desktop app's default originator then downstream attribution
cannot distinguish local Work launches from cloud-backed Work launches.
`thread/start.serviceName` already carries that launch signal, while
`SessionMeta.originator` is the durable thread-level value that survives
resume and fork.

This change converts the Desktop Work service names into an effective
originator at thread creation time, persists that originator with the
thread, and keeps using it for later model requests and memory writes.

## What changed

- Map `CODEX_WORK_LOCAL` and `CODEX_WORK_CLOUD` service names to
per-thread originators, while preserving
`CODEX_INTERNAL_ORIGINATOR_OVERRIDE` as the highest-precedence override.
- Persist the effective originator in `SessionMeta.originator`, read it
back on resume/fork, and inherit the parent originator for subagent
spawns when there is no persisted session metadata.
- Handle truncated `SpawnAgentForkMode::LastNTurns` forks by falling
back to the live parent originator when the forked history no longer
includes `SessionMeta`.
- Thread the per-thread originator through Responses headers,
websocket/compaction request paths, thread-store creation, rollout
metadata, and memory stage-one telemetry.

## Verification

- `just test -p codex-core
agent::control::tests::spawn_thread_subagent_inherits_parent_originator_without_fork
agent::control::tests::spawn_thread_subagent_fork_last_n_turns_inherits_parent_originator_without_session_meta
thread_manager::tests::originator_override_precedes_service_name_remapping`
- `just test -p codex-core
agent::control::tests::resume_thread_subagent_restores_stored_metadata_and_effective_multi_agent_mode`
- `just test -p codex-memories-write`
- `just fix -p codex-core -p codex-memories-write`
- `git diff --check`
2026-06-23 17:23:38 -07:00
jiayuhuang-openai 683bd170dc [codex] control automatic realtime handoff delivery (#27986)
## What

Built on the realtime speech-control plumbing merged in #27917.

- Add optional `codexResponseHandoffPrefix` to `thread/realtime/start`.
- Apply that prefix only to automatic V1 commentary sent through
`conversation.handoff.append`; final answers remain unprefixed.
- Add opt-in `clientManagedHandoffs`. When true, core suppresses
automatic response handoffs and completion output so delivery is
controlled by explicit client append APIs.
- Preserve existing automatic behavior by default.
`codexResponsesAsItems: true` continues to select item routing when
client-managed mode is disabled.

## Why

Voice clients need two delivery policies: automatic background context
with silent commentary instructions and fully client-owned handoffs.
Phase-aware prefixing keeps routine commentary silent without
suppressing the final answer, while client-managed mode lets an app
decide exactly which updates to append.

## Validation

- `just fmt`
- `cargo test -p codex-app-server-protocol
serialize_thread_realtime_start`
- `RUST_MIN_STACK=16777216 cargo test -p codex-core --test all
conversation_handoff_persists_across_item_done_until_turn_complete`
- `RUST_MIN_STACK=16777216 cargo test -p codex-app-server --test all
webrtc_v1_client_managed_handoffs_disable_automatic_output`
- `RUST_MIN_STACK=16777216 cargo test -p codex-app-server --test all
webrtc_v1_final_automatic_handoff_omits_silent_prefix`
- `cargo build -p codex-cli --bin codex`
- Local Codex Apps compatibility check: 43 focused webview tests passed,
and a live voice session routed through the source-built app-server.

The explicit `RUST_MIN_STACK` avoids a macOS Tokio test-worker stack
overflow seen with the default test environment.
2026-06-18 02:22:29 +00:00
guinness-oai 1d8ff89aa3 Add realtime speech append control (#27917)
## Why

Realtime voice harness tuning needs app-side control over what backend
Codex text is spoken. Backend orchestrator text is written for a reading
UI, so automatically speaking every preamble, progress update, or final
assistant message can make the realtime voice model too chatty.

For experimentation, clients need two simple controls: keep app/client
text-item injection on the existing item-create path, and add an
explicit speakable path that app code can call only when it wants
realtime to speak. Automatic Codex output also needs an opt-in way to
switch from the protocol's default speakable path to regular realtime
items, with a caller-provided prefix so prompt wording can be tuned
outside core.

The default remains unchanged: if a client omits the new start fields
and never calls `appendSpeech`, automatic backend output continues down
the existing speakable path for the selected realtime protocol.

## What Changed

- Adds experimental `thread/realtime/appendSpeech` for app-provided
speakable text.
- Keeps existing `thread/realtime/appendText` as the item-create API for
app-provided realtime text items.
- Adds `codexResponsesAsItems` / `codex_responses_as_items` on
`thread/realtime/start` to send automatic Codex responses with
`conversation.item.create` instead of the protocol's default speakable
output path.
- Adds `codexResponseItemPrefix` / `codex_response_item_prefix` so
clients can prepend experiment instructions to those automatic Codex
response items.
- Keeps literal `conversation.handoff.append` routing scoped to the v1
speakable path; v2 default speech uses its item/function-output plus
`response.create` behavior.
- Removes the earlier public silent-context API and hardcoded
silent-context prefix.
- Updates realtime tests to cover default automatic speakable behavior,
opt-in automatic item-create behavior, and explicit `appendSpeech`
behavior.

## Validation

- `cargo check -p codex-core -p codex-app-server -p codex-api`
- `just test -p codex-app-server realtime_conversation`
- `just test -p codex-core realtime_conversation` (50/51 passed in the
filtered parallel run; the lone failure passed when rerun in isolation)
- `just test -p codex-core
conversation_mirrors_assistant_message_text_to_realtime_handoff`
- `just test -p codex-api
e2e_connect_and_exchange_events_against_mock_ws_server`
- `just fix -p codex-core`
- `just fix -p codex-app-server`
- `cargo build -p codex-cli`
2026-06-15 16:15:58 -07:00
guinness-oai d6d03d42ea [codex] Fix realtime v1 websocket compatibility (#23771)
## Why

Realtime v1 websocket sessions now expect a slightly different boundary
shape for text input, completed input transcripts, and connection
headers. Codex was still using the older shape, so some v1 text appends
could be rejected before the existing conversation flow could handle
them.

## What changed

- Send v1 user text items with `input_text` content
- Accept v1 turn-marked input transcript events as completed transcripts
- Add the v1 alpha header only for v1 realtime sessions
- Cover the outbound text shape, transcript parsing, and versioned
headers

## Test plan

- `cargo test -p codex-api endpoint::realtime_websocket::methods::tests`
- `cargo test -p codex-core quicksilver_alpha_header`
2026-05-20 16:03:51 -07:00
guinness-oai 126bd6e7a8 Update realtime handoff transcript handling (#18597)
## Summary

This PR aims to improve integration between the realtime model and the
codex agent by sharing more context with each other. In particular, we
now share full realtime conversation transcript deltas in addition to
the delegation message.

realtime_conversation.rs now turns a handoff into:
```
<realtime_delegation>
  <input>...</input>
  <transcript_delta>...</transcript_delta>
</realtime_delegation>
```

## Implementation notes

The transcript is accumulated in the realtime websocket layer as parsed
realtime events arrive. When a background-agent handoff is requested,
the current transcript snapshot is copied onto the handoff event and
then serialized by `realtime_conversation.rs` into the hidden realtime
delegation envelope that Codex receives as user-turn context.

For Realtime V2, the session now explicitly enables input audio
transcription, and the parser handles the relevant input/output
transcript completion events so the snapshot includes both user speech
and realtime model responses. The delegation `<input>` remains the
actual handoff request, while `<transcript_delta>` carries the
surrounding conversation history for context.

Reviewers should note that the transcript payload is intended for Codex
context sharing, not UI rendering. The realtime delegation envelope
should stay hidden from the user-facing transcript surface, while still
being included in the background-agent turn so Codex can answer with the
same conversational context the realtime model had.
2026-04-20 14:04:09 -07:00
guinness-oai 6f5ddd408b Wrap delegated input text (#17868)
## Summary
- wrap routed delegation text in a small XML envelope before submitting
it as a user turn
- escape XML text content so the envelope stays well formed
- update focused coverage for the wrapper and the affected routed-turn
expectations
2026-04-14 19:58:58 -07:00
Ahmed IbrahimandCodex fbd7f9b986 [stack 2/4] Align main realtime v2 wire and runtime flow (#14830)
## Stack Position
2/4. Built on top of #14828.

## Base
- #14828

## Unblocks
- #14829
- #14827

## Scope
- Port the realtime v2 wire parsing, session, app-server, and
conversation runtime behavior onto the split websocket-method base.
- Branch runtime behavior directly on the current realtime session kind
instead of parser-derived flow flags.
- Keep regression coverage in the existing e2e suites.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-03-16 21:38:07 -07:00
Michael Bolin 0c8a36676a fix: move inline codex-rs/core unit tests into sibling files (#14444)
## Why
PR #13783 moved the `codex.rs` unit tests into `codex_tests.rs`. This
applies the same extraction pattern across the rest of `codex-rs/core`
so the production modules stay focused on runtime code instead of large
inline test blocks.

Keeping the tests in sibling files also makes follow-up edits easier to
review because product changes no longer have to share a file with
hundreds or thousands of lines of test scaffolding.

## What changed
- replaced each inline `mod tests { ... }` in `codex-rs/core/src/**`
with a path-based module declaration
- moved each extracted unit test module into a sibling `*_tests.rs`
file, using `mod_tests.rs` for `mod.rs` modules
- preserved the existing `cfg(...)` guards and module-local structure so
the refactor remains structural rather than behavioral

## Testing
- `cargo test -p codex-core --lib` (`1653 passed; 0 failed; 5 ignored`)
- `just fix -p codex-core`
- `cargo fmt --check`
- `cargo shear`
2026-03-12 08:16:36 -07:00