## What changed
- Add the optional `delegationAckFiller` field to `thread/realtime/start`.
- Forward explicit `true` or `false` values to V3 Frameless Bidi session payloads as `delegation.ack_filler`; leave the field absent when unspecified so the Realtime API default is preserved.
- Document that V1 and V2 ignore this setting.
## Testing
- Cover both boolean values in Frameless Bidi serialization and verify the WebRTC V3 session creation payload.
GitOrigin-RevId: 9fe8efd14e42738c3e86e3e9a8af81d0f4521244
## What changed
- Add an optional `initialItems` field to `thread/realtime/start` for seeding
realtime V3 session history with `user`, `developer`, and `assistant` text.
- Encode the items in Frameless Bidi session bootstrap payloads for websocket
and WebRTC transports while preserving existing payloads when the list is
omitted or empty.
- Reject non-empty initial items for V1 and V2, more than 128 items, any item
over 8,192 estimated tokens, or a combined total over 8,192 estimated tokens.
## Testing
- Cover protocol serialization, app-server forwarding, websocket and WebRTC
payloads, empty-list compatibility, version checks, and size limits.
GitOrigin-RevId: 09af367f9a07543e3f8d59b7fa5387f932dc5a6a
## What changed
- Add `codexResponseHandoffMode` to `thread/realtime/start` for V3 sessions, with `thinking` as the default and `commentary` and `bemTags` routing modes.
- Map BEM `analysis` and `commentary` output to the commentary channel and `final` output to the speakable channel, while preserving the original BEM envelope. Unrecognized output falls back to speakable.
- Mark explicit speech appends as speakable and stop adding the legacy `"Agent Final Message"` prefix to V3 handoffs. V1 and V2 continue to ignore the new setting.
## Testing
- Cover channel selection for every routing mode, streamed BEM header parsing, fallback behavior, explicit speech, and V1 compatibility.
GitOrigin-RevId: 1974578257488f2914b252c9c1990ba38831c96b
## What changed
- Add realtime conversation version `v3`, which preserves the V1 Codex Voice behavior while using Frameless Bidi `delegation.*` events.
- Translate audio, transcripts, handoffs, session context, and lifecycle events between the app server and the Frameless Bidi wire protocol.
- Support `v3` over WebSocket and WebRTC, including the Frameless `/live` endpoint, session configuration, headers, and default model selection.
- Update the app-server protocol schemas and documentation for the new version.
## Testing
- Add unit coverage for Frameless event parsing, outbound messages, context chunking, URL construction, and call creation.
- Add app-server end-to-end coverage for WebSocket delegation and WebRTC session startup.
GitOrigin-RevId: 79a3307bc209e1a54582ebd2febc07c909fca016
## Why
Frontend realtime voice continuity needs to replay a tiny
previous-session overlap as actual conversation items, including
assistant text. The app-server `thread/realtime/appendText` API already
carries a role through to the Rust realtime websocket layer, but the
shared role enum only accepted `user` and `developer`.
## What Changed
- Added `assistant` to `ConversationTextRole` and regenerated the
app-server schema/type fixtures.
- Added `output_text` as a realtime conversation content type.
- Updated realtime websocket item creation so assistant appendText emits
`content: [{ type: "output_text", text }]`, while user and developer
continue to emit `input_text`.
- Updated app-server docs and tests to cover assistant appendText
alongside the existing developer role behavior.
## Validation
- `just write-app-server-schema`
- `just fmt` (first sandboxed attempt failed because `uv` could not
access `~/.cache/uv`; reran with filesystem access and passed)
- `just test -p codex-api` passed: 126/126
- `just test -p codex-app-server-protocol` passed: 239/239, including
generated JSON/TypeScript fixture checks
- `just test -p codex-app-server` was started locally but stopped per
request after unrelated local sandbox/Seatbelt failures (`sandbox-exec:
sandbox_apply: Operation not permitted`) and one missing local `codex`
binary failure; CI should be faster and more authoritative for the full
suite.
## Summary
Add an explicit `user` or `developer` role to
`thread/realtime/appendText` and propagate it through the realtime input
queue into `conversation.item.create`. Older JSON clients that omit the
field continue to default to `user`.
This lets app-provided context such as memory retain developer authority
without bypassing app-server through a renderer-owned data channel. The
app-server schemas, API documentation, and focused protocol and
websocket coverage are updated with the new contract.
The Codex Apps consumer is tracked in
[openai/openai#1025261](https://github.com/openai/openai/pull/1025261).
## Why
Realtime v1 websocket sessions now expect a slightly different boundary
shape for text input, completed input transcripts, and connection
headers. Codex was still using the older shape, so some v1 text appends
could be rejected before the existing conversation flow could handle
them.
## What changed
- Send v1 user text items with `input_text` content
- Accept v1 turn-marked input transcript events as completed transcripts
- Add the v1 alpha header only for v1 realtime sessions
- Cover the outbound text shape, transcript parsing, and versioned
headers
## Test plan
- `cargo test -p codex-api endpoint::realtime_websocket::methods::tests`
- `cargo test -p codex-core quicksilver_alpha_header`
## Summary
This PR aims to improve integration between the realtime model and the
codex agent by sharing more context with each other. In particular, we
now share full realtime conversation transcript deltas in addition to
the delegation message.
realtime_conversation.rs now turns a handoff into:
```
<realtime_delegation>
<input>...</input>
<transcript_delta>...</transcript_delta>
</realtime_delegation>
```
## Implementation notes
The transcript is accumulated in the realtime websocket layer as parsed
realtime events arrive. When a background-agent handoff is requested,
the current transcript snapshot is copied onto the handoff event and
then serialized by `realtime_conversation.rs` into the hidden realtime
delegation envelope that Codex receives as user-turn context.
For Realtime V2, the session now explicitly enables input audio
transcription, and the parser handles the relevant input/output
transcript completion events so the snapshot includes both user speech
and realtime model responses. The delegation `<input>` remains the
actual handoff request, while `<transcript_delta>` carries the
surrounding conversation history for context.
Reviewers should note that the transcript payload is intended for Codex
context sharing, not UI rendering. The realtime delegation envelope
should stay hidden from the user-facing transcript surface, while still
being included in the background-agent turn so Codex can answer with the
same conversational context the realtime model had.
## Summary
- set the realtime v2 server VAD silence delay to 500ms
- update the default realtime 1.5 backend prompt to the v4 text
- keep the session payload and prompt rendering tests aligned with those
changes
## Why
- the VAD change gives the voice path a longer pause before ending the
user's turn
- the prompt change makes the default bundled realtime prompt match the
current v4 content
## Validation
- `cargo +1.93.0 test -p codex-core realtime_prompt --manifest-path
/tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml`
- `CARGO_TARGET_DIR=/tmp/codex-pr-v4-target cargo +1.93.0 test -p
codex-api
realtime_v2_session_update_includes_background_agent_tool_and_handoff_output_item
--manifest-path
/tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml`
- `CARGO_TARGET_DIR=/tmp/codex-pr-v4-target cargo +1.93.0 test -p
codex-app-server --test all
'suite::v2::realtime_conversation::realtime_webrtc_start_emits_sdp_notification'
--manifest-path /tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml
-- --exact`
- Add outputModality to thread/realtime/start and wire text/audio output
selection through app-server, core, API, and TUI.\n- Rename the realtime
transcript delta notification and add a separate transcript done
notification that forwards final text from item done without correlating
it with deltas.
Adds WebRTC startup to the experimental app-server
`thread/realtime/start` method with an optional transport enum. The
websocket path remains the default; WebRTC offers create the realtime
session through the shared start flow and emit the answer SDP via
`thread/realtime/sdp`.
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- reduce public module visibility across Rust crates, preferring private
or crate-private modules with explicit crate-root public exports
- update external call sites and tests to use the intended public crate
APIs instead of reaching through module trees
- add the module visibility guideline to AGENTS.md
## Validation
- `cargo check --workspace --all-targets --message-format=short` passed
before the final fix/format pass
- `just fix` completed successfully
- `just fmt` completed successfully
- `git diff --check` passed
## Stack Position
2/4. Built on top of #14828.
## Base
- #14828
## Unblocks
- #14829
- #14827
## Scope
- Port the realtime v2 wire parsing, session, app-server, and
conversation runtime behavior onto the split websocket-method base.
- Branch runtime behavior directly on the current realtime session kind
instead of parser-derived flow flags.
- Keep regression coverage in the existing e2e suites.
---------
Co-authored-by: Codex <noreply@openai.com>
- add experimental_realtime_ws_mode (conversational/transcription) and
plumb it into realtime conversation session config
- switch realtime websocket intent and session.update payload shape
based on mode
- update config schema and realtime/config tests
---------
Co-authored-by: Codex <noreply@openai.com>
- Advertise a `codex` function tool in realtime v2 session updates.
- Emit handoff replies as `function_call_output` items while keeping v1
behavior unchanged.
- Split realtime event parsing into explicit v1/v2 modules with shared
common helpers.
---------
Co-authored-by: Codex <noreply@openai.com>
- Add a feature-flagged realtime v2 parser on the existing
websocket/session pipeline.
- Wire parser selection from core feature flags and map the codex
handoff tool-call path into existing handoff events.
---------
Co-authored-by: Codex <noreply@openai.com>
- collect input/output transcript deltas into active handoff transcript
state
- attach and clear that transcript on each handoff, and regenerate
schema/tests
- migrate the realtime websocket transport to the new session and
handoff flow
- make the realtime model configurable in config.toml and use API-key
auth for the websocket
---------
Co-authored-by: Codex <noreply@openai.com>
- Introduce `RealtimeConversationManager` for realtime API management
- Add `op::conversation` to start conversation, insert audio, insert
text, and close conversation.
- emit conversation lifecycle and realtime events.
- Move shared realtime payload types into codex-protocol and add core
e2e websocket tests for start/replace/transport-close paths.
Things to consider:
- Should we use the same `op::` and `Events` channel to carry audio? I
think we should try this simple approach and later we can create
separate one if the channels got congested.
- Sending text updates to the client: we can start simple and later
restrict that.
- Provider auth isn't wired for now intentionally
## Summary
- add realtime websocket client transport in codex-api
- send session.create on connect with backend prompt and optional
conversation_id
- keep session.update for prompt changes after connect
- switch inbound event parsing to a tagged enum (typed variants instead
of optional field bag)
- add a websocket e2e integration test in
codex-rs/codex-api/tests/realtime_websocket_e2e.rs
## Why
This moves the realtime transport to an explicit session-create
handshake and improves protocol safety with typed inbound events.
## Testing
- Added e2e integration test coverage for session create + event flow in
the API crate.
jiayuhuang-openai