Commit Graph

13 Commits

Author SHA1 Message Date
Benjamin Carlsson
b01c3986fd Add live WebRTC voice conversations to the TUI (#43581)
## What changed

- Add feature-gated `/voice`, `/voice mute`, and `/voice stop` commands with local WebRTC audio and app-server signaling.
- Show live transcripts, conversation status, and microphone and speaker levels. Preserve captions and undelivered answers across thread switches.
- Speak final answers from voice handoffs while keeping delegated reasoning and commentary hidden and typed answers unspoken.
- Retry eligible startup failures once and clean up voice sessions on thread switches and disconnects.
- Stop voice and block late handoffs after a misalignment policy violation. Remove realtime event payloads and spoken text from receipt and debug logs.

## Testing

Add coverage for voice command mapping, rejected starts, disconnect cleanup, delegated final-answer speech, late handoff rejection, and transcript and footer rendering.

GitOrigin-RevId: 11def1f86b0e023b21d0d92072d6636ccad89815
2026-09-07 20:25:30 +00:00
Benjamin Carlsson
9daf7d22ca Add bounded GStreamer playback components to the voice host (#43244)
## What changed

- Add a `GstAudioSink` subclass backed by a bounded mono `F32LE` playback writer, with partial writes, cancellation on speaker epoch changes, and delay accounting for queued samples and pending device output.
- Recreate the CPAL output stream and discard buffered audio when speaker controls change the epoch, preventing stale audio from surviving suppression.
- Add GStreamer dependencies, adjust Windows Bazel native-tool selection, and allow the helper handshake time to load startup-linked libraries.
- Separate installed-client tests from protocol lifecycle tests and supply native startup libraries in test packages.

## Testing

Add unit coverage for partial writes, sample accounting, suppression cancellation, stale writers, device delay, invalid samples, and stalled consumption.

GitOrigin-RevId: c48c48d2fee00659cf6f8a4039a6745385cb77cd
2026-09-06 16:02:27 +00:00
Benjamin Carlsson
64e9a68987 Add a helper-backed realtime WebRTC session API (#43097)
## What changed

- Add `RealtimeWebrtcSession` and cloneable handles for startup, answer negotiation, audio controls, level meters, and error reporting.
- Open devices after answer negotiation and apply the latest controls before enabling audio. Preserve subsequent control transitions in order and close the session if the command queue fills.
- Cancel pending work on explicit close, external cancellation, or final handle drop, with a shared runtime keeping helper reaping alive.
- Add audio-state polling to consume microphone and speaker peaks and detect helper, device, or connection failures.
- Check packaged helper/runtime availability and initialize build information in the standalone TUI for the helper handshake.

## Testing

Add unit and helper-process integration tests covering startup control ordering, queue overflow, cancellation, helper reaping and loss, package availability, SDP debug redaction, and peak accumulation, clamping, and consumption.

GitOrigin-RevId: 458cd80353697f7411cee785f380d1d53aef8416
2026-09-05 20:39:46 +00:00
Benjamin Carlsson
19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00
Benjamin Carlsson
1b53f6a44e Add WebRTC negotiation to the voice host (#42676)
## What changed

- Add helper protocol requests to create a WebRTC offer, apply an answer, and report readiness once the ordered `oai-events` data channel opens.
- Support UDP and TCP candidates while bounding signaling size, remote candidates, inbound TCP streams, and WebRTC buffers. Redact session descriptions and peer errors from diagnostics.
- Close unexpected remote data channels and tear down the peer during helper shutdown, including when the parent pipe is lost during startup.

## Testing

- Exercise offer/answer negotiation and clean shutdown over real UDP and TCP sockets.
- Cover signaling validation, candidate and connection limits, early packet loss, slow TCP connection setup, and blocked-startup parent loss.

GitOrigin-RevId: 5f5e60a3a4d1d8eaca2e4724e66284f54061b6af
2026-09-04 05:07:02 +00:00
Benjamin Carlsson
d979df154c Initialize the packaged GStreamer runtime in the voice host (#42631)
## Why

The voice helper previously supported only handshake and shutdown, without
validating that its packaged native runtime could be initialized safely.

## What changed

- Add an `initializeRuntime` protocol exchange that loads GStreamer and the
  required plugins from physical package paths without opening audio devices.
- Restrict plugin discovery, registry access, and native library search paths,
  and keep loaded libraries alive until the helper exits.
- Give initialization a dedicated deadline and terminate the owned helper when
  initialization is cancelled or fails.
- Allow binary-only Rust targets to disable the default Bazel library target.

## Testing

- Cover helper-only packages, cancellation, and environment filtering.
- Add an ignored integration test for initialization from a relocated prepared
  runtime and rejection of duplicate initialization.

GitOrigin-RevId: 8fff68fe26e52cb2e0722dc98fb5f124eddb0d4d
2026-09-03 22:53:45 +00:00
Benjamin Carlsson
deb1471166 Read voice helper frames independently of pipe chunks (#42404)
## Why

Pipe output chunks can split one protocol frame or contain several frames. A
timed-out read must also retain any partial frame it already consumed so the next
exchange can continue decoding it.

## What changed

Add a stateful `MessageReader` that buffers partial input, preserves unread bytes
from coalesced chunks, and validates frame bounds before reading the payload. Use
it for voice helper handshake and shutdown responses.

## Testing

Add async tests for fragmented and coalesced frames, cancelled reads, oversized
headers, truncated and malformed input, and invalid pipe chunk sizes.

GitOrigin-RevId: 274d9698e05f2de2147235b6e4e1bd38dbb7c23b
2026-09-03 01:24:12 +00:00
Benjamin Carlsson
13bc770eaf Add installed voice host lifecycle support (#41902)
## What changed

- Add `VoiceHost` to resolve the packaged voice helper, launch it with an
  allowlisted environment, perform the protocol handshake, and enforce bounded
  shutdown and process cleanup.
- Preserve native executable path encoding in the pipe process APIs.
- Add `third_party/voice/assemble_package.py` to create a fresh package copy
  containing a target-compatible helper and a provenance manifest with file
  hashes.

## Testing

- Cover installed helper lifecycle, build matching, missing and symlinked
  helpers, non-UTF-8 package paths, environment filtering, package validation,
  target pairing, and failure cleanup.

GitOrigin-RevId: f893074b36ae6bb9bcedc00fbe7af6bb72745f4c
2026-08-31 19:44:44 +00:00
Benjamin Carlsson
d60560f14e Add the voice helper lifecycle foundation (#41897)
## What changed

- Add `codex-realtime-webrtc` with a length-prefixed, 256-byte JSON control protocol for `hello`, `ready`, `close`, and `closed` messages.
- Add a hardened `codex-voice-host` binary that validates protocol and build compatibility, handles orderly shutdown and parent pipe loss, and exposes its build commit.
- Keep this initial foundation limited to process lifecycle management; it does not open audio devices, load native plugins, negotiate WebRTC, or enable voice in the TUI.

## Testing

- Exercise the helper as a subprocess, covering handshake, explicit close, parent pipe loss, malformed frames, incompatible builds, and invalid message order.

GitOrigin-RevId: 6d8d7dd7ba5c4886879882d79314d235f6ab7cf4
2026-08-31 19:08:43 +00:00
Charlie Marsh
b93dcf341c Remove the unused realtime WebRTC crate (#33639)
## What changed

- Remove `codex-realtime-webrtc` from the Cargo workspace and Bazel build,
  along with its native WebRTC dependency graph.
- Use `blake3`'s pure Rust implementation on
  `x86_64-pc-windows-gnullvm`, where the Bazel lint toolchain cannot
  reliably materialize the crate's native x86 assembly archives.

GitOrigin-RevId: c382d888cd7ada7a17a7a0dcbdecbe00744f5055
2026-07-16 16:39:09 +00:00
Charlie Marsh
54ef99a365 Disable empty Cargo test targets (#21584)
## Summary

`cargo test` has entails both running standard Rust tests and doctests.
It turns out that the doctest discovery is fairly slow, and it's a cost
you pay even for crates that don't include any doctests.

This PR disables doctests with `doctest = false` for crates that lack
any doctests.

For the collection of crates below, this speeds up test execution by
>4x.

E.g., before this PR:

```
Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
  Time (mean ± σ):      1.849 s ±  4.455 s    [User: 0.752 s, System: 1.367 s]
  Range (min … max):    0.418 s … 14.529 s    10 runs
```

And after:

```
Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
  Time (mean ± σ):     428.6 ms ±   6.9 ms    [User: 187.7 ms, System: 219.7 ms]
  Range (min … max):   418.0 ms … 436.8 ms    10 runs
```

For a single crate, with >2x speedup, before:

```
Benchmark 1: cargo test -p codex-utils-string
  Time (mean ± σ):     491.1 ms ±   9.0 ms    [User: 229.8 ms, System: 234.9 ms]
  Range (min … max):   480.9 ms … 512.0 ms    10 runs
```

And after:

```
Benchmark 1: cargo test -p codex-utils-string
  Time (mean ± σ):     213.9 ms ±   4.3 ms    [User: 112.8 ms, System: 84.0 ms]
  Range (min … max):   206.8 ms … 221.0 ms    13 runs
```

Co-authored-by: Codex <noreply@openai.com>
2026-05-07 15:44:17 -07:00
Ahmed Ibrahim
19bd018300 Wire realtime WebRTC native media into Bazel (#17145)
- Builds codex-realtime-webrtc through the normal Bazel Rust macro so
native macOS WebRTC sources are included.\n- Shares the macOS -ObjC link
flag with Bazel targets that can link libwebrtc.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-08 15:15:55 -07:00
Ahmed Ibrahim
d90a348870 Add WebRTC media transport to realtime TUI (#17058)
Adds the `[realtime].transport = "webrtc"` TUI media path using a new
`codex-realtime-webrtc` crate, while leaving app-server as the
signaling/event source.\n\nLocal checks: fmt, diff-check, dependency
tree only; test signal should come from CI.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-08 10:26:55 -07:00