Commit Graph
15 Commits
Author SHA1 Message Date
Benjamin Carlsson da20788df9 Bundle Linux voice runtimes and improve audio reliability (#44714)
## Why

Linux voice needs system ALSA plugins and enough buffering to accommodate PipeWire graph cycles without losing capture samples. Voice startup failures also need actionable diagnostics without exposing native error details.

## What changed

- Build and bundle GNU voice helpers and runtimes with primary Linux musl release archives, and sign the archives. Keep Python wheels free of these libraries to preserve `manylinux_2_17` compatibility.
- Discover ALSA plugins in fixed system directories and increase Linux capture and playback buffering to support larger PipeWire graph cycles.
- Report voice failures by stage, preserve negotiation timeout classification, and discard native error sources. Suppress the misleading `requested` closure message after failure cleanup.
- Add explicit Windows MSVC, pkgconf, and CMake toolchain configuration and preserve host architecture in native build environments.

## Testing

Add coverage for Linux release assembly, ALSA plugin discovery, PipeWire capture and playback, classified startup failures, failure cleanup rendering, and Windows build environment handling.

GitOrigin-RevId: d805eace96a669ce3a4489f12e2db6f68f9f7f53
2026-09-11 03:16:01 +00:00
Benjamin Carlsson 28f43b0417 Keep voice sessions alive through mute and audio backlog (#44671)
## Why

Audio queue saturation and processing delays could terminate voice sessions. Muted sessions also need outgoing audio to keep the peer alive.

## What changed

- Drop stale or excess incoming, capture, render-reference, and playback audio so fresh audio can resume. Reset affected processing state while preserving queue bounds and mute generations.
- Send paced, generated Opus silence while muted, without using device or processing buffers.
- Add fixed helper exit stages and session failure categories to default TUI logs without logging raw native errors, SDP, or device error text. Preserve observed helper exit status for shutdown.

## Testing

Extend tests to cover decoded mute silence and RTP sequence continuity, recovery after stale audio and queue saturation, and shutdown after observing helper output closure. Verify the helper's exit stage when the parent pipe closes during blocked startup.

GitOrigin-RevId: 17d580a72acd4423fb0b4f5107d9b541b964d069
2026-09-10 23:25:10 +00:00
Benjamin Carlsson b01c3986fd Add live WebRTC voice conversations to the TUI (#43581)
## What changed

- Add feature-gated `/voice`, `/voice mute`, and `/voice stop` commands with local WebRTC audio and app-server signaling.
- Show live transcripts, conversation status, and microphone and speaker levels. Preserve captions and undelivered answers across thread switches.
- Speak final answers from voice handoffs while keeping delegated reasoning and commentary hidden and typed answers unspoken.
- Retry eligible startup failures once and clean up voice sessions on thread switches and disconnects.
- Stop voice and block late handoffs after a misalignment policy violation. Remove realtime event payloads and spoken text from receipt and debug logs.

## Testing

Add coverage for voice command mapping, rejected starts, disconnect cleanup, delegated final-answer speech, late handoff rejection, and transcript and footer rendering.

GitOrigin-RevId: 11def1f86b0e023b21d0d92072d6636ccad89815
2026-09-07 20:25:30 +00:00
Benjamin Carlsson 9daf7d22ca Add bounded GStreamer playback components to the voice host (#43244)
## What changed

- Add a `GstAudioSink` subclass backed by a bounded mono `F32LE` playback writer, with partial writes, cancellation on speaker epoch changes, and delay accounting for queued samples and pending device output.
- Recreate the CPAL output stream and discard buffered audio when speaker controls change the epoch, preventing stale audio from surviving suppression.
- Add GStreamer dependencies, adjust Windows Bazel native-tool selection, and allow the helper handshake time to load startup-linked libraries.
- Separate installed-client tests from protocol lifecycle tests and supply native startup libraries in test packages.

## Testing

Add unit coverage for partial writes, sample accounting, suppression cancellation, stale writers, device delay, invalid samples, and stalled consumption.

GitOrigin-RevId: c48c48d2fee00659cf6f8a4039a6745385cb77cd
2026-09-06 16:02:27 +00:00
Benjamin Carlsson 64e9a68987 Add a helper-backed realtime WebRTC session API (#43097)
## What changed

- Add `RealtimeWebrtcSession` and cloneable handles for startup, answer negotiation, audio controls, level meters, and error reporting.
- Open devices after answer negotiation and apply the latest controls before enabling audio. Preserve subsequent control transitions in order and close the session if the command queue fills.
- Cancel pending work on explicit close, external cancellation, or final handle drop, with a shared runtime keeping helper reaping alive.
- Add audio-state polling to consume microphone and speaker peaks and detect helper, device, or connection failures.
- Check packaged helper/runtime availability and initialize build information in the standalone TUI for the helper handshake.

## Testing

Add unit and helper-process integration tests covering startup control ordering, queue overflow, cancellation, helper reaping and loss, package availability, SDP debug redaction, and peak accumulation, clamping, and consumption.

GitOrigin-RevId: 458cd80353697f7411cee785f380d1d53aef8416
2026-09-05 20:39:46 +00:00
Benjamin Carlsson 19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00
Benjamin Carlsson 1b53f6a44e Add WebRTC negotiation to the voice host (#42676)
## What changed

- Add helper protocol requests to create a WebRTC offer, apply an answer, and report readiness once the ordered `oai-events` data channel opens.
- Support UDP and TCP candidates while bounding signaling size, remote candidates, inbound TCP streams, and WebRTC buffers. Redact session descriptions and peer errors from diagnostics.
- Close unexpected remote data channels and tear down the peer during helper shutdown, including when the parent pipe is lost during startup.

## Testing

- Exercise offer/answer negotiation and clean shutdown over real UDP and TCP sockets.
- Cover signaling validation, candidate and connection limits, early packet loss, slow TCP connection setup, and blocked-startup parent loss.

GitOrigin-RevId: 5f5e60a3a4d1d8eaca2e4724e66284f54061b6af
2026-09-04 05:07:02 +00:00
Benjamin Carlsson d979df154c Initialize the packaged GStreamer runtime in the voice host (#42631)
## Why

The voice helper previously supported only handshake and shutdown, without
validating that its packaged native runtime could be initialized safely.

## What changed

- Add an `initializeRuntime` protocol exchange that loads GStreamer and the
  required plugins from physical package paths without opening audio devices.
- Restrict plugin discovery, registry access, and native library search paths,
  and keep loaded libraries alive until the helper exits.
- Give initialization a dedicated deadline and terminate the owned helper when
  initialization is cancelled or fails.
- Allow binary-only Rust targets to disable the default Bazel library target.

## Testing

- Cover helper-only packages, cancellation, and environment filtering.
- Add an ignored integration test for initialization from a relocated prepared
  runtime and rejection of duplicate initialization.

GitOrigin-RevId: 8fff68fe26e52cb2e0722dc98fb5f124eddb0d4d
2026-09-03 22:53:45 +00:00
Benjamin Carlsson deb1471166 Read voice helper frames independently of pipe chunks (#42404)
## Why

Pipe output chunks can split one protocol frame or contain several frames. A
timed-out read must also retain any partial frame it already consumed so the next
exchange can continue decoding it.

## What changed

Add a stateful `MessageReader` that buffers partial input, preserves unread bytes
from coalesced chunks, and validates frame bounds before reading the payload. Use
it for voice helper handshake and shutdown responses.

## Testing

Add async tests for fragmented and coalesced frames, cancelled reads, oversized
headers, truncated and malformed input, and invalid pipe chunk sizes.

GitOrigin-RevId: 274d9698e05f2de2147235b6e4e1bd38dbb7c23b
2026-09-03 01:24:12 +00:00
Benjamin Carlsson 13bc770eaf Add installed voice host lifecycle support (#41902)
## What changed

- Add `VoiceHost` to resolve the packaged voice helper, launch it with an
  allowlisted environment, perform the protocol handshake, and enforce bounded
  shutdown and process cleanup.
- Preserve native executable path encoding in the pipe process APIs.
- Add `third_party/voice/assemble_package.py` to create a fresh package copy
  containing a target-compatible helper and a provenance manifest with file
  hashes.

## Testing

- Cover installed helper lifecycle, build matching, missing and symlinked
  helpers, non-UTF-8 package paths, environment filtering, package validation,
  target pairing, and failure cleanup.

GitOrigin-RevId: f893074b36ae6bb9bcedc00fbe7af6bb72745f4c
2026-08-31 19:44:44 +00:00
Benjamin Carlsson d60560f14e Add the voice helper lifecycle foundation (#41897)
## What changed

- Add `codex-realtime-webrtc` with a length-prefixed, 256-byte JSON control protocol for `hello`, `ready`, `close`, and `closed` messages.
- Add a hardened `codex-voice-host` binary that validates protocol and build compatibility, handles orderly shutdown and parent pipe loss, and exposes its build commit.
- Keep this initial foundation limited to process lifecycle management; it does not open audio devices, load native plugins, negotiate WebRTC, or enable voice in the TUI.

## Testing

- Exercise the helper as a subprocess, covering handshake, explicit close, parent pipe loss, malformed frames, incompatible builds, and invalid message order.

GitOrigin-RevId: 6d8d7dd7ba5c4886879882d79314d235f6ab7cf4
2026-08-31 19:08:43 +00:00
Charlie Marsh b93dcf341c Remove the unused realtime WebRTC crate (#33639)
## What changed

- Remove `codex-realtime-webrtc` from the Cargo workspace and Bazel build,
  along with its native WebRTC dependency graph.
- Use `blake3`'s pure Rust implementation on
  `x86_64-pc-windows-gnullvm`, where the Bazel lint toolchain cannot
  reliably materialize the crate's native x86 assembly archives.

GitOrigin-RevId: c382d888cd7ada7a17a7a0dcbdecbe00744f5055
2026-07-16 16:39:09 +00:00
Charlie MarshandCodex 54ef99a365 Disable empty Cargo test targets (#21584)
## Summary

`cargo test` has entails both running standard Rust tests and doctests.
It turns out that the doctest discovery is fairly slow, and it's a cost
you pay even for crates that don't include any doctests.

This PR disables doctests with `doctest = false` for crates that lack
any doctests.

For the collection of crates below, this speeds up test execution by
>4x.

E.g., before this PR:

```
Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
  Time (mean ± σ):      1.849 s ±  4.455 s    [User: 0.752 s, System: 1.367 s]
  Range (min … max):    0.418 s … 14.529 s    10 runs
```

And after:

```
Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
  Time (mean ± σ):     428.6 ms ±   6.9 ms    [User: 187.7 ms, System: 219.7 ms]
  Range (min … max):   418.0 ms … 436.8 ms    10 runs
```

For a single crate, with >2x speedup, before:

```
Benchmark 1: cargo test -p codex-utils-string
  Time (mean ± σ):     491.1 ms ±   9.0 ms    [User: 229.8 ms, System: 234.9 ms]
  Range (min … max):   480.9 ms … 512.0 ms    10 runs
```

And after:

```
Benchmark 1: cargo test -p codex-utils-string
  Time (mean ± σ):     213.9 ms ±   4.3 ms    [User: 112.8 ms, System: 84.0 ms]
  Range (min … max):   206.8 ms … 221.0 ms    13 runs
```

Co-authored-by: Codex <noreply@openai.com>
2026-05-07 15:44:17 -07:00
Ahmed IbrahimandCodex 19bd018300 Wire realtime WebRTC native media into Bazel (#17145)
- Builds codex-realtime-webrtc through the normal Bazel Rust macro so
native macOS WebRTC sources are included.\n- Shares the macOS -ObjC link
flag with Bazel targets that can link libwebrtc.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-08 15:15:55 -07:00
Ahmed IbrahimandCodex d90a348870 Add WebRTC media transport to realtime TUI (#17058)
Adds the `[realtime].transport = "webrtc"` TUI media path using a new
`codex-realtime-webrtc` crate, while leaving app-server as the
signaling/event source.\n\nLocal checks: fmt, diff-check, dependency
tree only; test signal should come from CI.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-08 10:26:55 -07:00