Commit Graph
7 Commits
Author SHA1 Message Date
Benjamin Carlsson fc81a7154b Add local audio device selection to TUI voice settings (#49437)
## Why

Voice conversations previously opened the system-default microphone and speaker. Users need to choose which local devices to use, including when connected to a remote app server.

## What changed

- Add input and output device pickers alongside voice selection, with back navigation and a System default option.
- Persist selections as `audio.microphone` and `audio.speaker` on the TUI's machine, excluding project configuration layers. Apply them to the next voice conversation.
- Enumerate devices without opening audio streams and pass the selected names to the voice helper at startup.
- Disable ambiguous device names in the picker and use the same eligibility filtering for discovery and named-device startup.

## Testing

Add tests for device preference persistence across project and remote transitions, picker navigation and duplicate names, discovery without capture, and forwarding selected devices to the helper.

GitOrigin-RevId: d48fb3a09fa9b982a408f5a7c96bdd81776af21d
2026-09-30 01:58:03 +00:00
Benjamin Carlsson da20788df9 Bundle Linux voice runtimes and improve audio reliability (#44714)
## Why

Linux voice needs system ALSA plugins and enough buffering to accommodate PipeWire graph cycles without losing capture samples. Voice startup failures also need actionable diagnostics without exposing native error details.

## What changed

- Build and bundle GNU voice helpers and runtimes with primary Linux musl release archives, and sign the archives. Keep Python wheels free of these libraries to preserve `manylinux_2_17` compatibility.
- Discover ALSA plugins in fixed system directories and increase Linux capture and playback buffering to support larger PipeWire graph cycles.
- Report voice failures by stage, preserve negotiation timeout classification, and discard native error sources. Suppress the misleading `requested` closure message after failure cleanup.
- Add explicit Windows MSVC, pkgconf, and CMake toolchain configuration and preserve host architecture in native build environments.

## Testing

Add coverage for Linux release assembly, ALSA plugin discovery, PipeWire capture and playback, classified startup failures, failure cleanup rendering, and Windows build environment handling.

GitOrigin-RevId: d805eace96a669ce3a4489f12e2db6f68f9f7f53
2026-09-11 03:16:01 +00:00
Benjamin Carlsson 28f43b0417 Keep voice sessions alive through mute and audio backlog (#44671)
## Why

Audio queue saturation and processing delays could terminate voice sessions. Muted sessions also need outgoing audio to keep the peer alive.

## What changed

- Drop stale or excess incoming, capture, render-reference, and playback audio so fresh audio can resume. Reset affected processing state while preserving queue bounds and mute generations.
- Send paced, generated Opus silence while muted, without using device or processing buffers.
- Add fixed helper exit stages and session failure categories to default TUI logs without logging raw native errors, SDP, or device error text. Preserve observed helper exit status for shutdown.

## Testing

Extend tests to cover decoded mute silence and RTP sequence continuity, recovery after stale audio and queue saturation, and shutdown after observing helper output closure. Verify the helper's exit stage when the parent pipe closes during blocked startup.

GitOrigin-RevId: 17d580a72acd4423fb0b4f5107d9b541b964d069
2026-09-10 23:25:10 +00:00
Benjamin Carlsson 9daf7d22ca Add bounded GStreamer playback components to the voice host (#43244)
## What changed

- Add a `GstAudioSink` subclass backed by a bounded mono `F32LE` playback writer, with partial writes, cancellation on speaker epoch changes, and delay accounting for queued samples and pending device output.
- Recreate the CPAL output stream and discard buffered audio when speaker controls change the epoch, preventing stale audio from surviving suppression.
- Add GStreamer dependencies, adjust Windows Bazel native-tool selection, and allow the helper handshake time to load startup-linked libraries.
- Separate installed-client tests from protocol lifecycle tests and supply native startup libraries in test packages.

## Testing

Add unit coverage for partial writes, sample accounting, suppression cancellation, stale writers, device delay, invalid samples, and stalled consumption.

GitOrigin-RevId: c48c48d2fee00659cf6f8a4039a6745385cb77cd
2026-09-06 16:02:27 +00:00
Benjamin Carlsson 64e9a68987 Add a helper-backed realtime WebRTC session API (#43097)
## What changed

- Add `RealtimeWebrtcSession` and cloneable handles for startup, answer negotiation, audio controls, level meters, and error reporting.
- Open devices after answer negotiation and apply the latest controls before enabling audio. Preserve subsequent control transitions in order and close the session if the command queue fills.
- Cancel pending work on explicit close, external cancellation, or final handle drop, with a shared runtime keeping helper reaping alive.
- Add audio-state polling to consume microphone and speaker peaks and detect helper, device, or connection failures.
- Check packaged helper/runtime availability and initialize build information in the standalone TUI for the helper handshake.

## Testing

Add unit and helper-process integration tests covering startup control ordering, queue overflow, cancellation, helper reaping and loss, package availability, SDP debug redaction, and peak accumulation, clamping, and consumption.

GitOrigin-RevId: 458cd80353697f7411cee785f380d1d53aef8416
2026-09-05 20:39:46 +00:00
Benjamin Carlsson 7dc7c7a756 Send processed microphone audio over RTP in voice-host (#43090)
## Why

`voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio.

## What changed

- Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets.
- Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps.
- Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps.
- Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms.

Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler.

## Testing

Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps.

GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
2026-09-05 20:02:55 +00:00
Benjamin Carlsson 19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00