Commit Graph
3 Commits
Author SHA1 Message Date
Benjamin Carlsson 9daf7d22ca Add bounded GStreamer playback components to the voice host (#43244)
## What changed

- Add a `GstAudioSink` subclass backed by a bounded mono `F32LE` playback writer, with partial writes, cancellation on speaker epoch changes, and delay accounting for queued samples and pending device output.
- Recreate the CPAL output stream and discard buffered audio when speaker controls change the epoch, preventing stale audio from surviving suppression.
- Add GStreamer dependencies, adjust Windows Bazel native-tool selection, and allow the helper handshake time to load startup-linked libraries.
- Separate installed-client tests from protocol lifecycle tests and supply native startup libraries in test packages.

## Testing

Add unit coverage for partial writes, sample accounting, suppression cancellation, stale writers, device delay, invalid samples, and stalled consumption.

GitOrigin-RevId: c48c48d2fee00659cf6f8a4039a6745385cb77cd
2026-09-06 16:02:27 +00:00
Benjamin Carlsson 7dc7c7a756 Send processed microphone audio over RTP in voice-host (#43090)
## Why

`voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio.

## What changed

- Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets.
- Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps.
- Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps.
- Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms.

Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler.

## Testing

Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps.

GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
2026-09-05 20:02:55 +00:00
Benjamin Carlsson 19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00