Commit Graph
4 Commits
Author SHA1 Message Date
Benjamin Carlsson a9896da3fe Connect voice-host RTP audio to speaker playback (#43248)
## Why

The voice host drained incoming RTP packets without playing them. Route received audio through decoding and speaker output while preserving speaker suppression boundaries.

## What changed

- Add a GStreamer pipeline with RTP jitter buffering, Opus decoding, conversion, and resampling into the native speaker sink.
- Rebuild playout when the speaker is re-enabled, reject packets received before the new pipeline started, and discard queued audio and echo references on speaker transitions while preserving capture history.
- Limit each receive pass to 64 packets so incoming media cannot indefinitely delay audio controls.
- Prevent retired sink failures from failing a new speaker generation, and retain the last coherent playback delay estimate during callback contention.

## Testing

Add tests for real Opus playback and stale packet rejection, combined microphone mute and speaker suppression, retired sink failures, delay contention, and echo-reference reset without losing capture history.

GitOrigin-RevId: c91c94a2ba2347ac98af4e7fd1dc331ae766cc45
2026-09-06 16:40:33 +00:00
Benjamin Carlsson 64e9a68987 Add a helper-backed realtime WebRTC session API (#43097)
## What changed

- Add `RealtimeWebrtcSession` and cloneable handles for startup, answer negotiation, audio controls, level meters, and error reporting.
- Open devices after answer negotiation and apply the latest controls before enabling audio. Preserve subsequent control transitions in order and close the session if the command queue fills.
- Cancel pending work on explicit close, external cancellation, or final handle drop, with a shared runtime keeping helper reaping alive.
- Add audio-state polling to consume microphone and speaker peaks and detect helper, device, or connection failures.
- Check packaged helper/runtime availability and initialize build information in the standalone TUI for the helper handshake.

## Testing

Add unit and helper-process integration tests covering startup control ordering, queue overflow, cancellation, helper reaping and loss, package availability, SDP debug redaction, and peak accumulation, clamping, and consumption.

GitOrigin-RevId: 458cd80353697f7411cee785f380d1d53aef8416
2026-09-05 20:39:46 +00:00
Benjamin Carlsson 7dc7c7a756 Send processed microphone audio over RTP in voice-host (#43090)
## Why

`voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio.

## What changed

- Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets.
- Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps.
- Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps.
- Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms.

Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler.

## Testing

Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps.

GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
2026-09-05 20:02:55 +00:00
Benjamin Carlsson 19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00