Commit Graph
4 Commits
Author SHA1 Message Date
Benjamin Carlsson b01c3986fd Add live WebRTC voice conversations to the TUI (#43581)
## What changed

- Add feature-gated `/voice`, `/voice mute`, and `/voice stop` commands with local WebRTC audio and app-server signaling.
- Show live transcripts, conversation status, and microphone and speaker levels. Preserve captions and undelivered answers across thread switches.
- Speak final answers from voice handoffs while keeping delegated reasoning and commentary hidden and typed answers unspoken.
- Retry eligible startup failures once and clean up voice sessions on thread switches and disconnects.
- Stop voice and block late handoffs after a misalignment policy violation. Remove realtime event payloads and spoken text from receipt and debug logs.

## Testing

Add coverage for voice command mapping, rejected starts, disconnect cleanup, delegated final-answer speech, late handoff rejection, and transcript and footer rendering.

GitOrigin-RevId: 11def1f86b0e023b21d0d92072d6636ccad89815
2026-09-07 20:25:30 +00:00
Benjamin Carlsson 64b482500d Add bounded incoming Opus RTP handling to the voice host (#43100)
## What changed

- Intercept incoming Opus RTP before the upstream track queue, preserving arrival timestamps without retaining a second queued copy.
- Limit outstanding media to 64 packets and 2 MiB, with a 64 KiB per-packet limit. Keep packet and byte permits until the received data is dropped.
- Reject unexpected payload types, changes in SSRC, oversized packets, and exhausted budgets. Report an error when a packet is more than one second old.
- Start with incoming audio suppressed and connect suppression to `speaker_suppressed`. Discard packets from earlier suppression epochs.
- Drain received packets in the host loop; decoding and playback are not implemented by this change.

## Testing

Add unit coverage for RTP bytes and arrival timestamps, budget ownership, suppression epochs, and invalid or oversized packets. Extend transport tests to verify incoming Opus delivery over UDP and TCP, including simulated packet loss and delayed TCP connection setup.

GitOrigin-RevId: 60237efafdf2acc1c9c4c5d2728e978b8131d9e6
2026-09-05 20:58:32 +00:00
Benjamin Carlsson 7dc7c7a756 Send processed microphone audio over RTP in voice-host (#43090)
## Why

`voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio.

## What changed

- Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets.
- Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps.
- Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps.
- Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms.

Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler.

## Testing

Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps.

GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
2026-09-05 20:02:55 +00:00
Benjamin Carlsson 1b53f6a44e Add WebRTC negotiation to the voice host (#42676)
## What changed

- Add helper protocol requests to create a WebRTC offer, apply an answer, and report readiness once the ordered `oai-events` data channel opens.
- Support UDP and TCP candidates while bounding signaling size, remote candidates, inbound TCP streams, and WebRTC buffers. Redact session descriptions and peer errors from diagnostics.
- Close unexpected remote data channels and tear down the peer during helper shutdown, including when the parent pipe is lost during startup.

## Testing

- Exercise offer/answer negotiation and clean shutdown over real UDP and TCP sockets.
- Cover signaling validation, candidate and connection limits, early packet loss, slow TCP connection setup, and blocked-startup parent loss.

GitOrigin-RevId: 5f5e60a3a4d1d8eaca2e4724e66284f54061b6af
2026-09-04 05:07:02 +00:00