Commit Graph

8 Commits

Author SHA1 Message Date
Benjamin Carlsson
e01f38c388 Require a prepared runtime when assembling voice helper packages (#43121)
## Why

The helper's native bindings require shared libraries before entering `main`, including for lifecycle calls. Helper-only packages cannot satisfy that requirement.

## What changed

Make `--runtime` mandatory in `third_party/voice/assemble_package.py` and require the `runtime` argument in `assemble()`. Always validate and include the prepared runtime, and update the voice host documentation to describe the requirement.

## Testing

Update assembly tests to supply prepared runtimes and verify that package manifests include runtime library hashes and `runtime.json` alongside the helper.

GitOrigin-RevId: ab5abf80ae935b4ace833fa59b74f670ba62c632
2026-09-05 23:09:26 +00:00
Benjamin Carlsson
7dc7c7a756 Send processed microphone audio over RTP in voice-host (#43090)
## Why

`voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio.

## What changed

- Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets.
- Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps.
- Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps.
- Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms.

Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler.

## Testing

Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps.

GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
2026-09-05 20:02:55 +00:00
Benjamin Carlsson
19b62211d9 Add opt-in local audio devices to the voice helper (#43079)
## What changed

- Add `openDevices` and `setAudioControls` to the helper protocol. Open the default microphone and speaker after runtime initialization and transport negotiation, initially muted and suppressed.
- Use CPAL on macOS, GNU Linux, and MSVC Windows, with native audio dependencies confined to `codex-voice-host`. Add ALSA development dependencies to Linux CI.
- Pack callback samples into preallocated, bounded queues and service devices every 5 ms. Reject incompatible callback configurations, invalidate queued audio on control transitions, and reject capture backlog after unmute using device timestamps.
- Handle recoverable stream underruns and overruns without terminating the helper; terminate on device failure, active audio queue overflow, or stale queued audio.

Device opening only establishes local device readiness. Capture and rendered audio references are drained locally; encoding, decoding, and peer audio are not connected by this change.

## Testing

Add unit tests for mute boundaries, stale playback invalidation, sample bounds, partial-block packing and timestamps, callback size and timing limits, startup silence, queue overflow, and recoverable stream errors.

GitOrigin-RevId: 108aea3d411ef6c12908fe5fb67aaf9cddf4475f
2026-09-05 19:11:52 +00:00
Benjamin Carlsson
1b53f6a44e Add WebRTC negotiation to the voice host (#42676)
## What changed

- Add helper protocol requests to create a WebRTC offer, apply an answer, and report readiness once the ordered `oai-events` data channel opens.
- Support UDP and TCP candidates while bounding signaling size, remote candidates, inbound TCP streams, and WebRTC buffers. Redact session descriptions and peer errors from diagnostics.
- Close unexpected remote data channels and tear down the peer during helper shutdown, including when the parent pipe is lost during startup.

## Testing

- Exercise offer/answer negotiation and clean shutdown over real UDP and TCP sockets.
- Cover signaling validation, candidate and connection limits, early packet loss, slow TCP connection setup, and blocked-startup parent loss.

GitOrigin-RevId: 5f5e60a3a4d1d8eaca2e4724e66284f54061b6af
2026-09-04 05:07:02 +00:00
Benjamin Carlsson
d979df154c Initialize the packaged GStreamer runtime in the voice host (#42631)
## Why

The voice helper previously supported only handshake and shutdown, without
validating that its packaged native runtime could be initialized safely.

## What changed

- Add an `initializeRuntime` protocol exchange that loads GStreamer and the
  required plugins from physical package paths without opening audio devices.
- Restrict plugin discovery, registry access, and native library search paths,
  and keep loaded libraries alive until the helper exits.
- Give initialization a dedicated deadline and terminate the owned helper when
  initialization is cancelled or fails.
- Allow binary-only Rust targets to disable the default Bazel library target.

## Testing

- Cover helper-only packages, cancellation, and environment filtering.
- Add an ignored integration test for initialization from a relocated prepared
  runtime and rejection of duplicate initialization.

GitOrigin-RevId: 8fff68fe26e52cb2e0722dc98fb5f124eddb0d4d
2026-09-03 22:53:45 +00:00
Benjamin Carlsson
dc0dc4f15d Package prepared runtimes with the voice host (#42332)
## What changed

- Add an optional `--runtime` input to `assemble_package.py`.
- Validate the runtime receipt, target, source manifest, plugin inventory, paths,
  and file hashes before copying only the declared runtime files.
- Preserve each platform's runtime layout and record the copied files and
  `runtime.json` in the package manifest.
- Recheck hashes after copying and remove the new output if assembly fails.

## Testing

Add package assembly tests for Linux, macOS, and Windows layouts, invalid
receipts and paths, modified inputs, symlinked directories, and copy-time
changes.

GitOrigin-RevId: 8c9609af8ced406926707b2b0e1dcd6d2251f94b
2026-09-02 19:13:32 +00:00
Benjamin Carlsson
13bc770eaf Add installed voice host lifecycle support (#41902)
## What changed

- Add `VoiceHost` to resolve the packaged voice helper, launch it with an
  allowlisted environment, perform the protocol handshake, and enforce bounded
  shutdown and process cleanup.
- Preserve native executable path encoding in the pipe process APIs.
- Add `third_party/voice/assemble_package.py` to create a fresh package copy
  containing a target-compatible helper and a provenance manifest with file
  hashes.

## Testing

- Cover installed helper lifecycle, build matching, missing and symlinked
  helpers, non-UTF-8 package paths, environment filtering, package validation,
  target pairing, and failure cleanup.

GitOrigin-RevId: f893074b36ae6bb9bcedc00fbe7af6bb72745f4c
2026-08-31 19:44:44 +00:00
Benjamin Carlsson
d60560f14e Add the voice helper lifecycle foundation (#41897)
## What changed

- Add `codex-realtime-webrtc` with a length-prefixed, 256-byte JSON control protocol for `hello`, `ready`, `close`, and `closed` messages.
- Add a hardened `codex-voice-host` binary that validates protocol and build compatibility, handles orderly shutdown and parent pipe loss, and exposes its build commit.
- Keep this initial foundation limited to process lifecycle management; it does not open audio devices, load native plugins, negotiate WebRTC, or enable voice in the TUI.

## Testing

- Exercise the helper as a subprocess, covering handshake, explicit close, parent pipe loss, malformed frames, incompatible builds, and invalid message order.

GitOrigin-RevId: 6d8d7dd7ba5c4886879882d79314d235f6ab7cf4
2026-08-31 19:08:43 +00:00