mirror of
https://github.com/openai/codex.git
synced 2026-10-25 18:28:38 +00:00
## What changed - Add an x64 and ARM64 MSVC runtime preparer that uses `dumpbin` to validate PE32+ DLL metadata, selects the declared GStreamer plugins and dependency closure, and copies them unchanged into a private `bin/` directory. - Reject malformed or unsupported PE metadata, path-bearing imports, delayed imports, managed DLLs, forwarded exports, undeclared dependencies, and case-insensitive DLL identity conflicts. - Keep `third_party/voice/sources.json` line endings stable across Windows checkouts so native build receipts remain valid. ## Testing Add native Windows tests for relocated DLL loading with a restricted search path, receipt and digest failures, duplicate identities, malformed PE headers, unsupported loader features, and cleanup after failed preparation. GitOrigin-RevId: 5375f21fc54f5d597f3d4e5f8fbca3df307fa2a8
150 lines
8.7 KiB
Markdown
150 lines
8.7 KiB
Markdown
# Native voice source inputs
|
|
|
|
This stage pins and prepares sources for a privately bundled, GStreamer-based
|
|
audio runtime, including its native dependencies and build tools. It does not
|
|
compile native libraries, link them into Codex or enable voice.
|
|
|
|
`sources.json` records the versions, URLs and SHA-256 digests of 11 archives:
|
|
|
|
| Purpose | Sources |
|
|
| --- | --- |
|
|
| GStreamer framework and plugin sources | `gstreamer`, `gst-plugins-base`, `gst-plugins-good` |
|
|
| Supporting native libraries | `glib`, `libffi`, `pcre2`, `zlib`, `proxy-libintl` |
|
|
| Audio codec | `opus` |
|
|
| Build tools, not runtime libraries | `meson`, `ninja` |
|
|
|
|
GLib also includes `gvdb` in its archive; it is recorded without a separate fetch.
|
|
These inputs do not include the complete platform toolchain for native builds.
|
|
|
|
Bazel uses standard `http_archive` rules to fetch, verify, unpack and cache the
|
|
archives from that manifest. Run from the repository root:
|
|
|
|
```sh
|
|
bazel build //third_party/voice:sources
|
|
```
|
|
|
|
For offline preparation without Bazel, use Python 3.12 or newer and an existing
|
|
archive directory whose filenames match the manifest:
|
|
|
|
```sh
|
|
python3 third_party/voice/prepare_sources.py --archives /path/to/archives --output /path/to/new-sources
|
|
python3 -m unittest discover -s third_party/voice -p 'test_*.py'
|
|
```
|
|
|
|
The adapter verifies archive digests and bounds, then extracts with Python's
|
|
`tarfile` data filter. It preserves links where supported and copies their archive
|
|
targets when link creation is unavailable, including on Windows. It refuses an
|
|
existing output directory and cleans up incomplete output. `prepared.json` records
|
|
successful preparation, not the integrity of later edits to the extracted tree.
|
|
|
|
Ordinary CLI builds do not run either path. The Bazel `:sources` target is
|
|
`manual`, so wildcard builds do not fetch these archives. `:source_inputs`
|
|
exports the manifest and adapter for standalone consumers; `:sources` exposes
|
|
extracted archives with Bazel build metadata. Neither target compiles libraries.
|
|
|
|
Checksums establish input identity, not security or license approval. Native
|
|
compilation, final Cargo/Bazel linking, installed packages, minimum OS support
|
|
and duplex audio validation remain separate stages. These inputs do not establish
|
|
a shared Opus build with Rust consumers or a reduced dependency count.
|
|
|
|
Rust `opus` 0.4.0 is available through Socket. Adding Rust transport dependencies
|
|
and establishing a shared Opus build remain separate integration work.
|
|
|
|
## Native build recipe
|
|
|
|
`build_native.py` runs the unmodified upstream build systems in a new output
|
|
directory, using the same archives. Specify the target and existing compiler,
|
|
CMake, make, pkg-config and shell paths explicitly. It requires a matching
|
|
native host: GNU Linux, macOS, or Windows MSVC, on x64 or ARM64.
|
|
|
|
On macOS, specify the existing release deployment target with
|
|
`--deployment-target`; the host OS version is not an acceptable default.
|
|
Windows requires the normal Visual Studio SDK environment, Cygwin GNU make,
|
|
bash/cygpath and Automake 1.18's standard `ar-lib` for upstream libffi,
|
|
native Windows pkgconf, and `--bootstrap-make` pointing to NMake.
|
|
The recipe does not install these build prerequisites or patch upstream sources.
|
|
The private CI bootstrap verifies the official Cygwin installer and native pkgconf
|
|
MSI hashes before use. It also verifies a retained Cygwin package snapshot against
|
|
pinned archive and member hashes before installing it offline using signed
|
|
metadata. The installed package/version set must exactly match the snapshot
|
|
manifest.
|
|
The MSI is administratively extracted into job storage without a system install.
|
|
Cygwin runs under x64 emulation on ARM64; the compiler probes and emitted DLLs
|
|
must still match the real native target. Native pkgconf relocates libffi's POSIX
|
|
prefix metadata; CI rejects residual Cygwin paths. These are build prerequisites,
|
|
not shipped runtime components or evidence of working voice.
|
|
|
|
CMake libraries use relative install runpaths (`$ORIGIN` on Linux and
|
|
`@loader_path` on Mac), with `@rpath` install names on Mac. Linux Meson links
|
|
use `$ORIGIN:$ORIGIN/..`, matching libraries in `lib/` and plugins in
|
|
`lib/gstreamer-1.0/`. Mac Meson and libffi still need packaging-time fixups;
|
|
these options do not make every Mac library relocatable at installation.
|
|
|
|
Outputs are under `prefix/`, build tools under `tools/`, and logs beside them.
|
|
`build-state.json` records completed commands and failures; `built.json` exists
|
|
only when every build/install command succeeds. Failed builds retain their logs
|
|
and must use a new output directory on retry. CMake compiler-identification logs
|
|
and the recorded tool/configuration inputs remain part of the build provenance.
|
|
|
|
The recipe disables optional plugins and Meson fallback dependency resolution,
|
|
with pkg-config restricted to this prefix. Only system ABI libraries/frameworks
|
|
may remain external; runtime closure inspection must verify that independently.
|
|
`//third_party/voice:build_inputs` exposes the recipe and source inputs to Bazel.
|
|
Neither this filegroup nor a successful prefix build proves final Cargo/Bazel
|
|
linkage, safe private runtime loading, or an installed voice-capable Codex package.
|
|
|
|
## Private macOS runtime projection
|
|
|
|
`macos_runtime.py --prefix <extracted-prefix> --receipts <native-ci-receipts>
|
|
--target <macOS-triple> --output <fresh-directory>` verifies the native receipt,
|
|
source manifest, per-file digests and Mach-O architecture before projecting the
|
|
seven explicit plugins and their declared library dependencies. It removes SDK
|
|
aliases and build-machine runpaths, rewrites private imports relative to each
|
|
loader, and regenerates only development ad-hoc signatures. Inputs are untouched.
|
|
The output must be new and outside the input directories; failures remove only
|
|
that new output. `runtime.json` records source and transformed file identities.
|
|
Xcode's `xcrun llvm-objdump` inspects Mach-O headers and load commands; the
|
|
preparer reads its output and enforces the package dependency policy rather than
|
|
decoding binary structures itself. `install_name_tool` still rewrites paths.
|
|
`runtime.py` owns shared receipt checks, dependency selection, verified copying,
|
|
and cleanup; each platform owns its binary format and loader changes. Output
|
|
containment uses filesystem identity, and copied bytes are checked again before
|
|
transformation so input changes cannot silently invalidate the source receipt.
|
|
|
|
This is a development-only payload, not a signed distribution package or proof of
|
|
audio behavior. Dynamic-only dependencies, native helper linkage, LGPL notices,
|
|
production signing/notarization, Windows/Linux loading and security approval remain
|
|
separate requirements. No microphone, device, plugin scanner or backend is started
|
|
by projection. Run its native relocation tests on macOS with Python 3.12 or newer.
|
|
|
|
## Private GNU Linux runtime preparation
|
|
|
|
`linux_runtime.py` takes the same prefix, receipts, target and output arguments.
|
|
It reads bounded ELF64 headers, segments and dynamic tables directly and accepts
|
|
x64/ARM64 GNU Linux libraries. The shared Python coordinator selects the seven
|
|
plugins and their declared dependencies without changing their bytes. The native
|
|
build must have emitted package-relative runpaths; older absolute paths are
|
|
rejected with a rebuild instruction. Output preserves `lib/gstreamer-1.0/` so
|
|
those relative paths remain valid. Loader audit/filter dependencies and
|
|
path-bearing imports are rejected. Native tests require Python 3.12, a C compiler
|
|
and `patchelf`; the latter constructs malformed inputs and is not needed during
|
|
preparation or shipped in the runtime. The output is development-only, uses the
|
|
host glibc, and does not establish musl or minimum-glibc support, dynamic-only
|
|
dependency closure, helper loading policy or working voice.
|
|
|
|
## Private Windows runtime preparation
|
|
|
|
`windows_runtime.py` takes the same arguments for x64/ARM64 MSVC build prefixes.
|
|
MSVC's existing `dumpbin` reads PE headers, dependencies and exports; the Python
|
|
adapter applies package policy without walking binary structures.
|
|
It checks bounded PE32+ import tables, uses case-insensitive DLL identities, and
|
|
copies the seven plugins and their declared dependencies into one private `bin/`
|
|
directory without changing DLL bytes. Delayed imports, managed DLLs and forwarded
|
|
exports are unsupported and rejected. Native tests require MSVC and Python 3.12;
|
|
they load the moved DLLs using only the DLL directory and System32 search flags.
|
|
This development payload expects the Windows Universal CRT and the matching
|
|
Microsoft Visual C++ runtime (`VCRUNTIME140.dll`) already installed. The latter
|
|
is not a guaranteed OS component. Release redistribution/licensing, Authenticode
|
|
policy and actual helper loading remain separate requirements; this script does
|
|
not install or redistribute Microsoft runtime files or enable voice.
|