## Why
Exec-server process and network-policy events need stable attribution to the
launching tool call and executor without recording process payloads.
## What changed
- Add optional `ExecMetadata` to `ExecParams` and propagate the thread and tool
call IDs from unified exec.
- Emit bounded OpenTelemetry events for process start, spawn failure, sandbox
denial, and exit. Correlate them with the launch trace and registry-issued
executor identity while excluding arguments, paths, environment values,
output, and error text.
- Preserve launch attribution across long-running processes and reconnects, and
attach the same metadata to network-policy audit events.
- Keep the protocol backward compatible when metadata is omitted, and prevent
invalid trace headers from inheriting an unrelated active span.
## Testing
Add coverage for metadata serialization and propagation, lifecycle event
fields, trace relationships, reconnect behavior, spawn failures, sandbox
denials, and network-policy attribution.
GitOrigin-RevId: 7aa480a7289c73cb95e2c124c35500bb6f0d5084
## What changed
- Add a `NoiseStreamHandler` abstraction that owns payload encoding, decoding,
and connection processing while the virtual stream handles framing,
encryption, and multiplexing.
- Reassemble authenticated payloads as opaque bytes, with the existing
JSON-RPC processor supplied as a handler implementation.
- Have the physical relay send a reset after the current stream instance
closes, including when its processor exits before the writer task.
## Testing
- Verify local JSON-RPC decoding creates the queued request span before queue
admission.
- Verify a processor exit resets the corresponding Noise harness stream.
GitOrigin-RevId: 270befcc8bfbf27e46c84d6a2c6e349cb98ea3f8
## What changed
- Start inbound exec-server request spans when messages enter the connection queue and carry them through dispatch and response handling.
- Record request outcomes for client-handled network policy callbacks, including errors and disconnections.
- Add the `exec_server_request_queue_duration_seconds` histogram, labeled by bounded route name, while excluding synchronous route setup time.
## Testing
- Cover span lifetime and trace-parent propagation across server and client queues.
- Verify queue-duration telemetry and outcome recording for completed, rejected, and cancelled requests.
GitOrigin-RevId: ed67fe5305048bdf283a26ec874337d549e3324f
## What changed
- Add optional W3C `traceparent` and `tracestate` fields to relay frames.
- Copy trace context from JSON-RPC requests onto relay data frames.
- For encrypted requests split across multiple Noise records, attach the context only to the first record while keeping the request payload encrypted.
## Testing
- Cover trace propagation for both Noise relay paths, including fragmented encrypted requests.
GitOrigin-RevId: a61bbbefef31e2e7e93a43f439c5f296700feb7b
## Why
Remote environment connections need to honor Codex's effective outbound proxy policy, including when a rendezvous connection reconnects.
## What changed
- Pass the configured `HttpClientFactory` into remote environment transports and use `WebSocketConnector` for exec-server and rendezvous WebSockets.
- Resolve proxy routes asynchronously so these connections can use the configured system proxy.
- Add connector options that preserve Tungstenite's default TLS behavior and enable `TCP_NODELAY` for latency-sensitive rendezvous traffic.
## Testing
- Verify prepared remote environments connect through a configured system proxy.
- Verify initial and reconnected encrypted relay peers use the system proxy.
- Cover default TLS selection and opt-in `TCP_NODELAY` behavior in the WebSocket client.
GitOrigin-RevId: 8a8da2116e37cb3a891269d0c0b037986fecdd3c
## Summary
- require a Pong within 60 seconds for established Noise Rendezvous
WebSockets on both the harness and executor
- bound steady-state WebSocket writes and harness event delivery so
backpressure cannot mask the deadline
- classify executor disconnects with bounded reasons and feed them into
the existing reconnect metric and structured log
- cover silent peers, responsive peers, continuous non-Pong traffic, and
local application backpressure
## Why
The existing periodic Pings did not track Pongs, so a half-open or
blackholed connection could remain stuck until the operating system's
TCP timeout. This adds the smallest explicit liveness contract without
new spans, RTT histograms, feature flags, or TCP diagnostics.
## Testing
- `just test -p codex-exec-server` on devbox `richard-6` — 300 passed, 2
skipped
- `just fix -p codex-exec-server`
- `just fmt`
- independent correctness, performance/security, and YAGNI reviews — no
findings
## Why
Exec-server JSON-RPC calls can cross local and remote transports, but
trace context stopped at the RPC boundary. That made client and server
work difficult to correlate when diagnosing latency or failures.
## What changed
- Propagate the current W3C trace context on outbound JSON-RPC requests.
- Parent inbound request spans from received trace context.
- Record the received JSON-RPC method on server spans and keep each span
open through response enqueue.
- Add only the OTEL dependencies required by the exec-server crate.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests **(this PR)**
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (153 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server`
## Why
The app-server and exec-server expose separate JSON-RPC APIs, but
exec-server currently sources its serialized protocol and envelope types
through app-server-oriented code. Giving each API an explicit owner
makes the crate boundary legible without introducing shared generic
envelopes.
## What changed
- Added `codex-exec-server-protocol` to own exec DTOs, process IDs, and
JSON-RPC envelopes.
- Updated exec-server clients, transports, handlers, and tests to use
the new crate.
- Exposed app-server's existing JSON-RPC types through a public `rpc`
module while retaining root re-exports.
- Preserved existing wire shapes, including exec `PathUri` behavior.
## Stack
This is PR 1 of 6. Next: [PR
#29721](https://github.com/openai/codex/pull/29721), which moves auth
mode below the app wire boundary.
## Validation
- Exec-server protocol and server coverage passed in the focused
protocol test runs.
- App-server protocol schema fixtures passed.
## Summary
- Initialize stderr tracing and the configured OpenTelemetry provider
for local and remote `codex exec-server` startup.
- Instrument the local and remote server entrypoints with a root runtime
span.
- Keep raw Noise environment, registration, and stream identifiers out
of exported spans while preserving them in local debug events.
- Keep telemetry setup in a focused CLI module instead of growing the
top-level command entrypoint.
## Stack
- Previous: none (`#27058` has merged)
- Next: #27466
## Validation
- `just test -p codex-exec-server --lib` (139 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
---------
Co-authored-by: Richard Lee <richardlee@openai.com>
## Why
The transport in
[openai/codex#26242](https://github.com/openai/codex/pull/26242) needs
to be used by every remote orchestrator-to-executor connection before
JSON-RPC traffic starts.
## Changes
- Generates one executor Noise identity when remote exec-server starts
and registers its public key.
- Creates a harness identity for each physical remote environment
connection.
- Fetches a fresh registry bundle before connecting and validates the
authenticated harness key before completing the executor handshake.
- Multiplexes encrypted logical streams over the existing executor
WebSocket.
- Adds bounded stream, handshake-failure, and reassembly state.
- Adds safe lifecycle diagnostics without logging keys, authorizations,
plaintext, or ciphertext.
- Covers reconnects, replay rejection, validation failure, framing
limits, and encrypted JSON-RPC tool traffic.
## Stack
1. [openai/codex#26242](https://github.com/openai/codex/pull/26242):
Noise channel and relay transport
2. **[openai/codex#26245](https://github.com/openai/codex/pull/26245)**:
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
Rendezvous forwards traffic between the orchestrator and exec-server.
The endpoints need to authenticate each other and encrypt that traffic
without trusting Rendezvous with plaintext or endpoint keys.
## Changes
- Adds a hybrid Noise IK channel through Clatter using X25519,
ML-KEM-768, AES-256-GCM, and SHA-256.
- Binds each handshake to `environment_id`, `executor_registration_id`,
and `stream_id`.
- Pins the registry-provided executor key and carries the harness
authorization inside the encrypted handshake.
- Orders relay frames before consuming Noise nonces and fragments large
JSON-RPC messages into bounded records.
- Bounds handshake payloads, frames, streams, and message reassembly.
Runtime activation is in
[openai/codex#26245](https://github.com/openai/codex/pull/26245).
## Stack
1. **[openai/codex#26242](https://github.com/openai/codex/pull/26242)**:
Noise channel and relay transport
2. [openai/codex#26245](https://github.com/openai/codex/pull/26245):
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- Oversized initiator payload regression coverage
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
The ws pump refactor removed the relay keepalive timers that had been
added to keep idle rendezvous connections alive. An idle relay could
therefore be closed by the rendezvous service or a load balancer,
disconnecting executor-backed MCP processes.
## What
- restore periodic WebSocket ping frames on both rendezvous relay
endpoints
- keep missed-tick behavior bounded with `MissedTickBehavior::Skip`
- cover the harness and remote-environment pumps with focused
traffic-after-keepalive tests
## Summary
- migrate exec-server remote registration naming from executor to
environment
- align CLI, public Rust exports, registry error messages, and relay
test fixtures with the environment registry contract
- keep the live registration path and response model consistent with
`/cloud/environment/{environment_id}/register`
## Verification
- `cargo test -p codex-exec-server
remote::tests::register_environment_posts_with_auth_provider_headers
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_environment_routes_independent_virtual_streams
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo check -p codex-cli --manifest-path
/Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
opened; will update after completion if needed)
## Why
Exec-server websocket handling had separate reader and writer tasks for
the same socket. That made websocket control-frame handling asymmetric:
the task reading frames could observe `Ping`, but the task allowed to
write frames was elsewhere. This PR moves each physical websocket onto
one always-running pump so the socket owner can handle application
frames and websocket control frames together.
## What changed
- Refactored direct exec-server websocket connections in `connection.rs`
to use one task that owns the websocket for outbound JSON-RPC, inbound
JSON-RPC, periodic keepalive pings, and `Ping` -> `Pong` replies.
- Refactored relay websocket handling in `relay.rs` the same way for
both the harness-side logical connection and the multiplexed executor
physical socket.
- Preserved the existing keepalive ownership policy: outbound direct
websocket clients still send periodic pings, inbound Axum accepts only
reply with pongs, and relay physical websocket endpoints keep their
existing periodic pings.
- Added focused websocket pump tests for ping/pong, binary JSON-RPC,
relay data, malformed relay text frames, and close/disconnect behavior.
- Reconnect behavior is intentionally left for a follow-up.
## Validation
- Devbox Bazel focused unit target:
- `//codex-rs/exec-server:exec-server-unit-tests
--test_filter='websocket_connection_|harness_connection_|multiplexed_executor_'`
## Why
Remote exec-server now needs one executor websocket to serve multiple
harness JSON-RPC sessions. Rendezvous routes by `stream_id`, and the
exec-server side needs to use the same stable relay frame contract
instead of a hand-rolled JSON shape.
The relay protocol also needs to make ownership boundaries clear:
harness and executor endpoints own sequencing, acks, retries, duplicate
suppression, segmentation, and reassembly; rendezvous only routes
frames.
## What Changed
- Add the checked-in `codex.exec_server.relay.v1.RelayMessageFrame`
proto plus generated prost bindings for `codex-exec-server`.
- Encode remote harness/executor relay traffic as binary protobuf
websocket frames while keeping local websocket JSON-RPC unchanged.
- Demux executor-side relay streams into independent
`ConnectionProcessor` sessions keyed by `stream_id`.
- Add a programmatic `RemoteExecutorConfig::with_bearer_token(...)`
constructor for non-CLI callers and integration tests.
- Add an integration test that starts the remote executor against a fake
registry/rendezvous websocket and verifies two virtual streams share one
executor websocket without cross-talk, including per-stream reset
behavior.
- Document the remote relay envelope, sequence ranges, `ack`/`ack_bits`,
and endpoint responsibilities in `exec-server/README.md`.
## Verification
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_executor_routes_independent_virtual_streams --
--exact`
- `cargo test -p codex-exec-server --test relay`
- `cargo test -p codex-exec-server` passed outside the sandbox. The
sandboxed run hit macOS `sandbox-exec: sandbox_apply: Operation not
permitted` in filesystem sandbox tests.
