## Why
Delegated HTTP requests need to honor the same outbound proxy policy as the
Codex process that starts the exec server.
## What changed
- Pass the configured `HttpClientFactory` through local and remote exec-server
startup and use route-aware client pools for delegated HTTP and local MCP
requests.
- Preserve per-request timeouts and follow-or-stop redirect behavior while
keeping request URLs and sensitive response headers out of diagnostics.
## Testing
- Cover configured system-proxy routing across the exec-server transport.
- Cover both redirect policies and verify that success and failure logs do not
expose request or response secrets.
GitOrigin-RevId: 4af6aec1d265c4db62dfcb6e1fb076fb31736137
## Why
The TUI must inspect the default execution environment before loading its final
configuration. Initializing the environment manager at that point can give
startup services the bootstrap HTTP policy instead of the effective policy after
managed requirements are applied.
## What changed
- Split environment discovery from manager construction so callers can inspect
the default environment without starting remote connections.
- Build the environment manager after final configuration loading and pass its
resolved `HttpClientFactory` through all construction paths.
- Add shared test support for managers that use the legacy default HTTP policy.
## Testing
- Cover connection-free environment discovery and explicit HTTP policy
propagation.
- Verify TUI startup services use the final managed `respect_system_proxy` value.
GitOrigin-RevId: 928fa31e6b4bcfbe1a121cade2f351427fdfa0f4
## Why
`local` identifies the environment owned by `EnvironmentManager`, but dynamic
remote environment registration previously allowed that ID to be replaced.
## What changed
Reject `local` in the shared environment ID validator used by dynamic remote
environment registration.
GitOrigin-RevId: 2622551f78989c16c432e6dd3d7da9bd8b0527c5
## What changed
- Let `exec-server` issue JSON-RPC requests to its client and correlate responses, with bounded concurrency, timeouts, and connection cleanup.
- When `request_policy_decisions` is enabled, forward proxy policy requests with the process ID and network destination, then apply the client's allow, deny, or ask decision.
- Fail closed for invalid inputs, unavailable or malformed responses, timeouts, process shutdown, and disconnects. Keep callbacks active while inherited process streams keep the proxy alive.
## Testing
- Cover response correlation, request limits, timeouts, disconnect cleanup, validation boundaries, decision forwarding, and process lifecycle behavior.
GitOrigin-RevId: 60e658102c50036589ca9de844760a8bffaa53b0
## What changed
- Make the generated Linux proxy socket directory readable inside restricted
`bubblewrap` sandboxes so the proxy bridge can connect.
- Route `WS_PROXY` and `WSS_PROXY` through the Linux managed proxy bridge.
- Remove inherited proxy attribution tokens from unscoped executions while
continuing to replace them with the current token for scoped executions.
## Testing
- Exercise `WSS_PROXY` routing with a minimal filesystem policy.
- Cover attribution-token removal and replacement during environment setup.
GitOrigin-RevId: 2f14ddb26baab0147a786bc354d93eb5aad059f3
## What changed
- Define the `network/policyRequest` RPC payloads for associating a network request with a process and returning an `allow`, `deny`, or `ask` decision.
- Cover HTTP, HTTPS CONNECT, and SOCKS5 TCP/UDP requests, with shared size limits for callback fields.
- Add the backward-compatible `requestPolicyDecisions` remote proxy option so executors can opt in to sending policy decisions to clients.
- Re-export the new protocol types from `codex-exec-server`.
## Testing
- Verify the request and decision JSON shapes.
- Verify that the proxy opt-in is omitted and defaults to `false` for legacy configurations, while an enabled value round-trips.
GitOrigin-RevId: 65b0a0723d1abb6e4f504a41d3a0dd41f5f21bf7
## Why
Elevated Windows sandboxes need stable managed-proxy ports while preserving the network policy and environment attribution of each sandboxed process.
## What changed
- Keep shared HTTP and SOCKS5 loopback ingress listeners alive across managed-proxy instances.
- Add a per-route restricting SID to elevated sandbox tokens and dispatch incoming connections to the matching proxy policy after attributing the client process.
- Reject connections without exactly one registered route, remove routes when their proxy handle is dropped, and keep unsandboxed Windows launches off the managed ingress.
- Provision the elevated sandbox with the configured proxy ports and local-binding setting, honoring the selected profile and CLI overrides.
## Testing
- Add Windows unit tests for TCP ownership attribution, route selection, restricting-token propagation, and setup settings.
- Add an end-to-end Windows test covering stable ports, isolated environment policies, HTTP and SOCKS5 routing, and route teardown.
GitOrigin-RevId: 783fac6e0f904dc9bb1955b75d4a5895e8bb9690
## Why
Default read-only protections for project metadata should apply when paths such
as `.git`, `.agents`, and `.codex` exist, without causing sandbox setup to
materialize missing paths as ACL targets.
## What changed
- Add an optional `missing_path_behavior` to filesystem sandbox entries and
mark default project-metadata protections with `skip`.
- Preserve the behavior through permission transforms and exec/MCP protocol
serialization while keeping existing path wire variants stable.
- Ignore skip-missing entries when projecting configuration or Windows sandbox
overrides, while retaining explicit metadata carveouts.
## Testing
- Cover protocol round trips for path and special-path entries.
- Verify default metadata protections and Windows explicit carveout handling.
GitOrigin-RevId: 6df13dadacdd131c44aab9f15a967c81051355c1
## What changed
Allocate Noise handshake input and output buffers from the actual message size
and the handshake-reported overhead instead of placing maximum-size arrays on
the stack. Truncate variable-length buffers to the bytes produced or read.
GitOrigin-RevId: d4776079ad8116b3cfe1ceaa37c04509d31d1436
## Why
Sandbox wrappers replaced the process launch command and did not carry an
`ExecParams.arg0` override through to the inner process.
## What changed
- Route sandboxed Unix launches with a custom `arg0` through a helper mode that
re-execs the requested program with the override.
- Expose the helper executable to the filesystem sandbox and dispatch its mode
from Codex and exec-server test binaries.
## Testing
Add coverage for the prepared sandbox command and an end-to-end remote process
that verifies both the custom `arg0` and filesystem restrictions.
GitOrigin-RevId: c9f8eef3906d184e670184c2eeed250d5895a9ca
## Why
Sandboxed process launch through the exec server was unsupported on Windows.
## What changed
- Add a shared native process launcher that selects the Windows sandbox session backend when required and otherwise launches PTY or pipe-based processes.
- Pass Windows permission, workspace-root, proxy, filesystem-override, and desktop settings through exec-server process preparation.
- Preserve piped stdin and inherited file descriptors across the shared launch path.
## Testing
- Verify that a remote Windows sandbox process accepts stdin while blocking a write under a read-only policy.
- Enable workspace-root sandbox tests on native Windows while continuing to skip environments that cannot emulate Windows sandbox semantics.
GitOrigin-RevId: 6719ccbe87ad0d5a57af56503e3f23f894c60a71
## What changed
- Remove unused APIs and test helpers across the Rust workspace.
- Route Linux sandbox test setup through the shared exec request path.
- Drop dependencies that became unnecessary after the cleanup.
GitOrigin-RevId: 48fd9b2b1575bad7add0819786b4d8ec88d8486f
## Why
Remote executions need managed-network proxy listeners in the executor so their
loopback proxy addresses are reachable by the launched process.
## What changed
- Add a capability-gated exec-server protocol field for executor-local proxy
launch configuration, including network policy, audit metadata, and execution
attribution.
- Start the proxy while preparing a remote process, replace inherited proxy
environment variables with its local addresses, and derive the sandbox
context from its listeners.
- Keep the proxy alive until inherited output streams close, then shut it down.
- Reject unsupported remote settings such as MITM and credential injection.
## Testing
- Cover configuration round trips and rejection of unsupported settings.
- Verify executor-local startup, blocked-domain enforcement, protocol
compatibility, and proxy lifetime through process closure.
GitOrigin-RevId: c984f54e3e600aa9ebcbf8cf4574046e2c199d11
## Why
Selected capability roots can contribute plugins, MCP servers, connectors, and
skills. Discovering each contribution separately requires repeated access to the
executor filesystem.
## What changed
- Add the `capabilityRoots/discoverV1` exec-server RPC to scan selected roots and
materialize recognized plugin manifests, configuration files, skill
instructions, and skill metadata in one bounded request.
- Add the opt-in `executor_capability_discovery` feature, with a thread-scoped
cache and per-step snapshot shared by MCP and skill discovery.
- Parse MCP, connector, and skill contributions from the materialized snapshot,
including serving cached skill instructions without another filesystem read.
## Testing
- Cover discovery limits, manifest precedence, root-local failures, cache reuse,
plugin contributions, and parity with the existing environment skill loader.
GitOrigin-RevId: f98fd2321cafb58c596db02da1f83c09d8eb375d
## What changed
- Let deferred environments provide selected capability roots with their ready signal.
- Validate that those roots have unique, non-empty IDs, belong to the registering environment, and stay within the root limit.
- Include roots from ready turn environments when resolving MCP contributions, and refresh the MCP runtime when the selected root set changes.
- Expose the exact ready root set to MCP contributors so executor plugins become available with their environment.
## Testing
- Cover ready-root propagation, validation failures, replacement isolation, reconnection, and MCP plugin availability refresh.
GitOrigin-RevId: ec3498aab1164824025094e96a9b1063b7b731ad
## What changed
- Track connected and disconnected states across initial remote exec-server connections and reconnection attempts.
- Emit experimental `thread/environment/connected` and `thread/environment/disconnected` app-server notifications for each thread selecting the environment. Each payload identifies the thread and environment; current state is not replayed when a thread starts.
- Stop forwarding connection events when an environment selection is removed or replaced.
## Testing
- Cover connection, disconnection, reconnection, shared-environment notifications, and replacement of a selected environment.
GitOrigin-RevId: 5dd767372363c4a2a8319fc16164be117d5bd20c
## What changed
Add named tracing spans around step environment snapshots, capability and
executor plugin resolution, `AGENTS.md` refreshes, and MCP runtime projection
and refresh operations. Skip recording function arguments to keep these spans
focused on timing and execution flow.
GitOrigin-RevId: 73a452ef1b5da6f7f2b00d24421a815c5d514eaf
## What changed
- Replace pending WebSocket URL registration with deferred Noise environment
registration that gates connection attempts on an explicit readiness signal.
- Reuse the Noise rendezvous transport after readiness so reconnects request a
fresh connection bundle.
- Preserve terminal errors for failed or dropped registrations and keep late
completion isolated from replacement environments.
## Testing
- Add coverage for readiness gating, registration failure and replacement,
eager Noise connections, and reconnection through a fresh rendezvous bundle.
GitOrigin-RevId: 83e23fa03a02e3b2bdf1a83fe26d7ac461f55cf3
## Why
Compact JSON arrays can expand into millions of heap values during decoding, and duplicate object keys make a message ambiguous.
## What changed
- Limit exec-server JSON-RPC messages to 256K JSON values and reject duplicate object keys.
- Cap `fs/read_directory` results and retained `process/read` output at 50,000 entries or chunks so locally produced responses remain within the decoder budget.
## Testing
Add coverage for all JSON-RPC variants, large scalar payloads, duplicate keys, compact array amplification, and retained process output at the chunk limit.
GitOrigin-RevId: e31d1f25ab0a7e2272015c98174fd2b7cdd669d7
## What changed
- Add the experimental `environment/status` request for inspecting a configured environment without starting or reconnecting it.
- Report `ready`, `pending`, `disconnected`, or `unknown`, including error details for disconnected and unknown environments.
- Probe ready remote environments over their existing exec-server connection.
## Testing
- Add an app-server integration test covering local and remote ready, pending, disconnected, and unknown environments.
GitOrigin-RevId: 397bba603aa9e0b59008ee6cb7cbde46e357652c
## What changed
- Add the initialized `environment/status` RPC, which reports `ready` when the exec server can handle requests.
- Expose environment IDs and `ready`, `pending`, or `disconnected` status through `EnvironmentManager` and `Environment`.
- Keep status checks non-mutating: they do not start or recover lazy remote environments, and probe only an existing connection.
## Testing
- Cover the status RPC over WebSocket and the in-process request processor.
- Verify that checking an unstarted stdio environment leaves it pending and that failed connections report as disconnected.
GitOrigin-RevId: 22febeb6a3457849292128a8991c6400c22b3fd8
## Why
Remote environment provisioning can finish after a thread starts, before an
exec-server WebSocket URL is available.
## What changed
- Add `EnvironmentManager::register_pending_environment` and a one-shot
`PendingEnvironmentRegistration` handle that resolves to either a validated
WebSocket URL or a terminal provisioning error.
- Let lazy remote exec-server clients wait for that result, while preserving
reconnection behavior after a successful registration.
- Keep replacement registrations isolated so completing an older handle does
not resolve the current environment with the same ID.
## Testing
Add coverage for successful connection and reconnection, provisioning and
dropped-registration failures, invalid URLs, replacement isolation, and the
deferred-executor startup flow.
GitOrigin-RevId: 5c05be2b72291b77a1f71176d7075b1ad63332a5
## What changed
- Pass configured workspace roots from core to the exec server so filesystem and process sandbox permissions are materialized against the intended roots.
- Preserve an explicitly empty workspace-root list instead of treating the sandbox working directory as an implicit root.
- Initialize filesystem sandbox contexts with their working directory as the default workspace root.
## Testing
- Add end-to-end coverage for patch and command writes inside and outside workspace roots.
- Verify remote filesystem and process sandboxes do not grant access through an empty workspace-root list.
GitOrigin-RevId: 6684c8f7de50970b45a29e2a6323df954c96f9f6
## Why
The process-start background task can move to a Tokio worker thread, where the
caller's thread-local tracing subscriber is otherwise unavailable. This can
break propagation of the caller's trace context to the exec-server request.
## What changed
- Attach both the current span and current tracing subscriber to spawned
process-start tasks.
- Run exec-server Bazel unit tests serially because their tracing setup uses
process-global state.
## Testing
Run the trace-context regression test on a multi-thread Tokio runtime so it
exercises propagation across the background task.
GitOrigin-RevId: a586eefd6983916d670fc5a90f0decd0468d273b
## What changed
Update the exact-size-limit stdio connection test to exercise both `\n` and
`\r\n` line endings, sizing the duplex buffer for each delimiter.
GitOrigin-RevId: 949e28bef3f1a34d379f3623f6aa7d99c0ecea80
## Why
Newline-delimited stdio input could buffer an unterminated JSON-RPC message without a per-message limit. Apply the same 64 MiB ceiling used by the other exec-server transports.
## What changed
- Read stdio messages with bounded lookahead and disconnect when a message exceeds the limit.
- Preserve LF and CRLF framing, including messages whose payload is exactly at the limit.
## Testing
Added tests for accepting a limit-sized CRLF message and rejecting an unterminated overlong message.
GitOrigin-RevId: 73eaf883324a7777960725037cc5b9c34720084f
## Why
Frame-count backpressure does not bound the amount of executor-controlled data
retained in streamed HTTP response queues, and a single body delta could exceed
the intended wire size.
## What changed
- Limit each decoded `http/request/bodyDelta` payload to 1 MiB, split locally
produced response chunks at that boundary, and reject oversized incoming
deltas.
- Apply a shared 16 MiB byte budget across queued HTTP response streams. Release
capacity as deltas are consumed and fail a stream when the budget is
exhausted.
## Testing
Added coverage for rejecting an oversized delta and for failing a stream after
its queued deltas exhaust the shared byte budget.
GitOrigin-RevId: be9205fef44b0ec96b84b31a8e059b1cb9cd3f3b
## Why
Sandboxed exec-server process requests can use a restricted filesystem
profile that does not expose the exec-server binary. On Linux, the outer
bubblewrap stage re-enters that binary with the `codex-linux-sandbox`
argv0 to install seccomp, so hiding the binary prevents the requested
process from starting.
## What changed
- add the configured `codex_self_exe` to the process permission profile
before constructing the outer platform sandbox
- add a Linux exec-server integration test that starts a real remote
process with restricted reads and verifies it can read an allowed
workspace file
## Test plan
- `just test -p codex-exec-server process_sandbox`
- `just test -p codex-exec-server --test exec_process
remote_process_keeps_sandbox_helper_visible_with_restricted_reads`
Filesystem helper requests currently turn symbolic `:workspace_roots`
permissions into a sandbox policy before applying the workspace roots
from the filesystem sandbox context. This can accidentally broaden
filesystem access to the cwd instead of limiting it to the selected
workspace roots.
Materialize project-root permissions using the context workspace roots
before deriving the filesystem sandbox policy. The same converted roots
are then reused when constructing the sandbox command, keeping policy
enforcement and process setup aligned.
Adds a remote filesystem integration test that verifies a file inside
the selected workspace root is readable while a sibling under the cwd is
denied.
## Validation
- `just test -p codex-exec-server
remote_read_file_materializes_environment_workspace_roots` (macOS,
outside the outer Seatbelt sandbox)
## Why
A thread can select skill roots that live in an executor environment.
`skills/list` needs a passive snapshot of the roots that are usable now:
it must not start an executor, wait for recovery, or reconnect a failed
environment.
The initial implementation checked the immutable first startup result.
After a successful connection later entered recovery or failed, that
result still looked successful. A read-only catalog request could then
wait for recovery or trigger a new connection while reading the
filesystem.
## What
- inspect readiness from the current exec-server connection state
- return roots only while their environment can serve a request
immediately
- omit environments that have not started, are connecting, or are
recovering
- return warnings for missing environments and terminal connection
failures
- add a fail-fast filesystem view that never starts, waits for, or
reconnects an environment
- expose the passive selected-root snapshot through `CodexThread`
## Behavior
- Local and currently connected environments are ready.
- Starting and recovering environments are omitted without a warning so
callers can retry later.
- Missing and terminally failed environments are omitted with a warning.
- A disconnect between readiness inspection and filesystem access fails
promptly instead of crossing into the normal recovery path.
- Normal model-turn and execution paths keep their existing reconnect
behavior.
## Design
The recovery policy is private to the exec-server client. Callers choose
the explicit fail-fast filesystem method; the existing client and
filesystem APIs remain reconnecting. This keeps the passive contract at
the transport boundary instead of plumbing timeout or retry flags
through the skills stack.
## Coverage
- a lazy stdio environment stays unstarted during passive inspection
- missing and terminally failed environments surface warnings
- a real websocket disconnect proves current readiness drops, a
previously acquired fail-fast filesystem handle returns promptly, and
readiness returns after recovery
## Scope
This PR only provides passive readiness and fail-fast filesystem
primitives. It does not add app-server API fields or notifications.
## Stack
- #31582 uses these primitives for experimental thread-scoped
`skills/list`.
- #30228 adds targeted invalidation notifications.
## Why
A malicious exec-server can send out-of-order process notifications
faster than missing events arrive. The client retained every future
event in a per-session reorder map, so many tiny events or a few large
output chunks could grow orchestrator memory without bound.
## What changed
- cap each session's pending process-event reorder state at 256 events
and 1 MiB
- reject individual process events larger than 1 MiB
- release byte accounting as events publish or a session fails
- fail and detach only the affected process session when a limit is
exceeded
- let the next expected event drain a full buffer, so the limits apply
to retained future state
- apply the same bounded insertion path during reconnect recovery while
safely handling dense tail output, missing exit events, newer live
notifications, and conflicting `Closed` sequences
Sequence distance is intentionally not capped because it does not affect
allocation; the event-count and byte limits are the resource-exhaustion
boundary.
## Tests
Focused exec-server tests cover oversized output, full count/byte
buffers, gap-closing delivery, dense recovery interleaved with newer
notifications, missing exit reconstruction, and conflicting terminal
sequences.
## Scope
This is limited to per-session process-event reorder state. It does not
introduce a scheduler, quota framework, or new public configuration.
## Why
An untrusted exec-server can stop reading requests, never answer them,
or send guessed responses before queued requests are written. Without
client-side admission, the orchestrator can retain unbounded RPC call
futures and request payloads.
We need a hard bound without adding a blanket timeout, because
individual operations already own their timeout and cleanup semantics.
## What changed
- hold one of 64 shared admission permits for the full lifetime of each
regular RPC call, so an early response cannot free capacity while the
call remains queued
- allow `process/terminate` and `fs/close` to use one additional cleanup
permit
- close the transport and fail pending calls if the cleanup permit is
also stuck, allowing teardown or recovery to release remote resources
- leave the wire format and existing timeout behavior unchanged
## Tests
- `rpc_client_call_has_no_implicit_deadline` verifies that ordinary
calls remain untimed
- `rpc_client_bounds_in_flight_calls_and_preserves_cleanup` covers the
regular-call cap, guessed early responses, cleanup admission, and the
cleanup circuit breaker
## Why
`fragmented_writes_yield_to_keepalive_and_queued_pong` deliberately
blocks WebSocket writes while exercising keepalive and queued-Pong
scheduling. It previously advanced those states with wall-clock sleeps.
Under a sufficiently delayed CI worker, those sleeps and scheduling gaps
could consume the test-only 100 ms Pong-watchdog budget, causing the
relay to exit and the next write-permit send to fail with
`TrySendError::Disconnected`.
The failure was therefore a timing flake in the harness test, not
evidence that the production relay mishandled a Pong.
## What changed
- Run this test with Tokio time paused.
- Advance the virtual clock through its two keepalive transitions
instead of sleeping in wall-clock time.
- Enable Tokio's `test-util` feature only for `codex-exec-server` dev
dependencies.
No production code or timeout values change.
## Review guide
The behavioral change is confined to `noise_relay/harness_tests.rs`; the
`Cargo.toml` change only exposes Tokio's paused-clock test APIs.
## Validation
- `just test -p codex-exec-server
fragmented_writes_yield_to_keepalive_and_queued_pong`
- `just fix -p codex-exec-server`
- `just bazel-lock-update` (no lockfile changes)
## Why
#31323 introduces `codex-http-client` and leaves compatibility
re-exports in `codex-client`. Low-level HTTP consumers should depend on
the crate that now owns those APIs rather than continuing through the
transitional compatibility layer.
This stacked follow-up makes that ownership explicit and moves the
repository toward enforcing the abstraction without mixing call-site
churn into the extraction itself.
## What changed
- Switched `codex-backend-client`, `codex-cloud-tasks`,
`codex-exec-server`, `codex-login`, and `codex-model-provider` from
`codex-client` to `codex-http-client` where they only use low-level HTTP
APIs.
- Added the direct dependency to `codex-api` for its custom-CA request
and websocket paths while retaining `codex-client` for higher-level
retry and transport policy.
- Updated imports and normalized login's internal client type name from
`CodexHttpClient` to `HttpClient`, while preserving its existing
`CodexRequestBuilder` re-export.
- Updated `Cargo.lock` to reflect the new direct dependency edges.
## Review guide
This PR is intentionally mechanical: 20 files and 92 changed lines, with
no runtime logic changes. The largest diff is
`codex-rs/login/src/auth/default_client.rs`, where the only
semantic-looking changes are type and import renames. The remaining
source changes replace `codex_client` import paths with
`codex_http_client`; the manifest and lockfile changes mirror those
imports.
## Test plan
- Compile-checked `codex-api`, `codex-backend-client`,
`codex-cloud-tasks`, `codex-exec-server`, `codex-login`, and
`codex-model-provider` together.
## Why
Make it easier to measure the performance of different parts of skill
loading.
## What
- Add spans for step-context capture, world-state construction, executor
catalog snapshot/root loading, and environment skill loading.
- Record the discovered environment skill count.
- Trace outbound exec-server requests with client kind and RPC method
fields.
- Update trace propagation tests to assert that requests keep the parent
trace id while creating their own child span.
## Summary
- require a Pong within 60 seconds for established Noise Rendezvous
WebSockets on both the harness and executor
- bound steady-state WebSocket writes and harness event delivery so
backpressure cannot mask the deadline
- classify executor disconnects with bounded reasons and feed them into
the existing reconnect metric and structured log
- cover silent peers, responsive peers, continuous non-Pong traffic, and
local application backpressure
## Why
The existing periodic Pings did not track Pongs, so a half-open or
blackholed connection could remain stuck until the operating system's
TCP timeout. This adds the smallest explicit liveness contract without
new spans, RTT histograms, feature flags, or TCP diagnostics.
## Testing
- `just test -p codex-exec-server` on devbox `richard-6` — 300 passed, 2
skipped
- `just fix -p codex-exec-server`
- `just fmt`
- independent correctness, performance/security, and YAGNI reviews — no
findings
## Summary
Disable Nagle unconditionally for both exec-server Rendezvous WebSocket
connections.
- pass `disable_nagle=true` at the executor and harness connection call
sites
- keep the existing signed URL, protocol, and connection flow unchanged
- add no feature flag, rollout schema, path variant, or
experiment-specific telemetry
The companion internal PR enables `TCP_NODELAY` on accepted Rendezvous
sockets: https://github.com/openai/openai/pull/1082463
## Why
Rendezvous carries small, latency-sensitive relay and JSON-RPC frames.
Three staging runs of 30 steady-state `process/read` calls per
configuration measured p50 improving from 139.1 ms to 81.5 ms and p95
from 162.0 ms to 95.8 ms with Nagle disabled.
The expected packet overhead is small at the current connection scale.
We will use existing latency, error, packet, and CPU monitoring and
revert normally if production regresses.
## Rollout and rollback
The client and accepted-socket changes can deploy independently. New
connections receive the setting as each side deploys. Rollback is a
normal code revert; there is no persisted assignment or gate state to
unwind.
## Validation
- `just test -p codex-exec-server --lib`: 164 passed
- `just fix -p codex-exec-server`: passed
- `just fmt`: passed
- independent final review found no actionable issue
## Why
App-server clients that configure named execution environments need to
discover an environment's shell and working directory before selecting
it for a thread or turn. Because the environment can run on a different
operating system than app-server, its working directory is represented
as a canonical `file:` URI rather than a host-local path string. The
probe also needs a bounded response time: an exec-server that completes
initialization but never answers `environment/info` must not hold the
environment serialization queue indefinitely.
## What changed
- Add an experimental `environment/info` app-server RPC for named
environments.
- Route the probe through the managed environment connection and return
target-native shell metadata plus the default working directory as a
`PathUri`.
- Return connection and protocol failures as JSON-RPC errors.
- Bound the exec-server probe response to 30 seconds and remove
timed-out calls from the pending-request table so later environment
mutations can proceed.
- Cover successful responses, omitted working directories, unknown
environments, connection failures, and pending-call cleanup.
## Protocol examples
Request:
```json
{
"id": 42,
"method": "environment/info",
"params": {
"environmentId": "remote-a"
}
}
```
Successful response:
```json
{
"id": 42,
"result": {
"shell": {
"name": "zsh",
"path": "/bin/zsh"
},
"cwd": "file:///workspace"
}
}
```
If the exec-server initializes but does not answer the probe within 30
seconds:
```json
{
"id": 42,
"error": {
"code": -32603,
"message": "failed to get info for environment `remote-a`: exec-server protocol error: timed out waiting for exec-server `environment/info` response after 30s"
}
}
```
## Testing
- App-server integration coverage for successful info (including omitted
`cwd`), unknown environments, and connection failures.
- Exec-server RPC coverage verifying a timed-out call is removed from
the pending-request table.
---------
Co-authored-by: Michael Bolin <mbolin@openai.com>
## Summary
- complete unified-exec processes from the ordered event stream instead
of issuing a final zero-wait `process/read`
- add optional executor sandbox-denial state to `process/exited`
- retain `process/read` as a retained-output and compatibility fallback
for receiver lag, sequence gaps, and legacy servers
- recover sandbox-denial state across transport reconnection
- cover the real `TestCodex` remote-exec path without adding a public
test-only event constructor
## Why
A successful one-shot tool call currently receives its output and
terminal notifications, then pays another wide-area `process/read` round
trip before returning. Staging traces showed that remote response wait
accounted for more than 99.8% of RPC time; local serialization,
queueing, and deserialization were below 0.6 ms.
## Measured impact
A direct staging A/B used the same build and route and changed only
completion mode. Each arm ran three times with 30 one-shot
`/usr/bin/true` calls per run. The table reports the median of the three
per-run percentiles.
| Metric | Final `process/read` | Pushed events | Change |
| --- | ---: | ---: | ---: |
| End-to-end completion p50 | 159.5 ms | 118.7 ms | -40.8 ms (-25.6%) |
| End-to-end completion p95 | 182.4 ms | 131.7 ms | -50.6 ms (-27.8%) |
| Completion-wait p50 | 80.1 ms | 41.5 ms | -38.5 ms (-48.1%) |
| Final `process/read` RPC p50 | 79.9 ms | eliminated | -79.9 ms |
TCP_NODELAY was enabled in both A/B arms, so its effect cancels out. The
successful, complete, in-order event path issued zero final
`process/read` calls.
## Compatibility and recovery
- new servers send `sandboxDenied` on `process/exited`
- legacy servers omit it, which triggers one compatibility
`process/read`
- broadcast lag or a sequence gap triggers a retained-output read
- recovery remains bounded by the server's existing 1 MiB
retained-output window
- complete, in-order event streams issue no completion read
- sandbox denial is attached to the exit event before consumers can
observe process completion
- server-first and client-first rollouts remain wire-compatible;
server-first realizes the latency win immediately
## Integration coverage
The `TestCodex` suite exercises four distinct remote-exec contracts:
- complete pushed output/exit/close with zero reads
- direct pushed sandbox denial with zero reads
- legacy missing denial metadata with exactly one compatibility read
- count-bounded replay eviction recovered from retained output without
duplication
## Validation
- `just test -p codex-core
exec_command_consumes_pushed_remote_process_events`: 4 passed
- `just test -p codex-core unified_exec::process_tests::`: 4 passed
- `just test -p codex-exec-server`: 294 passed, 2 skipped
- `just test -p codex-exec-server-protocol`: 5 passed
- `just test -p codex-rmcp-client`: 89 passed, 2 skipped
- focused Bazel `//codex-rs/core:core-all-test`: passed across 16 shards
- scoped `just fix` passed for core and exec-server
- `just fmt` passed
The complete workspace suite was not rerun; focused Cargo and Bazel
coverage passed for the changed behavior.
[Codex Thread
019ef1f9-36e2-7e91-9337-504f097b9dc1](https://codex-thread-link.openai.chatgpt-team.site/thread/019ef1f9-36e2-7e91-9337-504f097b9dc1)
## Why
Hosted plugin-service Streamable HTTP MCP traffic uses
`https://chatgpt.com/backend-api/ps/mcp` and depends on Cloudflare's
`__cflb` cookie for load-balancer affinity. The local and exec-server
`http/request` path built a fresh reqwest client for each request
without installing Codex's existing shared ChatGPT Cloudflare cookie
store, so affinity could be lost between calls.
This is an affinity-hardening change motivated by an incident
investigation. It does not establish the broader connector-cache
incident RCA or claim to fix that incident in full.
## What changed
- Install the existing process-local, strictly allowlisted ChatGPT
Cloudflare cookie store on the reqwest client used by
`ReqwestHttpClient`.
- Fresh clients now share allowed Cloudflare infrastructure cookies
within the process that originates the local or exec-server network
request.
- Keep the existing HTTPS ChatGPT-host and Cloudflare-cookie-name
restrictions. This does not introduce a general cookie jar or send
ChatGPT Cloudflare cookies to unrelated hosts.
## Test coverage
- `codex-client` unit coverage verifies that the existing strict store
accepts and returns `__cflb` for HTTPS ChatGPT URLs.
- The exec-server HTTPS integration test sends four independent
`http/request` calls through a local TLS-intercepting proxy and verifies
that:
- `Set-Cookie: __cflb=west` is sent on the next plugin-service request;
- a later `Set-Cookie: __cflb=central` replaces the stored value;
- non-Cloudflare session cookies are discarded;
- no stored ChatGPT Cloudflare cookie is sent to a non-ChatGPT host.
- `just test -p codex-client` — 38 passed.
- `just test -p codex-exec-server --test chatgpt_cloudflare_affinity` —
1 passed.
- `just bazel-lock-check` — passed.
## Non-goals
- No persistence of ChatGPT auth, account, session, residency, or
arbitrary cookies.
- No cookie persistence for third-party MCP servers.
- No special composition of caller-provided `Cookie` headers.
- No plugin-service, connector-cache, Habitat/habicache, routing,
redirect, or API-contract changes.
- No broader incident RCA conclusions.
## Why
This stack crosses World State, executor skills, selected plugin
metadata, MCP processes, connectors, dynamic environments, and resume.
This PR adds two end-to-end scenarios that validate those pieces
together.
Both tests enable `deferred_executor`, so they exercise the real
delayed-environment path.
## Scenario 1: availability across turns and resume
```text
1. Start a thread with one selected plugin root bound to E1.
2. E1 is unavailable.
- executor skill is absent
- selected MCP is absent
- connector has no selected-plugin attribution
3. Start E1 and register the same stable environment ID.
4. Start a new turn.
- the executor skill appears through World State
- its body beats a colliding host skill
- the selected MCP tool is advertised and executes inside E1
- the connector is attributed to the selected plugin
5. Start another turn without changing E1.
- the MCP PID stays the same, proving runtime reuse
6. Restart app-server and resume the thread.
- durable selected-root intent is restored
- skills, MCP, and connector attribution are restored
- a new MCP PID proves ephemeral process state was rebuilt
```
## Scenario 2: availability changes inside one turn
```text
1. Start a turn while E1 is unavailable.
2. The first model sample sees no executor skill, MCP, or selected connector.
3. The turn pauses on request_user_input.
4. Start E1 and register it while that same turn is still active.
5. Continue the turn.
6. The very next model sample sees:
- the executor skill catalog
- the selected MCP tool
- selected-plugin connector attribution
7. The model calls the MCP, and its output proves execution happened inside E1.
```
This second scenario specifically protects the aeon-style behavior:
capability state is captured again for every sampling step, not only at
the next user turn.
## Scope
These are integration tests only. They do not add a combinatorial matrix
for unsupported plugin-file mutation, environment generations, transport
disconnects, or delayed `required = true` executor MCPs.
Fixes distributed trace continuity across exec-server JSON-RPC HTTP
egress by adding an executor client span and injecting its W3C context
through a reusable `codex-otel` helper.
This preserves the caller trace across core/tool → executor →
provider/MCP instead of dropping parentage at raw reqwest.
Note that this doesn't include the websocket path, which is needed to
really get the full story but at least we cover the basic http path with
this change.
## Summary
- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**
## Validation
- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
## Summary
- retry ERS `409 environment_offline` responses inside the existing
exec-server recovery loop
- keep all other registry conflicts terminal
- add focused coverage for both cases
## Root cause
When an exec server disconnects and reconnects, the client already
starts recovery and calls ERS `/connect`. During the transient executor
presence gap, ERS can return `409 environment_offline`. The retry
classifier treated every 409 as terminal, so the first response aborted
the existing 25-second recovery window before the executor came back
online. That then caused active processes to be marked lost.
This change classifies only the structured `environment_offline`
conflict as retryable. Recovery continues with the existing bounded
deadline, exponential backoff, and jitter.
## Validation
- `just test -p codex-exec-server client::recovery::tests` — 4 passed
- `just fix -p codex-exec-server` — passed
- `just fmt` — passed
- Full `just test -p codex-exec-server` reached unrelated macOS
filesystem-sandbox integration failures because nested
`/usr/bin/sandbox-exec` is denied in this environment (`sandbox_apply:
Operation not permitted`).
## Summary
- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.
## Root cause
Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.
Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.
The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
## Why
`selectedCapabilityRoots` is durable thread intent: “use this capability
root from environment `worker`.”
The important product assumption is:
> One environment ID always names the same logical executor and stable
contents.
`worker` does not silently change from executor A to an unrelated
executor B. The process-local connection handle for `worker` can still
be replaced while Codex is running, though, for example when
`environment/add` registers a fresh handle for the same logical
environment.
The thread should persist only the stable selection. Each model step
should pair that selection with the exact ready handle captured for that
step.
## The boundary
```text
persisted thread intent
plugin@1 -> environment "worker"
|
| capture the current step
v
model-step view
unavailable, or
plugin@1 + worker's exact captured ready handle
```
The environment ID is the stable identity and cache key. The
`Arc<Environment>` is only a process-local handle retained so consumers
of one model step use the same captured environment. It is never
persisted and it does not imply different environment contents.
## What changes
### Persist the stable selection
Selected roots are written into `SessionMeta` and restored with the
thread. Forked subagents inherit the same selections, including
bounded-history forks.
Only stable data is persisted: root ID, environment ID, and root path.
### Capture readiness together with the exact handle
The environment snapshot records:
```rust
environment_id -> Some(Arc<Environment>) // ready in this step
environment_id -> None // still starting in this step
```
This prevents readiness and execution from coming from different
registry snapshots.
For example:
```text
step snapshot: worker -> handle A, ready
environment/add: worker -> fresh handle B for the same logical environment
current step: plugin@1 still uses captured handle A
```
Without carrying handle A in the snapshot, the resolver could combine “A
was ready” with handle B and treat B as ready before it had finished
starting.
This does not change cache invalidation. Stable capability metadata
remains identified by environment ID and capability root. Replacing a
process-local handle under the same stable environment ID does not
invalidate or rediscover that metadata.
### Resolve availability per model step
- A ready captured environment produces resolved roots using its
captured handle.
- A starting, missing, or failed environment is omitted from that step.
- A selected lazy environment that is outside the turn's captured
environment set is asked to start, and a later step can observe it as
ready.
- No capability files are scanned here.
Transient transport disconnects remain the remote client's reconnect
concern. This PR models initial attachment/readiness; it does not add
live socket-connectivity state.
## Example
```text
thread selection: plugin@1 -> environment "worker"
step 1: worker is starting -> plugin@1 unavailable
step 2: worker is ready -> plugin@1 resolves through worker's captured handle
step 3: fresh local handle -> current step remains pinned; a later step captures its own view
```
Temporary unavailability does not discard the durable selection. Later
PRs can retain stable metadata caches while projecting only currently
available capabilities into model-visible World State.
## Compatibility
The app-server request shape does not change. Older rollouts without
`selected_capability_roots` deserialize to an empty list.
## Stack
1. **This PR:** persist stable selected roots and resolve them through
an exact model-step handle.
2. #29960: cache stable skill metadata and project available skills into
World State.
3. #29946: cache stable plugin declarations and manage the separate live
MCP runtime.
## Why
#28522 routes selected-plugin HTTP MCP traffic through the owning
executor, but OAuth bootstrap and refresh still used host-local clients.
Executor-only servers therefore cannot complete discovery or login
through the same network boundary as the MCP connection.
## What changed
- adapt `codex_exec_server::HttpClient` to RMCP 1.8's `OAuthHttpClient`
contract
- let RMCP own discovery, dynamic registration, PKCE, token exchange,
and refresh
- route auth status, persisted-token startup, and app-server login
through the server runtime while preserving the existing local discovery
path
- add optional `threadId` to `mcpServer/oauth/login` and echo it in the
completion notification
- implement RMCP's redirect policy and 1 MiB OAuth response limit over
executor HTTP
- cover selected-thread OAuth discovery and login through an
executor-only route
Depends on #28522.
## Why
Exec-server JSON-RPC calls can cross local and remote transports, but
trace context stopped at the RPC boundary. That made client and server
work difficult to correlate when diagnosing latency or failures.
## What changed
- Propagate the current W3C trace context on outbound JSON-RPC requests.
- Parent inbound request spans from received trace context.
- Record the received JSON-RPC method on server spans and keep each span
open through response enqueue.
- Add only the OTEL dependencies required by the exec-server crate.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests **(this PR)**
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (153 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server`
Stack 1 of 3. Follow-ups: #29842 and #29844.
## What changes
Adds a general bounded `fs/walk` operation to the exec server.
The operation returns file and directory entries plus recoverable
per-path errors. It skips symlinks, preserves the existing filesystem
sandbox routing, and enforces depth, directory, entry, and response-size
limits.
This PR only defines and wires the filesystem operation. It does not
change any callers yet.
## Why
Start moving towards app-server tests defaulting to running against
remote & foreign OS executors. To do so we need a point of indirection
similar to core integration tests' `build_with_auto_env`, but with the
flexibility of letting tests control environment registration if they
need to.
## What
This adds:
- `TestAppServer::new_with_auto_env()` for constructing an app server
with a default environment defined by the test runner (e.g. bazel)
- `TestAppServer::auto_env_params()` for tests to easily acquire turn
env params tailored to the automatic environment
- `TestAppServer::send_thread_start_request_with_auto_env()` to make it
easy for tests to start a thread using the automatic environment
The above methods all fail if the test calling them has set up an
environment where the automatic environment configuration conflicts with
test-created state.
## Validation
Adds a couple of basic smoke tests to the app-server test suite.
Follow-ups will migrate more tests to use it.
