## Why
Remote executions need managed-network proxy listeners in the executor so their
loopback proxy addresses are reachable by the launched process.
## What changed
- Add a capability-gated exec-server protocol field for executor-local proxy
launch configuration, including network policy, audit metadata, and execution
attribution.
- Start the proxy while preparing a remote process, replace inherited proxy
environment variables with its local addresses, and derive the sandbox
context from its listeners.
- Keep the proxy alive until inherited output streams close, then shut it down.
- Reject unsupported remote settings such as MITM and credential injection.
## Testing
- Cover configuration round trips and rejection of unsupported settings.
- Verify executor-local startup, blocked-domain enforcement, protocol
compatibility, and proxy lifetime through process closure.
GitOrigin-RevId: c984f54e3e600aa9ebcbf8cf4574046e2c199d11
## Why
Selected capability roots can contribute plugins, MCP servers, connectors, and
skills. Discovering each contribution separately requires repeated access to the
executor filesystem.
## What changed
- Add the `capabilityRoots/discoverV1` exec-server RPC to scan selected roots and
materialize recognized plugin manifests, configuration files, skill
instructions, and skill metadata in one bounded request.
- Add the opt-in `executor_capability_discovery` feature, with a thread-scoped
cache and per-step snapshot shared by MCP and skill discovery.
- Parse MCP, connector, and skill contributions from the materialized snapshot,
including serving cached skill instructions without another filesystem read.
## Testing
- Cover discovery limits, manifest precedence, root-local failures, cache reuse,
plugin contributions, and parity with the existing environment skill loader.
GitOrigin-RevId: f98fd2321cafb58c596db02da1f83c09d8eb375d
## What changed
- Let deferred environments provide selected capability roots with their ready signal.
- Validate that those roots have unique, non-empty IDs, belong to the registering environment, and stay within the root limit.
- Include roots from ready turn environments when resolving MCP contributions, and refresh the MCP runtime when the selected root set changes.
- Expose the exact ready root set to MCP contributors so executor plugins become available with their environment.
## Testing
- Cover ready-root propagation, validation failures, replacement isolation, reconnection, and MCP plugin availability refresh.
GitOrigin-RevId: ec3498aab1164824025094e96a9b1063b7b731ad
## What changed
- Track connected and disconnected states across initial remote exec-server connections and reconnection attempts.
- Emit experimental `thread/environment/connected` and `thread/environment/disconnected` app-server notifications for each thread selecting the environment. Each payload identifies the thread and environment; current state is not replayed when a thread starts.
- Stop forwarding connection events when an environment selection is removed or replaced.
## Testing
- Cover connection, disconnection, reconnection, shared-environment notifications, and replacement of a selected environment.
GitOrigin-RevId: 5dd767372363c4a2a8319fc16164be117d5bd20c
## What changed
Add named tracing spans around step environment snapshots, capability and
executor plugin resolution, `AGENTS.md` refreshes, and MCP runtime projection
and refresh operations. Skip recording function arguments to keep these spans
focused on timing and execution flow.
GitOrigin-RevId: 73a452ef1b5da6f7f2b00d24421a815c5d514eaf
## What changed
- Replace pending WebSocket URL registration with deferred Noise environment
registration that gates connection attempts on an explicit readiness signal.
- Reuse the Noise rendezvous transport after readiness so reconnects request a
fresh connection bundle.
- Preserve terminal errors for failed or dropped registrations and keep late
completion isolated from replacement environments.
## Testing
- Add coverage for readiness gating, registration failure and replacement,
eager Noise connections, and reconnection through a fresh rendezvous bundle.
GitOrigin-RevId: 83e23fa03a02e3b2bdf1a83fe26d7ac461f55cf3
## Why
Compact JSON arrays can expand into millions of heap values during decoding, and duplicate object keys make a message ambiguous.
## What changed
- Limit exec-server JSON-RPC messages to 256K JSON values and reject duplicate object keys.
- Cap `fs/read_directory` results and retained `process/read` output at 50,000 entries or chunks so locally produced responses remain within the decoder budget.
## Testing
Add coverage for all JSON-RPC variants, large scalar payloads, duplicate keys, compact array amplification, and retained process output at the chunk limit.
GitOrigin-RevId: e31d1f25ab0a7e2272015c98174fd2b7cdd669d7
## What changed
- Add the experimental `environment/status` request for inspecting a configured environment without starting or reconnecting it.
- Report `ready`, `pending`, `disconnected`, or `unknown`, including error details for disconnected and unknown environments.
- Probe ready remote environments over their existing exec-server connection.
## Testing
- Add an app-server integration test covering local and remote ready, pending, disconnected, and unknown environments.
GitOrigin-RevId: 397bba603aa9e0b59008ee6cb7cbde46e357652c
## What changed
- Add the initialized `environment/status` RPC, which reports `ready` when the exec server can handle requests.
- Expose environment IDs and `ready`, `pending`, or `disconnected` status through `EnvironmentManager` and `Environment`.
- Keep status checks non-mutating: they do not start or recover lazy remote environments, and probe only an existing connection.
## Testing
- Cover the status RPC over WebSocket and the in-process request processor.
- Verify that checking an unstarted stdio environment leaves it pending and that failed connections report as disconnected.
GitOrigin-RevId: 22febeb6a3457849292128a8991c6400c22b3fd8
## Why
Remote environment provisioning can finish after a thread starts, before an
exec-server WebSocket URL is available.
## What changed
- Add `EnvironmentManager::register_pending_environment` and a one-shot
`PendingEnvironmentRegistration` handle that resolves to either a validated
WebSocket URL or a terminal provisioning error.
- Let lazy remote exec-server clients wait for that result, while preserving
reconnection behavior after a successful registration.
- Keep replacement registrations isolated so completing an older handle does
not resolve the current environment with the same ID.
## Testing
Add coverage for successful connection and reconnection, provisioning and
dropped-registration failures, invalid URLs, replacement isolation, and the
deferred-executor startup flow.
GitOrigin-RevId: 5c05be2b72291b77a1f71176d7075b1ad63332a5
## What changed
- Pass configured workspace roots from core to the exec server so filesystem and process sandbox permissions are materialized against the intended roots.
- Preserve an explicitly empty workspace-root list instead of treating the sandbox working directory as an implicit root.
- Initialize filesystem sandbox contexts with their working directory as the default workspace root.
## Testing
- Add end-to-end coverage for patch and command writes inside and outside workspace roots.
- Verify remote filesystem and process sandboxes do not grant access through an empty workspace-root list.
GitOrigin-RevId: 6684c8f7de50970b45a29e2a6323df954c96f9f6
## Why
The process-start background task can move to a Tokio worker thread, where the
caller's thread-local tracing subscriber is otherwise unavailable. This can
break propagation of the caller's trace context to the exec-server request.
## What changed
- Attach both the current span and current tracing subscriber to spawned
process-start tasks.
- Run exec-server Bazel unit tests serially because their tracing setup uses
process-global state.
## Testing
Run the trace-context regression test on a multi-thread Tokio runtime so it
exercises propagation across the background task.
GitOrigin-RevId: a586eefd6983916d670fc5a90f0decd0468d273b
## What changed
Update the exact-size-limit stdio connection test to exercise both `\n` and
`\r\n` line endings, sizing the duplex buffer for each delimiter.
GitOrigin-RevId: 949e28bef3f1a34d379f3623f6aa7d99c0ecea80
## Why
Newline-delimited stdio input could buffer an unterminated JSON-RPC message without a per-message limit. Apply the same 64 MiB ceiling used by the other exec-server transports.
## What changed
- Read stdio messages with bounded lookahead and disconnect when a message exceeds the limit.
- Preserve LF and CRLF framing, including messages whose payload is exactly at the limit.
## Testing
Added tests for accepting a limit-sized CRLF message and rejecting an unterminated overlong message.
GitOrigin-RevId: 73eaf883324a7777960725037cc5b9c34720084f
## Testing
Add a concurrent-stream regression test that fills the connection-wide queued
body budget across two HTTP responses. Verify that the overflowing stream
reports the byte-budget error while the other stream still drains successfully.
GitOrigin-RevId: 65efacb15f1368b1de3b98e7ef65ebca66a3840b
## Why
Frame-count backpressure does not bound the amount of executor-controlled data
retained in streamed HTTP response queues, and a single body delta could exceed
the intended wire size.
## What changed
- Limit each decoded `http/request/bodyDelta` payload to 1 MiB, split locally
produced response chunks at that boundary, and reject oversized incoming
deltas.
- Apply a shared 16 MiB byte budget across queued HTTP response streams. Release
capacity as deltas are consumed and fail a stream when the budget is
exhausted.
## Testing
Added coverage for rejecting an oversized delta and for failing a stream after
its queued deltas exhaust the shared byte budget.
GitOrigin-RevId: be9205fef44b0ec96b84b31a8e059b1cb9cd3f3b
## Why
Sandboxed exec-server process requests can use a restricted filesystem
profile that does not expose the exec-server binary. On Linux, the outer
bubblewrap stage re-enters that binary with the `codex-linux-sandbox`
argv0 to install seccomp, so hiding the binary prevents the requested
process from starting.
## What changed
- add the configured `codex_self_exe` to the process permission profile
before constructing the outer platform sandbox
- add a Linux exec-server integration test that starts a real remote
process with restricted reads and verifies it can read an allowed
workspace file
## Test plan
- `just test -p codex-exec-server process_sandbox`
- `just test -p codex-exec-server --test exec_process
remote_process_keeps_sandbox_helper_visible_with_restricted_reads`
Filesystem helper requests currently turn symbolic `:workspace_roots`
permissions into a sandbox policy before applying the workspace roots
from the filesystem sandbox context. This can accidentally broaden
filesystem access to the cwd instead of limiting it to the selected
workspace roots.
Materialize project-root permissions using the context workspace roots
before deriving the filesystem sandbox policy. The same converted roots
are then reused when constructing the sandbox command, keeping policy
enforcement and process setup aligned.
Adds a remote filesystem integration test that verifies a file inside
the selected workspace root is readable while a sibling under the cwd is
denied.
## Validation
- `just test -p codex-exec-server
remote_read_file_materializes_environment_workspace_roots` (macOS,
outside the outer Seatbelt sandbox)
## Why
A thread can select skill roots that live in an executor environment.
`skills/list` needs a passive snapshot of the roots that are usable now:
it must not start an executor, wait for recovery, or reconnect a failed
environment.
The initial implementation checked the immutable first startup result.
After a successful connection later entered recovery or failed, that
result still looked successful. A read-only catalog request could then
wait for recovery or trigger a new connection while reading the
filesystem.
## What
- inspect readiness from the current exec-server connection state
- return roots only while their environment can serve a request
immediately
- omit environments that have not started, are connecting, or are
recovering
- return warnings for missing environments and terminal connection
failures
- add a fail-fast filesystem view that never starts, waits for, or
reconnects an environment
- expose the passive selected-root snapshot through `CodexThread`
## Behavior
- Local and currently connected environments are ready.
- Starting and recovering environments are omitted without a warning so
callers can retry later.
- Missing and terminally failed environments are omitted with a warning.
- A disconnect between readiness inspection and filesystem access fails
promptly instead of crossing into the normal recovery path.
- Normal model-turn and execution paths keep their existing reconnect
behavior.
## Design
The recovery policy is private to the exec-server client. Callers choose
the explicit fail-fast filesystem method; the existing client and
filesystem APIs remain reconnecting. This keeps the passive contract at
the transport boundary instead of plumbing timeout or retry flags
through the skills stack.
## Coverage
- a lazy stdio environment stays unstarted during passive inspection
- missing and terminally failed environments surface warnings
- a real websocket disconnect proves current readiness drops, a
previously acquired fail-fast filesystem handle returns promptly, and
readiness returns after recovery
## Scope
This PR only provides passive readiness and fail-fast filesystem
primitives. It does not add app-server API fields or notifications.
## Stack
- #31582 uses these primitives for experimental thread-scoped
`skills/list`.
- #30228 adds targeted invalidation notifications.
## Why
A malicious exec-server can send out-of-order process notifications
faster than missing events arrive. The client retained every future
event in a per-session reorder map, so many tiny events or a few large
output chunks could grow orchestrator memory without bound.
## What changed
- cap each session's pending process-event reorder state at 256 events
and 1 MiB
- reject individual process events larger than 1 MiB
- release byte accounting as events publish or a session fails
- fail and detach only the affected process session when a limit is
exceeded
- let the next expected event drain a full buffer, so the limits apply
to retained future state
- apply the same bounded insertion path during reconnect recovery while
safely handling dense tail output, missing exit events, newer live
notifications, and conflicting `Closed` sequences
Sequence distance is intentionally not capped because it does not affect
allocation; the event-count and byte limits are the resource-exhaustion
boundary.
## Tests
Focused exec-server tests cover oversized output, full count/byte
buffers, gap-closing delivery, dense recovery interleaved with newer
notifications, missing exit reconstruction, and conflicting terminal
sequences.
## Scope
This is limited to per-session process-event reorder state. It does not
introduce a scheduler, quota framework, or new public configuration.
## Why
An untrusted exec-server can stop reading requests, never answer them,
or send guessed responses before queued requests are written. Without
client-side admission, the orchestrator can retain unbounded RPC call
futures and request payloads.
We need a hard bound without adding a blanket timeout, because
individual operations already own their timeout and cleanup semantics.
## What changed
- hold one of 64 shared admission permits for the full lifetime of each
regular RPC call, so an early response cannot free capacity while the
call remains queued
- allow `process/terminate` and `fs/close` to use one additional cleanup
permit
- close the transport and fail pending calls if the cleanup permit is
also stuck, allowing teardown or recovery to release remote resources
- leave the wire format and existing timeout behavior unchanged
## Tests
- `rpc_client_call_has_no_implicit_deadline` verifies that ordinary
calls remain untimed
- `rpc_client_bounds_in_flight_calls_and_preserves_cleanup` covers the
regular-call cap, guessed early responses, cleanup admission, and the
cleanup circuit breaker
Why
Filtering hidden directories after a walk is too late: their descendants
consume traversal limits, and canonical directory deduplication can let
a hidden path claim a target before a visible symlink reaches it.
What this changes
- Add an optional pruneHiddenDirectories walk option. It defaults to
false and is omitted from the wire when disabled.
- Return hidden directory entries, but do not traverse them or add their
canonical identities to the visited set.
- Cover the visible-symlink-to-hidden-directory case through both local
and remote filesystem implementations.
This is the small filesystem prerequisite for #31566. Skill-specific
behavior remains in that PR.
## Why
`fragmented_writes_yield_to_keepalive_and_queued_pong` deliberately
blocks WebSocket writes while exercising keepalive and queued-Pong
scheduling. It previously advanced those states with wall-clock sleeps.
Under a sufficiently delayed CI worker, those sleeps and scheduling gaps
could consume the test-only 100 ms Pong-watchdog budget, causing the
relay to exit and the next write-permit send to fail with
`TrySendError::Disconnected`.
The failure was therefore a timing flake in the harness test, not
evidence that the production relay mishandled a Pong.
## What changed
- Run this test with Tokio time paused.
- Advance the virtual clock through its two keepalive transitions
instead of sleeping in wall-clock time.
- Enable Tokio's `test-util` feature only for `codex-exec-server` dev
dependencies.
No production code or timeout values change.
## Review guide
The behavioral change is confined to `noise_relay/harness_tests.rs`; the
`Cargo.toml` change only exposes Tokio's paused-clock test APIs.
## Validation
- `just test -p codex-exec-server
fragmented_writes_yield_to_keepalive_and_queued_pong`
- `just fix -p codex-exec-server`
- `just bazel-lock-update` (no lockfile changes)
## Why
#31323 introduces `codex-http-client` and leaves compatibility
re-exports in `codex-client`. Low-level HTTP consumers should depend on
the crate that now owns those APIs rather than continuing through the
transitional compatibility layer.
This stacked follow-up makes that ownership explicit and moves the
repository toward enforcing the abstraction without mixing call-site
churn into the extraction itself.
## What changed
- Switched `codex-backend-client`, `codex-cloud-tasks`,
`codex-exec-server`, `codex-login`, and `codex-model-provider` from
`codex-client` to `codex-http-client` where they only use low-level HTTP
APIs.
- Added the direct dependency to `codex-api` for its custom-CA request
and websocket paths while retaining `codex-client` for higher-level
retry and transport policy.
- Updated imports and normalized login's internal client type name from
`CodexHttpClient` to `HttpClient`, while preserving its existing
`CodexRequestBuilder` re-export.
- Updated `Cargo.lock` to reflect the new direct dependency edges.
## Review guide
This PR is intentionally mechanical: 20 files and 92 changed lines, with
no runtime logic changes. The largest diff is
`codex-rs/login/src/auth/default_client.rs`, where the only
semantic-looking changes are type and import renames. The remaining
source changes replace `codex_client` import paths with
`codex_http_client`; the manifest and lockfile changes mirror those
imports.
## Test plan
- Compile-checked `codex-api`, `codex-backend-client`,
`codex-cloud-tasks`, `codex-exec-server`, `codex-login`, and
`codex-model-provider` together.
## Why
Make it easier to measure the performance of different parts of skill
loading.
## What
- Add spans for step-context capture, world-state construction, executor
catalog snapshot/root loading, and environment skill loading.
- Record the discovered environment skill count.
- Trace outbound exec-server requests with client kind and RPC method
fields.
- Update trace propagation tests to assert that requests keep the parent
trace id while creating their own child span.
## Summary
- require a Pong within 60 seconds for established Noise Rendezvous
WebSockets on both the harness and executor
- bound steady-state WebSocket writes and harness event delivery so
backpressure cannot mask the deadline
- classify executor disconnects with bounded reasons and feed them into
the existing reconnect metric and structured log
- cover silent peers, responsive peers, continuous non-Pong traffic, and
local application backpressure
## Why
The existing periodic Pings did not track Pongs, so a half-open or
blackholed connection could remain stuck until the operating system's
TCP timeout. This adds the smallest explicit liveness contract without
new spans, RTT histograms, feature flags, or TCP diagnostics.
## Testing
- `just test -p codex-exec-server` on devbox `richard-6` — 300 passed, 2
skipped
- `just fix -p codex-exec-server`
- `just fmt`
- independent correctness, performance/security, and YAGNI reviews — no
findings
## Summary
Disable Nagle unconditionally for both exec-server Rendezvous WebSocket
connections.
- pass `disable_nagle=true` at the executor and harness connection call
sites
- keep the existing signed URL, protocol, and connection flow unchanged
- add no feature flag, rollout schema, path variant, or
experiment-specific telemetry
The companion internal PR enables `TCP_NODELAY` on accepted Rendezvous
sockets: https://github.com/openai/openai/pull/1082463
## Why
Rendezvous carries small, latency-sensitive relay and JSON-RPC frames.
Three staging runs of 30 steady-state `process/read` calls per
configuration measured p50 improving from 139.1 ms to 81.5 ms and p95
from 162.0 ms to 95.8 ms with Nagle disabled.
The expected packet overhead is small at the current connection scale.
We will use existing latency, error, packet, and CPU monitoring and
revert normally if production regresses.
## Rollout and rollback
The client and accepted-socket changes can deploy independently. New
connections receive the setting as each side deploys. Rollback is a
normal code revert; there is no persisted assignment or gate state to
unwind.
## Validation
- `just test -p codex-exec-server --lib`: 164 passed
- `just fix -p codex-exec-server`: passed
- `just fmt`: passed
- independent final review found no actionable issue
## Why
App-server clients that configure named execution environments need to
discover an environment's shell and working directory before selecting
it for a thread or turn. Because the environment can run on a different
operating system than app-server, its working directory is represented
as a canonical `file:` URI rather than a host-local path string. The
probe also needs a bounded response time: an exec-server that completes
initialization but never answers `environment/info` must not hold the
environment serialization queue indefinitely.
## What changed
- Add an experimental `environment/info` app-server RPC for named
environments.
- Route the probe through the managed environment connection and return
target-native shell metadata plus the default working directory as a
`PathUri`.
- Return connection and protocol failures as JSON-RPC errors.
- Bound the exec-server probe response to 30 seconds and remove
timed-out calls from the pending-request table so later environment
mutations can proceed.
- Cover successful responses, omitted working directories, unknown
environments, connection failures, and pending-call cleanup.
## Protocol examples
Request:
```json
{
"id": 42,
"method": "environment/info",
"params": {
"environmentId": "remote-a"
}
}
```
Successful response:
```json
{
"id": 42,
"result": {
"shell": {
"name": "zsh",
"path": "/bin/zsh"
},
"cwd": "file:///workspace"
}
}
```
If the exec-server initializes but does not answer the probe within 30
seconds:
```json
{
"id": 42,
"error": {
"code": -32603,
"message": "failed to get info for environment `remote-a`: exec-server protocol error: timed out waiting for exec-server `environment/info` response after 30s"
}
}
```
## Testing
- App-server integration coverage for successful info (including omitted
`cwd`), unknown environments, and connection failures.
- Exec-server RPC coverage verifying a timed-out call is removed from
the pending-request table.
---------
Co-authored-by: Michael Bolin <mbolin@openai.com>
## Summary
- complete unified-exec processes from the ordered event stream instead
of issuing a final zero-wait `process/read`
- add optional executor sandbox-denial state to `process/exited`
- retain `process/read` as a retained-output and compatibility fallback
for receiver lag, sequence gaps, and legacy servers
- recover sandbox-denial state across transport reconnection
- cover the real `TestCodex` remote-exec path without adding a public
test-only event constructor
## Why
A successful one-shot tool call currently receives its output and
terminal notifications, then pays another wide-area `process/read` round
trip before returning. Staging traces showed that remote response wait
accounted for more than 99.8% of RPC time; local serialization,
queueing, and deserialization were below 0.6 ms.
## Measured impact
A direct staging A/B used the same build and route and changed only
completion mode. Each arm ran three times with 30 one-shot
`/usr/bin/true` calls per run. The table reports the median of the three
per-run percentiles.
| Metric | Final `process/read` | Pushed events | Change |
| --- | ---: | ---: | ---: |
| End-to-end completion p50 | 159.5 ms | 118.7 ms | -40.8 ms (-25.6%) |
| End-to-end completion p95 | 182.4 ms | 131.7 ms | -50.6 ms (-27.8%) |
| Completion-wait p50 | 80.1 ms | 41.5 ms | -38.5 ms (-48.1%) |
| Final `process/read` RPC p50 | 79.9 ms | eliminated | -79.9 ms |
TCP_NODELAY was enabled in both A/B arms, so its effect cancels out. The
successful, complete, in-order event path issued zero final
`process/read` calls.
## Compatibility and recovery
- new servers send `sandboxDenied` on `process/exited`
- legacy servers omit it, which triggers one compatibility
`process/read`
- broadcast lag or a sequence gap triggers a retained-output read
- recovery remains bounded by the server's existing 1 MiB
retained-output window
- complete, in-order event streams issue no completion read
- sandbox denial is attached to the exit event before consumers can
observe process completion
- server-first and client-first rollouts remain wire-compatible;
server-first realizes the latency win immediately
## Integration coverage
The `TestCodex` suite exercises four distinct remote-exec contracts:
- complete pushed output/exit/close with zero reads
- direct pushed sandbox denial with zero reads
- legacy missing denial metadata with exactly one compatibility read
- count-bounded replay eviction recovered from retained output without
duplication
## Validation
- `just test -p codex-core
exec_command_consumes_pushed_remote_process_events`: 4 passed
- `just test -p codex-core unified_exec::process_tests::`: 4 passed
- `just test -p codex-exec-server`: 294 passed, 2 skipped
- `just test -p codex-exec-server-protocol`: 5 passed
- `just test -p codex-rmcp-client`: 89 passed, 2 skipped
- focused Bazel `//codex-rs/core:core-all-test`: passed across 16 shards
- scoped `just fix` passed for core and exec-server
- `just fmt` passed
The complete workspace suite was not rerun; focused Cargo and Bazel
coverage passed for the changed behavior.
[Codex Thread
019ef1f9-36e2-7e91-9337-504f097b9dc1](https://codex-thread-link.openai.chatgpt-team.site/thread/019ef1f9-36e2-7e91-9337-504f097b9dc1)
## Why
Hosted plugin-service Streamable HTTP MCP traffic uses
`https://chatgpt.com/backend-api/ps/mcp` and depends on Cloudflare's
`__cflb` cookie for load-balancer affinity. The local and exec-server
`http/request` path built a fresh reqwest client for each request
without installing Codex's existing shared ChatGPT Cloudflare cookie
store, so affinity could be lost between calls.
This is an affinity-hardening change motivated by an incident
investigation. It does not establish the broader connector-cache
incident RCA or claim to fix that incident in full.
## What changed
- Install the existing process-local, strictly allowlisted ChatGPT
Cloudflare cookie store on the reqwest client used by
`ReqwestHttpClient`.
- Fresh clients now share allowed Cloudflare infrastructure cookies
within the process that originates the local or exec-server network
request.
- Keep the existing HTTPS ChatGPT-host and Cloudflare-cookie-name
restrictions. This does not introduce a general cookie jar or send
ChatGPT Cloudflare cookies to unrelated hosts.
## Test coverage
- `codex-client` unit coverage verifies that the existing strict store
accepts and returns `__cflb` for HTTPS ChatGPT URLs.
- The exec-server HTTPS integration test sends four independent
`http/request` calls through a local TLS-intercepting proxy and verifies
that:
- `Set-Cookie: __cflb=west` is sent on the next plugin-service request;
- a later `Set-Cookie: __cflb=central` replaces the stored value;
- non-Cloudflare session cookies are discarded;
- no stored ChatGPT Cloudflare cookie is sent to a non-ChatGPT host.
- `just test -p codex-client` — 38 passed.
- `just test -p codex-exec-server --test chatgpt_cloudflare_affinity` —
1 passed.
- `just bazel-lock-check` — passed.
## Non-goals
- No persistence of ChatGPT auth, account, session, residency, or
arbitrary cookies.
- No cookie persistence for third-party MCP servers.
- No special composition of caller-provided `Cookie` headers.
- No plugin-service, connector-cache, Habitat/habicache, routing,
redirect, or API-contract changes.
- No broader incident RCA conclusions.
## Why
This stack crosses World State, executor skills, selected plugin
metadata, MCP processes, connectors, dynamic environments, and resume.
This PR adds two end-to-end scenarios that validate those pieces
together.
Both tests enable `deferred_executor`, so they exercise the real
delayed-environment path.
## Scenario 1: availability across turns and resume
```text
1. Start a thread with one selected plugin root bound to E1.
2. E1 is unavailable.
- executor skill is absent
- selected MCP is absent
- connector has no selected-plugin attribution
3. Start E1 and register the same stable environment ID.
4. Start a new turn.
- the executor skill appears through World State
- its body beats a colliding host skill
- the selected MCP tool is advertised and executes inside E1
- the connector is attributed to the selected plugin
5. Start another turn without changing E1.
- the MCP PID stays the same, proving runtime reuse
6. Restart app-server and resume the thread.
- durable selected-root intent is restored
- skills, MCP, and connector attribution are restored
- a new MCP PID proves ephemeral process state was rebuilt
```
## Scenario 2: availability changes inside one turn
```text
1. Start a turn while E1 is unavailable.
2. The first model sample sees no executor skill, MCP, or selected connector.
3. The turn pauses on request_user_input.
4. Start E1 and register it while that same turn is still active.
5. Continue the turn.
6. The very next model sample sees:
- the executor skill catalog
- the selected MCP tool
- selected-plugin connector attribution
7. The model calls the MCP, and its output proves execution happened inside E1.
```
This second scenario specifically protects the aeon-style behavior:
capability state is captured again for every sampling step, not only at
the next user turn.
## Scope
These are integration tests only. They do not add a combinatorial matrix
for unsupported plugin-file mutation, environment generations, transport
disconnects, or delayed `required = true` executor MCPs.
Fixes distributed trace continuity across exec-server JSON-RPC HTTP
egress by adding an executor client span and injecting its W3C context
through a reusable `codex-otel` helper.
This preserves the caller trace across core/tool → executor →
provider/MCP instead of dropping parentage at raw reqwest.
Note that this doesn't include the websocket path, which is needed to
really get the full story but at least we cover the basic http path with
this change.
## Summary
- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**
## Validation
- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
## Summary
- retry ERS `409 environment_offline` responses inside the existing
exec-server recovery loop
- keep all other registry conflicts terminal
- add focused coverage for both cases
## Root cause
When an exec server disconnects and reconnects, the client already
starts recovery and calls ERS `/connect`. During the transient executor
presence gap, ERS can return `409 environment_offline`. The retry
classifier treated every 409 as terminal, so the first response aborted
the existing 25-second recovery window before the executor came back
online. That then caused active processes to be marked lost.
This change classifies only the structured `environment_offline`
conflict as retryable. Recovery continues with the existing bounded
deadline, exponential backoff, and jitter.
## Validation
- `just test -p codex-exec-server client::recovery::tests` — 4 passed
- `just fix -p codex-exec-server` — passed
- `just fmt` — passed
- Full `just test -p codex-exec-server` reached unrelated macOS
filesystem-sandbox integration failures because nested
`/usr/bin/sandbox-exec` is denied in this environment (`sandbox_apply:
Operation not permitted`).
## Summary
- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.
## Root cause
Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.
Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.
The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
## Why
`selectedCapabilityRoots` is durable thread intent: “use this capability
root from environment `worker`.”
The important product assumption is:
> One environment ID always names the same logical executor and stable
contents.
`worker` does not silently change from executor A to an unrelated
executor B. The process-local connection handle for `worker` can still
be replaced while Codex is running, though, for example when
`environment/add` registers a fresh handle for the same logical
environment.
The thread should persist only the stable selection. Each model step
should pair that selection with the exact ready handle captured for that
step.
## The boundary
```text
persisted thread intent
plugin@1 -> environment "worker"
|
| capture the current step
v
model-step view
unavailable, or
plugin@1 + worker's exact captured ready handle
```
The environment ID is the stable identity and cache key. The
`Arc<Environment>` is only a process-local handle retained so consumers
of one model step use the same captured environment. It is never
persisted and it does not imply different environment contents.
## What changes
### Persist the stable selection
Selected roots are written into `SessionMeta` and restored with the
thread. Forked subagents inherit the same selections, including
bounded-history forks.
Only stable data is persisted: root ID, environment ID, and root path.
### Capture readiness together with the exact handle
The environment snapshot records:
```rust
environment_id -> Some(Arc<Environment>) // ready in this step
environment_id -> None // still starting in this step
```
This prevents readiness and execution from coming from different
registry snapshots.
For example:
```text
step snapshot: worker -> handle A, ready
environment/add: worker -> fresh handle B for the same logical environment
current step: plugin@1 still uses captured handle A
```
Without carrying handle A in the snapshot, the resolver could combine “A
was ready” with handle B and treat B as ready before it had finished
starting.
This does not change cache invalidation. Stable capability metadata
remains identified by environment ID and capability root. Replacing a
process-local handle under the same stable environment ID does not
invalidate or rediscover that metadata.
### Resolve availability per model step
- A ready captured environment produces resolved roots using its
captured handle.
- A starting, missing, or failed environment is omitted from that step.
- A selected lazy environment that is outside the turn's captured
environment set is asked to start, and a later step can observe it as
ready.
- No capability files are scanned here.
Transient transport disconnects remain the remote client's reconnect
concern. This PR models initial attachment/readiness; it does not add
live socket-connectivity state.
## Example
```text
thread selection: plugin@1 -> environment "worker"
step 1: worker is starting -> plugin@1 unavailable
step 2: worker is ready -> plugin@1 resolves through worker's captured handle
step 3: fresh local handle -> current step remains pinned; a later step captures its own view
```
Temporary unavailability does not discard the durable selection. Later
PRs can retain stable metadata caches while projecting only currently
available capabilities into model-visible World State.
## Compatibility
The app-server request shape does not change. Older rollouts without
`selected_capability_roots` deserialize to an empty list.
## Stack
1. **This PR:** persist stable selected roots and resolve them through
an exact model-step handle.
2. #29960: cache stable skill metadata and project available skills into
World State.
3. #29946: cache stable plugin declarations and manage the separate live
MCP runtime.
## Why
#28522 routes selected-plugin HTTP MCP traffic through the owning
executor, but OAuth bootstrap and refresh still used host-local clients.
Executor-only servers therefore cannot complete discovery or login
through the same network boundary as the MCP connection.
## What changed
- adapt `codex_exec_server::HttpClient` to RMCP 1.8's `OAuthHttpClient`
contract
- let RMCP own discovery, dynamic registration, PKCE, token exchange,
and refresh
- route auth status, persisted-token startup, and app-server login
through the server runtime while preserving the existing local discovery
path
- add optional `threadId` to `mcpServer/oauth/login` and echo it in the
completion notification
- implement RMCP's redirect policy and 1 MiB OAuth response limit over
executor HTTP
- cover selected-thread OAuth discovery and login through an
executor-only route
Depends on #28522.
Stack 3 of 3. Stacked on #29842.
## What changes
Adds an opt-in `followDirectorySymlinks` setting to `fs/walk`.
When enabled, the walk follows directory symlinks but continues to
ignore symlinked files. Canonical directory identities prevent symlink
cycles, while normal paths keep their existing spelling.
Environment skill discovery enables the setting so symlinked skill
directories continue to work with the new single-RPC scan.
## Why
Exec-server JSON-RPC calls can cross local and remote transports, but
trace context stopped at the RPC boundary. That made client and server
work difficult to correlate when diagnosing latency or failures.
## What changed
- Propagate the current W3C trace context on outbound JSON-RPC requests.
- Parent inbound request spans from received trace context.
- Record the received JSON-RPC method on server spans and keep each span
open through response enqueue.
- Add only the OTEL dependencies required by the exec-server crate.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests **(this PR)**
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (153 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server`
Stack 1 of 3. Follow-ups: #29842 and #29844.
## What changes
Adds a general bounded `fs/walk` operation to the exec server.
The operation returns file and directory entries plus recoverable
per-path errors. It skips symlinks, preserves the existing filesystem
sandbox routing, and enforces depth, directory, entry, and response-size
limits.
This PR only defines and wires the filesystem operation. It does not
change any callers yet.
## Why
Start moving towards app-server tests defaulting to running against
remote & foreign OS executors. To do so we need a point of indirection
similar to core integration tests' `build_with_auto_env`, but with the
flexibility of letting tests control environment registration if they
need to.
## What
This adds:
- `TestAppServer::new_with_auto_env()` for constructing an app server
with a default environment defined by the test runner (e.g. bazel)
- `TestAppServer::auto_env_params()` for tests to easily acquire turn
env params tailored to the automatic environment
- `TestAppServer::send_thread_start_request_with_auto_env()` to make it
easy for tests to start a thread using the automatic environment
The above methods all fail if the test calling them has set up an
environment where the automatic environment configuration conflicts with
test-created state.
## Validation
Adds a couple of basic smoke tests to the app-server test suite.
Follow-ups will migrate more tests to use it.
## Why
The app-server and exec-server expose separate JSON-RPC APIs, but
exec-server currently sources its serialized protocol and envelope types
through app-server-oriented code. Giving each API an explicit owner
makes the crate boundary legible without introducing shared generic
envelopes.
## What changed
- Added `codex-exec-server-protocol` to own exec DTOs, process IDs, and
JSON-RPC envelopes.
- Updated exec-server clients, transports, handlers, and tests to use
the new crate.
- Exposed app-server's existing JSON-RPC types through a public `rpc`
module while retaining root re-exports.
- Preserved existing wire shapes, including exec `PathUri` behavior.
## Stack
This is PR 1 of 6. Next: [PR
#29721](https://github.com/openai/codex/pull/29721), which moves auth
mode below the app wire boundary.
## Validation
- Exec-server protocol and server coverage passed in the focused
protocol test runs.
- App-server protocol schema fixtures passed.
## Why
I'd originally added `PathUri` legacy path deserialization thinking we'd
want it for having `PathUri` in public app-server APIs. Since then we've
added `LegacyAppPathString` to handle the messy conversions that we need
for backcompat. It's confusing for `PathUri` to support deserializing
legacy paths when we don't yet want to actually expose app-server
callers or rollout storage to the new URI format.
Stacked on top of #29472 to avoid breaking compatibility in case those
types ended up stored somewhere for someone.
## What changed
- Parse deserialized `PathUri` values exclusively as valid `file:` URIs.
- Replace legacy acceptance coverage with rejection coverage for
top-level filesystem paths and sandbox working directories.
- Serialize CWDs in hand-built exec-server process requests as `PathUri`
values.
## Summary
- add the exec-server working directory to `environment/info` as an
optional `PathUri`
- populate it from the executor process's current directory
- preserve compatibility with older responses that omit `cwd`
## Why
Remote clients currently have no executor-native default working
directory. This forces callers such as app-server-backend to assume
`/workspace`, which fails for laptop environments. Reporting the cwd
alongside the detected shell lets clients use the path convention and
location of the actual executor.
## Impact
This is backward-compatible: the new response field is optional, and
clients can continue handling responses from older exec servers. A
follow-up app-server-backend change will consume the value for cwd-less
`command/exec` requests.
## Validation
- `just test -p codex-exec-server` (275 passed, 2 skipped)
## Why
Filesystem helpers intentionally run with a minimal environment that
excludes proxy variables. After filesystem operations started using the
Windows sandbox wrapper, the wrapper derived an empty proxy
configuration from that helper environment and compared it with the
persistent sandbox setup marker. When the marker contained proxy ports,
every filesystem operation appeared to require a firewall update, which
could launch elevated setup, show a UAC or loader dialog, and fail
operations such as `apply_patch` with error 1223.
Filesystem helpers do not use network access, so they should preserve
the proxy/firewall state established by normal sandboxed process
launches.
## What changed
- Add an explicit Windows sandbox proxy-settings mode for reconciling or
preserving persistent proxy state.
- Use preserve mode for filesystem helpers while normal process launches
continue to reconcile proxy settings from their environment.
- Carry the selected proxy state consistently through setup validation,
elevated setup, and non-elevated ACL refreshes.
- Cover wrapper argument propagation and marker-derived proxy
preservation.
## Validation
- `cargo build -p codex-cli --bin codex`
- `just test -p codex-windows-sandbox
preserving_proxy_settings_uses_the_existing_marker`
- `just test -p codex-windows-sandbox windows_wrapper_args_round_trip`
- `just test -p codex-windows-sandbox
setup_request_prefers_explicit_proxy_settings`
- `just test -p codex-sandboxing transform_for_direct_spawn_windows`
- `just test -p codex-exec-server fs_sandbox::tests`
- Ran the same sandboxed `fs/writeFile` reproduction against published
`0.142.0-alpha.6` and the new CLI. The published CLI launched elevated
setup and failed with `ShellExecuteExW ... 1223`; the new CLI completed
without elevation.
Related to #28359.
## Why
Managed network configures commands to use local HTTP and SOCKS proxies.
For commands delegated to the exec server, the proxy environment and the
sandbox policy were prepared separately. On macOS, that meant a command
could receive `HTTPS_PROXY=http://127.0.0.1:43123` while Seatbelt still
denied access to port `43123`.
## What changed
`NetworkProxy` now prepares the command environment and sandbox context
together from the same runtime snapshot:
```text
Prepared managed network
├── command environment: HTTPS_PROXY=http://127.0.0.1:43123
└── sandbox context: allow outbound to 127.0.0.1:43123
```
That context travels with remote exec requests. The exec server
preserves the managed proxy and CA environment, and macOS Seatbelt
allows only the prepared loopback proxy ports without enabling broad
network access or local binding.
The protocol field is optional and the existing enforcement flag remains
in place, preserving compatibility with callers that do not send the new
context.
## Why
Downstream refactors are producing confusing code with this
functionality having a very generic name. Encoding the specific
conversion approach in the method name makes it clearer.
## What
Rename `PathUri::from_path` to `PathUri::from_host_native_path` and
update its Rust call sites.
## Why
#29113 moved remote sandbox setup and enforcement to the exec server.
That gives the executor ownership of the platform-specific work: a Linux
executor chooses and runs a Linux sandbox even when the Codex
orchestrator is running on macOS or Windows.
It also means the orchestrator no longer knows which concrete sandbox
the executor selected. When that sandbox blocks a remote command, the
orchestrator currently sees only a failed process and can treat the
denial as an ordinary command failure. The existing sandbox approval and
retry path is then skipped.
This PR lets the executor report one portable fact:
> This command probably failed because the executor sandbox blocked it.
The executor keeps its concrete sandbox type private. The protocol sends
only the semantic result.
## Example
Suppose a local macOS Codex session asks a Linux devbox to write outside
the allowed workspace.
Before this PR:
```text
Linux sandbox blocks the write
-> remote process exits with "Permission denied"
-> local orchestrator sees an ordinary command failure
-> the normal sandbox approval and retry path can be skipped
```
With this PR:
```text
Linux sandbox blocks the write
-> executor reports sandboxDenied: true
-> unified exec returns UnifiedExecError::SandboxDenied
-> the existing approval prompt is shown
-> an approved retry runs through the existing unsandboxed retry path
```
## What changes
### The executor remembers its selected sandbox
The prepared remote process now retains the executor-selected
`SandboxType`. This value never crosses the executor boundary.
Commands started without a sandbox retain `SandboxType::None` and are
never reported as sandbox denials.
### The executor uses the existing denial heuristic
The existing local denial heuristic moves from `codex-core` into the
shared `codex-sandboxing` crate.
When a sandboxed remote process exits, the executor:
1. waits the same short output grace period used by local unified exec;
2. reads the output currently available in the existing retained output
buffer;
3. runs the existing heuristic using the exit code and common denial
messages;
4. stores the yes/no result before publishing the process exit.
This deliberately matches the old local unified-exec behavior. It does
not add a new streaming classifier, another output buffer, or stronger
output-retention guarantees.
### The protocol reports a portable boolean
`process/read` gains `sandboxDenied`:
```json
{
"exited": true,
"exitCode": 1,
"closed": false,
"sandboxDenied": true
}
```
The field defaults to `false` when an older executor omits it. The
response does not expose the executor sandbox implementation or
executor-native paths.
### Unified exec uses the existing error path
The exec-server client carries `sandboxDenied` into the unified process
state. If it is true, unified exec returns the existing `SandboxDenied`
error instead of trying to classify remote output using an
orchestrator-side sandbox type.
Remote process exit remains visible as soon as the process exits. This
PR does not wait for stdout or stderr to close and does not change the
existing process lifecycle.
## Scope
This PR is intentionally limited to matching the existing local
unified-exec behavior for the initial command execution path.
It does not add:
- incremental denial tracking across the full output stream;
- new denial handling for commands completed later through
`write_stdin`;
- new guarantees for preserving the semantic flag during the narrow
reconnect-recovery race.
Those can be considered separately if the same behavior is added for
local execution.
## Test coverage
One remote end-to-end integration test covers the complete intended
flow:
```text
remote read-only sandbox
-> denied write
-> executor reports the denial
-> Codex requests approval
-> user approves
-> retry succeeds on the remote executor
```
Existing lifecycle coverage continues to verify that remote process exit
is reported before late output streams close.
## Why
PR #29108 lets the orchestrator send sandbox intent with `process/start`
without wrapping the command for its own operating system.
This PR completes that boundary by making the executor interpret and
enforce the intent using its own filesystem paths and sandbox
implementation.
For example, a macOS TUI targeting a Linux devbox sends `/bin/bash -lc
pwd`. The Linux executor turns that into its own `codex-linux-sandbox
... /bin/bash -lc pwd` launch.
## What changes
- Keep `process/start` unchanged when no sandbox intent is present.
- Convert sandbox `PathUri` values into native paths on the executor.
- Bind symbolic `:workspace_roots` permissions to the executor's native
sandbox cwd.
- Select the sandbox implementation on the executor and wrap the
original command immediately before spawning it.
- Reject sandbox-required execution before spawning when the executor
cannot enforce the intent.
- Pass exec-server runtime paths into process creation so Linux can
locate `codex-linux-sandbox`.
The boundary is therefore:
```text
orchestrator executor
original argv + sandbox intent -> select and enforce local sandbox
```
This PR intentionally treats a denied remote command as an ordinary
command failure. Draft follow-up #29424 carries a semantic
`sandboxDenied` result back to unified exec for the existing approval
and retry flow.
## Platform scope
Linux and macOS use their existing direct-spawn sandbox transforms.
Windows sandboxed remote process launch is intentionally unsupported in
this PR. The current Windows direct-spawn wrapper does not correctly
preserve arbitrary argv, TTY behavior, or pass the full child
environment out of band. The executor rejects the request instead of
running it incorrectly or unsandboxed.
## Known follow-ups
- The transported permission profile can still contain
orchestrator-materialized helper or explicit paths. A `TODO(jif)` marks
where the executor boundary should receive pre-host-materialization
permission intent.
- The sandbox wrapper currently replaces a requested custom inner
`arg0`. A `TODO(jif)` marks where this must be preserved or rejected
explicitly.
- Draft PR #29424 contains the deferred sandbox-denial classification
and approval/retry behavior.
## Rollout assumption
This executor-sandbox stack is unreleased and its client and executor
are expected to move together. This PR does not add mixed-version
negotiation with older exec servers.
