## What changed
- Add pending, ready, and failed provisioning states for Noise environments.
- Preserve the same environment instance whether provisioning is reported before or after materialization, and reject conflicts with ordinary environments.
- Make readiness and failure reports idempotent while rejecting contradictory terminal transitions.
- Delay connection attempts until a provisioned environment is selected and provisioning succeeds.
## Testing
- Cover status reports before and after materialization, terminal failures, repeated and contradictory reports, and replacement between ordinary and deferred environments.
GitOrigin-RevId: 4360a8f2a80c1a99a1dc9257e5d77c07b72b8eb3
## Why
Filesystem denials and managed-network blocks did not share a structured event
shape, requiring downstream consumers to rediscover enforcement paths and parse
backend-specific output. See https://github.com/openai/codex/pull/17573.
## What changed
- Add normalized filesystem and network violation types in `codex-sandboxing`
and emit them through a shared tracing seam.
- Classify filesystem denials by backend and reason, retaining an optional path
and bounded output snippet, and preserve managed-network block context.
- Report the sandbox type through exec-server responses so unified exec can
classify remote denials without guessing; omitted values remain compatible
with older peers.
- Record violations from exec, apply-patch, shell-escalation, unified-exec, and
managed-network enforcement paths without changing denial behavior.
## Testing
- Cover filesystem classification, path extraction, `SIGSYS`, network event
conversion, protocol compatibility, and remote sandbox-type propagation.
GitOrigin-RevId: d673173b4fa6bdf0a24421194a8a61c81fab9c96
## What changed
- Forward remote managed-network policy requests to the controller-side decider when Guardian review is enabled.
- Bound callbacks using the configured permission-hook and Guardian review timeouts, restore trusted execution attribution, and recheck live baseline policy before requesting a decision.
- Cancel pending decisions when the process or execution scope ends, and propagate EOF across the sandbox proxy bridge with half-closes.
## Testing
- Cover callback enablement, strict allowlist behavior, live policy updates, trusted attribution, and execution-scope cancellation.
GitOrigin-RevId: 83b47657c8c4546dfe98eec8b272833e96a46b7e
## What changed
- Add client-side handling for exec-server network policy requests, including request validation, per-process decision routing, and allow, deny, or ask responses.
- Bound concurrent callbacks and fail closed when requests are invalid, exceed capacity, time out, or outlive their process session.
- Replace the callback opt-in flag with a launch-level `policyDecisionTimeoutMs` value and include transport overhead in the executor timeout.
- Preserve callback admission limits across connection recovery and clean up policy state when process startup is abandoned or a session ends.
## Testing
- Cover decision forwarding, invalid and excess requests, cancellation during process cleanup, abandoned starts, timeout propagation, and launch-config serialization.
GitOrigin-RevId: 6ee2a0139e0f8170cf1a5cdd646bd811b7d3c1f6
## What changed
- Preserve the current tracing span and subscriber when lazy remote environment
startup runs in a background task.
- Add spans for the remote connection, environment registry, Noise, and
rendezvous WebSocket stages, including relevant environment and registration
identifiers.
- Propagate the current trace context to environment registry connection and
harness-key validation requests.
GitOrigin-RevId: 0044c3eb94d59c6a6b92752d3d372195be4adb81
## Why
Remote environment connections need to honor Codex's effective outbound proxy policy, including when a rendezvous connection reconnects.
## What changed
- Pass the configured `HttpClientFactory` into remote environment transports and use `WebSocketConnector` for exec-server and rendezvous WebSockets.
- Resolve proxy routes asynchronously so these connections can use the configured system proxy.
- Add connector options that preserve Tungstenite's default TLS behavior and enable `TCP_NODELAY` for latency-sensitive rendezvous traffic.
## Testing
- Verify prepared remote environments connect through a configured system proxy.
- Verify initial and reconnected encrypted relay peers use the system proxy.
- Cover default TLS selection and opt-in `TCP_NODELAY` behavior in the WebSocket client.
GitOrigin-RevId: 8a8da2116e37cb3a891269d0c0b037986fecdd3c
## Why
Noise environment registry requests need to follow the exec server's effective
outbound proxy policy without exposing registry URLs or response headers in HTTP
diagnostics.
## What changed
- Build the registry client from the supplied `HttpClientFactory` and use a
route-aware API client with redirects and request logging disabled.
- Defer construction of the Noise connection provider until the outbound HTTP
policy is available.
- Map route-aware request failures into registry errors while retaining timeout
detection across response body reads.
## Testing
Add coverage for system-proxy routing, sensitive registry metadata redaction,
stalled response-body timeouts, and prepared Noise configuration validation.
GitOrigin-RevId: d312dfe037f72732085bf38109af44df76ed0b53
## What changed
- Remove unused APIs and test helpers across the Rust workspace.
- Route Linux sandbox test setup through the shared exec request path.
- Drop dependencies that became unnecessary after the cleanup.
GitOrigin-RevId: 48fd9b2b1575bad7add0819786b4d8ec88d8486f
## Why
Remote executions need managed-network proxy listeners in the executor so their
loopback proxy addresses are reachable by the launched process.
## What changed
- Add a capability-gated exec-server protocol field for executor-local proxy
launch configuration, including network policy, audit metadata, and execution
attribution.
- Start the proxy while preparing a remote process, replace inherited proxy
environment variables with its local addresses, and derive the sandbox
context from its listeners.
- Keep the proxy alive until inherited output streams close, then shut it down.
- Reject unsupported remote settings such as MITM and credential injection.
## Testing
- Cover configuration round trips and rejection of unsupported settings.
- Verify executor-local startup, blocked-domain enforcement, protocol
compatibility, and proxy lifetime through process closure.
GitOrigin-RevId: c984f54e3e600aa9ebcbf8cf4574046e2c199d11
## Why
Selected capability roots can contribute plugins, MCP servers, connectors, and
skills. Discovering each contribution separately requires repeated access to the
executor filesystem.
## What changed
- Add the `capabilityRoots/discoverV1` exec-server RPC to scan selected roots and
materialize recognized plugin manifests, configuration files, skill
instructions, and skill metadata in one bounded request.
- Add the opt-in `executor_capability_discovery` feature, with a thread-scoped
cache and per-step snapshot shared by MCP and skill discovery.
- Parse MCP, connector, and skill contributions from the materialized snapshot,
including serving cached skill instructions without another filesystem read.
## Testing
- Cover discovery limits, manifest precedence, root-local failures, cache reuse,
plugin contributions, and parity with the existing environment skill loader.
GitOrigin-RevId: f98fd2321cafb58c596db02da1f83c09d8eb375d
## What changed
- Track connected and disconnected states across initial remote exec-server connections and reconnection attempts.
- Emit experimental `thread/environment/connected` and `thread/environment/disconnected` app-server notifications for each thread selecting the environment. Each payload identifies the thread and environment; current state is not replayed when a thread starts.
- Stop forwarding connection events when an environment selection is removed or replaced.
## Testing
- Cover connection, disconnection, reconnection, shared-environment notifications, and replacement of a selected environment.
GitOrigin-RevId: 5dd767372363c4a2a8319fc16164be117d5bd20c
## What changed
- Replace pending WebSocket URL registration with deferred Noise environment
registration that gates connection attempts on an explicit readiness signal.
- Reuse the Noise rendezvous transport after readiness so reconnects request a
fresh connection bundle.
- Preserve terminal errors for failed or dropped registrations and keep late
completion isolated from replacement environments.
## Testing
- Add coverage for readiness gating, registration failure and replacement,
eager Noise connections, and reconnection through a fresh rendezvous bundle.
GitOrigin-RevId: 83e23fa03a02e3b2bdf1a83fe26d7ac461f55cf3
## What changed
- Add the initialized `environment/status` RPC, which reports `ready` when the exec server can handle requests.
- Expose environment IDs and `ready`, `pending`, or `disconnected` status through `EnvironmentManager` and `Environment`.
- Keep status checks non-mutating: they do not start or recover lazy remote environments, and probe only an existing connection.
## Testing
- Cover the status RPC over WebSocket and the in-process request processor.
- Verify that checking an unstarted stdio environment leaves it pending and that failed connections report as disconnected.
GitOrigin-RevId: 22febeb6a3457849292128a8991c6400c22b3fd8
## Why
Remote environment provisioning can finish after a thread starts, before an
exec-server WebSocket URL is available.
## What changed
- Add `EnvironmentManager::register_pending_environment` and a one-shot
`PendingEnvironmentRegistration` handle that resolves to either a validated
WebSocket URL or a terminal provisioning error.
- Let lazy remote exec-server clients wait for that result, while preserving
reconnection behavior after a successful registration.
- Keep replacement registrations isolated so completing an older handle does
not resolve the current environment with the same ID.
## Testing
Add coverage for successful connection and reconnection, provisioning and
dropped-registration failures, invalid URLs, replacement isolation, and the
deferred-executor startup flow.
GitOrigin-RevId: 5c05be2b72291b77a1f71176d7075b1ad63332a5
## Why
The process-start background task can move to a Tokio worker thread, where the
caller's thread-local tracing subscriber is otherwise unavailable. This can
break propagation of the caller's trace context to the exec-server request.
## What changed
- Attach both the current span and current tracing subscriber to spawned
process-start tasks.
- Run exec-server Bazel unit tests serially because their tracing setup uses
process-global state.
## Testing
Run the trace-context regression test on a multi-thread Tokio runtime so it
exercises propagation across the background task.
GitOrigin-RevId: a586eefd6983916d670fc5a90f0decd0468d273b
## Why
Frame-count backpressure does not bound the amount of executor-controlled data
retained in streamed HTTP response queues, and a single body delta could exceed
the intended wire size.
## What changed
- Limit each decoded `http/request/bodyDelta` payload to 1 MiB, split locally
produced response chunks at that boundary, and reject oversized incoming
deltas.
- Apply a shared 16 MiB byte budget across queued HTTP response streams. Release
capacity as deltas are consumed and fail a stream when the budget is
exhausted.
## Testing
Added coverage for rejecting an oversized delta and for failing a stream after
its queued deltas exhaust the shared byte budget.
GitOrigin-RevId: be9205fef44b0ec96b84b31a8e059b1cb9cd3f3b
## Why
A thread can select skill roots that live in an executor environment.
`skills/list` needs a passive snapshot of the roots that are usable now:
it must not start an executor, wait for recovery, or reconnect a failed
environment.
The initial implementation checked the immutable first startup result.
After a successful connection later entered recovery or failed, that
result still looked successful. A read-only catalog request could then
wait for recovery or trigger a new connection while reading the
filesystem.
## What
- inspect readiness from the current exec-server connection state
- return roots only while their environment can serve a request
immediately
- omit environments that have not started, are connecting, or are
recovering
- return warnings for missing environments and terminal connection
failures
- add a fail-fast filesystem view that never starts, waits for, or
reconnects an environment
- expose the passive selected-root snapshot through `CodexThread`
## Behavior
- Local and currently connected environments are ready.
- Starting and recovering environments are omitted without a warning so
callers can retry later.
- Missing and terminally failed environments are omitted with a warning.
- A disconnect between readiness inspection and filesystem access fails
promptly instead of crossing into the normal recovery path.
- Normal model-turn and execution paths keep their existing reconnect
behavior.
## Design
The recovery policy is private to the exec-server client. Callers choose
the explicit fail-fast filesystem method; the existing client and
filesystem APIs remain reconnecting. This keeps the passive contract at
the transport boundary instead of plumbing timeout or retry flags
through the skills stack.
## Coverage
- a lazy stdio environment stays unstarted during passive inspection
- missing and terminally failed environments surface warnings
- a real websocket disconnect proves current readiness drops, a
previously acquired fail-fast filesystem handle returns promptly, and
readiness returns after recovery
## Scope
This PR only provides passive readiness and fail-fast filesystem
primitives. It does not add app-server API fields or notifications.
## Stack
- #31582 uses these primitives for experimental thread-scoped
`skills/list`.
- #30228 adds targeted invalidation notifications.
## Why
A malicious exec-server can send out-of-order process notifications
faster than missing events arrive. The client retained every future
event in a per-session reorder map, so many tiny events or a few large
output chunks could grow orchestrator memory without bound.
## What changed
- cap each session's pending process-event reorder state at 256 events
and 1 MiB
- reject individual process events larger than 1 MiB
- release byte accounting as events publish or a session fails
- fail and detach only the affected process session when a limit is
exceeded
- let the next expected event drain a full buffer, so the limits apply
to retained future state
- apply the same bounded insertion path during reconnect recovery while
safely handling dense tail output, missing exit events, newer live
notifications, and conflicting `Closed` sequences
Sequence distance is intentionally not capped because it does not affect
allocation; the event-count and byte limits are the resource-exhaustion
boundary.
## Tests
Focused exec-server tests cover oversized output, full count/byte
buffers, gap-closing delivery, dense recovery interleaved with newer
notifications, missing exit reconstruction, and conflicting terminal
sequences.
## Scope
This is limited to per-session process-event reorder state. It does not
introduce a scheduler, quota framework, or new public configuration.
## Why
An untrusted exec-server can stop reading requests, never answer them,
or send guessed responses before queued requests are written. Without
client-side admission, the orchestrator can retain unbounded RPC call
futures and request payloads.
We need a hard bound without adding a blanket timeout, because
individual operations already own their timeout and cleanup semantics.
## What changed
- hold one of 64 shared admission permits for the full lifetime of each
regular RPC call, so an early response cannot free capacity while the
call remains queued
- allow `process/terminate` and `fs/close` to use one additional cleanup
permit
- close the transport and fail pending calls if the cleanup permit is
also stuck, allowing teardown or recovery to release remote resources
- leave the wire format and existing timeout behavior unchanged
## Tests
- `rpc_client_call_has_no_implicit_deadline` verifies that ordinary
calls remain untimed
- `rpc_client_bounds_in_flight_calls_and_preserves_cleanup` covers the
regular-call cap, guessed early responses, cleanup admission, and the
cleanup circuit breaker
## Why
Make it easier to measure the performance of different parts of skill
loading.
## What
- Add spans for step-context capture, world-state construction, executor
catalog snapshot/root loading, and environment skill loading.
- Record the discovered environment skill count.
- Trace outbound exec-server requests with client kind and RPC method
fields.
- Update trace propagation tests to assert that requests keep the parent
trace id while creating their own child span.
## Why
App-server clients that configure named execution environments need to
discover an environment's shell and working directory before selecting
it for a thread or turn. Because the environment can run on a different
operating system than app-server, its working directory is represented
as a canonical `file:` URI rather than a host-local path string. The
probe also needs a bounded response time: an exec-server that completes
initialization but never answers `environment/info` must not hold the
environment serialization queue indefinitely.
## What changed
- Add an experimental `environment/info` app-server RPC for named
environments.
- Route the probe through the managed environment connection and return
target-native shell metadata plus the default working directory as a
`PathUri`.
- Return connection and protocol failures as JSON-RPC errors.
- Bound the exec-server probe response to 30 seconds and remove
timed-out calls from the pending-request table so later environment
mutations can proceed.
- Cover successful responses, omitted working directories, unknown
environments, connection failures, and pending-call cleanup.
## Protocol examples
Request:
```json
{
"id": 42,
"method": "environment/info",
"params": {
"environmentId": "remote-a"
}
}
```
Successful response:
```json
{
"id": 42,
"result": {
"shell": {
"name": "zsh",
"path": "/bin/zsh"
},
"cwd": "file:///workspace"
}
}
```
If the exec-server initializes but does not answer the probe within 30
seconds:
```json
{
"id": 42,
"error": {
"code": -32603,
"message": "failed to get info for environment `remote-a`: exec-server protocol error: timed out waiting for exec-server `environment/info` response after 30s"
}
}
```
## Testing
- App-server integration coverage for successful info (including omitted
`cwd`), unknown environments, and connection failures.
- Exec-server RPC coverage verifying a timed-out call is removed from
the pending-request table.
---------
Co-authored-by: Michael Bolin <mbolin@openai.com>
## Summary
- complete unified-exec processes from the ordered event stream instead
of issuing a final zero-wait `process/read`
- add optional executor sandbox-denial state to `process/exited`
- retain `process/read` as a retained-output and compatibility fallback
for receiver lag, sequence gaps, and legacy servers
- recover sandbox-denial state across transport reconnection
- cover the real `TestCodex` remote-exec path without adding a public
test-only event constructor
## Why
A successful one-shot tool call currently receives its output and
terminal notifications, then pays another wide-area `process/read` round
trip before returning. Staging traces showed that remote response wait
accounted for more than 99.8% of RPC time; local serialization,
queueing, and deserialization were below 0.6 ms.
## Measured impact
A direct staging A/B used the same build and route and changed only
completion mode. Each arm ran three times with 30 one-shot
`/usr/bin/true` calls per run. The table reports the median of the three
per-run percentiles.
| Metric | Final `process/read` | Pushed events | Change |
| --- | ---: | ---: | ---: |
| End-to-end completion p50 | 159.5 ms | 118.7 ms | -40.8 ms (-25.6%) |
| End-to-end completion p95 | 182.4 ms | 131.7 ms | -50.6 ms (-27.8%) |
| Completion-wait p50 | 80.1 ms | 41.5 ms | -38.5 ms (-48.1%) |
| Final `process/read` RPC p50 | 79.9 ms | eliminated | -79.9 ms |
TCP_NODELAY was enabled in both A/B arms, so its effect cancels out. The
successful, complete, in-order event path issued zero final
`process/read` calls.
## Compatibility and recovery
- new servers send `sandboxDenied` on `process/exited`
- legacy servers omit it, which triggers one compatibility
`process/read`
- broadcast lag or a sequence gap triggers a retained-output read
- recovery remains bounded by the server's existing 1 MiB
retained-output window
- complete, in-order event streams issue no completion read
- sandbox denial is attached to the exit event before consumers can
observe process completion
- server-first and client-first rollouts remain wire-compatible;
server-first realizes the latency win immediately
## Integration coverage
The `TestCodex` suite exercises four distinct remote-exec contracts:
- complete pushed output/exit/close with zero reads
- direct pushed sandbox denial with zero reads
- legacy missing denial metadata with exactly one compatibility read
- count-bounded replay eviction recovered from retained output without
duplication
## Validation
- `just test -p codex-core
exec_command_consumes_pushed_remote_process_events`: 4 passed
- `just test -p codex-core unified_exec::process_tests::`: 4 passed
- `just test -p codex-exec-server`: 294 passed, 2 skipped
- `just test -p codex-exec-server-protocol`: 5 passed
- `just test -p codex-rmcp-client`: 89 passed, 2 skipped
- focused Bazel `//codex-rs/core:core-all-test`: passed across 16 shards
- scoped `just fix` passed for core and exec-server
- `just fmt` passed
The complete workspace suite was not rerun; focused Cargo and Bazel
coverage passed for the changed behavior.
## Why
Exec-server JSON-RPC calls can cross local and remote transports, but
trace context stopped at the RPC boundary. That made client and server
work difficult to correlate when diagnosing latency or failures.
## What changed
- Propagate the current W3C trace context on outbound JSON-RPC requests.
- Parent inbound request spans from received trace context.
- Record the received JSON-RPC method on server spans and keep each span
open through response enqueue.
- Add only the OTEL dependencies required by the exec-server crate.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests **(this PR)**
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (153 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server`
Stack 1 of 3. Follow-ups: #29842 and #29844.
## What changes
Adds a general bounded `fs/walk` operation to the exec server.
The operation returns file and directory entries plus recoverable
per-path errors. It skips symlinks, preserves the existing filesystem
sandbox routing, and enforces depth, directory, entry, and response-size
limits.
This PR only defines and wires the filesystem operation. It does not
change any callers yet.
## Why
The app-server and exec-server expose separate JSON-RPC APIs, but
exec-server currently sources its serialized protocol and envelope types
through app-server-oriented code. Giving each API an explicit owner
makes the crate boundary legible without introducing shared generic
envelopes.
## What changed
- Added `codex-exec-server-protocol` to own exec DTOs, process IDs, and
JSON-RPC envelopes.
- Updated exec-server clients, transports, handlers, and tests to use
the new crate.
- Exposed app-server's existing JSON-RPC types through a public `rpc`
module while retaining root re-exports.
- Preserved existing wire shapes, including exec `PathUri` behavior.
## Stack
This is PR 1 of 6. Next: [PR
#29721](https://github.com/openai/codex/pull/29721), which moves auth
mode below the app wire boundary.
## Validation
- Exec-server protocol and server coverage passed in the focused
protocol test runs.
- App-server protocol schema fixtures passed.
## Why
#29113 moved remote sandbox setup and enforcement to the exec server.
That gives the executor ownership of the platform-specific work: a Linux
executor chooses and runs a Linux sandbox even when the Codex
orchestrator is running on macOS or Windows.
It also means the orchestrator no longer knows which concrete sandbox
the executor selected. When that sandbox blocks a remote command, the
orchestrator currently sees only a failed process and can treat the
denial as an ordinary command failure. The existing sandbox approval and
retry path is then skipped.
This PR lets the executor report one portable fact:
> This command probably failed because the executor sandbox blocked it.
The executor keeps its concrete sandbox type private. The protocol sends
only the semantic result.
## Example
Suppose a local macOS Codex session asks a Linux devbox to write outside
the allowed workspace.
Before this PR:
```text
Linux sandbox blocks the write
-> remote process exits with "Permission denied"
-> local orchestrator sees an ordinary command failure
-> the normal sandbox approval and retry path can be skipped
```
With this PR:
```text
Linux sandbox blocks the write
-> executor reports sandboxDenied: true
-> unified exec returns UnifiedExecError::SandboxDenied
-> the existing approval prompt is shown
-> an approved retry runs through the existing unsandboxed retry path
```
## What changes
### The executor remembers its selected sandbox
The prepared remote process now retains the executor-selected
`SandboxType`. This value never crosses the executor boundary.
Commands started without a sandbox retain `SandboxType::None` and are
never reported as sandbox denials.
### The executor uses the existing denial heuristic
The existing local denial heuristic moves from `codex-core` into the
shared `codex-sandboxing` crate.
When a sandboxed remote process exits, the executor:
1. waits the same short output grace period used by local unified exec;
2. reads the output currently available in the existing retained output
buffer;
3. runs the existing heuristic using the exit code and common denial
messages;
4. stores the yes/no result before publishing the process exit.
This deliberately matches the old local unified-exec behavior. It does
not add a new streaming classifier, another output buffer, or stronger
output-retention guarantees.
### The protocol reports a portable boolean
`process/read` gains `sandboxDenied`:
```json
{
"exited": true,
"exitCode": 1,
"closed": false,
"sandboxDenied": true
}
```
The field defaults to `false` when an older executor omits it. The
response does not expose the executor sandbox implementation or
executor-native paths.
### Unified exec uses the existing error path
The exec-server client carries `sandboxDenied` into the unified process
state. If it is true, unified exec returns the existing `SandboxDenied`
error instead of trying to classify remote output using an
orchestrator-side sandbox type.
Remote process exit remains visible as soon as the process exits. This
PR does not wait for stdout or stderr to close and does not change the
existing process lifecycle.
## Scope
This PR is intentionally limited to matching the existing local
unified-exec behavior for the initial command execution path.
It does not add:
- incremental denial tracking across the full output stream;
- new denial handling for commands completed later through
`write_stdin`;
- new guarantees for preserving the semantic flag during the narrow
reconnect-recovery race.
Those can be considered separately if the same behavior is added for
local execution.
## Test coverage
One remote end-to-end integration test covers the complete intended
flow:
```text
remote read-only sandbox
-> denied write
-> executor reports the denial
-> Codex requests approval
-> user approves
-> retry succeeds on the remote executor
```
Existing lifecycle coverage continues to verify that remote process exit
is reported before late output streams close.
## Why
Remote environments can be registered before their exec-server is first
used. Starting the connection at registration time uses that startup
window, while sharing one startup result prevents background work and
capability calls from opening competing connections.
Keep initial startup simple: each environment makes one connection
attempt using its configured transport timeout. A failed initial attempt
is final for that environment, while an environment that disconnects
after connecting can still recover on a later operation.
## What changed
- Start URL and Noise environments in the background when they are added
to `EnvironmentManager`. Provider snapshots are fully validated before
connection work begins.
- Share one initial connection attempt and its saved result across
metadata, process, filesystem, and HTTP callers.
- Keep configured stdio environments lazy until first use so
registration does not launch a process.
- Tie background startup work to the environment lifetime so replacing
or dropping an environment cancels unfinished work.
- After an established client disconnects, share one fresh connection
attempt across concurrent callers. A failed attempt fails the current
operation without permanently preventing a later attempt.
- Store the shared lazy client directly on `Environment` and expose
small methods for starting, observing, and awaiting startup.
## Test plan
- `just test -p codex-exec-server`
- `just test -p codex-app-server
turn_start_resolves_sticky_thread_local_environment_and_turn_overrides`
## Summary
Remote stdio MCP servers send tool calls by writing JSON-RPC bytes
through `process/write`.
When the exec-server websocket drops at the wrong time, the remote
process can survive session recovery, but the stdin write can still fail
back to RMCP as a transport send error. RMCP then closes the stdio MCP
transport, so tools like `node_repl` are lost even though the
process/session recovery path is working.
This changes `process/write` to be safe to retry across exec-server
recovery:
- adds a required `writeId` to `process/write`
- retries remote `Session::write` with the same `writeId` after
reconnect
- remembers accepted write ids per process so duplicate retries return
`Accepted` without writing the same bytes to child stdin again
- covers both the client retry path and server-side write id dedupe with
tests
In simple terms:
```text
before:
write to MCP stdin -> websocket closes -> write errors -> RMCP closes node_repl
after:
write to MCP stdin -> websocket closes -> reconnect -> retry same writeId
server either writes once or recognizes it already did
```
Supersedes #28288 (closed).
## Why
A short WebSocket interruption currently ends every client-side process
handle, even though exec-server keeps the server session and its
processes alive for a short time.
This is especially visible for executor-backed stdio MCP servers: a
temporary connection loss becomes a permanent `Transport closed` error.
The server already has the information needed to resume the session, but
the client opens a fresh session instead of using it.
This change reconnects below the process and MCP layers. Existing
process handles stay valid, missed output is recovered, and the same
server-side processes continue running.
## State machine
One logical `ExecServerClient` stays alive while its underlying RPC
connection changes generations.
```text
transport closes
+------------------------------------------------+
| v
+-------------+ +-------------+
| Connected | | Recovering |
+-------------+ +-------------+
^ |
| session resumed, processes caught up | retryable error
+------------------------------------------------+ loops until deadline
|
| deadline or permanent error
v
+-------------+
| Failed |
+-------------+
```
### `Connected`
- New RPC calls use the current connection.
- Process notifications are published in sequence order.
- A disconnect only starts recovery if it came from the current
connection generation. Late events from older generations cannot replace
the active connection.
### `Recovering`
- New calls wait instead of choosing a half-connected RPC client.
- Existing process handles, wake subscriptions, and event subscriptions
stay open.
- Streaming HTTP response bodies fail immediately because their byte
streams cannot be resumed safely.
- Recovery first waits for process starts that were already in flight. A
start whose result became ambiguous is cleaned up after reconnection
instead of being silently adopted.
- The client reconnects with the learned `session_id`. The server may
briefly report that the old connection is still attached, so that error
is retried until the detach finishes.
- The notification consumer starts before the resume handshake
completes. This prevents a busy process from filling the notification
queue and blocking the initialize response.
- Before installing the new connection, the client catches up every
recoverable process with `process/read`.
### `Failed`
- Recovery stops after 25 seconds or after a permanent error.
- Waiting calls are released with one stable disconnect error.
- Existing process sessions receive a terminal failure instead of
waiting forever.
## Recovering process events
Output, exit, and close events share one sequence. During normal
operation, the client buffers early events until every lower sequence
has been published.
After reconnection, the client reads each process starting after its
last published sequence:
1. Retained output chunks are inserted by sequence number.
2. Exit and close state are reconstructed in their sequence positions.
3. Events already received as live notifications are ignored as
duplicates.
4. Newly contiguous events are published in order.
5. If the server no longer retains enough output to fill a sequence gap,
only that process is terminated and failed. The recovered connection
remains usable for other processes.
The server reports its full next event sequence for unbounded reads,
including exit and close events. Closed processes remain readable for
the same 30-second window used to retain detached sessions.
## Other details
- Detached server sessions are retained for 30 seconds, leaving margin
around the client's 25-second recovery deadline.
- Session attach and detach update the active notification sender under
the same attachment lock, so an old connection cannot clear a newly
attached sender.
- A dedicated error code distinguishes the temporary "session is still
attached" race from permanent initialization errors.
- Process starts are identity-checked on both client and server. Cleanup
from an older start cannot remove a newer process that reused the same
ID.
- Mutating requests that were already in flight when the transport
closed are not replayed, because the client cannot know whether the
server applied them. Requests started after recovery is known wait for
the replacement connection.
- We assume the server/client version stays in sync (on the before/after
this PR)
## User impact
Long-running commands and stdio MCP servers can survive a temporary
exec-server WebSocket interruption without changing process IDs or
losing output produced during the outage.
## Why
`fs/readFile` buffers the entire file in one response, which makes large
remote reads expensive and prevents callers from applying backpressure.
We need an opt-in streaming path with bounded block sizes while
preserving the existing single-call API for small and sandboxed reads.
## What changed
- Add `ExecServerClient::stream`, returning a named `FileReadStream`
that implements `futures::Stream` and yields immutable 1 MiB byte
blocks.
- Add internal `fs/open`, `fs/readBlock`, and `fs/close` RPCs.
`fs/readBlock` accepts an explicit offset and length.
- Keep unsandboxed files open between block reads, cap open handles per
connection, and clean them up on EOF, error, stream drop, explicit
close, or connection shutdown.
- Reject platform-sandboxed streaming opens instead of turning the
one-shot sandbox helper into a persistent server. Existing `fs/readFile`
behavior is unchanged.
## Testing
- `just test -p codex-exec-server`
- Integration coverage for 1 MiB chunking, exact block-boundary EOF,
sandbox rejection, and continued reads from the opened file after path
replacement.
- Handle-manager coverage for non-sequential offsets, variable block
lengths, the 128-handle limit, and capacity release after close.
## Why
Rendezvous forwards traffic between the orchestrator and exec-server.
The endpoints need to authenticate each other and encrypt that traffic
without trusting Rendezvous with plaintext or endpoint keys.
## Changes
- Adds a hybrid Noise IK channel through Clatter using X25519,
ML-KEM-768, AES-256-GCM, and SHA-256.
- Binds each handshake to `environment_id`, `executor_registration_id`,
and `stream_id`.
- Pins the registry-provided executor key and carries the harness
authorization inside the encrypted handshake.
- Orders relay frames before consuming Noise nonces and fragments large
JSON-RPC messages into bounded records.
- Bounds handshake payloads, frames, streams, and message reassembly.
Runtime activation is in
[openai/codex#26245](https://github.com/openai/codex/pull/26245).
## Stack
1. **[openai/codex#26242](https://github.com/openai/codex/pull/26242)**:
Noise channel and relay transport
2. [openai/codex#26245](https://github.com/openai/codex/pull/26245):
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- Oversized initiator payload regression coverage
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
Path composition is already handled by `PathUri`, leaving `fs/join` and
`fs/parent` as redundant exec-server protocol surface. Because
app-server and exec-server are deployed atomically, these obsolete
methods can be removed without a compatibility shim.
This removes the protocol constants and payloads, public client APIs,
server registrations and handlers, and endpoint-only tests. Existing
in-process `PathUri` join/parent coverage remains.
## Validation
- `just test -p codex-exec-server` (215 passed, 2 skipped)
## Why
A long-running unified exec process started with `tty: false` could not
be interrupted via `write_stdin`: ordinary non-TTY stdin writes are
rejected once stdin is closed, but an exact U+0003 payload should still
map to a process interrupt. The interrupt should flow through the same
process lifecycle path as a real signal so Codex preserves
process-reported output and exit metadata instead of fabricating a
Ctrl-C exit code or tearing down the session early.
## What Changed
- Add `process/signal` to exec-server with `ProcessSignal::Interrupt`
and an empty response.
- Add a non-consuming `ProcessHandle::signal` path for spawned
processes; on Unix it sends SIGINT to the process group and leaves
terminate/hard-kill unchanged.
- Route non-TTY U+0003 `write_stdin` through `process.signal(...)`
instead of `terminate`, then let the normal post-write collection path
drain output and observe exit.
- Add exec-server coverage where a shell `trap INT` handler prints the
signal and exits with its own code.
- Add unified exec coverage where a `tty: false` process traps SIGINT,
emits output, and exits with its own code.
## Validation
- `just test -p codex-exec-server
exec_process_signal_interrupts_process`
- `just test -p codex-exec-server`
- `just test -p codex-core
write_stdin_ctrl_c_interrupts_non_tty_session`
## Why
Shell detection needs to be available through the `Environment`
abstraction so callers can ask the selected local or remote environment
for shell metadata without adding a separate HTTP endpoint or parallel
info-source path. This keeps shell metadata shaped like the existing
environment-owned filesystem capability and lets remote environments
answer through exec-server JSON-RPC.
## What changed
- Added `environment/info` to the exec-server protocol/client/server and
exposed `Environment::info()`.
- Added local and remote environment info providers on `Environment`,
following the existing capability-provider pattern used for filesystem
access.
- Moved the shared shell detection logic into `codex-shell-command` and
kept core shell APIs as wrappers around that implementation.
- Returned shell metadata as `EnvironmentInfo { shell: ShellInfo }`
using the existing shell detection path.
- Added a remote environment test that calls `Environment::info()`
through an exec-server-backed environment.
## Validation
- `git diff --check`
- `just test -p codex-shell-command`
- `just test -p codex-core -E 'test(/shell::tests::/)'`\n- `just test -p
codex-exec-server environment`
## Summary
- add executor filesystem canonicalization as a bound-path operation
- route remote canonicalization through the exec-server filesystem RPC
surface
- keep path normalization attached to the filesystem that owns the path
## Stack
- 2/5 in the skills path authority stack extracted from
https://github.com/openai/codex/pull/25098
- follows merged https://github.com/openai/codex/pull/25121
## Validation
- `cd
/Users/starr/code/codex-worktrees/pr-25098-restack-review-pr1b/codex-rs
&& just fmt`
- Not run: tests/checks (not requested)
- GitHub CI pending on rewritten head
## Summary
- Bump the workspace Rust toolchain from `1.93.0` to `1.95.0` across
Cargo, Bazel, CI, release workflows, devcontainers, and the Codex
environment config.
- Refresh `MODULE.bazel.lock` so the Bazel Rust toolchain artifacts
match the new version.
- Leave purpose-specific toolchains unchanged, including the
`argument-comment-lint` nightly and the upstream `rusty_v8` `1.91.0`
build pin.
- Includes fixes for new lints from `just fix` and a few codex-authored
fixes for lints without a suggestion.
## Summary
- replace the one-shot lazy remote exec-server cache with a
lock-protected current client
- when the cached websocket client is already disconnected, create one
fresh websocket client/session on the next `get()`
- keep existing disconnect failure behavior for old process sessions and
HTTP body streams; do not add session resume or request retry
## Why
The prior PR direction was trying to grow into session restore: resume
the old `session_id`, preserve existing process handles, and add
reconnect retry policy. That is more machinery than we want for this
slice.
For now, the useful minimum is simpler: later fresh remote operations
should not be stuck behind a dead cached websocket client, but anything
already attached to the dead connection should fail loudly through the
existing disconnect path. The server already has detached-session
cleanup via its existing TTL, so this PR does not need to add
client-side session preservation.
## What Changed
- `LazyRemoteExecServerClient::get()` now keeps the current concrete
client in a small mutex-protected cache plus one async connect lock.
- If that cached client is still connected, `get()` returns it.
- If that cached websocket client has observed the transport close,
`get()` creates a brand-new websocket client with a brand-new
exec-server session and replaces the cache.
- If that cached client is stdio-backed, behavior stays one-shot: the
dead client is returned and later work surfaces the existing disconnect
error.
- No `resume_session_id`, backoff, request replay, or existing
`RemoteExecProcess` rebinding is added here.
- Added focused websocket coverage that proves two concurrent `get()`
calls after disconnect share one fresh replacement client/session.
## Summary
- migrate exec-server remote registration naming from executor to
environment
- align CLI, public Rust exports, registry error messages, and relay
test fixtures with the environment registry contract
- keep the live registration path and response model consistent with
`/cloud/environment/{environment_id}/register`
## Verification
- `cargo test -p codex-exec-server
remote::tests::register_environment_posts_with_auth_provider_headers
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_environment_routes_independent_virtual_streams
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo check -p codex-cli --manifest-path
/Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
opened; will update after completion if needed)
## Why
The environment-backed exec-server transport currently hardcodes 5
second connect and initialize timeouts in `client_transport.rs`. That is
short for SSH-backed stdio environments and remote websocket
environments, and there is currently no way to raise those values from
`CODEX_HOME/environments.toml`.
This stacked follow-up raises the default environment transport timeouts
and lets each configured environment override them in
`environments.toml`.
## What Changed
- raise the default environment transport connect and initialize
timeouts from 5s to 10s
- store concrete timeout values on `ExecServerTransportParams` instead
of hardcoding them in `connect_for_transport(...)`
- add `connect_timeout_sec` and `initialize_timeout_sec` to
`[[environments]]` entries in `environments.toml`
- apply parse-time defaults so runtime transport code receives fully
resolved timeout values
- reject `connect_timeout_sec` on stdio environments because it only
applies to websocket transports
- extend parser tests to cover the new fields and defaults
## Stack
- base: https://github.com/openai/codex/pull/21794
- this PR: configurable environment transport timeouts
## Validation
- `cd
/Users/starr/code/codex-worktrees/exec-env-timeouts-config-20260508/codex-rs
&& just fmt`
- not run: tests
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
Configured environments need to connect to exec-server instances that
are not necessarily already listening on a websocket URL. A
command-backed stdio transport lets Codex start an exec-server process,
speak JSON-RPC over its stdio streams, and clean up that child process
with the client lifetime.
**Stack position:** this is PR 2 of 5. It builds on the server-side
stdio listener from PR 1 and provides the client transport used by later
environment/config PRs.
## What Changed
- Add `ExecServerTransport` variants for websocket URLs and stdio shell
commands.
- Add stdio command connection support for `ExecServerClient`.
- Move websocket/stdio transport setup into `client_transport.rs` so
`client.rs` stays focused on shared JSON-RPC client, session, HTTP, and
notification behavior.
- Tie stdio child process cleanup to the JSON-RPC connection lifetime
with a RAII lifetime guard.
- Keep existing websocket environment behavior by adapting URL-backed
remotes to `ExecServerTransport::WebSocketUrl`.
## Stack
- 1. https://github.com/openai/codex/pull/20663 - Add stdio exec-server
listener
- **2. This PR:** https://github.com/openai/codex/pull/20664 - Add stdio
exec-server client transport
- 3. https://github.com/openai/codex/pull/20665 - Make environment
providers own default selection
- 4. https://github.com/openai/codex/pull/20666 - Add CODEX_HOME
environments TOML provider
- 5. https://github.com/openai/codex/pull/20667 - Load configured
environments from CODEX_HOME
Split from original draft: https://github.com/openai/codex/pull/20508
## Validation
Not run locally; this was split out of the original draft stack and then
refactored to separate transport setup from the base client.
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
This PR adds the first `codex-rs` milestone for remote-exec e2e: a local
`codex exec-server` can now register itself with
`codex-cloud-environments` and attach to the returned rendezvous
websocket.
At a high level, `codex exec-server --cloud ...` now:
- loads ChatGPT auth from normal Codex config
- registers an executor with `codex-cloud-environments`
- receives a signed rendezvous websocket URL
- serves the existing exec-server JSON-RPC protocol over that websocket
## What Changed
- Added `--cloud`, `--cloud-base-url`, `--cloud-environment-id`, and
`--cloud-name` to `codex exec-server`
- Added a new `exec-server/src/cloud.rs` module that handles:
- registration requests
- auth/header setup
- bounded auth retry on `401/403`
- reconnect/backoff after websocket disconnects
- Reused the existing `ConnectionProcessor` / `ExecServerHandler` path
so cloud mode serves the same exec/filesystem RPC surface as local
websocket mode
- Added cloud-specific error variants and minimal docs for the new mode
## Testing
Manual e2e test that fully goes through exec server flow with our codex
cloud agent as orchestrator
### Why
The RMCP layer needs a Streamable HTTP client that can talk either
directly over `reqwest` or through the executor HTTP runner without
duplicating MCP session logic higher in the stack. This PR adds that
client-side transport boundary so remote Streamable HTTP MCP can reuse
the same RMCP flow as the local path.
### What
- Add a shared `rmcp-client/src/streamable_http/` module with:
- `transport_client.rs` for the local-or-remote transport enum
- `local_client.rs` for the direct `reqwest` implementation
- `remote_client.rs` for the executor-backed implementation
- `common.rs` for the small shared Streamable HTTP helpers
- Teach `RmcpClient` to build Streamable HTTP transports in either local
or remote mode while keeping the existing OAuth ownership in RMCP.
- Translate remote POST, GET, and DELETE session operations into
executor `http/request` calls.
- Preserve RMCP session expiry handling and reconnect behavior for the
remote transport.
- Add remote transport coverage in
`rmcp-client/tests/streamable_http_remote.rs` and keep the shared test
support in `rmcp-client/tests/streamable_http_test_support.rs`.
### Verification
- `cargo check -p codex-rmcp-client`
- online CI
### Stack
1. #18581 protocol
2. #18582 runner
3. #18583 RMCP client
4. #18584 manager wiring and local/remote coverage
---------
Co-authored-by: Codex <noreply@openai.com>
### Why
Remote streamable HTTP MCP needs a transport-shaped executor primitive
before the MCP client can move network I/O to the executor. This layer
keeps the executor unaware of MCP and gives later PRs an ordered
streaming surface for response bodies.
### What
- Add typed `http/request` and `http/request/bodyDelta` protocol
payloads.
- Add executor client helpers for buffered and streamed HTTP responses.
- Route body-delta notifications to request-scoped streams with sequence
validation and cleanup when a stream finishes or is dropped.
- Document the new protocol constants, transport structs, public client
methods, body-stream lifecycle, and request-scoped routing helpers.
- Add in-memory JSON-RPC client coverage for streamed HTTP response-body
notifications, with comments spelling out what the test proves and each
setup/exercise/assert phase.
### Stack
1. #18581 protocol
2. #18582 runner
3. #18583 RMCP client
4. #18584 manager wiring and local/remote coverage
### Verification
- `just fmt`
- `cargo check -p codex-exec-server -p codex-rmcp-client --tests`
- `cargo check -p codex-core --test all` compile-only
- `git diff --check`
- Online full CI is running from the `full-ci` branch, including the
remote Rust test job.
Co-authored-by: Codex <noreply@openai.com>
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- Reject new exec-server client operations once the transport has
disconnected.
- Convert pending RPC calls into closed errors instead of synthetic
server errors.
- Cover pending read and later write behavior after remote executor
disconnect.
## Verification
- `just fmt`
- `cargo check -p codex-exec-server`
## Stack
```text
@ #18027 [6/6] Fail exec client operations after disconnect
│
o #18212 [5/6] Wire executor-backed MCP stdio
│
o #18087 [4/6] Abstract MCP stdio server launching
│
o #18020 [3/6] Add pushed exec process events
│
o #18086 [2/6] Support piped stdin in exec process API
│
o #18085 [1/6] Add MCP server environment config
│
o main
```
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- Add a pushed `ExecProcessEvent` stream alongside retained
`process/read` output.
- Publish local and remote output, exit, close, and failure events.
- Cover the event stream with shared local/remote exec process tests.
## Testing
- `cargo check -p codex-exec-server`
- `cargo check -p codex-rmcp-client`
- Not run: `cargo test` per repo instruction; CI will cover.
## Stack
```text
o #18027 [6/6] Fail exec client operations after disconnect
│
o #18212 [5/6] Wire executor-backed MCP stdio
│
o #18087 [4/6] Abstract MCP stdio server launching
│
@ #18020 [3/6] Add pushed exec process events
│
o #18086 [2/6] Support piped stdin in exec process API
│
o #18085 [1/6] Add MCP server environment config
│
o main
```
---------
Co-authored-by: Codex <noreply@openai.com>
This introduces session-scoped ownership for exec-server so ws
disconnects no longer immediately kill running remote exec processes,
and it prepares the protocol for reconnect-based resume.
- add session_id / resume_session_id to the exec-server initialize
handshake
- move process ownership under a shared session registry
- detach sessions on websocket disconnect and expire them after a TTL
instead of killing processes immediately (we will resume based on this)
- allow a new connection to resume an existing session and take over
notifications/ownership
- I use UUID to make them not predictable as we don't have auth for now
- make detached-session expiry authoritative at resume time so teardown
wins at the TTL boundary
- reject long-poll process/read calls that get resumed out from under an
older attachment
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- add optional `sandboxPolicy` support to the app-server filesystem
request surface
- thread sandbox-aware filesystem options through app-server and
exec-server adapters
- enforce sandboxed read/write access in the filesystem abstraction with
focused local and remote coverage
## Validation
- `cargo test -p codex-app-server-protocol`
- `cargo test -p codex-exec-server file_system`
- `cargo test -p codex-app-server suite::v2::fs`
---------
Co-authored-by: Codex <noreply@openai.com>
This PR partially rebase `unified_exec` on the `exec-server` and adapt
the `exec-server` accordingly.
## What changed in `exec-server`
1. Replaced the old "broadcast-driven; process-global" event model with
process-scoped session events. The goal is to be able to have dedicated
handler for each process.
2. Add to protocol contract to support explicit lifecycle status and
stream ordering:
- `WriteResponse` now returns `WriteStatus` (Accepted, UnknownProcess,
StdinClosed, Starting) instead of a bool.
- Added seq fields to output/exited notifications.
- Added terminal process/closed notification.
3. Demultiplexed remote notifications into per-process channels. Same as
for the event sys
4. Local and remote backends now both implement ExecBackend.
5. Local backend wraps internal process ID/operations into per-process
ExecProcess objects.
6. Remote backend registers a session channel before launch and
unregisters on failed launch.
## What changed in `unified_exec`
1. Added unified process-state model and backend-neutral process
wrapper. This will probably disappear in the future, but it makes it
easier to keep the work flowing on both side.
- `UnifiedExecProcess` now handles both local PTY sessions and remote
exec-server processes through a shared `ProcessHandle`.
- Added `ProcessState` to track has_exited, exit_code, and terminal
failure message consistently across backends.
2. Routed write and lifecycle handling through process-level methods.
## Some rationals
1. The change centralizes execution transport in exec-server while
preserving policy and orchestration ownership in core, avoiding
duplicated launch approval logic. This comes from internal discussion.
2. Session-scoped events remove coupling/cross-talk between processes
and make stream ordering and terminal state explicit (seq, closed,
failed).
3. The failure-path surfacing (remote launch failures, write failures,
transport disconnects) makes command tool output and cleanup behavior
deterministic
## Follow-ups:
* Unify the concept of thread ID behind an obfuscated struct
* FD handling
* Full zsh-fork compatibility
* Full network sandboxing compatibility
* Handle ws disconnection
