Supersedes #28288 (closed).
## Why
A short WebSocket interruption currently ends every client-side process
handle, even though exec-server keeps the server session and its
processes alive for a short time.
This is especially visible for executor-backed stdio MCP servers: a
temporary connection loss becomes a permanent `Transport closed` error.
The server already has the information needed to resume the session, but
the client opens a fresh session instead of using it.
This change reconnects below the process and MCP layers. Existing
process handles stay valid, missed output is recovered, and the same
server-side processes continue running.
## State machine
One logical `ExecServerClient` stays alive while its underlying RPC
connection changes generations.
```text
transport closes
+------------------------------------------------+
| v
+-------------+ +-------------+
| Connected | | Recovering |
+-------------+ +-------------+
^ |
| session resumed, processes caught up | retryable error
+------------------------------------------------+ loops until deadline
|
| deadline or permanent error
v
+-------------+
| Failed |
+-------------+
```
### `Connected`
- New RPC calls use the current connection.
- Process notifications are published in sequence order.
- A disconnect only starts recovery if it came from the current
connection generation. Late events from older generations cannot replace
the active connection.
### `Recovering`
- New calls wait instead of choosing a half-connected RPC client.
- Existing process handles, wake subscriptions, and event subscriptions
stay open.
- Streaming HTTP response bodies fail immediately because their byte
streams cannot be resumed safely.
- Recovery first waits for process starts that were already in flight. A
start whose result became ambiguous is cleaned up after reconnection
instead of being silently adopted.
- The client reconnects with the learned `session_id`. The server may
briefly report that the old connection is still attached, so that error
is retried until the detach finishes.
- The notification consumer starts before the resume handshake
completes. This prevents a busy process from filling the notification
queue and blocking the initialize response.
- Before installing the new connection, the client catches up every
recoverable process with `process/read`.
### `Failed`
- Recovery stops after 25 seconds or after a permanent error.
- Waiting calls are released with one stable disconnect error.
- Existing process sessions receive a terminal failure instead of
waiting forever.
## Recovering process events
Output, exit, and close events share one sequence. During normal
operation, the client buffers early events until every lower sequence
has been published.
After reconnection, the client reads each process starting after its
last published sequence:
1. Retained output chunks are inserted by sequence number.
2. Exit and close state are reconstructed in their sequence positions.
3. Events already received as live notifications are ignored as
duplicates.
4. Newly contiguous events are published in order.
5. If the server no longer retains enough output to fill a sequence gap,
only that process is terminated and failed. The recovered connection
remains usable for other processes.
The server reports its full next event sequence for unbounded reads,
including exit and close events. Closed processes remain readable for
the same 30-second window used to retain detached sessions.
## Other details
- Detached server sessions are retained for 30 seconds, leaving margin
around the client's 25-second recovery deadline.
- Session attach and detach update the active notification sender under
the same attachment lock, so an old connection cannot clear a newly
attached sender.
- A dedicated error code distinguishes the temporary "session is still
attached" race from permanent initialization errors.
- Process starts are identity-checked on both client and server. Cleanup
from an older start cannot remove a newer process that reused the same
ID.
- Mutating requests that were already in flight when the transport
closed are not replayed, because the client cannot know whether the
server applied them. Requests started after recovery is known wait for
the replacement connection.
- We assume the server/client version stays in sync (on the before/after
this PR)
## User impact
Long-running commands and stdio MCP servers can survive a temporary
exec-server WebSocket interruption without changing process IDs or
losing output produced during the outage.
## Why
`fs/readFile` buffers the entire file in one response, which makes large
remote reads expensive and prevents callers from applying backpressure.
We need an opt-in streaming path with bounded block sizes while
preserving the existing single-call API for small and sandboxed reads.
## What changed
- Add `ExecServerClient::stream`, returning a named `FileReadStream`
that implements `futures::Stream` and yields immutable 1 MiB byte
blocks.
- Add internal `fs/open`, `fs/readBlock`, and `fs/close` RPCs.
`fs/readBlock` accepts an explicit offset and length.
- Keep unsandboxed files open between block reads, cap open handles per
connection, and clean them up on EOF, error, stream drop, explicit
close, or connection shutdown.
- Reject platform-sandboxed streaming opens instead of turning the
one-shot sandbox helper into a persistent server. Existing `fs/readFile`
behavior is unchanged.
## Testing
- `just test -p codex-exec-server`
- Integration coverage for 1 MiB chunking, exact block-boundary EOF,
sandbox rejection, and continued reads from the opened file after path
replacement.
- Handle-manager coverage for non-sequential offsets, variable block
lengths, the 128-handle limit, and capacity release after close.
## Why
Rendezvous forwards traffic between the orchestrator and exec-server.
The endpoints need to authenticate each other and encrypt that traffic
without trusting Rendezvous with plaintext or endpoint keys.
## Changes
- Adds a hybrid Noise IK channel through Clatter using X25519,
ML-KEM-768, AES-256-GCM, and SHA-256.
- Binds each handshake to `environment_id`, `executor_registration_id`,
and `stream_id`.
- Pins the registry-provided executor key and carries the harness
authorization inside the encrypted handshake.
- Orders relay frames before consuming Noise nonces and fragments large
JSON-RPC messages into bounded records.
- Bounds handshake payloads, frames, streams, and message reassembly.
Runtime activation is in
[openai/codex#26245](https://github.com/openai/codex/pull/26245).
## Stack
1. **[openai/codex#26242](https://github.com/openai/codex/pull/26242)**:
Noise channel and relay transport
2. [openai/codex#26245](https://github.com/openai/codex/pull/26245):
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- Oversized initiator payload regression coverage
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
Path composition is already handled by `PathUri`, leaving `fs/join` and
`fs/parent` as redundant exec-server protocol surface. Because
app-server and exec-server are deployed atomically, these obsolete
methods can be removed without a compatibility shim.
This removes the protocol constants and payloads, public client APIs,
server registrations and handlers, and endpoint-only tests. Existing
in-process `PathUri` join/parent coverage remains.
## Validation
- `just test -p codex-exec-server` (215 passed, 2 skipped)
## Why
A long-running unified exec process started with `tty: false` could not
be interrupted via `write_stdin`: ordinary non-TTY stdin writes are
rejected once stdin is closed, but an exact U+0003 payload should still
map to a process interrupt. The interrupt should flow through the same
process lifecycle path as a real signal so Codex preserves
process-reported output and exit metadata instead of fabricating a
Ctrl-C exit code or tearing down the session early.
## What Changed
- Add `process/signal` to exec-server with `ProcessSignal::Interrupt`
and an empty response.
- Add a non-consuming `ProcessHandle::signal` path for spawned
processes; on Unix it sends SIGINT to the process group and leaves
terminate/hard-kill unchanged.
- Route non-TTY U+0003 `write_stdin` through `process.signal(...)`
instead of `terminate`, then let the normal post-write collection path
drain output and observe exit.
- Add exec-server coverage where a shell `trap INT` handler prints the
signal and exits with its own code.
- Add unified exec coverage where a `tty: false` process traps SIGINT,
emits output, and exits with its own code.
## Validation
- `just test -p codex-exec-server
exec_process_signal_interrupts_process`
- `just test -p codex-exec-server`
- `just test -p codex-core
write_stdin_ctrl_c_interrupts_non_tty_session`
## Why
Shell detection needs to be available through the `Environment`
abstraction so callers can ask the selected local or remote environment
for shell metadata without adding a separate HTTP endpoint or parallel
info-source path. This keeps shell metadata shaped like the existing
environment-owned filesystem capability and lets remote environments
answer through exec-server JSON-RPC.
## What changed
- Added `environment/info` to the exec-server protocol/client/server and
exposed `Environment::info()`.
- Added local and remote environment info providers on `Environment`,
following the existing capability-provider pattern used for filesystem
access.
- Moved the shared shell detection logic into `codex-shell-command` and
kept core shell APIs as wrappers around that implementation.
- Returned shell metadata as `EnvironmentInfo { shell: ShellInfo }`
using the existing shell detection path.
- Added a remote environment test that calls `Environment::info()`
through an exec-server-backed environment.
## Validation
- `git diff --check`
- `just test -p codex-shell-command`
- `just test -p codex-core -E 'test(/shell::tests::/)'`\n- `just test -p
codex-exec-server environment`
## Summary
- add executor filesystem canonicalization as a bound-path operation
- route remote canonicalization through the exec-server filesystem RPC
surface
- keep path normalization attached to the filesystem that owns the path
## Stack
- 2/5 in the skills path authority stack extracted from
https://github.com/openai/codex/pull/25098
- follows merged https://github.com/openai/codex/pull/25121
## Validation
- `cd
/Users/starr/code/codex-worktrees/pr-25098-restack-review-pr1b/codex-rs
&& just fmt`
- Not run: tests/checks (not requested)
- GitHub CI pending on rewritten head
## Summary
- Bump the workspace Rust toolchain from `1.93.0` to `1.95.0` across
Cargo, Bazel, CI, release workflows, devcontainers, and the Codex
environment config.
- Refresh `MODULE.bazel.lock` so the Bazel Rust toolchain artifacts
match the new version.
- Leave purpose-specific toolchains unchanged, including the
`argument-comment-lint` nightly and the upstream `rusty_v8` `1.91.0`
build pin.
- Includes fixes for new lints from `just fix` and a few codex-authored
fixes for lints without a suggestion.
## Summary
- replace the one-shot lazy remote exec-server cache with a
lock-protected current client
- when the cached websocket client is already disconnected, create one
fresh websocket client/session on the next `get()`
- keep existing disconnect failure behavior for old process sessions and
HTTP body streams; do not add session resume or request retry
## Why
The prior PR direction was trying to grow into session restore: resume
the old `session_id`, preserve existing process handles, and add
reconnect retry policy. That is more machinery than we want for this
slice.
For now, the useful minimum is simpler: later fresh remote operations
should not be stuck behind a dead cached websocket client, but anything
already attached to the dead connection should fail loudly through the
existing disconnect path. The server already has detached-session
cleanup via its existing TTL, so this PR does not need to add
client-side session preservation.
## What Changed
- `LazyRemoteExecServerClient::get()` now keeps the current concrete
client in a small mutex-protected cache plus one async connect lock.
- If that cached client is still connected, `get()` returns it.
- If that cached websocket client has observed the transport close,
`get()` creates a brand-new websocket client with a brand-new
exec-server session and replaces the cache.
- If that cached client is stdio-backed, behavior stays one-shot: the
dead client is returned and later work surfaces the existing disconnect
error.
- No `resume_session_id`, backoff, request replay, or existing
`RemoteExecProcess` rebinding is added here.
- Added focused websocket coverage that proves two concurrent `get()`
calls after disconnect share one fresh replacement client/session.
## Summary
- migrate exec-server remote registration naming from executor to
environment
- align CLI, public Rust exports, registry error messages, and relay
test fixtures with the environment registry contract
- keep the live registration path and response model consistent with
`/cloud/environment/{environment_id}/register`
## Verification
- `cargo test -p codex-exec-server
remote::tests::register_environment_posts_with_auth_provider_headers
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_environment_routes_independent_virtual_streams
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo check -p codex-cli --manifest-path
/Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
opened; will update after completion if needed)
## Why
The environment-backed exec-server transport currently hardcodes 5
second connect and initialize timeouts in `client_transport.rs`. That is
short for SSH-backed stdio environments and remote websocket
environments, and there is currently no way to raise those values from
`CODEX_HOME/environments.toml`.
This stacked follow-up raises the default environment transport timeouts
and lets each configured environment override them in
`environments.toml`.
## What Changed
- raise the default environment transport connect and initialize
timeouts from 5s to 10s
- store concrete timeout values on `ExecServerTransportParams` instead
of hardcoding them in `connect_for_transport(...)`
- add `connect_timeout_sec` and `initialize_timeout_sec` to
`[[environments]]` entries in `environments.toml`
- apply parse-time defaults so runtime transport code receives fully
resolved timeout values
- reject `connect_timeout_sec` on stdio environments because it only
applies to websocket transports
- extend parser tests to cover the new fields and defaults
## Stack
- base: https://github.com/openai/codex/pull/21794
- this PR: configurable environment transport timeouts
## Validation
- `cd
/Users/starr/code/codex-worktrees/exec-env-timeouts-config-20260508/codex-rs
&& just fmt`
- not run: tests
---------
Co-authored-by: Codex <noreply@openai.com>
## Why
Configured environments need to connect to exec-server instances that
are not necessarily already listening on a websocket URL. A
command-backed stdio transport lets Codex start an exec-server process,
speak JSON-RPC over its stdio streams, and clean up that child process
with the client lifetime.
**Stack position:** this is PR 2 of 5. It builds on the server-side
stdio listener from PR 1 and provides the client transport used by later
environment/config PRs.
## What Changed
- Add `ExecServerTransport` variants for websocket URLs and stdio shell
commands.
- Add stdio command connection support for `ExecServerClient`.
- Move websocket/stdio transport setup into `client_transport.rs` so
`client.rs` stays focused on shared JSON-RPC client, session, HTTP, and
notification behavior.
- Tie stdio child process cleanup to the JSON-RPC connection lifetime
with a RAII lifetime guard.
- Keep existing websocket environment behavior by adapting URL-backed
remotes to `ExecServerTransport::WebSocketUrl`.
## Stack
- 1. https://github.com/openai/codex/pull/20663 - Add stdio exec-server
listener
- **2. This PR:** https://github.com/openai/codex/pull/20664 - Add stdio
exec-server client transport
- 3. https://github.com/openai/codex/pull/20665 - Make environment
providers own default selection
- 4. https://github.com/openai/codex/pull/20666 - Add CODEX_HOME
environments TOML provider
- 5. https://github.com/openai/codex/pull/20667 - Load configured
environments from CODEX_HOME
Split from original draft: https://github.com/openai/codex/pull/20508
## Validation
Not run locally; this was split out of the original draft stack and then
refactored to separate transport setup from the base client.
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
This PR adds the first `codex-rs` milestone for remote-exec e2e: a local
`codex exec-server` can now register itself with
`codex-cloud-environments` and attach to the returned rendezvous
websocket.
At a high level, `codex exec-server --cloud ...` now:
- loads ChatGPT auth from normal Codex config
- registers an executor with `codex-cloud-environments`
- receives a signed rendezvous websocket URL
- serves the existing exec-server JSON-RPC protocol over that websocket
## What Changed
- Added `--cloud`, `--cloud-base-url`, `--cloud-environment-id`, and
`--cloud-name` to `codex exec-server`
- Added a new `exec-server/src/cloud.rs` module that handles:
- registration requests
- auth/header setup
- bounded auth retry on `401/403`
- reconnect/backoff after websocket disconnects
- Reused the existing `ConnectionProcessor` / `ExecServerHandler` path
so cloud mode serves the same exec/filesystem RPC surface as local
websocket mode
- Added cloud-specific error variants and minimal docs for the new mode
## Testing
Manual e2e test that fully goes through exec server flow with our codex
cloud agent as orchestrator
### Why
The RMCP layer needs a Streamable HTTP client that can talk either
directly over `reqwest` or through the executor HTTP runner without
duplicating MCP session logic higher in the stack. This PR adds that
client-side transport boundary so remote Streamable HTTP MCP can reuse
the same RMCP flow as the local path.
### What
- Add a shared `rmcp-client/src/streamable_http/` module with:
- `transport_client.rs` for the local-or-remote transport enum
- `local_client.rs` for the direct `reqwest` implementation
- `remote_client.rs` for the executor-backed implementation
- `common.rs` for the small shared Streamable HTTP helpers
- Teach `RmcpClient` to build Streamable HTTP transports in either local
or remote mode while keeping the existing OAuth ownership in RMCP.
- Translate remote POST, GET, and DELETE session operations into
executor `http/request` calls.
- Preserve RMCP session expiry handling and reconnect behavior for the
remote transport.
- Add remote transport coverage in
`rmcp-client/tests/streamable_http_remote.rs` and keep the shared test
support in `rmcp-client/tests/streamable_http_test_support.rs`.
### Verification
- `cargo check -p codex-rmcp-client`
- online CI
### Stack
1. #18581 protocol
2. #18582 runner
3. #18583 RMCP client
4. #18584 manager wiring and local/remote coverage
---------
Co-authored-by: Codex <noreply@openai.com>
### Why
Remote streamable HTTP MCP needs a transport-shaped executor primitive
before the MCP client can move network I/O to the executor. This layer
keeps the executor unaware of MCP and gives later PRs an ordered
streaming surface for response bodies.
### What
- Add typed `http/request` and `http/request/bodyDelta` protocol
payloads.
- Add executor client helpers for buffered and streamed HTTP responses.
- Route body-delta notifications to request-scoped streams with sequence
validation and cleanup when a stream finishes or is dropped.
- Document the new protocol constants, transport structs, public client
methods, body-stream lifecycle, and request-scoped routing helpers.
- Add in-memory JSON-RPC client coverage for streamed HTTP response-body
notifications, with comments spelling out what the test proves and each
setup/exercise/assert phase.
### Stack
1. #18581 protocol
2. #18582 runner
3. #18583 RMCP client
4. #18584 manager wiring and local/remote coverage
### Verification
- `just fmt`
- `cargo check -p codex-exec-server -p codex-rmcp-client --tests`
- `cargo check -p codex-core --test all` compile-only
- `git diff --check`
- Online full CI is running from the `full-ci` branch, including the
remote Rust test job.
Co-authored-by: Codex <noreply@openai.com>
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- Reject new exec-server client operations once the transport has
disconnected.
- Convert pending RPC calls into closed errors instead of synthetic
server errors.
- Cover pending read and later write behavior after remote executor
disconnect.
## Verification
- `just fmt`
- `cargo check -p codex-exec-server`
## Stack
```text
@ #18027 [6/6] Fail exec client operations after disconnect
│
o #18212 [5/6] Wire executor-backed MCP stdio
│
o #18087 [4/6] Abstract MCP stdio server launching
│
o #18020 [3/6] Add pushed exec process events
│
o #18086 [2/6] Support piped stdin in exec process API
│
o #18085 [1/6] Add MCP server environment config
│
o main
```
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- Add a pushed `ExecProcessEvent` stream alongside retained
`process/read` output.
- Publish local and remote output, exit, close, and failure events.
- Cover the event stream with shared local/remote exec process tests.
## Testing
- `cargo check -p codex-exec-server`
- `cargo check -p codex-rmcp-client`
- Not run: `cargo test` per repo instruction; CI will cover.
## Stack
```text
o #18027 [6/6] Fail exec client operations after disconnect
│
o #18212 [5/6] Wire executor-backed MCP stdio
│
o #18087 [4/6] Abstract MCP stdio server launching
│
@ #18020 [3/6] Add pushed exec process events
│
o #18086 [2/6] Support piped stdin in exec process API
│
o #18085 [1/6] Add MCP server environment config
│
o main
```
---------
Co-authored-by: Codex <noreply@openai.com>
This introduces session-scoped ownership for exec-server so ws
disconnects no longer immediately kill running remote exec processes,
and it prepares the protocol for reconnect-based resume.
- add session_id / resume_session_id to the exec-server initialize
handshake
- move process ownership under a shared session registry
- detach sessions on websocket disconnect and expire them after a TTL
instead of killing processes immediately (we will resume based on this)
- allow a new connection to resume an existing session and take over
notifications/ownership
- I use UUID to make them not predictable as we don't have auth for now
- make detached-session expiry authoritative at resume time so teardown
wins at the TTL boundary
- reject long-poll process/read calls that get resumed out from under an
older attachment
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- add optional `sandboxPolicy` support to the app-server filesystem
request surface
- thread sandbox-aware filesystem options through app-server and
exec-server adapters
- enforce sandboxed read/write access in the filesystem abstraction with
focused local and remote coverage
## Validation
- `cargo test -p codex-app-server-protocol`
- `cargo test -p codex-exec-server file_system`
- `cargo test -p codex-app-server suite::v2::fs`
---------
Co-authored-by: Codex <noreply@openai.com>
This PR partially rebase `unified_exec` on the `exec-server` and adapt
the `exec-server` accordingly.
## What changed in `exec-server`
1. Replaced the old "broadcast-driven; process-global" event model with
process-scoped session events. The goal is to be able to have dedicated
handler for each process.
2. Add to protocol contract to support explicit lifecycle status and
stream ordering:
- `WriteResponse` now returns `WriteStatus` (Accepted, UnknownProcess,
StdinClosed, Starting) instead of a bool.
- Added seq fields to output/exited notifications.
- Added terminal process/closed notification.
3. Demultiplexed remote notifications into per-process channels. Same as
for the event sys
4. Local and remote backends now both implement ExecBackend.
5. Local backend wraps internal process ID/operations into per-process
ExecProcess objects.
6. Remote backend registers a session channel before launch and
unregisters on failed launch.
## What changed in `unified_exec`
1. Added unified process-state model and backend-neutral process
wrapper. This will probably disappear in the future, but it makes it
easier to keep the work flowing on both side.
- `UnifiedExecProcess` now handles both local PTY sessions and remote
exec-server processes through a shared `ProcessHandle`.
- Added `ProcessState` to track has_exited, exit_code, and terminal
failure message consistently across backends.
2. Routed write and lifecycle handling through process-level methods.
## Some rationals
1. The change centralizes execution transport in exec-server while
preserving policy and orchestration ownership in core, avoiding
duplicated launch approval logic. This comes from internal discussion.
2. Session-scoped events remove coupling/cross-talk between processes
and make stream ordering and terminal state explicit (seq, closed,
failed).
3. The failure-path surfacing (remote launch failures, write failures,
transport disconnects) makes command tool output and cleanup behavior
deterministic
## Follow-ups:
* Unify the concept of thread ID behind an obfuscated struct
* FD handling
* Full zsh-fork compatibility
* Full network sandboxing compatibility
* Handle ws disconnection
## Summary
- match the exec-process structure to filesystem PR #15232
- expose `ExecProcess` on `Environment`
- make `LocalProcess` the real implementation and `RemoteProcess` a thin
network proxy over `ExecServerClient`
- make `ProcessHandler` a thin RPC adapter delegating to `LocalProcess`
- add a shared local/remote process test
## Validation
- `just fmt`
- `CARGO_TARGET_DIR=~/.cache/cargo-target/codex cargo test -p
codex-exec-server`
- `just fix -p codex-exec-server`
---------
Co-authored-by: Codex <noreply@openai.com>
Stacked PR 2/3, based on the stub PR.
Adds the exec RPC implementation and process/event flow in exec-server
only.
---------
Co-authored-by: Codex <noreply@openai.com>
Summary
- delete the deprecated stdio transport plumbing from the exec server
stack
- add a basic `exec_server()` harness plus test utilities to start a
server, send requests, and await events
- refresh exec-server dependencies, configs, and documentation to
reflect the new flow
Testing
- Not run (not requested)
---------
Co-authored-by: starr-openai <starr@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Stacked PR 1/3.
This is the initialize-only exec-server stub slice: binary/client
scaffolding and protocol docs, without exec/filesystem implementation.
---------
Co-authored-by: Codex <noreply@openai.com>
