## Why
Remote environment registration can return an explicit `503` `registration_conflict` after the registry finishes its own recovery attempts. Retrying ambiguous failures is unsafe because a timed-out request may still have replaced a newer registration.
## What changed
- Retry only explicit `registration_conflict` responses with jittered backoff, both during initial registration and after a rejected relay connection.
- Apply the existing connection timeout to the complete registration attempt, including response-body diagnostics.
- Keep retry waits and in-flight requests cancellable while preserving the Noise identity and initialized session across successful recovery.
## Testing
Added unit coverage for retry eligibility and timeout handling, plus relay tests for identity and session preservation and prompt shutdown during registration.
GitOrigin-RevId: 374e6fdac4755f377cbea9400c76a228b790945a
## What changed
- Retry transient environment-registry failures while opening the initial Noise rendezvous connection, including timeouts, interrupted response bodies, retryable HTTP statuses, and temporarily offline environments.
- Bound registry attempts with per-request and overall timeouts, exponential backoff, and a retry limit while returning permanent registry errors immediately.
- Preserve the existing one-time bundle refresh after an unauthorized WebSocket handshake, with a fresh registry retry window.
- Treat malformed complete JSON responses as terminal rather than retryable transport failures.
## Testing
- Cover transient and permanent registry responses, stalled requests, retry limits and deadlines, interrupted bodies, malformed JSON, unauthorized bundle refresh, and session-resume behavior.
GitOrigin-RevId: 5929274103702ad477e7a638ab489319a43a93fa
## What changed
- Add `codex exec-server forward --connect ws://HOST:PORT` to register an
existing WebSocket exec-server as a remote environment.
- Open a separate destination WebSocket for each authenticated Noise relay
stream and forward complete payloads unchanged in both directions.
- Preserve large messages through WebSocket fragmentation and coordinate
disconnects, close acknowledgements, and pending output delivery.
## Testing
- Cover CLI validation, opaque payload forwarding, disconnect handling, and
destination close behavior.
- Verify commands and large file transfers through the complete relay and
forwarding path.
GitOrigin-RevId: 6e2dccff05b9e3571865611f4d432b0e2b4cdbac
## What changed
- Add `x-cluster-name` from `OPENAI_CLUSTER` and
`x-openai-internal-caller` from `DD_SERVICE` to rendezvous WebSocket
handshakes when the environment values are nonempty and valid HTTP header
values.
- Apply the rendezvous headers to both client and remote executor connections
while preserving the existing W3C trace context headers.
GitOrigin-RevId: fead0244ba24fa208990b92d2170a9ae76ff2ceb
## Why
Remote exec-server registry requests need to refresh managed credentials before sending a request. Static auth-header resolution cannot perform the asynchronous token exchange required by workload identity.
## What changed
- Add asynchronous auth-header resolution to `AuthProvider`, with the existing static-header behavior as the default.
- Resolve fresh managed credentials for each remote environment registry request while preserving the expected account and workspace identity.
- Load the cloud configuration bundle during remote exec-server startup when workload identity is selected.
## Testing
- Update the managed-auth and environment-registry auth tests to exercise asynchronous header resolution.
GitOrigin-RevId: 5d60f1127467aaacdb5d1a8f3d92278bc4bf2e29
## Why
Sequential dispatch lets a long-running request block unrelated health checks
and cleanup on the same connection.
## What changed
- Add `--concurrent-requests <COUNT>` for local and remote exec-server
connections, while retaining sequential dispatch when the option is omitted
or set to `1`.
- Preserve handshake ordering before enabling concurrent dispatch.
- Reserve separate capacity for status, signal, terminate, and close requests so
they remain responsive when ordinary request capacity is saturated.
- Drain queued client responses during disconnect and cancel outstanding
request tasks during connection shutdown.
## Testing
- Cover CLI parsing and concurrency-limit validation.
- Verify default sequential behavior, pipelined handshake ordering, concurrent
request progress, control-request responsiveness, and disconnect handling.
GitOrigin-RevId: 48e4b092e318204ed635543f01f9ee0e7df095fc
## What changed
- Add `--exit-on-stdin-close` and the `CODEX_EXEC_SERVER_EXIT_ON_STDIN_CLOSE` environment variable as opt-in controls for remote exec servers.
- Gracefully drain active sessions and processes when the parent closes stdin, then flush telemetry before exiting.
- Remove the parent-lifetime environment variable from child process environments.
## Testing
- Cover parent disconnects after signal-listener failures.
- Exercise remote shutdown end to end, including child termination and final telemetry metrics.
- Verify that explicitly disabling the environment variable preserves local exec-server behavior.
GitOrigin-RevId: 63063bc097b54684c370bd545cd32d17c4e55d90
## What changed
- Preserve the current tracing span and subscriber when lazy remote environment
startup runs in a background task.
- Add spans for the remote connection, environment registry, Noise, and
rendezvous WebSocket stages, including relevant environment and registration
identifiers.
- Propagate the current trace context to environment registry connection and
harness-key validation requests.
GitOrigin-RevId: 0044c3eb94d59c6a6b92752d3d372195be4adb81
## What changed
- Rename `ReqwestHttpClient` to `RouteAwareHttpClient` to reflect that delegated HTTP uses Codex's shared route-aware transport.
- Use `codex_http_client` response and error types plus transport-neutral `http` and `url` types, removing the exec server's direct `reqwest` dependency.
## Testing
- Cover fragment stripping and Unicode hostname normalization for delegated HTTP requests.
GitOrigin-RevId: 8b0fc60a76004feb57198bfb4afb1371c9ceb1bd
## Why
Remote environment connections need to honor Codex's effective outbound proxy policy, including when a rendezvous connection reconnects.
## What changed
- Pass the configured `HttpClientFactory` into remote environment transports and use `WebSocketConnector` for exec-server and rendezvous WebSockets.
- Resolve proxy routes asynchronously so these connections can use the configured system proxy.
- Add connector options that preserve Tungstenite's default TLS behavior and enable `TCP_NODELAY` for latency-sensitive rendezvous traffic.
## Testing
- Verify prepared remote environments connect through a configured system proxy.
- Verify initial and reconnected encrypted relay peers use the system proxy.
- Cover default TLS selection and opt-in `TCP_NODELAY` behavior in the WebSocket client.
GitOrigin-RevId: 8a8da2116e37cb3a891269d0c0b037986fecdd3c
## Why
Noise environment registry requests need to follow the exec server's effective
outbound proxy policy without exposing registry URLs or response headers in HTTP
diagnostics.
## What changed
- Build the registry client from the supplied `HttpClientFactory` and use a
route-aware API client with redirects and request logging disabled.
- Defer construction of the Noise connection provider until the outbound HTTP
policy is available.
- Map route-aware request failures into registry errors while retaining timeout
detection across response body reads.
## Testing
Add coverage for system-proxy routing, sensitive registry metadata redaction,
stalled response-body timeouts, and prepared Noise configuration validation.
GitOrigin-RevId: d312dfe037f72732085bf38109af44df76ed0b53
## Why
Delegated HTTP requests need to honor the same outbound proxy policy as the
Codex process that starts the exec server.
## What changed
- Pass the configured `HttpClientFactory` through local and remote exec-server
startup and use route-aware client pools for delegated HTTP and local MCP
requests.
- Preserve per-request timeouts and follow-or-stop redirect behavior while
keeping request URLs and sensitive response headers out of diagnostics.
## Testing
- Cover configured system-proxy routing across the exec-server transport.
- Cover both redirect policies and verify that success and failure logs do not
expose request or response secrets.
GitOrigin-RevId: 4af6aec1d265c4db62dfcb6e1fb076fb31736137
## Summary
- require a Pong within 60 seconds for established Noise Rendezvous
WebSockets on both the harness and executor
- bound steady-state WebSocket writes and harness event delivery so
backpressure cannot mask the deadline
- classify executor disconnects with bounded reasons and feed them into
the existing reconnect metric and structured log
- cover silent peers, responsive peers, continuous non-Pong traffic, and
local application backpressure
## Why
The existing periodic Pings did not track Pongs, so a half-open or
blackholed connection could remain stuck until the operating system's
TCP timeout. This adds the smallest explicit liveness contract without
new spans, RTT histograms, feature flags, or TCP diagnostics.
## Testing
- `just test -p codex-exec-server` on devbox `richard-6` — 300 passed, 2
skipped
- `just fix -p codex-exec-server`
- `just fmt`
- independent correctness, performance/security, and YAGNI reviews — no
findings
## Summary
Disable Nagle unconditionally for both exec-server Rendezvous WebSocket
connections.
- pass `disable_nagle=true` at the executor and harness connection call
sites
- keep the existing signed URL, protocol, and connection flow unchanged
- add no feature flag, rollout schema, path variant, or
experiment-specific telemetry
The companion internal PR enables `TCP_NODELAY` on accepted Rendezvous
sockets: https://github.com/openai/openai/pull/1082463
## Why
Rendezvous carries small, latency-sensitive relay and JSON-RPC frames.
Three staging runs of 30 steady-state `process/read` calls per
configuration measured p50 improving from 139.1 ms to 81.5 ms and p95
from 162.0 ms to 95.8 ms with Nagle disabled.
The expected packet overhead is small at the current connection scale.
We will use existing latency, error, packet, and CPU monitoring and
revert normally if production regresses.
## Rollout and rollback
The client and accepted-socket changes can deploy independently. New
connections receive the setting as each side deploys. Rollback is a
normal code revert; there is no persisted assignment or gate state to
unwind.
## Validation
- `just test -p codex-exec-server --lib`: 164 passed
- `just fix -p codex-exec-server`: passed
- `just fmt`: passed
- independent final review found no actionable issue
## Summary
- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**
## Validation
- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
## Summary
- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.
## Root cause
Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.
Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.
The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
## Summary
- Initialize stderr tracing and the configured OpenTelemetry provider
for local and remote `codex exec-server` startup.
- Instrument the local and remote server entrypoints with a root runtime
span.
- Keep raw Noise environment, registration, and stream identifiers out
of exported spans while preserving them in local debug events.
- Keep telemetry setup in a focused CLI module instead of growing the
top-level command entrypoint.
## Stack
- Previous: none (`#27058` has merged)
- Next: #27466
## Validation
- `just test -p codex-exec-server --lib` (139 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
---------
Co-authored-by: Richard Lee <richardlee@openai.com>
## Why
Codex can run a remote exec server through the Noise relay, but the
normal
environment-manager path could not establish an
environment-registry-backed
harness connection. Signed rendezvous URLs and harness authorizations
are
short-lived, so reconnects must fetch a fresh bundle instead of
retaining
stale connection credentials. A stalled registry request must also fail
within
the regular remote connection deadline, without exposing these
credentials in
debug logs.
Issue: N/A (internal environment-service integration).
## What Changed
- Add environment-manager configuration for a registry-backed Noise
rendezvous
environment.
- Request a fresh bundle from
`/cloud/environment/{environment_id}/connect` for every physical harness
connection, using the existing 10-second remote connection timeout.
- Share the Environment Registry register, connect, and validate wire
payloads
through `codex-exec-server` and `codex-core-api`.
- Redact the signed rendezvous URL and harness authorization from the
public
connect response's `Debug` output.
- Add focused coverage for registry bundle retrieval, stalled requests,
and
credential redaction.
## Why
Services that proxy the exec-server environment registry endpoints need
to deserialize and forward the same Noise registration and harness-key
validation payloads. Those wire models currently live as private,
serialize-only structs in `exec-server`, which forces consumers to
duplicate the contract.
## What changed
- Add owned serde models for registration and harness-key validation
requests and responses.
- Use those models in the existing exec-server registry client.
- Re-export the models from `codex-exec-server` and `codex-core-api`.
- Keep the harness authorization request free of a derived `Debug`
implementation so it is not accidentally logged.
## Testing
- Focused exec-server registration and harness-key validation tests: 2
passed.
- `cargo check -p codex-core-api`
The full `codex-exec-server` suite compiled and ran 254 tests: 222
passed, while 32 existing filesystem sandbox tests could not run under
the nested macOS sandbox (`sandbox_apply: Operation not permitted`).
Co-authored-by: Codex <noreply@openai.com>
## Why
The transport in
[openai/codex#26242](https://github.com/openai/codex/pull/26242) needs
to be used by every remote orchestrator-to-executor connection before
JSON-RPC traffic starts.
## Changes
- Generates one executor Noise identity when remote exec-server starts
and registers its public key.
- Creates a harness identity for each physical remote environment
connection.
- Fetches a fresh registry bundle before connecting and validates the
authenticated harness key before completing the executor handshake.
- Multiplexes encrypted logical streams over the existing executor
WebSocket.
- Adds bounded stream, handshake-failure, and reassembly state.
- Adds safe lifecycle diagnostics without logging keys, authorizations,
plaintext, or ciphertext.
- Covers reconnects, replay rejection, validation failure, framing
limits, and encrypted JSON-RPC tool traffic.
## Stack
1. [openai/codex#26242](https://github.com/openai/codex/pull/26242):
Noise channel and relay transport
2. **[openai/codex#26245](https://github.com/openai/codex/pull/26245)**:
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Overview
Allow remote `codex exec-server` registration to use existing API-key
auth while restricting where those credentials can be sent.
- Accept `CodexAuth::ApiKey` for the normal `--remote` registration
path.
- Restrict API-key remote registration to HTTPS `openai.com` and
`openai.org` hosts and subdomains, with explicit HTTP loopback support
for local development.
- Disable registry registration redirects so credentials cannot be
forwarded to an unvalidated destination.
- Retain `--use-agent-identity-auth` as the explicit Agent Identity
path.
- Document remote registration using `CODEX_API_KEY`.
## Big picture
Callers can now provide an API key directly to `exec-server`
registration without first establishing ChatGPT login state:
```sh
CODEX_API_KEY="$OPENAI_API_KEY" \
codex exec-server \
--remote "https://<host>.openai.org/api" \
--environment-id "$ENVIRONMENT_ID"
```
## Validation
- `cargo fmt --all` (`just fmt` is not installed on this host)
- `cargo test -p codex-cli -p codex-exec-server`
## Summary
- migrate exec-server remote registration naming from executor to
environment
- align CLI, public Rust exports, registry error messages, and relay
test fixtures with the environment registry contract
- keep the live registration path and response model consistent with
`/cloud/environment/{environment_id}/register`
## Verification
- `cargo test -p codex-exec-server
remote::tests::register_environment_posts_with_auth_provider_headers
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_environment_routes_independent_virtual_streams
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo check -p codex-cli --manifest-path
/Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
opened; will update after completion if needed)
This updates remote `exec-server` registration to use normal Codex auth
instead of a registry-issued credential. The registry request is built
from the existing auth-provider path, which preserves the biscuit-only
registry contract introduced in
[openai/openai#924101](https://github.com/openai/openai/pull/924101)
while removing the old remote registry bearer env var and its direct
transport assumptions.
The default remote flow uses persisted ChatGPT auth from the normal
Codex config/storage path. This PR also includes the containerized Agent
Identity path needed by
[openai/openai#924260](https://github.com/openai/openai/pull/924260):
remote `exec-server` accepts `--allow-agent-identity-auth`, permits
Agent Identity auth loaded from `CODEX_ACCESS_TOKEN` only when that flag
is present, and reuses the existing Agent task registration plus derived
`AgentAssertion` header generation. API-key auth remains unsupported,
and Agent Identity stays opt-in.
Validation performed beyond normal presubmit coverage:
- `cargo fmt --all --check`
- `cargo check -p codex-cli`
- `cargo test -p codex-exec-server`
- `cargo test -p codex-cli exec_server_agent_identity_auth_flag_`
- `cargo test -p codex-cli remote_exec_server_auth_mode_`
I also attempted `cargo test -p codex-cli`. The new CLI tests passed
inside that run, but the suite ended on an unrelated local
marketplace-state failure in
`plugin_list_excludes_unconfigured_repo_local_marketplaces`.
## Why
Remote exec-server now needs one executor websocket to serve multiple
harness JSON-RPC sessions. Rendezvous routes by `stream_id`, and the
exec-server side needs to use the same stable relay frame contract
instead of a hand-rolled JSON shape.
The relay protocol also needs to make ownership boundaries clear:
harness and executor endpoints own sequencing, acks, retries, duplicate
suppression, segmentation, and reassembly; rendezvous only routes
frames.
## What Changed
- Add the checked-in `codex.exec_server.relay.v1.RelayMessageFrame`
proto plus generated prost bindings for `codex-exec-server`.
- Encode remote harness/executor relay traffic as binary protobuf
websocket frames while keeping local websocket JSON-RPC unchanged.
- Demux executor-side relay streams into independent
`ConnectionProcessor` sessions keyed by `stream_id`.
- Add a programmatic `RemoteExecutorConfig::with_bearer_token(...)`
constructor for non-CLI callers and integration tests.
- Add an integration test that starts the remote executor against a fake
registry/rendezvous websocket and verifies two virtual streams share one
executor websocket without cross-talk, including per-stream reset
behavior.
- Document the remote relay envelope, sequence ranges, `ack`/`ack_bits`,
and endpoint responsibilities in `exec-server/README.md`.
## Verification
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_executor_routes_independent_virtual_streams --
--exact`
- `cargo test -p codex-exec-server --test relay`
- `cargo test -p codex-exec-server` passed outside the sandbox. The
sandboxed run hit macOS `sandbox-exec: sandbox_apply: Operation not
permitted` in filesystem sandbox tests.
## Why
`codex exec-server` should keep the existing public `ws://IP:PORT` URL
shape while serving that websocket connection through an HTTP upgrade
path internally. That keeps the client-facing configuration simple and
allows the listener to work through intermediate HTTP-aware
infrastructure.
## What changed
- keep the emitted and configured exec-server URL as `ws://IP:PORT`
- serve that websocket endpoint through Axum HTTP upgrade handling on
`/`
- expose `GET /readyz` from the same listener for readiness checks
- route upgraded Axum websocket streams through the shared JSON-RPC
connection machinery
- initialize the rustls crypto provider before websocket client
connections
- preserve inbound binary websocket JSON-RPC parsing for compatibility
with the prior transport behavior
## Verification
- `cargo test -p codex-exec-server --test health --test process --test
websocket --test initialize --test exec_process`
## Summary
Support registry-backed remote executors end to end so downstream
services can resolve an executor id into an exec-server URL and make
that environment available to Codex without relying on the legacy cloud
environments flow.
## What changed
- switch remote executor registration to the executor registry bootstrap
contract
- allow named remote environments to be inserted into
`EnvironmentManager` at runtime
- add the experimental app-server RPC `environment/add` so initialized
experimental clients can register those remote environments for later
`thread/start` and `turn/start` selection
## Validation
Ran focused validation locally:
- `cargo test -p codex-exec-server environment_manager_`
- `cargo test -p codex-exec-server
register_executor_posts_with_bearer_token_header`
- `cargo test -p codex-app-server-protocol`
## Summary
This PR adds the first `codex-rs` milestone for remote-exec e2e: a local
`codex exec-server` can now register itself with
`codex-cloud-environments` and attach to the returned rendezvous
websocket.
At a high level, `codex exec-server --cloud ...` now:
- loads ChatGPT auth from normal Codex config
- registers an executor with `codex-cloud-environments`
- receives a signed rendezvous websocket URL
- serves the existing exec-server JSON-RPC protocol over that websocket
## What Changed
- Added `--cloud`, `--cloud-base-url`, `--cloud-environment-id`, and
`--cloud-name` to `codex exec-server`
- Added a new `exec-server/src/cloud.rs` module that handles:
- registration requests
- auth/header setup
- bounded auth retry on `401/403`
- reconnect/backoff after websocket disconnects
- Reused the existing `ConnectionProcessor` / `ExecServerHandler` path
so cloud mode serves the same exec/filesystem RPC surface as local
websocket mode
- Added cloud-specific error variants and minimal docs for the new mode
## Testing
Manual e2e test that fully goes through exec server flow with our codex
cloud agent as orchestrator
