## Summary
- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**
## Validation
- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
## Summary
- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.
## Root cause
Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.
Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.
The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
## Summary
- Initialize stderr tracing and the configured OpenTelemetry provider
for local and remote `codex exec-server` startup.
- Instrument the local and remote server entrypoints with a root runtime
span.
- Keep raw Noise environment, registration, and stream identifiers out
of exported spans while preserving them in local debug events.
- Keep telemetry setup in a focused CLI module instead of growing the
top-level command entrypoint.
## Stack
- Previous: none (`#27058` has merged)
- Next: #27466
## Validation
- `just test -p codex-exec-server --lib` (139 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
---------
Co-authored-by: Richard Lee <richardlee@openai.com>
## Why
Codex can run a remote exec server through the Noise relay, but the
normal
environment-manager path could not establish an
environment-registry-backed
harness connection. Signed rendezvous URLs and harness authorizations
are
short-lived, so reconnects must fetch a fresh bundle instead of
retaining
stale connection credentials. A stalled registry request must also fail
within
the regular remote connection deadline, without exposing these
credentials in
debug logs.
Issue: N/A (internal environment-service integration).
## What Changed
- Add environment-manager configuration for a registry-backed Noise
rendezvous
environment.
- Request a fresh bundle from
`/cloud/environment/{environment_id}/connect` for every physical harness
connection, using the existing 10-second remote connection timeout.
- Share the Environment Registry register, connect, and validate wire
payloads
through `codex-exec-server` and `codex-core-api`.
- Redact the signed rendezvous URL and harness authorization from the
public
connect response's `Debug` output.
- Add focused coverage for registry bundle retrieval, stalled requests,
and
credential redaction.
## Why
Services that proxy the exec-server environment registry endpoints need
to deserialize and forward the same Noise registration and harness-key
validation payloads. Those wire models currently live as private,
serialize-only structs in `exec-server`, which forces consumers to
duplicate the contract.
## What changed
- Add owned serde models for registration and harness-key validation
requests and responses.
- Use those models in the existing exec-server registry client.
- Re-export the models from `codex-exec-server` and `codex-core-api`.
- Keep the harness authorization request free of a derived `Debug`
implementation so it is not accidentally logged.
## Testing
- Focused exec-server registration and harness-key validation tests: 2
passed.
- `cargo check -p codex-core-api`
The full `codex-exec-server` suite compiled and ran 254 tests: 222
passed, while 32 existing filesystem sandbox tests could not run under
the nested macOS sandbox (`sandbox_apply: Operation not permitted`).
Co-authored-by: Codex <noreply@openai.com>
## Why
The transport in
[openai/codex#26242](https://github.com/openai/codex/pull/26242) needs
to be used by every remote orchestrator-to-executor connection before
JSON-RPC traffic starts.
## Changes
- Generates one executor Noise identity when remote exec-server starts
and registers its public key.
- Creates a harness identity for each physical remote environment
connection.
- Fetches a fresh registry bundle before connecting and validates the
authenticated harness key before completing the executor handshake.
- Multiplexes encrypted logical streams over the existing executor
WebSocket.
- Adds bounded stream, handshake-failure, and reassembly state.
- Adds safe lifecycle diagnostics without logging keys, authorizations,
plaintext, or ciphertext.
- Covers reconnects, replay rejection, validation failure, framing
limits, and encrypted JSON-RPC tool traffic.
## Stack
1. [openai/codex#26242](https://github.com/openai/codex/pull/26242):
Noise channel and relay transport
2. **[openai/codex#26245](https://github.com/openai/codex/pull/26245)**:
remote registration and runtime activation
## Verification
- `just test -p codex-exec-server`
- `just fix -p codex-exec-server`
- `just bazel-lock-check`
- `cargo shear`
---------
Co-authored-by: Codex <noreply@openai.com>
## Overview
Allow remote `codex exec-server` registration to use existing API-key
auth while restricting where those credentials can be sent.
- Accept `CodexAuth::ApiKey` for the normal `--remote` registration
path.
- Restrict API-key remote registration to HTTPS `openai.com` and
`openai.org` hosts and subdomains, with explicit HTTP loopback support
for local development.
- Disable registry registration redirects so credentials cannot be
forwarded to an unvalidated destination.
- Retain `--use-agent-identity-auth` as the explicit Agent Identity
path.
- Document remote registration using `CODEX_API_KEY`.
## Big picture
Callers can now provide an API key directly to `exec-server`
registration without first establishing ChatGPT login state:
```sh
CODEX_API_KEY="$OPENAI_API_KEY" \
codex exec-server \
--remote "https://<host>.openai.org/api" \
--environment-id "$ENVIRONMENT_ID"
```
## Validation
- `cargo fmt --all` (`just fmt` is not installed on this host)
- `cargo test -p codex-cli -p codex-exec-server`
## Summary
- migrate exec-server remote registration naming from executor to
environment
- align CLI, public Rust exports, registry error messages, and relay
test fixtures with the environment registry contract
- keep the live registration path and response model consistent with
`/cloud/environment/{environment_id}/register`
## Verification
- `cargo test -p codex-exec-server
remote::tests::register_environment_posts_with_auth_provider_headers
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_environment_routes_independent_virtual_streams
--manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
- `cargo check -p codex-cli --manifest-path
/Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
opened; will update after completion if needed)
This updates remote `exec-server` registration to use normal Codex auth
instead of a registry-issued credential. The registry request is built
from the existing auth-provider path, which preserves the biscuit-only
registry contract introduced in
[openai/openai#924101](https://github.com/openai/openai/pull/924101)
while removing the old remote registry bearer env var and its direct
transport assumptions.
The default remote flow uses persisted ChatGPT auth from the normal
Codex config/storage path. This PR also includes the containerized Agent
Identity path needed by
[openai/openai#924260](https://github.com/openai/openai/pull/924260):
remote `exec-server` accepts `--allow-agent-identity-auth`, permits
Agent Identity auth loaded from `CODEX_ACCESS_TOKEN` only when that flag
is present, and reuses the existing Agent task registration plus derived
`AgentAssertion` header generation. API-key auth remains unsupported,
and Agent Identity stays opt-in.
Validation performed beyond normal presubmit coverage:
- `cargo fmt --all --check`
- `cargo check -p codex-cli`
- `cargo test -p codex-exec-server`
- `cargo test -p codex-cli exec_server_agent_identity_auth_flag_`
- `cargo test -p codex-cli remote_exec_server_auth_mode_`
I also attempted `cargo test -p codex-cli`. The new CLI tests passed
inside that run, but the suite ended on an unrelated local
marketplace-state failure in
`plugin_list_excludes_unconfigured_repo_local_marketplaces`.
## Why
Remote exec-server now needs one executor websocket to serve multiple
harness JSON-RPC sessions. Rendezvous routes by `stream_id`, and the
exec-server side needs to use the same stable relay frame contract
instead of a hand-rolled JSON shape.
The relay protocol also needs to make ownership boundaries clear:
harness and executor endpoints own sequencing, acks, retries, duplicate
suppression, segmentation, and reassembly; rendezvous only routes
frames.
## What Changed
- Add the checked-in `codex.exec_server.relay.v1.RelayMessageFrame`
proto plus generated prost bindings for `codex-exec-server`.
- Encode remote harness/executor relay traffic as binary protobuf
websocket frames while keeping local websocket JSON-RPC unchanged.
- Demux executor-side relay streams into independent
`ConnectionProcessor` sessions keyed by `stream_id`.
- Add a programmatic `RemoteExecutorConfig::with_bearer_token(...)`
constructor for non-CLI callers and integration tests.
- Add an integration test that starts the remote executor against a fake
registry/rendezvous websocket and verifies two virtual streams share one
executor websocket without cross-talk, including per-stream reset
behavior.
- Document the remote relay envelope, sequence ranges, `ack`/`ack_bits`,
and endpoint responsibilities in `exec-server/README.md`.
## Verification
- `cargo test -p codex-exec-server --test relay
multiplexed_remote_executor_routes_independent_virtual_streams --
--exact`
- `cargo test -p codex-exec-server --test relay`
- `cargo test -p codex-exec-server` passed outside the sandbox. The
sandboxed run hit macOS `sandbox-exec: sandbox_apply: Operation not
permitted` in filesystem sandbox tests.
## Why
`codex exec-server` should keep the existing public `ws://IP:PORT` URL
shape while serving that websocket connection through an HTTP upgrade
path internally. That keeps the client-facing configuration simple and
allows the listener to work through intermediate HTTP-aware
infrastructure.
## What changed
- keep the emitted and configured exec-server URL as `ws://IP:PORT`
- serve that websocket endpoint through Axum HTTP upgrade handling on
`/`
- expose `GET /readyz` from the same listener for readiness checks
- route upgraded Axum websocket streams through the shared JSON-RPC
connection machinery
- initialize the rustls crypto provider before websocket client
connections
- preserve inbound binary websocket JSON-RPC parsing for compatibility
with the prior transport behavior
## Verification
- `cargo test -p codex-exec-server --test health --test process --test
websocket --test initialize --test exec_process`
## Summary
Support registry-backed remote executors end to end so downstream
services can resolve an executor id into an exec-server URL and make
that environment available to Codex without relying on the legacy cloud
environments flow.
## What changed
- switch remote executor registration to the executor registry bootstrap
contract
- allow named remote environments to be inserted into
`EnvironmentManager` at runtime
- add the experimental app-server RPC `environment/add` so initialized
experimental clients can register those remote environments for later
`thread/start` and `turn/start` selection
## Validation
Ran focused validation locally:
- `cargo test -p codex-exec-server environment_manager_`
- `cargo test -p codex-exec-server
register_executor_posts_with_bearer_token_header`
- `cargo test -p codex-app-server-protocol`
## Summary
This PR adds the first `codex-rs` milestone for remote-exec e2e: a local
`codex exec-server` can now register itself with
`codex-cloud-environments` and attach to the returned rendezvous
websocket.
At a high level, `codex exec-server --cloud ...` now:
- loads ChatGPT auth from normal Codex config
- registers an executor with `codex-cloud-environments`
- receives a signed rendezvous websocket URL
- serves the existing exec-server JSON-RPC protocol over that websocket
## What Changed
- Added `--cloud`, `--cloud-base-url`, `--cloud-environment-id`, and
`--cloud-name` to `codex exec-server`
- Added a new `exec-server/src/cloud.rs` module that handles:
- registration requests
- auth/header setup
- bounded auth retry on `401/403`
- reconnect/backoff after websocket disconnects
- Reused the existing `ConnectionProcessor` / `ExecServerHandler` path
so cloud mode serves the same exec/filesystem RPC surface as local
websocket mode
- Added cloud-specific error variants and minimal docs for the new mode
## Testing
Manual e2e test that fully goes through exec server flow with our codex
cloud agent as orchestrator
