Commit Graph
11 Commits
Author SHA1 Message Date
virajrai-oai e18a20d6eb Add correlated diagnostics for Noise relay handshakes (#55498)
## What changed

- Record executor handshake rejections and pending-validation resets in promptly completed diagnostic spans, with classified reasons and registry HTTP status when available. Normalize stream IDs as UUIDs and cap diagnostics at 100 per minute per relay connection, followed by a suppression marker.
- Add elapsed time and connection identifiers to harness handshake diagnostics, including initialization timeouts, and classify WebSocket read failures from typed errors.
- Forward valid stream UUIDs to harness key validation requests through the `x-codex-noise-stream-id` header.

## Testing

Extend tests to cover diagnostic rate limits, timeout correlation, prompt span export, typed WebSocket error classification, and optional stream ID headers. Verify that diagnostic output excludes authorization secrets, registry response text, reset reasons, and invalid stream IDs.

GitOrigin-RevId: 2c224e2d2828b300164868b10806add410e148b2
2026-10-20 07:56:07 +00:00
virajrai-oai 34420d4264 Preserve disconnect reasons in Noise handshake errors (#54637)
## Why

Noise handshake failures before connection readiness returned a generic error even when the transport had reported a specific disconnect reason. No `RpcClient` was reading the connection event channel at that point.

## What changed

Read the pending disconnect event when a Noise rendezvous handshake fails and append its reason to the handshake error when available.

## Testing

Add a regression test covering unexpected text frames and relay stream resets, checking that each produces the expected transport reason in the handshake error.

GitOrigin-RevId: 9b8ef9c8dcba75b46d51dc67784af46d4ca74b85
2026-10-16 15:43:21 +00:00
aaronl-openai 7efd184c24 Add observers for initial exec-server connection attempts (#52724)
## What changed

Expose `Environment::observe_connection_attempts` and `ConnectionAttemptOutcome` to report elapsed time and success, failure, or cancellation for initial connections. Measure from the end of provisioning through client installation, including connection retries.

The first registered observer wins. Past attempts and reconnections are omitted.

## Testing

Extend tests to assert failure followed by success when startup is retried, and a single cancellation notification when an environment is replaced during startup.

GitOrigin-RevId: 88300f38dcf9428b824c5ff36aa1b40d7d466b6f
2026-10-09 23:28:38 +00:00
richardopenai 985cf47a4e Allow provisioned executors more time to come online (#48575)
## Why

A provisioned executor can still be resuming after readiness is reported. Initial connection attempts can exhaust the ordinary registry retry limits before the executor comes online.

## What changed

Retry `environment_offline` registry responses during initial Noise rendezvous connections for provisioned environments until a fixed five-minute deadline, starting after provisioning succeeds. Keep the existing retry limits for other registry errors and stop on permanent failures.

## Testing

Add tests covering the five-minute deadline, ordinary limits for other errors and stalled requests, and termination on a later permanent error. Verify that readiness and info requests remain pending through an extended offline period and succeed using the same environment handle once the executor comes online.

GitOrigin-RevId: e22211499d9ec2f3590e75224eb39e2e4bf3538d
2026-09-27 00:01:17 +00:00
viyatb-oai 328feb0c29 Track executor registrations across connection refresh and recovery (#46555)
## Why

An executor can renew its registration while retaining its Noise identity. Connection refresh must distinguish registrations even when their keys match, and a stale registry lookup must not retire a session that recovery has already renewed.

## What changed

- Track the installed session's registration and expose it through `Environment::cached_executor_registration_id()` without connecting. Publish registration changes only when the new connection is installed.
- Require both registration and executor key to match before refresh reuses a session. Retry stale lookups and prevent recovery from installing a connection after retirement.
- Pin session recovery to the original Noise key while allowing registration renewal to preserve the session and running processes.
- Expose `CodexThread::active_turn_environment_selections()` so hosts can authorize steering against the active turn's selections, including starting and failed environments, independently of later settings updates.

## Testing

Add regression coverage for registration replacement, renewal with a running process, stale refresh lookups, missing sessions, and rejection of changed Noise keys. Extend active-turn tests to check selection snapshots across settings updates, deferred startup, and turn completion.

GitOrigin-RevId: 4395bc6c57f137691887e6585f52dcc5b958101c
2026-09-19 01:27:02 +00:00
bkotsopoulos 781c183c3b Bound Noise handshakes by the exec server initialization timeout (#42623)
## What changed

- Wait for the authenticated Noise handshake before sending the JSON-RPC
  `initialize` request.
- Share the configured initialization timeout across the handshake and RPC,
  while preserving the configured duration in timeout diagnostics.
- Record the handshake and initialization RPC as child spans of the existing
  initialization span, and terminate transport tasks when handshake readiness
  fails.

## Testing

- Cover stalled handshake timeouts and cleanup of the underlying WebSocket.
- Cover timeout reporting after the handshake has consumed part of the budget.

GitOrigin-RevId: c3eb5f7cfedd8d155fde5b7614fc2eb399ce337c
2026-09-03 22:16:19 +00:00
Rasmus Rygaard eb49f491c6 Add explicit remote executor connection refresh (#40710)
## Why

Planned executor replacement needs a fresh session without waiting for the old
session's transient-disconnect recovery to finish.

## What changed

- Add `Environment::refresh_connection` for remote Noise registry-backed
  environments. It performs a fresh registry lookup, reuses a healthy session
  when the executor identity is unchanged, and connects to a replacement when
  it has changed.
- Retire superseded sessions and connection attempts so they cannot publish
  stale state, accept late RPC results, or replay outstanding work.
- Preserve the existing environment and filesystem handles while replacing the
  underlying client, and require a live status probe before refresh succeeds.

## Testing

Add coverage for replacement and session reuse, recovery and connection races,
lookup and handshake failures, handle preservation, and late RPC responses.

GitOrigin-RevId: f1d11208cbfe8af8feb25f6b6b8100da82169a99
2026-08-25 21:23:17 +00:00
richardopenai 2151d3a5b7 Reset registry retries when refreshing Noise bundles (#39852)
## Why

Refreshing a Noise rendezvous bundle after an unauthorized handshake inherited
the initial registry lookup's retry count. If that lookup had exhausted its
retries, a transient registry error during the refresh failed immediately.

## What changed

Reset the registry retry count along with the operation deadline before
requesting a replacement bundle after a `401` handshake response.

## Testing

Extend the initial Noise connection test to exhaust the first lookup's retry
budget, receive a `401`, then verify that an offline replacement lookup is
retried and connects successfully.

GitOrigin-RevId: bcfad50ea32d69855f1c2c4c461022b3ecbbd941
2026-08-21 03:47:37 +00:00
richardopenai 5663754f62 Retry transient registry failures during initial exec connection (#39777)
## What changed

- Retry transient environment-registry failures while opening the initial Noise rendezvous connection, including timeouts, interrupted response bodies, retryable HTTP statuses, and temporarily offline environments.
- Bound registry attempts with per-request and overall timeouts, exponential backoff, and a retry limit while returning permanent registry errors immediately.
- Preserve the existing one-time bundle refresh after an unauthorized WebSocket handshake, with a fresh registry retry window.
- Treat malformed complete JSON responses as terminal rather than retryable transport failures.

## Testing

- Cover transient and permanent registry responses, stalled requests, retry limits and deadlines, interrupted bodies, malformed JSON, unauthorized bundle refresh, and session-resume behavior.

GitOrigin-RevId: 5929274103702ad477e7a638ab489319a43a93fa
2026-08-20 18:48:17 +00:00
Celia Chen 94ebae725e Route exec-server WebSockets through configured proxies (#35056)
## Why

Remote environment connections need to honor Codex's effective outbound proxy policy, including when a rendezvous connection reconnects.

## What changed

- Pass the configured `HttpClientFactory` into remote environment transports and use `WebSocketConnector` for exec-server and rendezvous WebSockets.
- Resolve proxy routes asynchronously so these connections can use the configured system proxy.
- Add connector options that preserve Tungstenite's default TLS behavior and enable `TCP_NODELAY` for latency-sensitive rendezvous traffic.

## Testing

- Verify prepared remote environments connect through a configured system proxy.
- Verify initial and reconnected encrypted relay peers use the system proxy.
- Cover default TLS selection and opt-in `TCP_NODELAY` behavior in the WebSocket client.

GitOrigin-RevId: 8a8da2116e37cb3a891269d0c0b037986fecdd3c
2026-07-24 00:16:42 +00:00
Anton Panasenko ac3fe64100 Refresh signed exec-server URLs on reconnect (#28374)
## Summary

- add a provider API that supplies a fresh signed WebSocket URL for each
remote exec-server connection
- refresh the signed URL after disconnects and retry once when a
handshake returns `401 Unauthorized`
- allow `EnvironmentManager` consumers to register remote environments
backed by the URL provider

## Tests

- `just test -p codex-exec-server -E
'test(remote_websocket_client_refreshes_url_after_unauthorized_handshake)
| test(remote_websocket_client_refreshes_url_after_disconnect)'` — 2
passed
- `cargo check -p codex-core-api` — passed
- `just fix -p codex-exec-server` — passed
- `just fix -p codex-core-api` — no test targets; no-op
- `just fmt` — passed
- `just test -p codex-exec-server` — 187 passed; 32 unrelated macOS
sandbox tests could not invoke nested `sandbox-exec` (`Operation not
permitted`)
2026-06-17 20:58:48 -07:00