## Why
The existing request duration metric starts at dispatch, leaving out time spent waiting before dispatch.
## What changed
Add `exec_server_request_total_duration_seconds` to measure time from decoded receipt until response enqueue or disconnection, including queueing. Record it alongside the existing duration metric with the same `method` and `result` labels for success, error, and disconnection paths. Preserve existing dispatch and queue timing semantics.
## Testing
Add coverage for successful requests, route errors, unknown methods, response delivery failures, and disconnection during execution. Extend admission-wait coverage to verify total timing includes queueing and route setup without double-counting, and assert each completion and duration is recorded once.
GitOrigin-RevId: ed0f719751b821b8f9f9556914b0f67dc6c5ad8f
## Why
Shell snapshot failures were grouped under `capture_failed`, making failure causes and retry attempts indistinguishable in metrics.
## What changed
- Add `purpose` (`execution` or `prewarm`), `attempt`, `shell`, and `sandbox` labels to capture counters and durations.
- Report bounded failure reasons such as `spawn_failed`, `timeout`, and `nonzero_exit` while preserving the original RPC errors.
- Include `failure_reason` on duration metrics as well as counters.
## Testing
Update the bounded retry and single-flight test to assert per-attempt metric labels for execution and prewarm captures, including failure reasons on both counters and durations.
GitOrigin-RevId: 907dc98736c5b8f91006909dacc43278efb43abf
## Why
Exec-server process and network-policy events need stable attribution to the
launching tool call and executor without recording process payloads.
## What changed
- Add optional `ExecMetadata` to `ExecParams` and propagate the thread and tool
call IDs from unified exec.
- Emit bounded OpenTelemetry events for process start, spawn failure, sandbox
denial, and exit. Correlate them with the launch trace and registry-issued
executor identity while excluding arguments, paths, environment values,
output, and error text.
- Preserve launch attribution across long-running processes and reconnects, and
attach the same metadata to network-policy audit events.
- Keep the protocol backward compatible when metadata is omitted, and prevent
invalid trace headers from inheriting an unrelated active span.
## Testing
Add coverage for metadata serialization and propagation, lifecycle event
fields, trace relationships, reconnect behavior, spawn failures, sandbox
denials, and network-policy attribution.
GitOrigin-RevId: 7aa480a7289c73cb95e2c124c35500bb6f0d5084
## Why
Code-mode callbacks can outlive their initial request and run in separate tasks,
so their trace context must be preserved explicitly.
## What changed
- Preserve the execution context for code-mode callbacks and add spans for
nested tool invocations in the runtime and dispatch broker.
- Propagate each tool invocation span as the parent of its streamed gRPC
callback, falling back to the outer execution context when needed.
- Add a `codex.exec_server.process` span for each process lifetime, including
its process ID and completion result, without retaining the request span.
GitOrigin-RevId: a6059e34d895416f5517e51dad6ca0078355adca
## What changed
- Emit `codex.shell_snapshot` and `codex.shell_snapshot.duration_ms` for each
exec-server capture attempt, including success and failure tags.
- Label exec-server metrics as `v2` and existing core snapshot metrics as `v1`
so the two implementations can be distinguished.
- Use the exec server's metrics client when available and fall back to the
configured global client for local execution.
## Testing
- Extend the bounded-retry and single-flight snapshot test to verify metric
counts and tags for first-attempt success, retries, recovery, and exhaustion.
GitOrigin-RevId: 94f73e58a4a1a90c518fa102f138ca7ee29b5627
## What changed
- Start inbound exec-server request spans when messages enter the connection queue and carry them through dispatch and response handling.
- Record request outcomes for client-handled network policy callbacks, including errors and disconnections.
- Add the `exec_server_request_queue_duration_seconds` histogram, labeled by bounded route name, while excluding synchronous route setup time.
## Testing
- Cover span lifetime and trace-parent propagation across server and client queues.
- Verify queue-duration telemetry and outcome recording for completed, rejected, and cancelled requests.
GitOrigin-RevId: ed67fe5305048bdf283a26ec874337d549e3324f
## Summary
- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**
## Validation
- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
## Summary
- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.
## Root cause
Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.
Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.
The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.
## Stack
Review and land this stack in order:
1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle
## Validation
- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
