## What changed
- Replace Rust `1.99.1` pins with `1.95.0` across Cargo, Bazel, CI, release workflows, and Windows setup, and update the Bazel lockfile.
- Adjust Rust APIs, borrowing, and lint-related code for the pinned toolchain, and box MCP tool dispatch to bound compiler async-layout query depth.
- Update the halfwidth sound mark snapshot and allow optional symbol hash suffixes in the code mode host package smoke test.
GitOrigin-RevId: 75460c3d0465dc601811e619bce373a4f7ff2912
## What changed
- Update Cargo and Bazel toolchain pins, CI and release workflows, and Windows setup from Rust `1.95.0` to `1.99.1`.
- Adapt Rust code to updated APIs and lint requirements, including `try_update` and `as_chunks`, and refresh the halfwidth sound marks snapshot.
- Update `webpki-root-certs` from `1.0.5` to `1.0.10` in the Cargo and Bazel lockfiles.
GitOrigin-RevId: e1aaed63b7e72bb8c8bbe25b32c6329c9a37b6c3
## Why
Some metrics backends require cumulative metrics, while configured OTLP exporters previously always used delta temporality.
## What changed
Read `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` for configured HTTP and gRPC metrics exporters. Accept `delta`, `cumulative`, and `lowmemory` case-insensitively, retaining delta for unset, empty, or unrecognized values. Document the setting and preserve existing temporality for the built-in Statsig exporter and runtime snapshots.
## Testing
Update the HTTP loopback test to set the preference in an isolated subprocess and assert that the exported counter uses cumulative aggregation temporality.
GitOrigin-RevId: ca212eea20054b947005d323186c8b9a819ace9d
## Why
AWS credential discovery, analytics, and telemetry used clients that did not share the application's network policy. These requests need to honor destination restrictions and permission revocation.
## What changed
- Route AWS credential and region HTTP requests through the shared HTTP client, and check the signing destination before loading credentials. Skip discovery for static access keys with an explicit region.
- Apply account-scoped policy to Bedrock authentication. Require unrestricted policy for credential exporters and reauthentication commands, and cancel active work when permission is revoked. Document that AWS profile `credential_process` network traffic remains outside the application's HTTP policy.
- Send analytics through the authenticated account's HTTP client factory.
- Guard OTLP log, trace, and metric exports with revocable permits, disable them under destination restrictions, and make managed HTTP exports cancellable. Suppress global Statsig settings while managed policy is active.
## Testing
Add coverage for metadata credential request cancellation, allowed and denied SigV4 destinations, AWS credential precedence and endpoint configuration, and blocked credential exporter recovery. Update telemetry tests for account transitions and managed-policy Statsig suppression.
GitOrigin-RevId: d8a2018bfbbe971ec430699b0d3f86a7a9e1e072
## What changed
- Record `exec_server_client_requests_total` for every client RPC call attempt,
labeled by protocol method.
- Count attempts before local admission so rejected, timed-out, cancelled, and
transport-failed calls are included, while notifications and responses are not.
- Export the counter to configured OpenTelemetry collectors while excluding it
from the built-in Statsig metrics set.
## Testing
- Cover every RPC call entry point, successful responses, local and transport
failures, timeouts, cancellation, notifications, and disabled metrics.
- Verify that the OTLP HTTP exporter retains the counter.
GitOrigin-RevId: d266e840d5871aa2c34bed8ce44f3345e2b4650f
## Why
Creating the telemetry shutdown thread during shutdown can fail under resource
pressure, including when the native thread guard page cannot be allocated.
## What changed
- Start and verify a dedicated shutdown worker when `OtelProvider` is created.
- Send the provider to the prepared worker for bounded shutdown, while preserving
timeout behavior and avoiding a potentially blocking destructor if worker
preparation failed.
- Rename the fallible provider constructor to `try_new`.
## Testing
Add Unix regression coverage that injects guard-page allocation failures, plus
coverage for worker preparation failure, successful shutdown, and timeouts.
GitOrigin-RevId: 3656298078a800a7fa392437c2ee4a68753092e3
## What changed
- Exclude conversation turn count, response-engine latency, and turn token usage metrics from the built-in Statsig exporter, alongside the existing API and tool-call exclusions.
- Apply the Statsig exclusion list consistently to counters, histograms, gauges, observable gauges, and duration histograms.
- Preserve all excluded metrics for custom OTLP exporters and extend the loopback coverage to verify they are exported.
GitOrigin-RevId: e49f9aa1433126a3de8dbed76a4d2688f2831af8
## Why
A stalled telemetry exporter can block shutdown indefinitely when the interactive TUI exits.
## What changed
- Add `OtelProvider::shutdown_with_timeout` to run exporter shutdown on a detached thread and report completion, worker failure, or timeout.
- Give interactive TUI telemetry shutdown a 500 ms time budget and warn if cleanup does not finish successfully.
## Testing
- Cover successful shutdown on current-thread and multi-thread Tokio runtimes, worker creation failure, and timeout behavior.
- Verify that OTLP HTTP shutdown returns on time when a collector stalls.
GitOrigin-RevId: 5b5f31a841fbc5788715cb3065d522c6dafa994f
## What changed
- Treat `codex.api_request` and `codex.api_request.duration_ms` as runtime-only metrics for Statsig exporters, matching the existing tool-call metric behavior.
- Keep both metrics available through other OTLP exporters.
## Testing
- Extend the Statsig filtering test to cover both API request metrics.
- Verify the OTLP HTTP exporter still sends the API request count and duration.
GitOrigin-RevId: f0f800ade7979e4da72ae59641bf42aa8a3ebd5a
## What changed
- Treat `codex.tool.call` and `codex.tool.call.duration_ms` as runtime-only metrics when using the built-in Statsig exporter.
- Continue exporting both metrics through explicitly configured OTLP exporters.
## Testing
- Verify the Statsig configuration omits the tool-call metrics while retaining unrelated metrics.
- Extend the OTLP HTTP loopback test to cover the tool-call counter and duration histogram.
GitOrigin-RevId: 46dd2642063b34e9a4c69b67d58b45721609fcf2
The workspace denies `clippy::expect_used` in production. Although
`clippy.toml` allows `expect` in tests, Bazel Clippy compiles
integration-test helper code in a way that does not receive that
exemption, which encouraged verbose `unwrap_or_else(... panic!(...))`
and equivalent `match`/`let else` forms.
This allows `clippy::expect_used` once at each integration-test crate
root (including aggregated suites and test-support libraries), then
replaces manual panic-based Result and Option unwraps with
`expect`/`expect_err`. Standalone `tests/*.rs` files remain their own
crate roots. Intentional assertion and unexpected-variant panics remain
unchanged, and the production `expect_used = "deny"` lint remains in
place.
The cleanup is mechanical and net-negative in line count.
## Why
The generic OTLP HTTP paths for log events and trace events need
end-to-end coverage before exec-server relies on them.
## What changed
- Adds loopback coverage for exporting `codex_otel.log_only` events to
`/v1/logs`.
- Verifies `codex_otel.trace_safe` events are present in the exported
trace payload.
This is a test-only PR. It does not change OTEL runtime behavior or
metric APIs.
## Related work
- #26091: counter descriptions
- #27057: gauge instruments
- #27058: second-based duration histograms
This PR is independent and can land directly on `main`.
## Validation
- `just test -p codex-otel`
- `just fix -p codex-otel`
- `just fmt`
## Why
Exec-server observability needs current-value measurements in addition
to counters. The reusable OTEL client should expose that primitive
without coupling it to exec-server runtime behavior.
## What changed
- Adds integer gauge instruments, with optional descriptions.
- Caches gauges by name and description so instrument metadata remains
part of the declaration identity.
- Covers gauge values, descriptions, merged attributes, and OTLP HTTP
export.
This PR only adds the gauge primitive. It does not add second-based
duration histograms or exec-server adoption.
## Stack
1. #26091: counter descriptions
2. **#27057: gauge instruments**
3. #27058: second-based duration histograms
Related independent coverage: #27059 tests OTLP HTTP log and trace event
export.
## Validation
- `just test -p codex-otel`
- `just fix -p codex-otel`
- `just fmt`
Add Codex config for static trace span attributes and structured W3C
tracestate field upserts. The config flows through OtelSettings so
callers can attach trace metadata without touching every span call site.
Apply span attributes with an SDK span processor so every exported
trace span carries the configured metadata. Model tracestate as nested
member fields so configured keys can be upserted while unrelated
propagated state in the same member is preserved.
Validate configured tracestate before installing provider-global state,
including header-unsafe values the SDK does not reject by itself. This
keeps Codex from propagating malformed trace context from config.
Update the config schema, public docs, and OTLP loopback coverage for
config parsing, span export, propagation, and invalid-header rejection.
## Summary
- reduce public module visibility across Rust crates, preferring private
or crate-private modules with explicit crate-root public exports
- update external call sites and tests to use the intended public crate
APIs instead of reaching through module trees
- add the module visibility guideline to AGENTS.md
## Validation
- `cargo check --workspace --all-targets --message-format=short` passed
before the final fix/format pass
- `just fix` completed successfully
- `just fmt` completed successfully
- `git diff --check` passed
## Why
`argument-comment-lint` was green in CI even though the repo still had
many uncommented literal arguments. The main gap was target coverage:
the repo wrapper did not force Cargo to inspect test-only call sites, so
examples like the `latest_session_lookup_params(true, ...)` tests in
`codex-rs/tui_app_server/src/lib.rs` never entered the blocking CI path.
This change cleans up the existing backlog, makes the default repo lint
path cover all Cargo targets, and starts rolling that stricter CI
enforcement out on the platform where it is currently validated.
## What changed
- mechanically fixed existing `argument-comment-lint` violations across
the `codex-rs` workspace, including tests, examples, and benches
- updated `tools/argument-comment-lint/run-prebuilt-linter.sh` and
`tools/argument-comment-lint/run.sh` so non-`--fix` runs default to
`--all-targets` unless the caller explicitly narrows the target set
- fixed both wrappers so forwarded cargo arguments after `--` are
preserved with a single separator
- documented the new default behavior in
`tools/argument-comment-lint/README.md`
- updated `rust-ci` so the macOS lint lane keeps the plain wrapper
invocation and therefore enforces `--all-targets`, while Linux and
Windows temporarily pass `-- --lib --bins`
That temporary CI split keeps the stricter all-targets check where it is
already cleaned up, while leaving room to finish the remaining Linux-
and Windows-specific target-gated cleanup before enabling
`--all-targets` on those runners. The Linux and Windows failures on the
intermediate revision were caused by the wrapper forwarding bug, not by
additional lint findings in those lanes.
## Validation
- `bash -n tools/argument-comment-lint/run.sh`
- `bash -n tools/argument-comment-lint/run-prebuilt-linter.sh`
- shell-level wrapper forwarding check for `-- --lib --bins`
- shell-level wrapper forwarding check for `-- --tests`
- `just argument-comment-lint`
- `cargo test` in `tools/argument-comment-lint`
- `cargo test -p codex-terminal-detection`
## Follow-up
- Clean up remaining Linux-only target-gated callsites, then switch the
Linux lint lane back to the plain wrapper invocation.
- Clean up remaining Windows-only target-gated callsites, then switch
the Windows lint lane back to the plain wrapper invocation.
## Summary
This PR fixes OTLP HTTP trace export in runtimes where the previous
exporter setup was unreliable, especially around app-server usage. It
also removes the old `codex_otel::otel_provider` compatibility shim and
switches remaining call sites over to the crate-root
`codex_otel::OtelProvider` export.
## What changed
- Use a runtime-safe OTLP HTTP trace exporter path for Tokio runtimes.
- Add an async HTTP client path for trace export when we are already
inside a multi-thread Tokio runtime.
- Make provider shutdown flush traces before tearing down the tracer
provider.
- Add loopback coverage that verifies traces are actually sent to
`/v1/traces`:
- outside Tokio
- inside a multi-thread Tokio runtime
- inside a current-thread Tokio runtime
- Remove the `codex_otel::otel_provider` shim and update remaining
imports.
## Why
I hit cases where spans were being created correctly but never made it
to the collector. The issue turned out to be in exporter/runtime
behavior rather than the span plumbing itself. This PR narrows that gap
and gives us regression coverage for the actual export path.
Add metrics capabilities to Codex. The `README.md` is up to date.
This will not be merged with the metrics before this PR of course:
https://github.com/openai/codex/pull/8350
Anthony Tafoya