Commit Graph
11 Commits
Author SHA1 Message Date
Leonardo Di Donato 9b738582b1 Expose originating tool call metadata in command environments (#52119)
## Why

Command processes need thread and tool call labels for correlation, without stale values from inherited environments or cached shell snapshots.

## What changed

- Export the originating `exec_command` call ID as `CODEX_TOOL_CALL_ID`; later `write_stdin` calls leave it unchanged.
- Apply executor metadata as `CODEX_THREAD_ID` and `CODEX_TOOL_CALL_ID` after environment policies, overlays, and snapshot replay. Clear omitted fields when metadata is present; without metadata, preserve thread context and remove the call ID.
- Exclude both IDs from newly captured snapshots and omit the call ID from capture and validation helpers.
- Preserve empty call IDs, omit NUL-containing IDs, and handle case-insensitive environment names on Windows. Document that these labels support correlation, not authorization.

## Testing

Add regression coverage for metadata precedence, missing and invalid IDs, snapshot capture and replay, local and remote execution with pipes and TTYs, sandboxed remote execution, and preservation of the originating call ID across later input.

GitOrigin-RevId: 1027212b5d495d8e7cfac0f498c3134e060aa897
2026-10-08 13:45:19 +00:00
jif e32365a2c6 Allow larger shell snapshots when replaying from a file (#51350)
## Why

File replay can accommodate shell state larger than the 512 KiB budget used for environment replay.

## What changed

Allow up to 4 MiB of shell state and 8 MiB of capture output for file replay. Keep captured environments capped at 512 KiB and preserve the 512 KiB combined state-and-environment limit for environment replay. Report distinct failures for oversized capture output, state, and environments.

## Testing

Add replay-specific size-limit tests and enlarge the Bash integration-test snapshot to exercise file replay beyond the environment replay budget.

GitOrigin-RevId: 0210848eb398d75426cb8b4b3bfaa0bce25264b8
2026-10-06 12:10:58 +00:00
jif 588f616e8b Measure shell snapshot use and wait time per command (#51347)
## What changed

Add `codex.shell_snapshot.command` and `codex.shell_snapshot.wait_ms` metrics for eligible command preparations in the unified exec runtime and exec server. Tag observations with snapshot version, availability state, and whether replay was selected (`used`) or normal shell startup was used (`fallback`). Exclude background prewarming from command observations; `used` measures replay selection, not successful restoration or command completion.

## Testing

Extend snapshot retry tests to verify command counts and wait observations across fallback and recovery, excluding prewarm captures. Extend credential-broker approval tests to verify one `protected` snapshot observation for login-shell commands.

GitOrigin-RevId: d3d9f370b6bbba6dfc930a44044ce09373841799
2026-10-06 11:43:45 +00:00
jif c98e263fb5 Replay exec-server shell snapshots through unnamed files (#48078)
## Why

Large shell snapshots need a replay path that does not add their state to the child process environment.

## What changed

- Keep captured state cached in memory and give each Bash or Zsh launch an independent, read-only file descriptor to source. Unlink the file before writing shell state, create it under the sandbox-protected daemon directory, and close the carrier descriptor before restoring functions.
- Probe descriptor-path access inside the capture sandbox. Retain environment replay for `sh` and when descriptor paths are unavailable; use normal shell startup if preparing the replay file fails.
- Group options and aliases during capture so sourcing preserves their parsing behavior.

## Testing

Add coverage for concurrent replays with 480 KiB payloads, pipe and TTY execution, stdin preservation, descriptor cleanup, sandbox protection, blocked descriptor-path fallback, and alias/option parsing equivalence.

GitOrigin-RevId: 8103d5c37ce4d1885628a4efba3551549f12bea8
2026-09-25 12:23:15 +00:00
Winston Howes 1530f828cb Preserve complete shell snapshot exports through filtering and replay (#43907)
## Why

Line-based export parsing can truncate multiline values or mistake their contents for declarations. Snapshot replay also needs to restore Bash options before parsing functions that use extended glob syntax.

## What changed

- Capture shell state, aliases, and complete export records with NUL-delimited boundaries, separating capture, credential preparation, and rendering.
- Apply credential and environment policy to whole exports, preserving multiline values and making credential aliases follow allowed overrides.
- Restore Bash `shopt` options before functions, including for Bash running as `sh`.
- Support non-evaluating POSIX `ENV` path expansion and preserve native environment metadata and unset exports.
- Move core and exec-server snapshot consumers to the shared capture parser. Allow capture overhead while enforcing the exec-server's 512 KiB state-and-environment limit before filtering.

## Testing

Add regression coverage for complete capture records, multiline export filtering and replay, credential alias overrides, `ENV` expansion, and `PATH` export state. Extend exec-server tests to cover Bash-backed `sh`, `extglob`, `nocasematch`, `set -u`, and large environments.

GitOrigin-RevId: 7589ab4d137529a395dcee10b5162718de24a621
2026-09-08 20:41:01 +00:00
jif c84003c7e1 Add diagnostic labels to shell snapshot capture metrics (#43454)
## Why

Shell snapshot failures were grouped under `capture_failed`, making failure causes and retry attempts indistinguishable in metrics.

## What changed

- Add `purpose` (`execution` or `prewarm`), `attempt`, `shell`, and `sandbox` labels to capture counters and durations.
- Report bounded failure reasons such as `spawn_failed`, `timeout`, and `nonzero_exit` while preserving the original RPC errors.
- Include `failure_reason` on duration metrics as well as counters.

## Testing

Update the bounded retry and single-flight test to assert per-attempt metric labels for execution and prewarm captures, including failure reasons on both counters and durations.

GitOrigin-RevId: 907dc98736c5b8f91006909dacc43278efb43abf
2026-09-07 12:28:32 +00:00
viyatb-oai f53c91be2c Add attributed exec process lifecycle telemetry (#42373)
## Why

Exec-server process and network-policy events need stable attribution to the
launching tool call and executor without recording process payloads.

## What changed

- Add optional `ExecMetadata` to `ExecParams` and propagate the thread and tool
  call IDs from unified exec.
- Emit bounded OpenTelemetry events for process start, spawn failure, sandbox
  denial, and exit. Correlate them with the launch trace and registry-issued
  executor identity while excluding arguments, paths, environment values,
  output, and error text.
- Preserve launch attribution across long-running processes and reconnects, and
  attach the same metadata to network-policy audit events.
- Keep the protocol backward compatible when metadata is omitted, and prevent
  invalid trace headers from inheriting an unrelated active span.

## Testing

Add coverage for metadata serialization and propagation, lifecycle event
fields, trace relationships, reconnect behavior, spawn failures, sandbox
denials, and network-policy attribution.

GitOrigin-RevId: 7aa480a7289c73cb95e2c124c35500bb6f0d5084
2026-09-02 21:58:58 +00:00
jif 612e6491d5 Prewarm shell snapshots for eligible turns (#42137)
## Why

Shell Snapshot V2 previously captured the login-shell environment lazily when a
command started, adding that work to the command path.

## What changed

- Start one asynchronous snapshot capture after turn hooks accept the turn, so
  eligible local `exec_command` calls can reuse it without blocking the model.
- Use the turn's resolved tools, permissions, sandbox, environment, and shell
  configuration, and skip prewarming for unsupported or network-managed cases.
- Cancel outstanding prewarming during shutdown. Keep failed speculative
  captures retryable by real commands without consuming their retry budget.

## Testing

Added coverage for non-blocking capture and reuse, failure recovery, sandbox and
permission isolation, shutdown cancellation, hook-stopped turns, eligibility
gates, and concurrent capture retries.

GitOrigin-RevId: ff24a4f0328c06653e09d03fd0a3db1eccd65030
2026-09-01 21:16:12 +00:00
jif e16d098c00 Instrument exec-server shell snapshot capture (#40620)
## What changed

- Emit `codex.shell_snapshot` and `codex.shell_snapshot.duration_ms` for each
  exec-server capture attempt, including success and failure tags.
- Label exec-server metrics as `v2` and existing core snapshot metrics as `v1`
  so the two implementations can be distinguished.
- Use the exec server's metrics client when available and fall back to the
  configured global client for local execution.

## Testing

- Extend the bounded-retry and single-flight snapshot test to verify metric
  counts and tags for first-attempt success, retries, recovery, and exhaustion.

GitOrigin-RevId: 94f73e58a4a1a90c518fa102f138ca7ee29b5627
2026-08-25 14:34:25 +00:00
jif 0fe877b4de Retry failed shell snapshot captures (#40447)
## Why

A failed shell snapshot capture was cached for the lifetime of its cache entry, so later commands continued using the original command even after a transient profile error was fixed.

## What changed

- Retry failed captures after a one-second backoff, up to three total attempts.
- Share each attempt across concurrent requests and preserve command fallback during failures and backoff.
- Keep the final failure cached after the retry budget is exhausted.

## Testing

Cover recovery and retry exhaustion for concurrent requests, local and remote execution, pipes and TTYs, and Bash and zsh where available.

GitOrigin-RevId: b432ef7c9eb35a3a0a703ac675b4db6954c43a80
2026-08-24 18:09:47 +00:00
jif d0cc662b8c Cache shell snapshots in the exec server (#39756)
## What changed

- Add the `shellSnapshotV2` executor capability and an optional shell snapshot request to `ExecParams`.
- Capture and restore Unix shell state and profile exports from an in-memory, attachment-scoped cache for `bash`, `zsh`, and `sh`.
- Apply environment policies, runtime `PATH` entries, sandbox context, and live managed-proxy settings when preparing restored commands.
- Bound snapshot size, capture time, scope length, and cache capacity, and fall back to the original command when capture fails.

## Testing

- Cover local, remote, TTY, sandboxed, and supported-shell execution, plus environment filtering, proxy handling, in-memory reuse, and capture failure fallback.

GitOrigin-RevId: 624f747972c249c88c6f10f42cf0af97b75b5541
2026-08-20 17:39:06 +00:00