diff --git a/.codex/skills/babysit-pr/SKILL.md b/.codex/skills/babysit-pr/SKILL.md index 303bb030fa..4718f55b50 100644 --- a/.codex/skills/babysit-pr/SKILL.md +++ b/.codex/skills/babysit-pr/SKILL.md @@ -1,6 +1,6 @@ --- name: babysit-pr -description: Babysit a GitHub pull request after creation by continuously polling CI checks/workflow runs, new review comments, and mergeability state until the PR is ready to merge (or merged/closed). Diagnose failures, retry likely flaky failures up to 3 times, auto-fix/push branch-related issues when appropriate, and stop only when user help is required (for example CI infrastructure issues, exhausted flaky retries, or ambiguous/blocking situations). Use when the user asks Codex to monitor a PR, watch CI, handle review comments, or keep an eye on failures and feedback on an open PR. +description: Babysit a GitHub pull request after creation by continuously polling review comments, CI checks/workflow runs, and mergeability state until the PR is merged/closed or user help is required. Diagnose failures, retry likely flaky failures up to 3 times, auto-fix/push branch-related issues when appropriate, and keep watching open PRs so fresh review feedback is surfaced promptly. Use when the user asks Codex to monitor a PR, watch CI, handle review comments, or keep an eye on failures and feedback on an open PR. --- # PR Babysitter @@ -9,8 +9,8 @@ description: Babysit a GitHub pull request after creation by continuously pollin Babysit a PR persistently until one of these terminal outcomes occurs: - The PR is merged or closed. -- CI is successful, there are no unaddressed review comments surfaced by the watcher, required review approval is not blocking merge, and there are no potential merge conflicts (PR is mergeable / not reporting conflict risk). - A situation requires user help (for example CI infrastructure issues, repeated flaky failures after retry budget is exhausted, permission problems, or ambiguity that cannot be resolved safely). +- Optional handoff milestone: the PR is currently green + mergeable + review-clean. Treat this as a progress state, not a watcher stop, so late-arriving review comments are still surfaced promptly while the PR remains open. Do not stop merely because a single snapshot returns `idle` while checks are still pending. @@ -24,7 +24,7 @@ Accept any of the following: ## Core Workflow 1. When the user asks to "monitor"/"watch"/"babysit" a PR, start with the watcher's continuous mode (`--watch`) unless you are intentionally doing a one-shot diagnostic snapshot. -2. Run the watcher script to snapshot PR/CI/review state (or consume each streamed snapshot from `--watch`). +2. Run the watcher script to snapshot PR/review/CI state (or consume each streamed snapshot from `--watch`). 3. Inspect the `actions` list in the JSON response. 4. If `diagnose_ci_failure` is present, inspect failed run logs and classify the failure. 5. If the failure is likely caused by the current branch, patch code locally, commit, and push. @@ -33,10 +33,10 @@ Accept any of the following: 8. If a review item from another author is non-actionable, already addressed, or not valid, post one reply on the comment/thread explaining that decision (for example answering the question or explaining why no change is needed). If the watcher later surfaces your own reply, treat that self-authored item as already handled and do not reply again. 9. If the failure is likely flaky/unrelated and `retry_failed_checks` is present, rerun failed jobs with `--retry-failed-now`. 10. If both actionable review feedback and `retry_failed_checks` are present, prioritize review feedback first; a new commit will retrigger CI, so avoid rerunning flaky checks on the old SHA unless you intentionally defer the review change. -11. On every loop, verify mergeability / merge-conflict status (for example via `gh pr view`) in addition to CI and review state. +11. On every loop, look for newly surfaced review feedback before acting on CI failures or mergeability state, then verify mergeability / merge-conflict status (for example via `gh pr view`) alongside CI. 12. After any push or rerun action, immediately return to step 1 and continue polling on the updated SHA/state. 13. If you had been using `--watch` before pausing to patch/commit/push, relaunch `--watch` yourself in the same turn immediately after the push (do not wait for the user to re-invoke the skill). -14. Repeat polling until the PR is green + review-clean + mergeable, `stop_pr_closed` appears, or a user-help-required blocker is reached. +14. Repeat polling until `stop_pr_closed` appears or a user-help-required blocker is reached. A green + review-clean + mergeable PR is a progress milestone, not a reason to stop the watcher while the PR is still open. 15. Maintain terminal/session ownership: while babysitting is active, keep consuming watcher output in the same turn; do not leave a detached `--watch` process running and then end the turn as if monitoring were complete. ## Commands @@ -131,7 +131,7 @@ Use this loop in a live Codex session: 8. Retry failed checks only when `retry_failed_checks` is present and you are not about to replace the current SHA with a review/CI fix commit. 9. If you pushed a commit, resolved a review thread, replied to a review comment, or triggered a rerun, report the action briefly and continue polling (do not stop). 10. After a review-fix push, proactively restart continuous monitoring (`--watch`) in the same turn unless a strict stop condition has already been reached. -11. If everything is passing, mergeable, not blocked on required review approval, and there are no unaddressed review items, report success and stop. +11. If everything is passing, mergeable, not blocked on required review approval, and there are no unaddressed review items, report that the PR is currently ready to merge but keep the watcher running so new review comments are surfaced quickly while the PR remains open. 12. If blocked on a user-help-required issue (infra outage, exhausted flaky retries, unclear reviewer request, permissions), report the blocker and stop. 13. Otherwise sleep according to the polling cadence below and repeat. @@ -141,19 +141,18 @@ Do not hand control back to the user after a review-fix push just because a new If a `--watch` process is still running and no strict stop condition has been reached, the babysitting task is still in progress; keep streaming/consuming watcher output instead of ending the turn. ## Polling Cadence -Use adaptive polling and continue monitoring even after CI turns green: +Keep review polling aggressive and continue monitoring even after CI turns green: - While CI is not green (pending/running/queued or failing): poll every 1 minute. -- After CI turns green: start at every 1 minute, then back off exponentially when there is no change (for example 1m, 2m, 4m, 8m, 16m, 32m), capping at every 1 hour. -- Reset the green-state polling interval back to 1 minute whenever anything changes (new commit/SHA, check status changes, new review comments, mergeability changes, review decision changes). -- If CI stops being green again (new commit, rerun, or regression): return to 1-minute polling. +- After CI turns green: keep polling at the base cadence while the PR remains open so newly posted review comments are surfaced promptly instead of waiting on a long green-state backoff. +- Reset the cadence immediately whenever anything changes (new commit/SHA, check status changes, new review comments, mergeability changes, review decision changes). +- If CI stops being green again (new commit, rerun, or regression): stay on the base polling cadence. - If any poll shows the PR is merged or otherwise closed: stop polling immediately and report the terminal state. ## Stop Conditions (Strict) Stop only when one of the following is true: - PR merged or closed (stop as soon as a poll/snapshot confirms this). -- PR is ready to merge: CI succeeded, no surfaced unaddressed review comments, not blocked on required review approval, and no merge conflict risk. - User intervention is required and Codex cannot safely proceed alone. Keep polling when: @@ -162,14 +161,14 @@ Keep polling when: - CI is still running/queued. - Review state is quiet but CI is not terminal. - CI is green but mergeability is unknown/pending. -- CI is green and mergeable, but the PR is still open and you are waiting for possible new review comments or merge-conflict changes per the green-state cadence. -- The PR is green but blocked on review approval (`REVIEW_REQUIRED` / similar); continue polling on the green-state cadence and surface any new review comments without asking for confirmation to keep watching. +- CI is green and mergeable, but the PR is still open and you are waiting for possible new review comments or merge-conflict changes. +- The PR is green but blocked on review approval (`REVIEW_REQUIRED` / similar); continue polling at the base cadence and surface any new review comments without asking for confirmation to keep watching. ## Output Expectations Provide concise progress updates while monitoring and a final summary that includes: - During long unchanged monitoring periods, avoid emitting a full update on every poll; summarize only status changes plus occasional heartbeat updates. -- Treat push confirmations, intermediate CI snapshots, and review-action updates as progress updates only; do not emit the final summary or end the babysitting session unless a strict stop condition is met. +- Treat push confirmations, intermediate CI snapshots, ready-to-merge snapshots, and review-action updates as progress updates only; do not emit the final summary or end the babysitting session unless a strict stop condition is met. - A user request to "monitor" is not satisfied by a couple of sample polls; remain in the loop until a strict stop condition or an explicit user interruption. - A review-fix commit + push is not a completion event; immediately resume live monitoring (`--watch`) in the same turn and continue reporting progress updates. - When CI first transitions to all green for the current SHA, emit a one-time celebratory progress update (do not repeat it on every green poll). Preferred style: `🚀 CI is all green! 33/33 passed. Still on watch for review approval.` diff --git a/.codex/skills/babysit-pr/agents/openai.yaml b/.codex/skills/babysit-pr/agents/openai.yaml index 2faca770e0..b68a7287a2 100644 --- a/.codex/skills/babysit-pr/agents/openai.yaml +++ b/.codex/skills/babysit-pr/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "PR Babysitter" - short_description: "Watch PR CI, reviews, and merge conflicts" - default_prompt: "Babysit the current PR: monitor CI, reviewer comments, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); fix valid issues, push updates, and rerun flaky failures up to 3 times. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Continue polling autonomously after any push/rerun until a strict terminal stop condition is reached or the user interrupts." + short_description: "Watch PR review comments, CI, and merge conflicts" + default_prompt: "Babysit the current PR: monitor reviewer comments, CI, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); surface new review feedback before acting on CI or mergeability work, fix valid issues, push updates, and rerun flaky failures up to 3 times. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Do not treat a green + mergeable PR as a terminal stop while it is still open; continue polling autonomously after any push/rerun so newly posted review comments are surfaced until a strict terminal stop condition is reached or the user interrupts." diff --git a/.codex/skills/babysit-pr/scripts/gh_pr_watch.py b/.codex/skills/babysit-pr/scripts/gh_pr_watch.py index 80aefd58d9..2650770b2a 100755 --- a/.codex/skills/babysit-pr/scripts/gh_pr_watch.py +++ b/.codex/skills/babysit-pr/scripts/gh_pr_watch.py @@ -45,7 +45,6 @@ MERGE_CONFLICT_OR_BLOCKING_STATES = { "DRAFT", "UNKNOWN", } -GREEN_STATE_MAX_POLL_SECONDS = 60 * 60 class GhCommandError(RuntimeError): @@ -578,7 +577,7 @@ def recommend_actions(pr, checks_summary, failed_runs, new_review_items, retries return unique_actions(actions) if is_pr_ready_to_merge(pr, checks_summary, new_review_items): - actions.append("stop_ready_to_merge") + actions.append("ready_to_merge") return unique_actions(actions) if new_review_items: @@ -606,12 +605,6 @@ def collect_snapshot(args): if not state.get("started_at"): state["started_at"] = int(time.time()) - # `gh pr checks -R ` requires an explicit PR/branch/url argument. - # After resolving `--pr auto`, reuse the concrete PR number. - checks = get_pr_checks(str(pr["number"]), repo=pr["repo"]) - checks_summary = summarize_checks(checks) - workflow_runs = get_workflow_runs_for_sha(pr["repo"], pr["head_sha"]) - failed_runs = failed_runs_from_workflow_runs(workflow_runs, pr["head_sha"]) authenticated_login = get_authenticated_login() new_review_items = fetch_new_review_items( pr, @@ -619,6 +612,15 @@ def collect_snapshot(args): fresh_state=fresh_state, authenticated_login=authenticated_login, ) + # Surface review feedback before drilling into CI and mergeability details. + # That keeps the babysitter responsive to new comments even when other + # actions are also available. + # `gh pr checks -R ` requires an explicit PR/branch/url argument. + # After resolving `--pr auto`, reuse the concrete PR number. + checks = get_pr_checks(str(pr["number"]), repo=pr["repo"]) + checks_summary = summarize_checks(checks) + workflow_runs = get_workflow_runs_for_sha(pr["repo"], pr["head_sha"]) + failed_runs = failed_runs_from_workflow_runs(workflow_runs, pr["head_sha"]) retries_used = current_retry_count(state, pr["head_sha"]) actions = recommend_actions( @@ -761,7 +763,6 @@ def run_watch(args): if ( "stop_pr_closed" in actions or "stop_exhausted_retries" in actions - or "stop_ready_to_merge" in actions ): print_event("stop", {"actions": snapshot.get("actions"), "pr": snapshot.get("pr")}) return 0 @@ -769,13 +770,13 @@ def run_watch(args): current_change_key = snapshot_change_key(snapshot) changed = current_change_key != last_change_key green = is_ci_green(snapshot) + pr = snapshot.get("pr") or {} + pr_open = not bool(pr.get("closed")) and not bool(pr.get("merged")) - if not green: + if not green or pr_open: poll_seconds = args.poll_seconds elif changed or last_change_key is None: poll_seconds = args.poll_seconds - else: - poll_seconds = min(poll_seconds * 2, GREEN_STATE_MAX_POLL_SECONDS) last_change_key = current_change_key time.sleep(poll_seconds) diff --git a/.codex/skills/babysit-pr/scripts/test_gh_pr_watch.py b/.codex/skills/babysit-pr/scripts/test_gh_pr_watch.py new file mode 100644 index 0000000000..c6a5d25682 --- /dev/null +++ b/.codex/skills/babysit-pr/scripts/test_gh_pr_watch.py @@ -0,0 +1,155 @@ +import argparse +import importlib.util +from pathlib import Path + +import pytest + + +MODULE_PATH = Path(__file__).with_name("gh_pr_watch.py") +MODULE_SPEC = importlib.util.spec_from_file_location("gh_pr_watch", MODULE_PATH) +gh_pr_watch = importlib.util.module_from_spec(MODULE_SPEC) +assert MODULE_SPEC.loader is not None +MODULE_SPEC.loader.exec_module(gh_pr_watch) + + +def sample_pr(): + return { + "number": 123, + "url": "https://github.com/openai/codex/pull/123", + "repo": "openai/codex", + "head_sha": "abc123", + "head_branch": "feature", + "state": "OPEN", + "merged": False, + "closed": False, + "mergeable": "MERGEABLE", + "merge_state_status": "CLEAN", + "review_decision": "", + } + + +def sample_checks(**overrides): + checks = { + "pending_count": 0, + "failed_count": 0, + "passed_count": 12, + "all_terminal": True, + } + checks.update(overrides) + return checks + + +def test_collect_snapshot_fetches_review_items_before_ci(monkeypatch, tmp_path): + call_order = [] + pr = sample_pr() + + monkeypatch.setattr(gh_pr_watch, "resolve_pr", lambda *args, **kwargs: pr) + monkeypatch.setattr(gh_pr_watch, "load_state", lambda path: ({}, True)) + monkeypatch.setattr( + gh_pr_watch, + "get_authenticated_login", + lambda: call_order.append("auth") or "octocat", + ) + monkeypatch.setattr( + gh_pr_watch, + "fetch_new_review_items", + lambda *args, **kwargs: call_order.append("review") or [], + ) + monkeypatch.setattr( + gh_pr_watch, + "get_pr_checks", + lambda *args, **kwargs: call_order.append("checks") or [], + ) + monkeypatch.setattr( + gh_pr_watch, + "summarize_checks", + lambda checks: call_order.append("summarize") or sample_checks(), + ) + monkeypatch.setattr( + gh_pr_watch, + "get_workflow_runs_for_sha", + lambda *args, **kwargs: call_order.append("workflow") or [], + ) + monkeypatch.setattr( + gh_pr_watch, + "failed_runs_from_workflow_runs", + lambda *args, **kwargs: call_order.append("failed_runs") or [], + ) + monkeypatch.setattr( + gh_pr_watch, + "recommend_actions", + lambda *args, **kwargs: call_order.append("recommend") or ["idle"], + ) + monkeypatch.setattr(gh_pr_watch, "save_state", lambda *args, **kwargs: None) + + args = argparse.Namespace( + pr="123", + repo=None, + state_file=str(tmp_path / "watcher-state.json"), + max_flaky_retries=3, + ) + + gh_pr_watch.collect_snapshot(args) + + assert call_order.index("review") < call_order.index("checks") + assert call_order.index("review") < call_order.index("workflow") + + +def test_recommend_actions_prioritizes_review_comments(): + actions = gh_pr_watch.recommend_actions( + sample_pr(), + sample_checks(failed_count=1), + [{"run_id": 99}], + [{"kind": "review_comment", "id": "1"}], + 0, + 3, + ) + + assert actions == [ + "process_review_comment", + "diagnose_ci_failure", + "retry_failed_checks", + ] + + +def test_run_watch_keeps_polling_open_ready_to_merge_pr(monkeypatch): + sleeps = [] + events = [] + snapshot = { + "pr": sample_pr(), + "checks": sample_checks(), + "failed_runs": [], + "new_review_items": [], + "actions": ["ready_to_merge"], + "retry_state": { + "current_sha_retries_used": 0, + "max_flaky_retries": 3, + }, + } + + monkeypatch.setattr( + gh_pr_watch, + "collect_snapshot", + lambda args: (snapshot, Path("/tmp/codex-babysit-pr-state.json")), + ) + monkeypatch.setattr( + gh_pr_watch, + "print_event", + lambda event, payload: events.append((event, payload)), + ) + + class StopWatch(Exception): + pass + + def fake_sleep(seconds): + sleeps.append(seconds) + if len(sleeps) >= 2: + raise StopWatch + + monkeypatch.setattr(gh_pr_watch.time, "sleep", fake_sleep) + + with pytest.raises(StopWatch): + gh_pr_watch.run_watch(argparse.Namespace(poll_seconds=30)) + + assert sleeps == [30, 30] + assert [event for event, _ in events] == ["snapshot", "snapshot"] diff --git a/.github/actions/setup-bazel-ci/action.yml b/.github/actions/setup-bazel-ci/action.yml index 34bbd40b19..a1e46d2d04 100644 --- a/.github/actions/setup-bazel-ci/action.yml +++ b/.github/actions/setup-bazel-ci/action.yml @@ -60,8 +60,72 @@ runs: # Use the shortest available drive to reduce argv/path length issues, # but avoid the drive root because some Windows test launchers mis-handle # MANIFEST paths there. - $bazelOutputUserRoot = if (Test-Path 'D:\') { 'D:\b' } else { 'C:\b' } + $hasDDrive = Test-Path 'D:\' + $bazelOutputUserRoot = if ($hasDDrive) { 'D:\b' } else { 'C:\b' } + $repoContentsCache = Join-Path $env:RUNNER_TEMP "bazel-repo-contents-cache-$env:GITHUB_RUN_ID-$env:GITHUB_JOB" "BAZEL_OUTPUT_USER_ROOT=$bazelOutputUserRoot" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append + "BAZEL_REPO_CONTENTS_CACHE=$repoContentsCache" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append + if (-not $hasDDrive) { + $repositoryCache = Join-Path $env:USERPROFILE '.cache\bazel-repo-cache' + "BAZEL_REPOSITORY_CACHE=$repositoryCache" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append + } + + - name: Expose MSVC SDK environment (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: | + # Bazel exec-side Rust build scripts do not reliably inherit the MSVC developer + # shell on GitHub-hosted Windows runners, so discover the latest VS install and + # ask `VsDevCmd.bat` to materialize the x64/x64 compiler + SDK environment. + $vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" + if (-not (Test-Path $vswhere)) { + throw "vswhere.exe not found" + } + + $installPath = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2>$null + if (-not $installPath) { + throw "Could not locate a Visual Studio installation with VC tools" + } + + $vsDevCmd = Join-Path $installPath 'Common7\Tools\VsDevCmd.bat' + if (-not (Test-Path $vsDevCmd)) { + throw "VsDevCmd.bat not found at $vsDevCmd" + } + + # Keep the export surface explicit: these are the paths and SDK roots that the + # MSVC toolchain probes need later when Bazel runs Windows exec-platform build + # scripts such as `aws-lc-sys`. + $varsToExport = @( + 'INCLUDE', + 'LIB', + 'LIBPATH', + 'PATH', + 'UCRTVersion', + 'UniversalCRTSdkDir', + 'VCINSTALLDIR', + 'VCToolsInstallDir', + 'WindowsLibPath', + 'WindowsSdkBinPath', + 'WindowsSdkDir', + 'WindowsSDKLibVersion', + 'WindowsSDKVersion' + ) + + # `VsDevCmd.bat` is a batch file, so invoke it under `cmd.exe`, suppress its + # banner, then dump the resulting environment with `set`. Re-export only the + # approved keys into `GITHUB_ENV` so later steps inherit the same MSVC context. + $envLines = & cmd.exe /c ('"{0}" -no_logo -arch=x64 -host_arch=x64 >nul && set' -f $vsDevCmd) + foreach ($line in $envLines) { + if ($line -notmatch '^(.*?)=(.*)$') { + continue + } + + $name = $matches[1] + $value = $matches[2] + if ($varsToExport -contains $name) { + "$name=$value" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append + } + } - name: Enable Git long paths (Windows) if: runner.os == 'Windows' diff --git a/.github/scripts/run-argument-comment-lint-bazel.sh b/.github/scripts/run-argument-comment-lint-bazel.sh new file mode 100755 index 0000000000..e2f494d620 --- /dev/null +++ b/.github/scripts/run-argument-comment-lint-bazel.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash + +set -euo pipefail + +ci_config=ci-linux +case "${RUNNER_OS:-}" in + macOS) + ci_config=ci-macos + ;; + Windows) + ci_config=ci-windows + ;; +esac + +bazel_lint_args=("$@") +if [[ "${RUNNER_OS:-}" == "Windows" ]]; then + has_host_platform_override=0 + for arg in "${bazel_lint_args[@]}"; do + if [[ "$arg" == --host_platform=* ]]; then + has_host_platform_override=1 + break + fi + done + + if [[ $has_host_platform_override -eq 0 ]]; then + # The nightly Windows lint toolchain is registered with an MSVC exec + # platform even though the lint target platform stays on `windows-gnullvm`. + # Override the host platform here so the exec-side helper binaries actually + # match the registered toolchain set. + bazel_lint_args+=("--host_platform=//:local_windows_msvc") + fi + + # Native Windows lint runs need exec-side Rust helper binaries and proc-macros + # to use rust-lld instead of the C++ linker path. The default `none` + # preference resolves to `cc` when a cc_toolchain is present, which currently + # routes these exec actions through clang++ with an argument shape it cannot + # consume. + bazel_lint_args+=("--@rules_rust//rust/settings:toolchain_linker_preference=rust") + + # Some Rust top-level targets are still intentionally incompatible with the + # local Windows MSVC exec platform. Skip those explicit targets so the native + # lint aspect can run across the compatible crate graph instead of failing the + # whole build after analysis. + bazel_lint_args+=("--skip_incompatible_explicit_targets") +fi + +bazel_startup_args=() +if [[ -n "${BAZEL_OUTPUT_USER_ROOT:-}" ]]; then + bazel_startup_args+=("--output_user_root=${BAZEL_OUTPUT_USER_ROOT}") +fi + +run_bazel() { + if [[ "${RUNNER_OS:-}" == "Windows" ]]; then + MSYS2_ARG_CONV_EXCL='*' bazel "$@" + return + fi + + bazel "$@" +} + +run_bazel_with_startup_args() { + if [[ ${#bazel_startup_args[@]} -gt 0 ]]; then + run_bazel "${bazel_startup_args[@]}" "$@" + return + fi + + run_bazel "$@" +} + +read_query_labels() { + local query="$1" + local query_stdout + local query_stderr + query_stdout="$(mktemp)" + query_stderr="$(mktemp)" + + if ! run_bazel_with_startup_args \ + --noexperimental_remote_repo_contents_cache \ + query \ + --keep_going \ + --output=label \ + "$query" >"$query_stdout" 2>"$query_stderr"; then + cat "$query_stderr" >&2 + rm -f "$query_stdout" "$query_stderr" + exit 1 + fi + + cat "$query_stdout" + rm -f "$query_stdout" "$query_stderr" +} + +final_build_targets=(//codex-rs/...) +if [[ "${RUNNER_OS:-}" == "Windows" ]]; then + # Bazel's local Windows platform currently lacks a default test toolchain for + # `rust_test`, so target the concrete Rust crate rules directly. The lint + # aspect still walks their crate graph, which preserves incremental reuse for + # non-test code while avoiding non-Rust wrapper targets such as platform_data. + final_build_targets=() + while IFS= read -r label; do + [[ -n "$label" ]] || continue + final_build_targets+=("$label") + done < <(read_query_labels 'kind("rust_(library|binary|proc_macro) rule", //codex-rs/...)') + + if [[ ${#final_build_targets[@]} -eq 0 ]]; then + echo "Failed to discover Windows Bazel lint targets." >&2 + exit 1 + fi +fi + +./.github/scripts/run-bazel-ci.sh \ + -- \ + build \ + "${bazel_lint_args[@]}" \ + -- \ + "${final_build_targets[@]}" diff --git a/.github/scripts/run-bazel-ci.sh b/.github/scripts/run-bazel-ci.sh index cf50135c2e..d08b32e3d4 100755 --- a/.github/scripts/run-bazel-ci.sh +++ b/.github/scripts/run-bazel-ci.sh @@ -41,6 +41,15 @@ if [[ -n "${BAZEL_OUTPUT_USER_ROOT:-}" ]]; then bazel_startup_args+=("--output_user_root=${BAZEL_OUTPUT_USER_ROOT}") fi +run_bazel() { + if [[ "${RUNNER_OS:-}" == "Windows" ]]; then + MSYS2_ARG_CONV_EXCL='*' bazel "$@" + return + fi + + bazel "$@" +} + ci_config=ci-linux case "${RUNNER_OS:-}" in macOS) @@ -60,7 +69,7 @@ print_bazel_test_log_tails() { bazel_info_cmd+=("${bazel_startup_args[@]}") fi - testlogs_dir="$("${bazel_info_cmd[@]}" info bazel-testlogs 2>/dev/null || echo bazel-testlogs)" + testlogs_dir="$(run_bazel "${bazel_info_cmd[@]:1}" info bazel-testlogs 2>/dev/null || echo bazel-testlogs)" local failed_targets=() while IFS= read -r target; do @@ -126,6 +135,41 @@ if [[ $remote_download_toplevel -eq 1 ]]; then post_config_bazel_args+=(--remote_download_toplevel) fi +if [[ -n "${BAZEL_REPO_CONTENTS_CACHE:-}" ]]; then + # Windows self-hosted runners can run multiple Bazel jobs concurrently. Give + # each job its own repo contents cache so they do not fight over the shared + # path configured in `ci-windows`. + post_config_bazel_args+=("--repo_contents_cache=${BAZEL_REPO_CONTENTS_CACHE}") +fi + +if [[ -n "${BAZEL_REPOSITORY_CACHE:-}" ]]; then + post_config_bazel_args+=("--repository_cache=${BAZEL_REPOSITORY_CACHE}") +fi + +if [[ "${RUNNER_OS:-}" == "Windows" ]]; then + windows_action_env_vars=( + INCLUDE + LIB + LIBPATH + PATH + UCRTVersion + UniversalCRTSdkDir + VCINSTALLDIR + VCToolsInstallDir + WindowsLibPath + WindowsSdkBinPath + WindowsSdkDir + WindowsSDKLibVersion + WindowsSDKVersion + ) + + for env_var in "${windows_action_env_vars[@]}"; do + if [[ -n "${!env_var:-}" ]]; then + post_config_bazel_args+=("--action_env=${env_var}" "--host_action_env=${env_var}") + fi + done +fi + bazel_console_log="$(mktemp)" trap 'rm -f "$bazel_console_log"' EXIT @@ -149,7 +193,7 @@ if [[ -n "${BUILDBUDDY_API_KEY:-}" ]]; then bazel_run_args+=("${post_config_bazel_args[@]}") fi set +e - "${bazel_cmd[@]}" \ + run_bazel "${bazel_cmd[@]:1}" \ --noexperimental_remote_repo_contents_cache \ "${bazel_run_args[@]}" \ -- \ @@ -184,7 +228,7 @@ else bazel_run_args+=("${post_config_bazel_args[@]}") fi set +e - "${bazel_cmd[@]}" \ + run_bazel "${bazel_cmd[@]:1}" \ --noexperimental_remote_repo_contents_cache \ "${bazel_run_args[@]}" \ -- \ diff --git a/.github/scripts/verify_cargo_workspace_manifests.py b/.github/scripts/verify_cargo_workspace_manifests.py new file mode 100644 index 0000000000..ad146bc286 --- /dev/null +++ b/.github/scripts/verify_cargo_workspace_manifests.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 + +"""Verify that codex-rs crates inherit workspace metadata, lints, and names. + +This keeps `cargo clippy` aligned with the workspace lint policy by ensuring +each crate opts into `[lints] workspace = true`, and it also checks the crate +name conventions for top-level `codex-rs/*` crates and `codex-rs/utils/*` +crates. +""" + +from __future__ import annotations + +import sys +import tomllib +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CARGO_RS_ROOT = ROOT / "codex-rs" +WORKSPACE_PACKAGE_FIELDS = ("version", "edition", "license") +TOP_LEVEL_NAME_EXCEPTIONS = { + "windows-sandbox-rs": "codex-windows-sandbox", +} +UTILITY_NAME_EXCEPTIONS = { + "path-utils": "codex-utils-path", +} + + +def main() -> int: + failures = [ + (path.relative_to(ROOT), errors) + for path in cargo_manifests() + if (errors := manifest_errors(path)) + ] + if not failures: + return 0 + + print( + "Cargo manifests under codex-rs must inherit workspace package metadata and " + "opt into workspace lints." + ) + print( + "Cargo only applies `codex-rs/Cargo.toml` `[workspace.lints.clippy]` " + "entries to a crate when that crate declares:" + ) + print() + print("[lints]") + print("workspace = true") + print() + print( + "Without that opt-in, `cargo clippy` can miss violations that Bazel clippy " + "catches." + ) + print() + print( + "Package-name checks apply to `codex-rs//Cargo.toml` and " + "`codex-rs/utils//Cargo.toml`." + ) + print() + for path, errors in failures: + print(f"{path}:") + for error in errors: + print(f" - {error}") + + return 1 + + +def manifest_errors(path: Path) -> list[str]: + manifest = load_manifest(path) + package = manifest.get("package") + if not isinstance(package, dict): + return [] + + errors = [] + for field in WORKSPACE_PACKAGE_FIELDS: + if not is_workspace_reference(package.get(field)): + errors.append(f"set `{field}.workspace = true` in `[package]`") + + lints = manifest.get("lints") + if not (isinstance(lints, dict) and lints.get("workspace") is True): + errors.append("add `[lints]` with `workspace = true`") + + expected_name = expected_package_name(path) + if expected_name is not None: + actual_name = package.get("name") + if actual_name != expected_name: + errors.append( + f"set `[package].name` to `{expected_name}` (found `{actual_name}`)" + ) + + return errors + + +def expected_package_name(path: Path) -> str | None: + parts = path.relative_to(CARGO_RS_ROOT).parts + if len(parts) == 2 and parts[1] == "Cargo.toml": + directory = parts[0] + return TOP_LEVEL_NAME_EXCEPTIONS.get( + directory, + directory if directory.startswith("codex-") else f"codex-{directory}", + ) + if len(parts) == 3 and parts[0] == "utils" and parts[2] == "Cargo.toml": + directory = parts[1] + return UTILITY_NAME_EXCEPTIONS.get(directory, f"codex-utils-{directory}") + return None + + +def is_workspace_reference(value: object) -> bool: + return isinstance(value, dict) and value.get("workspace") is True + + +def load_manifest(path: Path) -> dict: + return tomllib.loads(path.read_text()) + + +def cargo_manifests() -> list[Path]: + return sorted( + path + for path in CARGO_RS_ROOT.rglob("Cargo.toml") + if path != CARGO_RS_ROOT / "Cargo.toml" + ) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c53900401..d32a8fd0a1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,6 +14,9 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - name: Verify codex-rs Cargo manifests inherit workspace settings + run: python3 .github/scripts/verify_cargo_workspace_manifests.py + - name: Setup pnpm uses: pnpm/action-setup@a8198c4bff370c8506180b035930dea56dbd5288 # v5 with: diff --git a/.github/workflows/rust-ci-full.yml b/.github/workflows/rust-ci-full.yml index a09c7fd433..6d36b5f2eb 100644 --- a/.github/workflows/rust-ci-full.yml +++ b/.github/workflows/rust-ci-full.yml @@ -99,23 +99,13 @@ jobs: run: | sudo DEBIAN_FRONTEND=noninteractive apt-get update sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends pkg-config libcap-dev - - name: Install nightly argument-comment-lint toolchain - if: ${{ runner.os == 'Windows' }} - shell: bash - run: | - rustup toolchain install nightly-2025-09-18 \ - --profile minimal \ - --component llvm-tools-preview \ - --component rustc-dev \ - --component rust-src \ - --no-self-update - rustup default nightly-2025-09-18 - name: Run argument comment lint on codex-rs via Bazel if: ${{ runner.os != 'Windows' }} env: BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }} shell: bash run: | + bazel_targets="$(./tools/argument-comment-lint/list-bazel-targets.sh)" ./.github/scripts/run-bazel-ci.sh \ -- \ build \ @@ -123,11 +113,18 @@ jobs: --keep_going \ --build_metadata=COMMIT_SHA=${GITHUB_SHA} \ -- \ - //codex-rs/... - - name: Run argument comment lint on codex-rs via packaged wrapper + ${bazel_targets} + - name: Run argument comment lint on codex-rs via Bazel if: ${{ runner.os == 'Windows' }} + env: + BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }} shell: bash - run: python3 ./tools/argument-comment-lint/run-prebuilt-linter.py + run: | + ./.github/scripts/run-argument-comment-lint-bazel.sh \ + --config=argument-comment-lint \ + --platforms=//:local_windows \ + --keep_going \ + --build_metadata=COMMIT_SHA=${GITHUB_SHA} # --- CI to validate on different os/targets -------------------------------- lint_build: diff --git a/.github/workflows/rust-ci.yml b/.github/workflows/rust-ci.yml index 2594fe7750..ba8c64764a 100644 --- a/.github/workflows/rust-ci.yml +++ b/.github/workflows/rust-ci.yml @@ -159,23 +159,13 @@ jobs: run: | sudo DEBIAN_FRONTEND=noninteractive apt-get update sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends pkg-config libcap-dev - - name: Install nightly argument-comment-lint toolchain - if: ${{ runner.os == 'Windows' }} - shell: bash - run: | - rustup toolchain install nightly-2025-09-18 \ - --profile minimal \ - --component llvm-tools-preview \ - --component rustc-dev \ - --component rust-src \ - --no-self-update - rustup default nightly-2025-09-18 - name: Run argument comment lint on codex-rs via Bazel if: ${{ runner.os != 'Windows' }} env: BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }} shell: bash run: | + bazel_targets="$(./tools/argument-comment-lint/list-bazel-targets.sh)" ./.github/scripts/run-bazel-ci.sh \ -- \ build \ @@ -183,11 +173,18 @@ jobs: --keep_going \ --build_metadata=COMMIT_SHA=${GITHUB_SHA} \ -- \ - //codex-rs/... - - name: Run argument comment lint on codex-rs via packaged wrapper + ${bazel_targets} + - name: Run argument comment lint on codex-rs via Bazel if: ${{ runner.os == 'Windows' }} + env: + BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }} shell: bash - run: python3 ./tools/argument-comment-lint/run-prebuilt-linter.py + run: | + ./.github/scripts/run-argument-comment-lint-bazel.sh \ + --config=argument-comment-lint \ + --platforms=//:local_windows \ + --keep_going \ + --build_metadata=COMMIT_SHA=${GITHUB_SHA} # --- Gatherer job that you mark as the ONLY required status ----------------- results: diff --git a/AGENTS.md b/AGENTS.md index ff36b0e19c..f5065f29b1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,7 @@ In the codex-rs folder where the rust code lives: - Do not add these comments for string or char literals unless the comment adds real clarity; those literals are intentionally exempt from the lint. - If you add one of these comments, the parameter name must exactly match the callee signature. - When possible, make `match` statements exhaustive and avoid wildcard arms. +- Newly added traits should include doc comments that explain their role and how implementations are expected to use them. - When writing tests, prefer comparing the equality of entire objects over fields one by one. - When making a change that adds or changes an API, ensure that the documentation in the `docs/` folder is up to date if applicable. - If you change `ConfigToml` or nested config types, run `just write-config-schema` to update `codex-rs/core/config.schema.json`. diff --git a/BUILD.bazel b/BUILD.bazel index 0be4b711e6..3f59ff1160 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -17,12 +17,19 @@ platform( platform( name = "local_windows", constraint_values = [ - # We just need to pick one of the ABIs. Do the same one we target. "@rules_rs//rs/experimental/platforms/constraints:windows_gnullvm", ], parents = ["@platforms//host"], ) +platform( + name = "local_windows_msvc", + constraint_values = [ + "@rules_rs//rs/experimental/platforms/constraints:windows_msvc", + ], + parents = ["@platforms//host"], +) + alias( name = "rbe", actual = "@rbe_platform", diff --git a/MODULE.bazel b/MODULE.bazel index e71f92ca77..d1a32df5f3 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -82,6 +82,13 @@ rules_rust = use_extension("@rules_rs//rs/experimental:rules_rust.bzl", "rules_r rules_rust.patch( patches = [ "//patches:rules_rust_windows_gnullvm_build_script.patch", + "//patches:rules_rust_windows_exec_msvc_build_script_env.patch", + "//patches:rules_rust_windows_bootstrap_process_wrapper_linker.patch", + "//patches:rules_rust_windows_msvc_direct_link_args.patch", + "//patches:rules_rust_windows_exec_bin_target.patch", + "//patches:rules_rust_windows_exec_std.patch", + "//patches:rules_rust_windows_exec_rustc_dev_rlib.patch", + "//patches:rules_rust_repository_set_exec_constraints.patch", ], strip = 1, ) @@ -96,6 +103,35 @@ nightly_rust.toolchain( dev_components = True, edition = "2024", ) +# Keep Windows exec tools on MSVC so Bazel helper binaries link correctly, but +# lint crate targets as `windows-gnullvm` to preserve the repo's actual cfgs. +nightly_rust.repository_set( + name = "rust_windows_x86_64", + dev_components = True, + edition = "2024", + exec_triple = "x86_64-pc-windows-msvc", + exec_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@rules_rs//rs/experimental/platforms/constraints:windows_msvc", + ], + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@rules_rs//rs/experimental/platforms/constraints:windows_msvc", + ], + target_triple = "x86_64-pc-windows-msvc", + versions = ["nightly/2025-09-18"], +) +nightly_rust.repository_set( + name = "rust_windows_x86_64", + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@rules_rs//rs/experimental/platforms/constraints:windows_gnullvm", + ], + target_triple = "x86_64-pc-windows-gnullvm", +) use_repo(nightly_rust, "rust_toolchains") toolchains = use_extension("@rules_rs//rs/experimental/toolchains:module_extension.bzl", "toolchains") @@ -163,6 +199,8 @@ crate.annotation( patch_args = ["-p1"], patches = [ "//patches:aws-lc-sys_memcmp_check.patch", + "//patches:aws-lc-sys_windows_msvc_prebuilt_nasm.patch", + "//patches:aws-lc-sys_windows_msvc_memcmp_probe.patch", ], ) diff --git a/codex-rs/Cargo.lock b/codex-rs/Cargo.lock index 3dd569c6b0..967b020d18 100644 --- a/codex-rs/Cargo.lock +++ b/codex-rs/Cargo.lock @@ -1337,10 +1337,12 @@ checksum = "e9b18233253483ce2f65329a24072ec414db782531bdbb7d0bbc4bd2ce6b7e21" name = "codex-analytics" version = "0.0.0" dependencies = [ + "codex-app-server-protocol", "codex-git-utils", "codex-login", "codex-plugin", "codex-protocol", + "os_info", "pretty_assertions", "serde", "serde_json", @@ -2613,6 +2615,7 @@ dependencies = [ name = "codex-tools" version = "0.0.0" dependencies = [ + "codex-app-server-protocol", "codex-code-mode", "codex-protocol", "pretty_assertions", diff --git a/codex-rs/analytics/Cargo.toml b/codex-rs/analytics/Cargo.toml index 9633eebb14..2239c4bc36 100644 --- a/codex-rs/analytics/Cargo.toml +++ b/codex-rs/analytics/Cargo.toml @@ -13,10 +13,12 @@ path = "src/lib.rs" workspace = true [dependencies] +codex-app-server-protocol = { workspace = true } codex-git-utils = { workspace = true } codex-login = { workspace = true } codex-plugin = { workspace = true } codex-protocol = { workspace = true } +os_info = { workspace = true } serde = { workspace = true, features = ["derive"] } sha1 = { workspace = true } tokio = { workspace = true, features = [ diff --git a/codex-rs/analytics/src/analytics_client.rs b/codex-rs/analytics/src/analytics_client.rs deleted file mode 100644 index ac8d10b6c1..0000000000 --- a/codex-rs/analytics/src/analytics_client.rs +++ /dev/null @@ -1,797 +0,0 @@ -use codex_git_utils::collect_git_info; -use codex_git_utils::get_git_repo_root; -use codex_login::AuthManager; -use codex_login::default_client::create_client; -use codex_login::default_client::originator; -use codex_plugin::PluginTelemetryMetadata; -use codex_protocol::protocol::SkillScope; -use serde::Serialize; -use sha1::Digest; -use sha1::Sha1; -use std::collections::HashSet; -use std::path::Path; -use std::path::PathBuf; -use std::sync::Arc; -use std::sync::Mutex; -use std::time::Duration; -use tokio::sync::mpsc; - -#[derive(Clone)] -pub struct TrackEventsContext { - pub model_slug: String, - pub thread_id: String, - pub turn_id: String, -} - -pub fn build_track_events_context( - model_slug: String, - thread_id: String, - turn_id: String, -) -> TrackEventsContext { - TrackEventsContext { - model_slug, - thread_id, - turn_id, - } -} - -#[derive(Clone, Debug)] -pub struct SkillInvocation { - pub skill_name: String, - pub skill_scope: SkillScope, - pub skill_path: PathBuf, - pub invocation_type: InvocationType, -} - -#[derive(Clone, Copy, Debug, Serialize)] -#[serde(rename_all = "lowercase")] -pub enum InvocationType { - Explicit, - Implicit, -} - -pub struct AppInvocation { - pub connector_id: Option, - pub app_name: Option, - pub invocation_type: Option, -} - -#[derive(Clone)] -pub(crate) struct AnalyticsEventsQueue { - sender: mpsc::Sender, - app_used_emitted_keys: Arc>>, - plugin_used_emitted_keys: Arc>>, -} - -#[derive(Clone)] -pub struct AnalyticsEventsClient { - queue: AnalyticsEventsQueue, - analytics_enabled: Option, -} - -impl AnalyticsEventsQueue { - pub(crate) fn new(auth_manager: Arc, base_url: String) -> Self { - let (sender, mut receiver) = mpsc::channel(ANALYTICS_EVENTS_QUEUE_SIZE); - tokio::spawn(async move { - while let Some(job) = receiver.recv().await { - match job { - TrackEventsJob::SkillInvocations(job) => { - send_track_skill_invocations(&auth_manager, &base_url, job).await; - } - TrackEventsJob::AppMentioned(job) => { - send_track_app_mentioned(&auth_manager, &base_url, job).await; - } - TrackEventsJob::AppUsed(job) => { - send_track_app_used(&auth_manager, &base_url, job).await; - } - TrackEventsJob::PluginUsed(job) => { - send_track_plugin_used(&auth_manager, &base_url, job).await; - } - TrackEventsJob::PluginInstalled(job) => { - send_track_plugin_installed(&auth_manager, &base_url, job).await; - } - TrackEventsJob::PluginUninstalled(job) => { - send_track_plugin_uninstalled(&auth_manager, &base_url, job).await; - } - TrackEventsJob::PluginEnabled(job) => { - send_track_plugin_enabled(&auth_manager, &base_url, job).await; - } - TrackEventsJob::PluginDisabled(job) => { - send_track_plugin_disabled(&auth_manager, &base_url, job).await; - } - } - } - }); - Self { - sender, - app_used_emitted_keys: Arc::new(Mutex::new(HashSet::new())), - plugin_used_emitted_keys: Arc::new(Mutex::new(HashSet::new())), - } - } - - fn try_send(&self, job: TrackEventsJob) { - if self.sender.try_send(job).is_err() { - //TODO: add a metric for this - tracing::warn!("dropping analytics events: queue is full"); - } - } - - fn should_enqueue_app_used(&self, tracking: &TrackEventsContext, app: &AppInvocation) -> bool { - let Some(connector_id) = app.connector_id.as_ref() else { - return true; - }; - let mut emitted = self - .app_used_emitted_keys - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - if emitted.len() >= ANALYTICS_EVENT_DEDUPE_MAX_KEYS { - emitted.clear(); - } - emitted.insert((tracking.turn_id.clone(), connector_id.clone())) - } - - fn should_enqueue_plugin_used( - &self, - tracking: &TrackEventsContext, - plugin: &PluginTelemetryMetadata, - ) -> bool { - let mut emitted = self - .plugin_used_emitted_keys - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - if emitted.len() >= ANALYTICS_EVENT_DEDUPE_MAX_KEYS { - emitted.clear(); - } - emitted.insert((tracking.turn_id.clone(), plugin.plugin_id.as_key())) - } -} - -impl AnalyticsEventsClient { - pub fn new( - auth_manager: Arc, - base_url: String, - analytics_enabled: Option, - ) -> Self { - Self { - queue: AnalyticsEventsQueue::new(Arc::clone(&auth_manager), base_url), - analytics_enabled, - } - } - - pub fn track_skill_invocations( - &self, - tracking: TrackEventsContext, - invocations: Vec, - ) { - track_skill_invocations( - &self.queue, - self.analytics_enabled, - Some(tracking), - invocations, - ); - } - - pub fn track_app_mentioned(&self, tracking: TrackEventsContext, mentions: Vec) { - track_app_mentioned( - &self.queue, - self.analytics_enabled, - Some(tracking), - mentions, - ); - } - - pub fn track_app_used(&self, tracking: TrackEventsContext, app: AppInvocation) { - track_app_used(&self.queue, self.analytics_enabled, Some(tracking), app); - } - - pub fn track_plugin_used(&self, tracking: TrackEventsContext, plugin: PluginTelemetryMetadata) { - track_plugin_used(&self.queue, self.analytics_enabled, Some(tracking), plugin); - } - - pub fn track_plugin_installed(&self, plugin: PluginTelemetryMetadata) { - track_plugin_management( - &self.queue, - self.analytics_enabled, - PluginManagementEventType::Installed, - plugin, - ); - } - - pub fn track_plugin_uninstalled(&self, plugin: PluginTelemetryMetadata) { - track_plugin_management( - &self.queue, - self.analytics_enabled, - PluginManagementEventType::Uninstalled, - plugin, - ); - } - - pub fn track_plugin_enabled(&self, plugin: PluginTelemetryMetadata) { - track_plugin_management( - &self.queue, - self.analytics_enabled, - PluginManagementEventType::Enabled, - plugin, - ); - } - - pub fn track_plugin_disabled(&self, plugin: PluginTelemetryMetadata) { - track_plugin_management( - &self.queue, - self.analytics_enabled, - PluginManagementEventType::Disabled, - plugin, - ); - } -} - -enum TrackEventsJob { - SkillInvocations(TrackSkillInvocationsJob), - AppMentioned(TrackAppMentionedJob), - AppUsed(TrackAppUsedJob), - PluginUsed(TrackPluginUsedJob), - PluginInstalled(TrackPluginManagementJob), - PluginUninstalled(TrackPluginManagementJob), - PluginEnabled(TrackPluginManagementJob), - PluginDisabled(TrackPluginManagementJob), -} - -struct TrackSkillInvocationsJob { - analytics_enabled: Option, - tracking: TrackEventsContext, - invocations: Vec, -} - -struct TrackAppMentionedJob { - analytics_enabled: Option, - tracking: TrackEventsContext, - mentions: Vec, -} - -struct TrackAppUsedJob { - analytics_enabled: Option, - tracking: TrackEventsContext, - app: AppInvocation, -} - -struct TrackPluginUsedJob { - analytics_enabled: Option, - tracking: TrackEventsContext, - plugin: PluginTelemetryMetadata, -} - -struct TrackPluginManagementJob { - analytics_enabled: Option, - plugin: PluginTelemetryMetadata, -} - -#[derive(Clone, Copy)] -enum PluginManagementEventType { - Installed, - Uninstalled, - Enabled, - Disabled, -} - -const ANALYTICS_EVENTS_QUEUE_SIZE: usize = 256; -const ANALYTICS_EVENTS_TIMEOUT: Duration = Duration::from_secs(10); -const ANALYTICS_EVENT_DEDUPE_MAX_KEYS: usize = 4096; - -#[derive(Serialize)] -struct TrackEventsRequest { - events: Vec, -} - -#[derive(Serialize)] -#[serde(untagged)] -enum TrackEventRequest { - SkillInvocation(SkillInvocationEventRequest), - AppMentioned(CodexAppMentionedEventRequest), - AppUsed(CodexAppUsedEventRequest), - PluginUsed(CodexPluginUsedEventRequest), - PluginInstalled(CodexPluginEventRequest), - PluginUninstalled(CodexPluginEventRequest), - PluginEnabled(CodexPluginEventRequest), - PluginDisabled(CodexPluginEventRequest), -} - -#[derive(Serialize)] -struct SkillInvocationEventRequest { - event_type: &'static str, - skill_id: String, - skill_name: String, - event_params: SkillInvocationEventParams, -} - -#[derive(Serialize)] -struct SkillInvocationEventParams { - product_client_id: Option, - skill_scope: Option, - repo_url: Option, - thread_id: Option, - invoke_type: Option, - model_slug: Option, -} - -#[derive(Serialize)] -struct CodexAppMetadata { - connector_id: Option, - thread_id: Option, - turn_id: Option, - app_name: Option, - product_client_id: Option, - invoke_type: Option, - model_slug: Option, -} - -#[derive(Serialize)] -struct CodexAppMentionedEventRequest { - event_type: &'static str, - event_params: CodexAppMetadata, -} - -#[derive(Serialize)] -struct CodexAppUsedEventRequest { - event_type: &'static str, - event_params: CodexAppMetadata, -} - -#[derive(Serialize)] -struct CodexPluginMetadata { - plugin_id: Option, - plugin_name: Option, - marketplace_name: Option, - has_skills: Option, - mcp_server_count: Option, - connector_ids: Option>, - product_client_id: Option, -} - -#[derive(Serialize)] -struct CodexPluginUsedMetadata { - #[serde(flatten)] - plugin: CodexPluginMetadata, - thread_id: Option, - turn_id: Option, - model_slug: Option, -} - -#[derive(Serialize)] -struct CodexPluginEventRequest { - event_type: &'static str, - event_params: CodexPluginMetadata, -} - -#[derive(Serialize)] -struct CodexPluginUsedEventRequest { - event_type: &'static str, - event_params: CodexPluginUsedMetadata, -} - -pub(crate) fn track_skill_invocations( - queue: &AnalyticsEventsQueue, - analytics_enabled: Option, - tracking: Option, - invocations: Vec, -) { - if analytics_enabled == Some(false) { - return; - } - let Some(tracking) = tracking else { - return; - }; - if invocations.is_empty() { - return; - } - let job = TrackEventsJob::SkillInvocations(TrackSkillInvocationsJob { - analytics_enabled, - tracking, - invocations, - }); - queue.try_send(job); -} - -pub(crate) fn track_app_mentioned( - queue: &AnalyticsEventsQueue, - analytics_enabled: Option, - tracking: Option, - mentions: Vec, -) { - if analytics_enabled == Some(false) { - return; - } - let Some(tracking) = tracking else { - return; - }; - if mentions.is_empty() { - return; - } - let job = TrackEventsJob::AppMentioned(TrackAppMentionedJob { - analytics_enabled, - tracking, - mentions, - }); - queue.try_send(job); -} - -pub(crate) fn track_app_used( - queue: &AnalyticsEventsQueue, - analytics_enabled: Option, - tracking: Option, - app: AppInvocation, -) { - if analytics_enabled == Some(false) { - return; - } - let Some(tracking) = tracking else { - return; - }; - if !queue.should_enqueue_app_used(&tracking, &app) { - return; - } - let job = TrackEventsJob::AppUsed(TrackAppUsedJob { - analytics_enabled, - tracking, - app, - }); - queue.try_send(job); -} - -pub(crate) fn track_plugin_used( - queue: &AnalyticsEventsQueue, - analytics_enabled: Option, - tracking: Option, - plugin: PluginTelemetryMetadata, -) { - if analytics_enabled == Some(false) { - return; - } - let Some(tracking) = tracking else { - return; - }; - if !queue.should_enqueue_plugin_used(&tracking, &plugin) { - return; - } - let job = TrackEventsJob::PluginUsed(TrackPluginUsedJob { - analytics_enabled, - tracking, - plugin, - }); - queue.try_send(job); -} - -fn track_plugin_management( - queue: &AnalyticsEventsQueue, - analytics_enabled: Option, - event_type: PluginManagementEventType, - plugin: PluginTelemetryMetadata, -) { - if analytics_enabled == Some(false) { - return; - } - let job = TrackPluginManagementJob { - analytics_enabled, - plugin, - }; - let job = match event_type { - PluginManagementEventType::Installed => TrackEventsJob::PluginInstalled(job), - PluginManagementEventType::Uninstalled => TrackEventsJob::PluginUninstalled(job), - PluginManagementEventType::Enabled => TrackEventsJob::PluginEnabled(job), - PluginManagementEventType::Disabled => TrackEventsJob::PluginDisabled(job), - }; - queue.try_send(job); -} - -async fn send_track_skill_invocations( - auth_manager: &AuthManager, - base_url: &str, - job: TrackSkillInvocationsJob, -) { - let TrackSkillInvocationsJob { - analytics_enabled, - tracking, - invocations, - } = job; - let mut events = Vec::with_capacity(invocations.len()); - for invocation in invocations { - let skill_scope = match invocation.skill_scope { - SkillScope::User => "user", - SkillScope::Repo => "repo", - SkillScope::System => "system", - SkillScope::Admin => "admin", - }; - let repo_root = get_git_repo_root(invocation.skill_path.as_path()); - let repo_url = if let Some(root) = repo_root.as_ref() { - collect_git_info(root) - .await - .and_then(|info| info.repository_url) - } else { - None - }; - let skill_id = skill_id_for_local_skill( - repo_url.as_deref(), - repo_root.as_deref(), - invocation.skill_path.as_path(), - invocation.skill_name.as_str(), - ); - events.push(TrackEventRequest::SkillInvocation( - SkillInvocationEventRequest { - event_type: "skill_invocation", - skill_id, - skill_name: invocation.skill_name.clone(), - event_params: SkillInvocationEventParams { - thread_id: Some(tracking.thread_id.clone()), - invoke_type: Some(invocation.invocation_type), - model_slug: Some(tracking.model_slug.clone()), - product_client_id: Some(originator().value), - repo_url, - skill_scope: Some(skill_scope.to_string()), - }, - }, - )); - } - - send_track_events(auth_manager, analytics_enabled, base_url, events).await; -} - -async fn send_track_app_mentioned( - auth_manager: &AuthManager, - base_url: &str, - job: TrackAppMentionedJob, -) { - let TrackAppMentionedJob { - analytics_enabled, - tracking, - mentions, - } = job; - let events = mentions - .into_iter() - .map(|mention| { - let event_params = codex_app_metadata(&tracking, mention); - TrackEventRequest::AppMentioned(CodexAppMentionedEventRequest { - event_type: "codex_app_mentioned", - event_params, - }) - }) - .collect::>(); - - send_track_events(auth_manager, analytics_enabled, base_url, events).await; -} - -async fn send_track_app_used(auth_manager: &AuthManager, base_url: &str, job: TrackAppUsedJob) { - let TrackAppUsedJob { - analytics_enabled, - tracking, - app, - } = job; - let event_params = codex_app_metadata(&tracking, app); - let events = vec![TrackEventRequest::AppUsed(CodexAppUsedEventRequest { - event_type: "codex_app_used", - event_params, - })]; - - send_track_events(auth_manager, analytics_enabled, base_url, events).await; -} - -async fn send_track_plugin_used( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginUsedJob, -) { - let TrackPluginUsedJob { - analytics_enabled, - tracking, - plugin, - } = job; - let events = vec![TrackEventRequest::PluginUsed(CodexPluginUsedEventRequest { - event_type: "codex_plugin_used", - event_params: codex_plugin_used_metadata(&tracking, plugin), - })]; - - send_track_events(auth_manager, analytics_enabled, base_url, events).await; -} - -async fn send_track_plugin_installed( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginManagementJob, -) { - send_track_plugin_management_event(auth_manager, base_url, job, "codex_plugin_installed").await; -} - -async fn send_track_plugin_uninstalled( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginManagementJob, -) { - send_track_plugin_management_event(auth_manager, base_url, job, "codex_plugin_uninstalled") - .await; -} - -async fn send_track_plugin_enabled( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginManagementJob, -) { - send_track_plugin_management_event(auth_manager, base_url, job, "codex_plugin_enabled").await; -} - -async fn send_track_plugin_disabled( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginManagementJob, -) { - send_track_plugin_management_event(auth_manager, base_url, job, "codex_plugin_disabled").await; -} - -async fn send_track_plugin_management_event( - auth_manager: &AuthManager, - base_url: &str, - job: TrackPluginManagementJob, - event_type: &'static str, -) { - let TrackPluginManagementJob { - analytics_enabled, - plugin, - } = job; - let event_params = codex_plugin_metadata(plugin); - let event = CodexPluginEventRequest { - event_type, - event_params, - }; - let events = vec![match event_type { - "codex_plugin_installed" => TrackEventRequest::PluginInstalled(event), - "codex_plugin_uninstalled" => TrackEventRequest::PluginUninstalled(event), - "codex_plugin_enabled" => TrackEventRequest::PluginEnabled(event), - "codex_plugin_disabled" => TrackEventRequest::PluginDisabled(event), - _ => unreachable!("unknown plugin management event type"), - }]; - - send_track_events(auth_manager, analytics_enabled, base_url, events).await; -} - -fn codex_app_metadata(tracking: &TrackEventsContext, app: AppInvocation) -> CodexAppMetadata { - CodexAppMetadata { - connector_id: app.connector_id, - thread_id: Some(tracking.thread_id.clone()), - turn_id: Some(tracking.turn_id.clone()), - app_name: app.app_name, - product_client_id: Some(originator().value), - invoke_type: app.invocation_type, - model_slug: Some(tracking.model_slug.clone()), - } -} - -fn codex_plugin_metadata(plugin: PluginTelemetryMetadata) -> CodexPluginMetadata { - let capability_summary = plugin.capability_summary; - CodexPluginMetadata { - plugin_id: Some(plugin.plugin_id.as_key()), - plugin_name: Some(plugin.plugin_id.plugin_name), - marketplace_name: Some(plugin.plugin_id.marketplace_name), - has_skills: capability_summary - .as_ref() - .map(|summary| summary.has_skills), - mcp_server_count: capability_summary - .as_ref() - .map(|summary| summary.mcp_server_names.len()), - connector_ids: capability_summary.map(|summary| { - summary - .app_connector_ids - .into_iter() - .map(|connector_id| connector_id.0) - .collect() - }), - product_client_id: Some(originator().value), - } -} - -fn codex_plugin_used_metadata( - tracking: &TrackEventsContext, - plugin: PluginTelemetryMetadata, -) -> CodexPluginUsedMetadata { - CodexPluginUsedMetadata { - plugin: codex_plugin_metadata(plugin), - thread_id: Some(tracking.thread_id.clone()), - turn_id: Some(tracking.turn_id.clone()), - model_slug: Some(tracking.model_slug.clone()), - } -} - -async fn send_track_events( - auth_manager: &AuthManager, - analytics_enabled: Option, - base_url: &str, - events: Vec, -) { - if analytics_enabled == Some(false) { - return; - } - if events.is_empty() { - return; - } - let Some(auth) = auth_manager.auth().await else { - return; - }; - if !auth.is_chatgpt_auth() { - return; - } - let access_token = match auth.get_token() { - Ok(token) => token, - Err(_) => return, - }; - let Some(account_id) = auth.get_account_id() else { - return; - }; - - let base_url = base_url.trim_end_matches('/'); - let url = format!("{base_url}/codex/analytics-events/events"); - let payload = TrackEventsRequest { events }; - - let response = create_client() - .post(&url) - .timeout(ANALYTICS_EVENTS_TIMEOUT) - .bearer_auth(&access_token) - .header("chatgpt-account-id", &account_id) - .header("Content-Type", "application/json") - .json(&payload) - .send() - .await; - - match response { - Ok(response) if response.status().is_success() => {} - Ok(response) => { - let status = response.status(); - let body = response.text().await.unwrap_or_default(); - tracing::warn!("events failed with status {status}: {body}"); - } - Err(err) => { - tracing::warn!("failed to send events request: {err}"); - } - } -} - -pub(crate) fn skill_id_for_local_skill( - repo_url: Option<&str>, - repo_root: Option<&Path>, - skill_path: &Path, - skill_name: &str, -) -> String { - let path = normalize_path_for_skill_id(repo_url, repo_root, skill_path); - let prefix = if let Some(url) = repo_url { - format!("repo_{url}") - } else { - "personal".to_string() - }; - let raw_id = format!("{prefix}_{path}_{skill_name}"); - let mut hasher = Sha1::new(); - hasher.update(raw_id.as_bytes()); - format!("{:x}", hasher.finalize()) -} - -/// Returns a normalized path for skill ID construction. -/// -/// - Repo-scoped skills use a path relative to the repo root. -/// - User/admin/system skills use an absolute path. -fn normalize_path_for_skill_id( - repo_url: Option<&str>, - repo_root: Option<&Path>, - skill_path: &Path, -) -> String { - let resolved_path = - std::fs::canonicalize(skill_path).unwrap_or_else(|_| skill_path.to_path_buf()); - match (repo_url, repo_root) { - (Some(_), Some(root)) => { - let resolved_root = std::fs::canonicalize(root).unwrap_or_else(|_| root.to_path_buf()); - resolved_path - .strip_prefix(&resolved_root) - .unwrap_or(resolved_path.as_path()) - .to_string_lossy() - .replace('\\', "/") - } - _ => resolved_path.to_string_lossy().replace('\\', "/"), - } -} - -#[cfg(test)] -#[path = "analytics_client_tests.rs"] -mod tests; diff --git a/codex-rs/analytics/src/analytics_client_tests.rs b/codex-rs/analytics/src/analytics_client_tests.rs index 4f49a7c713..3a98fbb837 100644 --- a/codex-rs/analytics/src/analytics_client_tests.rs +++ b/codex-rs/analytics/src/analytics_client_tests.rs @@ -1,16 +1,47 @@ -use super::AnalyticsEventsQueue; -use super::AppInvocation; -use super::CodexAppMentionedEventRequest; -use super::CodexAppUsedEventRequest; -use super::CodexPluginEventRequest; -use super::CodexPluginUsedEventRequest; -use super::InvocationType; -use super::TrackEventRequest; -use super::TrackEventsContext; -use super::codex_app_metadata; -use super::codex_plugin_metadata; -use super::codex_plugin_used_metadata; -use super::normalize_path_for_skill_id; +use crate::client::AnalyticsEventsQueue; +use crate::events::AppServerRpcTransport; +use crate::events::CodexAppMentionedEventRequest; +use crate::events::CodexAppServerClientMetadata; +use crate::events::CodexAppUsedEventRequest; +use crate::events::CodexPluginEventRequest; +use crate::events::CodexPluginUsedEventRequest; +use crate::events::CodexRuntimeMetadata; +use crate::events::ThreadInitializationMode; +use crate::events::ThreadInitializedEvent; +use crate::events::ThreadInitializedEventParams; +use crate::events::TrackEventRequest; +use crate::events::codex_app_metadata; +use crate::events::codex_plugin_metadata; +use crate::events::codex_plugin_used_metadata; +use crate::facts::AnalyticsFact; +use crate::facts::AppInvocation; +use crate::facts::AppMentionedInput; +use crate::facts::AppUsedInput; +use crate::facts::CustomAnalyticsFact; +use crate::facts::InvocationType; +use crate::facts::PluginState; +use crate::facts::PluginStateChangedInput; +use crate::facts::PluginUsedInput; +use crate::facts::SkillInvocation; +use crate::facts::SkillInvokedInput; +use crate::facts::TrackEventsContext; +use crate::reducer::AnalyticsReducer; +use crate::reducer::normalize_path_for_skill_id; +use crate::reducer::skill_id_for_local_skill; +use codex_app_server_protocol::ApprovalsReviewer as AppServerApprovalsReviewer; +use codex_app_server_protocol::AskForApproval as AppServerAskForApproval; +use codex_app_server_protocol::ClientInfo; +use codex_app_server_protocol::ClientResponse; +use codex_app_server_protocol::InitializeCapabilities; +use codex_app_server_protocol::InitializeParams; +use codex_app_server_protocol::RequestId; +use codex_app_server_protocol::SandboxPolicy as AppServerSandboxPolicy; +use codex_app_server_protocol::SessionSource as AppServerSessionSource; +use codex_app_server_protocol::Thread; +use codex_app_server_protocol::ThreadResumeResponse; +use codex_app_server_protocol::ThreadStartResponse; +use codex_app_server_protocol::ThreadStatus as AppServerThreadStatus; +use codex_login::default_client::DEFAULT_ORIGINATOR; use codex_login::default_client::originator; use codex_plugin::AppConnectorId; use codex_plugin::PluginCapabilitySummary; @@ -24,6 +55,61 @@ use std::sync::Arc; use std::sync::Mutex; use tokio::sync::mpsc; +fn sample_thread(thread_id: &str, ephemeral: bool) -> Thread { + Thread { + id: thread_id.to_string(), + preview: "first prompt".to_string(), + ephemeral, + model_provider: "openai".to_string(), + created_at: 1, + updated_at: 2, + status: AppServerThreadStatus::Idle, + path: None, + cwd: PathBuf::from("/tmp"), + cli_version: "0.0.0".to_string(), + source: AppServerSessionSource::Exec, + agent_nickname: None, + agent_role: None, + git_info: None, + name: None, + turns: Vec::new(), + } +} + +fn sample_thread_start_response(thread_id: &str, ephemeral: bool, model: &str) -> ClientResponse { + ClientResponse::ThreadStart { + request_id: RequestId::Integer(1), + response: ThreadStartResponse { + thread: sample_thread(thread_id, ephemeral), + model: model.to_string(), + model_provider: "openai".to_string(), + service_tier: None, + cwd: PathBuf::from("/tmp"), + approval_policy: AppServerAskForApproval::OnFailure, + approvals_reviewer: AppServerApprovalsReviewer::User, + sandbox: AppServerSandboxPolicy::DangerFullAccess, + reasoning_effort: None, + }, + } +} + +fn sample_thread_resume_response(thread_id: &str, ephemeral: bool, model: &str) -> ClientResponse { + ClientResponse::ThreadResume { + request_id: RequestId::Integer(2), + response: ThreadResumeResponse { + thread: sample_thread(thread_id, ephemeral), + model: model.to_string(), + model_provider: "openai".to_string(), + service_tier: None, + cwd: PathBuf::from("/tmp"), + approval_policy: AppServerAskForApproval::OnFailure, + approvals_reviewer: AppServerApprovalsReviewer::User, + sandbox: AppServerSandboxPolicy::DangerFullAccess, + reasoning_effort: None, + }, + } +} + fn expected_absolute_path(path: &PathBuf) -> String { std::fs::canonicalize(path) .unwrap_or_else(|_| path.to_path_buf()) @@ -194,6 +280,171 @@ fn app_used_dedupe_is_keyed_by_turn_and_connector() { assert_eq!(queue.should_enqueue_app_used(&turn_2, &app), true); } +#[test] +fn thread_initialized_event_serializes_expected_shape() { + let event = TrackEventRequest::ThreadInitialized(ThreadInitializedEvent { + event_type: "codex_thread_initialized", + event_params: ThreadInitializedEventParams { + thread_id: "thread-0".to_string(), + app_server_client: CodexAppServerClientMetadata { + product_client_id: DEFAULT_ORIGINATOR.to_string(), + client_name: Some("codex-tui".to_string()), + client_version: Some("1.0.0".to_string()), + rpc_transport: AppServerRpcTransport::Stdio, + experimental_api_enabled: Some(true), + }, + runtime: CodexRuntimeMetadata { + codex_rs_version: "0.1.0".to_string(), + runtime_os: "macos".to_string(), + runtime_os_version: "15.3.1".to_string(), + runtime_arch: "aarch64".to_string(), + }, + model: "gpt-5".to_string(), + ephemeral: true, + thread_source: Some("user"), + initialization_mode: ThreadInitializationMode::New, + subagent_source: None, + parent_thread_id: None, + created_at: 1, + }, + }); + + let payload = serde_json::to_value(&event).expect("serialize thread initialized event"); + + assert_eq!( + payload, + json!({ + "event_type": "codex_thread_initialized", + "event_params": { + "thread_id": "thread-0", + "app_server_client": { + "product_client_id": DEFAULT_ORIGINATOR, + "client_name": "codex-tui", + "client_version": "1.0.0", + "rpc_transport": "stdio", + "experimental_api_enabled": true + }, + "runtime": { + "codex_rs_version": "0.1.0", + "runtime_os": "macos", + "runtime_os_version": "15.3.1", + "runtime_arch": "aarch64" + }, + "model": "gpt-5", + "ephemeral": true, + "thread_source": "user", + "initialization_mode": "new", + "subagent_source": null, + "parent_thread_id": null, + "created_at": 1 + } + }) + ); +} + +#[tokio::test] +async fn initialize_caches_client_and_thread_lifecycle_publishes_once_initialized() { + let mut reducer = AnalyticsReducer::default(); + let mut events = Vec::new(); + + reducer + .ingest( + AnalyticsFact::Response { + connection_id: 7, + response: Box::new(sample_thread_start_response( + "thread-no-client", + /*ephemeral*/ false, + "gpt-5", + )), + }, + &mut events, + ) + .await; + assert!(events.is_empty(), "thread events should require initialize"); + + reducer + .ingest( + AnalyticsFact::Initialize { + connection_id: 7, + params: InitializeParams { + client_info: ClientInfo { + name: "codex-tui".to_string(), + title: None, + version: "1.0.0".to_string(), + }, + capabilities: Some(InitializeCapabilities { + experimental_api: false, + opt_out_notification_methods: None, + }), + }, + product_client_id: DEFAULT_ORIGINATOR.to_string(), + runtime: CodexRuntimeMetadata { + codex_rs_version: "0.99.0".to_string(), + runtime_os: "linux".to_string(), + runtime_os_version: "24.04".to_string(), + runtime_arch: "x86_64".to_string(), + }, + rpc_transport: AppServerRpcTransport::Websocket, + }, + &mut events, + ) + .await; + assert!(events.is_empty(), "initialize should not publish by itself"); + + reducer + .ingest( + AnalyticsFact::Response { + connection_id: 7, + response: Box::new(sample_thread_resume_response( + "thread-1", /*ephemeral*/ true, "gpt-5", + )), + }, + &mut events, + ) + .await; + + let payload = serde_json::to_value(&events).expect("serialize events"); + assert_eq!(payload.as_array().expect("events array").len(), 1); + assert_eq!(payload[0]["event_type"], "codex_thread_initialized"); + assert_eq!( + payload[0]["event_params"]["app_server_client"]["product_client_id"], + DEFAULT_ORIGINATOR + ); + assert_eq!( + payload[0]["event_params"]["app_server_client"]["client_name"], + "codex-tui" + ); + assert_eq!( + payload[0]["event_params"]["app_server_client"]["client_version"], + "1.0.0" + ); + assert_eq!( + payload[0]["event_params"]["app_server_client"]["rpc_transport"], + "websocket" + ); + assert_eq!( + payload[0]["event_params"]["app_server_client"]["experimental_api_enabled"], + false + ); + assert_eq!( + payload[0]["event_params"]["runtime"]["codex_rs_version"], + "0.99.0" + ); + assert_eq!(payload[0]["event_params"]["runtime"]["runtime_os"], "linux"); + assert_eq!( + payload[0]["event_params"]["runtime"]["runtime_os_version"], + "24.04" + ); + assert_eq!( + payload[0]["event_params"]["runtime"]["runtime_arch"], + "x86_64" + ); + assert_eq!(payload[0]["event_params"]["initialization_mode"], "resumed"); + assert_eq!(payload[0]["event_params"]["thread_source"], "user"); + assert_eq!(payload[0]["event_params"]["subagent_source"], json!(null)); + assert_eq!(payload[0]["event_params"]["parent_thread_id"], json!(null)); +} + #[test] fn plugin_used_event_serializes_expected_shape() { let tracking = TrackEventsContext { @@ -280,6 +531,145 @@ fn plugin_used_dedupe_is_keyed_by_turn_and_plugin() { assert_eq!(queue.should_enqueue_plugin_used(&turn_2, &plugin), true); } +#[tokio::test] +async fn reducer_ingests_skill_invoked_fact() { + let mut reducer = AnalyticsReducer::default(); + let mut events = Vec::new(); + let tracking = TrackEventsContext { + model_slug: "gpt-5".to_string(), + thread_id: "thread-1".to_string(), + turn_id: "turn-1".to_string(), + }; + let skill_path = PathBuf::from("/Users/abc/.codex/skills/doc/SKILL.md"); + let expected_skill_id = skill_id_for_local_skill( + /*repo_url*/ None, + /*repo_root*/ None, + skill_path.as_path(), + "doc", + ); + + reducer + .ingest( + AnalyticsFact::Custom(CustomAnalyticsFact::SkillInvoked(SkillInvokedInput { + tracking, + invocations: vec![SkillInvocation { + skill_name: "doc".to_string(), + skill_scope: codex_protocol::protocol::SkillScope::User, + skill_path, + invocation_type: InvocationType::Explicit, + }], + })), + &mut events, + ) + .await; + + let payload = serde_json::to_value(&events).expect("serialize events"); + assert_eq!( + payload, + json!([{ + "event_type": "skill_invocation", + "skill_id": expected_skill_id, + "skill_name": "doc", + "event_params": { + "product_client_id": originator().value, + "skill_scope": "user", + "repo_url": null, + "thread_id": "thread-1", + "invoke_type": "explicit", + "model_slug": "gpt-5" + } + }]) + ); +} + +#[tokio::test] +async fn reducer_ingests_app_and_plugin_facts() { + let mut reducer = AnalyticsReducer::default(); + let mut events = Vec::new(); + let tracking = TrackEventsContext { + model_slug: "gpt-5".to_string(), + thread_id: "thread-1".to_string(), + turn_id: "turn-1".to_string(), + }; + + reducer + .ingest( + AnalyticsFact::Custom(CustomAnalyticsFact::AppMentioned(AppMentionedInput { + tracking: tracking.clone(), + mentions: vec![AppInvocation { + connector_id: Some("calendar".to_string()), + app_name: Some("Calendar".to_string()), + invocation_type: Some(InvocationType::Explicit), + }], + })), + &mut events, + ) + .await; + reducer + .ingest( + AnalyticsFact::Custom(CustomAnalyticsFact::AppUsed(AppUsedInput { + tracking: tracking.clone(), + app: AppInvocation { + connector_id: Some("drive".to_string()), + app_name: Some("Drive".to_string()), + invocation_type: Some(InvocationType::Implicit), + }, + })), + &mut events, + ) + .await; + reducer + .ingest( + AnalyticsFact::Custom(CustomAnalyticsFact::PluginUsed(PluginUsedInput { + tracking, + plugin: sample_plugin_metadata(), + })), + &mut events, + ) + .await; + + let payload = serde_json::to_value(&events).expect("serialize events"); + assert_eq!(payload.as_array().expect("events array").len(), 3); + assert_eq!(payload[0]["event_type"], "codex_app_mentioned"); + assert_eq!(payload[1]["event_type"], "codex_app_used"); + assert_eq!(payload[2]["event_type"], "codex_plugin_used"); +} + +#[tokio::test] +async fn reducer_ingests_plugin_state_changed_fact() { + let mut reducer = AnalyticsReducer::default(); + let mut events = Vec::new(); + + reducer + .ingest( + AnalyticsFact::Custom(CustomAnalyticsFact::PluginStateChanged( + PluginStateChangedInput { + plugin: sample_plugin_metadata(), + state: PluginState::Disabled, + }, + )), + &mut events, + ) + .await; + + let payload = serde_json::to_value(&events).expect("serialize events"); + assert_eq!( + payload, + json!([{ + "event_type": "codex_plugin_disabled", + "event_params": { + "plugin_id": "sample@test", + "plugin_name": "sample", + "marketplace_name": "test", + "has_skills": true, + "mcp_server_count": 2, + "connector_ids": ["calendar", "drive"], + "product_client_id": originator().value + } + }]) + ); +} + fn sample_plugin_metadata() -> PluginTelemetryMetadata { PluginTelemetryMetadata { plugin_id: PluginId::parse("sample@test").expect("valid plugin id"), diff --git a/codex-rs/analytics/src/client.rs b/codex-rs/analytics/src/client.rs new file mode 100644 index 0000000000..b8a543a3b2 --- /dev/null +++ b/codex-rs/analytics/src/client.rs @@ -0,0 +1,272 @@ +use crate::events::AppServerRpcTransport; +use crate::events::TrackEventRequest; +use crate::events::TrackEventsRequest; +use crate::events::current_runtime_metadata; +use crate::facts::AnalyticsFact; +use crate::facts::AppInvocation; +use crate::facts::AppMentionedInput; +use crate::facts::AppUsedInput; +use crate::facts::CustomAnalyticsFact; +use crate::facts::PluginState; +use crate::facts::PluginStateChangedInput; +use crate::facts::SkillInvocation; +use crate::facts::SkillInvokedInput; +use crate::facts::TrackEventsContext; +use crate::reducer::AnalyticsReducer; +use codex_app_server_protocol::ClientResponse; +use codex_app_server_protocol::InitializeParams; +use codex_login::AuthManager; +use codex_login::default_client::create_client; +use codex_plugin::PluginTelemetryMetadata; +use std::collections::HashSet; +use std::sync::Arc; +use std::sync::Mutex; +use std::time::Duration; +use tokio::sync::mpsc; + +const ANALYTICS_EVENTS_QUEUE_SIZE: usize = 256; +const ANALYTICS_EVENTS_TIMEOUT: Duration = Duration::from_secs(10); +const ANALYTICS_EVENT_DEDUPE_MAX_KEYS: usize = 4096; + +#[derive(Clone)] +pub(crate) struct AnalyticsEventsQueue { + pub(crate) sender: mpsc::Sender, + pub(crate) app_used_emitted_keys: Arc>>, + pub(crate) plugin_used_emitted_keys: Arc>>, +} + +#[derive(Clone)] +pub struct AnalyticsEventsClient { + queue: AnalyticsEventsQueue, + analytics_enabled: Option, +} + +impl AnalyticsEventsQueue { + pub(crate) fn new(auth_manager: Arc, base_url: String) -> Self { + let (sender, mut receiver) = mpsc::channel(ANALYTICS_EVENTS_QUEUE_SIZE); + tokio::spawn(async move { + let mut reducer = AnalyticsReducer::default(); + while let Some(input) = receiver.recv().await { + let mut events = Vec::new(); + reducer.ingest(input, &mut events).await; + send_track_events(&auth_manager, &base_url, events).await; + } + }); + Self { + sender, + app_used_emitted_keys: Arc::new(Mutex::new(HashSet::new())), + plugin_used_emitted_keys: Arc::new(Mutex::new(HashSet::new())), + } + } + + fn try_send(&self, input: AnalyticsFact) { + if self.sender.try_send(input).is_err() { + //TODO: add a metric for this + tracing::warn!("dropping analytics events: queue is full"); + } + } + + pub(crate) fn should_enqueue_app_used( + &self, + tracking: &TrackEventsContext, + app: &AppInvocation, + ) -> bool { + let Some(connector_id) = app.connector_id.as_ref() else { + return true; + }; + let mut emitted = self + .app_used_emitted_keys + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if emitted.len() >= ANALYTICS_EVENT_DEDUPE_MAX_KEYS { + emitted.clear(); + } + emitted.insert((tracking.turn_id.clone(), connector_id.clone())) + } + + pub(crate) fn should_enqueue_plugin_used( + &self, + tracking: &TrackEventsContext, + plugin: &PluginTelemetryMetadata, + ) -> bool { + let mut emitted = self + .plugin_used_emitted_keys + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if emitted.len() >= ANALYTICS_EVENT_DEDUPE_MAX_KEYS { + emitted.clear(); + } + emitted.insert((tracking.turn_id.clone(), plugin.plugin_id.as_key())) + } +} + +impl AnalyticsEventsClient { + pub fn new( + auth_manager: Arc, + base_url: String, + analytics_enabled: Option, + ) -> Self { + Self { + queue: AnalyticsEventsQueue::new(Arc::clone(&auth_manager), base_url), + analytics_enabled, + } + } + + pub fn track_skill_invocations( + &self, + tracking: TrackEventsContext, + invocations: Vec, + ) { + if invocations.is_empty() { + return; + } + self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::SkillInvoked( + SkillInvokedInput { + tracking, + invocations, + }, + ))); + } + + pub fn track_initialize( + &self, + connection_id: u64, + params: InitializeParams, + product_client_id: String, + rpc_transport: AppServerRpcTransport, + ) { + self.record_fact(AnalyticsFact::Initialize { + connection_id, + params, + product_client_id, + runtime: current_runtime_metadata(), + rpc_transport, + }); + } + + pub fn track_app_mentioned(&self, tracking: TrackEventsContext, mentions: Vec) { + if mentions.is_empty() { + return; + } + self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::AppMentioned( + AppMentionedInput { tracking, mentions }, + ))); + } + + pub fn track_app_used(&self, tracking: TrackEventsContext, app: AppInvocation) { + if !self.queue.should_enqueue_app_used(&tracking, &app) { + return; + } + self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::AppUsed( + AppUsedInput { tracking, app }, + ))); + } + + pub fn track_plugin_used(&self, tracking: TrackEventsContext, plugin: PluginTelemetryMetadata) { + if !self.queue.should_enqueue_plugin_used(&tracking, &plugin) { + return; + } + self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::PluginUsed( + crate::facts::PluginUsedInput { tracking, plugin }, + ))); + } + + pub fn track_plugin_installed(&self, plugin: PluginTelemetryMetadata) { + self.record_fact(AnalyticsFact::Custom( + CustomAnalyticsFact::PluginStateChanged(PluginStateChangedInput { + plugin, + state: PluginState::Installed, + }), + )); + } + + pub fn track_plugin_uninstalled(&self, plugin: PluginTelemetryMetadata) { + self.record_fact(AnalyticsFact::Custom( + CustomAnalyticsFact::PluginStateChanged(PluginStateChangedInput { + plugin, + state: PluginState::Uninstalled, + }), + )); + } + + pub fn track_plugin_enabled(&self, plugin: PluginTelemetryMetadata) { + self.record_fact(AnalyticsFact::Custom( + CustomAnalyticsFact::PluginStateChanged(PluginStateChangedInput { + plugin, + state: PluginState::Enabled, + }), + )); + } + + pub fn track_plugin_disabled(&self, plugin: PluginTelemetryMetadata) { + self.record_fact(AnalyticsFact::Custom( + CustomAnalyticsFact::PluginStateChanged(PluginStateChangedInput { + plugin, + state: PluginState::Disabled, + }), + )); + } + + pub(crate) fn record_fact(&self, input: AnalyticsFact) { + if self.analytics_enabled == Some(false) { + return; + } + self.queue.try_send(input); + } + + pub fn track_response(&self, connection_id: u64, response: ClientResponse) { + self.record_fact(AnalyticsFact::Response { + connection_id, + response: Box::new(response), + }); + } +} + +async fn send_track_events( + auth_manager: &AuthManager, + base_url: &str, + events: Vec, +) { + if events.is_empty() { + return; + } + let Some(auth) = auth_manager.auth().await else { + return; + }; + if !auth.is_chatgpt_auth() { + return; + } + let access_token = match auth.get_token() { + Ok(token) => token, + Err(_) => return, + }; + let Some(account_id) = auth.get_account_id() else { + return; + }; + + let base_url = base_url.trim_end_matches('/'); + let url = format!("{base_url}/codex/analytics-events/events"); + let payload = TrackEventsRequest { events }; + + let response = create_client() + .post(&url) + .timeout(ANALYTICS_EVENTS_TIMEOUT) + .bearer_auth(&access_token) + .header("chatgpt-account-id", &account_id) + .header("Content-Type", "application/json") + .json(&payload) + .send() + .await; + + match response { + Ok(response) if response.status().is_success() => {} + Ok(response) => { + let status = response.status(); + let body = response.text().await.unwrap_or_default(); + tracing::warn!("events failed with status {status}: {body}"); + } + Err(err) => { + tracing::warn!("failed to send events request: {err}"); + } + } +} diff --git a/codex-rs/analytics/src/events.rs b/codex-rs/analytics/src/events.rs new file mode 100644 index 0000000000..36efb01a69 --- /dev/null +++ b/codex-rs/analytics/src/events.rs @@ -0,0 +1,230 @@ +use crate::facts::AppInvocation; +use crate::facts::InvocationType; +use crate::facts::PluginState; +use crate::facts::TrackEventsContext; +use codex_login::default_client::originator; +use codex_plugin::PluginTelemetryMetadata; +use codex_protocol::protocol::SessionSource; +use serde::Serialize; + +#[derive(Clone, Copy, Debug, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AppServerRpcTransport { + Stdio, + Websocket, + InProcess, +} + +#[derive(Clone, Copy, Debug, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum ThreadInitializationMode { + New, + Forked, + Resumed, +} + +#[derive(Serialize)] +pub(crate) struct TrackEventsRequest { + pub(crate) events: Vec, +} + +#[derive(Serialize)] +#[serde(untagged)] +pub(crate) enum TrackEventRequest { + SkillInvocation(SkillInvocationEventRequest), + ThreadInitialized(ThreadInitializedEvent), + AppMentioned(CodexAppMentionedEventRequest), + AppUsed(CodexAppUsedEventRequest), + PluginUsed(CodexPluginUsedEventRequest), + PluginInstalled(CodexPluginEventRequest), + PluginUninstalled(CodexPluginEventRequest), + PluginEnabled(CodexPluginEventRequest), + PluginDisabled(CodexPluginEventRequest), +} + +#[derive(Serialize)] +pub(crate) struct SkillInvocationEventRequest { + pub(crate) event_type: &'static str, + pub(crate) skill_id: String, + pub(crate) skill_name: String, + pub(crate) event_params: SkillInvocationEventParams, +} + +#[derive(Serialize)] +pub(crate) struct SkillInvocationEventParams { + pub(crate) product_client_id: Option, + pub(crate) skill_scope: Option, + pub(crate) repo_url: Option, + pub(crate) thread_id: Option, + pub(crate) invoke_type: Option, + pub(crate) model_slug: Option, +} + +#[derive(Clone, Serialize)] +pub(crate) struct CodexAppServerClientMetadata { + pub(crate) product_client_id: String, + pub(crate) client_name: Option, + pub(crate) client_version: Option, + pub(crate) rpc_transport: AppServerRpcTransport, + pub(crate) experimental_api_enabled: Option, +} + +#[derive(Clone, Serialize)] +pub(crate) struct CodexRuntimeMetadata { + pub(crate) codex_rs_version: String, + pub(crate) runtime_os: String, + pub(crate) runtime_os_version: String, + pub(crate) runtime_arch: String, +} + +#[derive(Serialize)] +pub(crate) struct ThreadInitializedEventParams { + pub(crate) thread_id: String, + pub(crate) app_server_client: CodexAppServerClientMetadata, + pub(crate) runtime: CodexRuntimeMetadata, + pub(crate) model: String, + pub(crate) ephemeral: bool, + pub(crate) thread_source: Option<&'static str>, + pub(crate) initialization_mode: ThreadInitializationMode, + pub(crate) subagent_source: Option, + pub(crate) parent_thread_id: Option, + pub(crate) created_at: u64, +} + +#[derive(Serialize)] +pub(crate) struct ThreadInitializedEvent { + pub(crate) event_type: &'static str, + pub(crate) event_params: ThreadInitializedEventParams, +} + +#[derive(Serialize)] +pub(crate) struct CodexAppMetadata { + pub(crate) connector_id: Option, + pub(crate) thread_id: Option, + pub(crate) turn_id: Option, + pub(crate) app_name: Option, + pub(crate) product_client_id: Option, + pub(crate) invoke_type: Option, + pub(crate) model_slug: Option, +} + +#[derive(Serialize)] +pub(crate) struct CodexAppMentionedEventRequest { + pub(crate) event_type: &'static str, + pub(crate) event_params: CodexAppMetadata, +} + +#[derive(Serialize)] +pub(crate) struct CodexAppUsedEventRequest { + pub(crate) event_type: &'static str, + pub(crate) event_params: CodexAppMetadata, +} + +#[derive(Serialize)] +pub(crate) struct CodexPluginMetadata { + pub(crate) plugin_id: Option, + pub(crate) plugin_name: Option, + pub(crate) marketplace_name: Option, + pub(crate) has_skills: Option, + pub(crate) mcp_server_count: Option, + pub(crate) connector_ids: Option>, + pub(crate) product_client_id: Option, +} + +#[derive(Serialize)] +pub(crate) struct CodexPluginUsedMetadata { + #[serde(flatten)] + pub(crate) plugin: CodexPluginMetadata, + pub(crate) thread_id: Option, + pub(crate) turn_id: Option, + pub(crate) model_slug: Option, +} + +#[derive(Serialize)] +pub(crate) struct CodexPluginEventRequest { + pub(crate) event_type: &'static str, + pub(crate) event_params: CodexPluginMetadata, +} + +#[derive(Serialize)] +pub(crate) struct CodexPluginUsedEventRequest { + pub(crate) event_type: &'static str, + pub(crate) event_params: CodexPluginUsedMetadata, +} + +pub(crate) fn plugin_state_event_type(state: PluginState) -> &'static str { + match state { + PluginState::Installed => "codex_plugin_installed", + PluginState::Uninstalled => "codex_plugin_uninstalled", + PluginState::Enabled => "codex_plugin_enabled", + PluginState::Disabled => "codex_plugin_disabled", + } +} + +pub(crate) fn codex_app_metadata( + tracking: &TrackEventsContext, + app: AppInvocation, +) -> CodexAppMetadata { + CodexAppMetadata { + connector_id: app.connector_id, + thread_id: Some(tracking.thread_id.clone()), + turn_id: Some(tracking.turn_id.clone()), + app_name: app.app_name, + product_client_id: Some(originator().value), + invoke_type: app.invocation_type, + model_slug: Some(tracking.model_slug.clone()), + } +} + +pub(crate) fn codex_plugin_metadata(plugin: PluginTelemetryMetadata) -> CodexPluginMetadata { + let capability_summary = plugin.capability_summary; + CodexPluginMetadata { + plugin_id: Some(plugin.plugin_id.as_key()), + plugin_name: Some(plugin.plugin_id.plugin_name), + marketplace_name: Some(plugin.plugin_id.marketplace_name), + has_skills: capability_summary + .as_ref() + .map(|summary| summary.has_skills), + mcp_server_count: capability_summary + .as_ref() + .map(|summary| summary.mcp_server_names.len()), + connector_ids: capability_summary.map(|summary| { + summary + .app_connector_ids + .into_iter() + .map(|connector_id| connector_id.0) + .collect() + }), + product_client_id: Some(originator().value), + } +} + +pub(crate) fn codex_plugin_used_metadata( + tracking: &TrackEventsContext, + plugin: PluginTelemetryMetadata, +) -> CodexPluginUsedMetadata { + CodexPluginUsedMetadata { + plugin: codex_plugin_metadata(plugin), + thread_id: Some(tracking.thread_id.clone()), + turn_id: Some(tracking.turn_id.clone()), + model_slug: Some(tracking.model_slug.clone()), + } +} + +pub(crate) fn thread_source_name(thread_source: &SessionSource) -> Option<&'static str> { + match thread_source { + SessionSource::Cli | SessionSource::VSCode | SessionSource::Exec => Some("user"), + SessionSource::SubAgent(_) => Some("subagent"), + SessionSource::Mcp | SessionSource::Custom(_) | SessionSource::Unknown => None, + } +} + +pub(crate) fn current_runtime_metadata() -> CodexRuntimeMetadata { + let os_info = os_info::get(); + CodexRuntimeMetadata { + codex_rs_version: env!("CARGO_PKG_VERSION").to_string(), + runtime_os: std::env::consts::OS.to_string(), + runtime_os_version: os_info.version().to_string(), + runtime_arch: std::env::consts::ARCH.to_string(), + } +} diff --git a/codex-rs/analytics/src/facts.rs b/codex-rs/analytics/src/facts.rs new file mode 100644 index 0000000000..31b8516e85 --- /dev/null +++ b/codex-rs/analytics/src/facts.rs @@ -0,0 +1,116 @@ +use crate::events::AppServerRpcTransport; +use crate::events::CodexRuntimeMetadata; +use codex_app_server_protocol::ClientRequest; +use codex_app_server_protocol::ClientResponse; +use codex_app_server_protocol::InitializeParams; +use codex_app_server_protocol::RequestId; +use codex_app_server_protocol::ServerNotification; +use codex_plugin::PluginTelemetryMetadata; +use codex_protocol::protocol::SkillScope; +use serde::Serialize; +use std::path::PathBuf; + +#[derive(Clone)] +pub struct TrackEventsContext { + pub model_slug: String, + pub thread_id: String, + pub turn_id: String, +} + +pub fn build_track_events_context( + model_slug: String, + thread_id: String, + turn_id: String, +) -> TrackEventsContext { + TrackEventsContext { + model_slug, + thread_id, + turn_id, + } +} + +#[derive(Clone, Debug)] +pub struct SkillInvocation { + pub skill_name: String, + pub skill_scope: SkillScope, + pub skill_path: PathBuf, + pub invocation_type: InvocationType, +} + +#[derive(Clone, Copy, Debug, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum InvocationType { + Explicit, + Implicit, +} + +pub struct AppInvocation { + pub connector_id: Option, + pub app_name: Option, + pub invocation_type: Option, +} + +#[allow(dead_code)] +pub(crate) enum AnalyticsFact { + Initialize { + connection_id: u64, + params: InitializeParams, + product_client_id: String, + runtime: CodexRuntimeMetadata, + rpc_transport: AppServerRpcTransport, + }, + Request { + connection_id: u64, + request_id: RequestId, + request: Box, + }, + Response { + connection_id: u64, + response: Box, + }, + Notification(Box), + // Facts that do not naturally exist on the app-server protocol surface, or + // would require non-trivial protocol reshaping on this branch. + Custom(CustomAnalyticsFact), +} + +pub(crate) enum CustomAnalyticsFact { + SkillInvoked(SkillInvokedInput), + AppMentioned(AppMentionedInput), + AppUsed(AppUsedInput), + PluginUsed(PluginUsedInput), + PluginStateChanged(PluginStateChangedInput), +} + +pub(crate) struct SkillInvokedInput { + pub tracking: TrackEventsContext, + pub invocations: Vec, +} + +pub(crate) struct AppMentionedInput { + pub tracking: TrackEventsContext, + pub mentions: Vec, +} + +pub(crate) struct AppUsedInput { + pub tracking: TrackEventsContext, + pub app: AppInvocation, +} + +pub(crate) struct PluginUsedInput { + pub tracking: TrackEventsContext, + pub plugin: PluginTelemetryMetadata, +} + +pub(crate) struct PluginStateChangedInput { + pub plugin: PluginTelemetryMetadata, + pub state: PluginState, +} + +#[derive(Clone, Copy)] +pub(crate) enum PluginState { + Installed, + Uninstalled, + Enabled, + Disabled, +} diff --git a/codex-rs/analytics/src/lib.rs b/codex-rs/analytics/src/lib.rs index e36f5d71b5..6f927d09c0 100644 --- a/codex-rs/analytics/src/lib.rs +++ b/codex-rs/analytics/src/lib.rs @@ -1,8 +1,15 @@ -mod analytics_client; +mod client; +mod events; +mod facts; +mod reducer; -pub use analytics_client::AnalyticsEventsClient; -pub use analytics_client::AppInvocation; -pub use analytics_client::InvocationType; -pub use analytics_client::SkillInvocation; -pub use analytics_client::TrackEventsContext; -pub use analytics_client::build_track_events_context; +pub use client::AnalyticsEventsClient; +pub use events::AppServerRpcTransport; +pub use facts::AppInvocation; +pub use facts::InvocationType; +pub use facts::SkillInvocation; +pub use facts::TrackEventsContext; +pub use facts::build_track_events_context; + +#[cfg(test)] +mod analytics_client_tests; diff --git a/codex-rs/analytics/src/reducer.rs b/codex-rs/analytics/src/reducer.rs new file mode 100644 index 0000000000..d83cddcbbc --- /dev/null +++ b/codex-rs/analytics/src/reducer.rs @@ -0,0 +1,305 @@ +use crate::events::AppServerRpcTransport; +use crate::events::CodexAppMentionedEventRequest; +use crate::events::CodexAppServerClientMetadata; +use crate::events::CodexAppUsedEventRequest; +use crate::events::CodexPluginEventRequest; +use crate::events::CodexPluginUsedEventRequest; +use crate::events::CodexRuntimeMetadata; +use crate::events::SkillInvocationEventParams; +use crate::events::SkillInvocationEventRequest; +use crate::events::ThreadInitializationMode; +use crate::events::ThreadInitializedEvent; +use crate::events::ThreadInitializedEventParams; +use crate::events::TrackEventRequest; +use crate::events::codex_app_metadata; +use crate::events::codex_plugin_metadata; +use crate::events::codex_plugin_used_metadata; +use crate::events::plugin_state_event_type; +use crate::events::thread_source_name; +use crate::facts::AnalyticsFact; +use crate::facts::AppMentionedInput; +use crate::facts::AppUsedInput; +use crate::facts::CustomAnalyticsFact; +use crate::facts::PluginState; +use crate::facts::PluginStateChangedInput; +use crate::facts::PluginUsedInput; +use crate::facts::SkillInvokedInput; +use codex_app_server_protocol::ClientResponse; +use codex_app_server_protocol::InitializeParams; +use codex_git_utils::collect_git_info; +use codex_git_utils::get_git_repo_root; +use codex_login::default_client::originator; +use codex_protocol::protocol::SessionSource; +use codex_protocol::protocol::SkillScope; +use sha1::Digest; +use std::collections::HashMap; +use std::path::Path; + +#[derive(Default)] +pub(crate) struct AnalyticsReducer { + connections: HashMap, +} + +struct ConnectionState { + app_server_client: CodexAppServerClientMetadata, + runtime: CodexRuntimeMetadata, +} + +impl AnalyticsReducer { + pub(crate) async fn ingest(&mut self, input: AnalyticsFact, out: &mut Vec) { + match input { + AnalyticsFact::Initialize { + connection_id, + params, + product_client_id, + runtime, + rpc_transport, + } => { + self.ingest_initialize( + connection_id, + params, + product_client_id, + runtime, + rpc_transport, + ); + } + AnalyticsFact::Request { + connection_id: _connection_id, + request_id: _request_id, + request: _request, + } => {} + AnalyticsFact::Response { + connection_id, + response, + } => { + self.ingest_response(connection_id, *response, out); + } + AnalyticsFact::Notification(_notification) => {} + AnalyticsFact::Custom(input) => match input { + CustomAnalyticsFact::SkillInvoked(input) => { + self.ingest_skill_invoked(input, out).await; + } + CustomAnalyticsFact::AppMentioned(input) => { + self.ingest_app_mentioned(input, out); + } + CustomAnalyticsFact::AppUsed(input) => { + self.ingest_app_used(input, out); + } + CustomAnalyticsFact::PluginUsed(input) => { + self.ingest_plugin_used(input, out); + } + CustomAnalyticsFact::PluginStateChanged(input) => { + self.ingest_plugin_state_changed(input, out); + } + }, + } + } + + fn ingest_initialize( + &mut self, + connection_id: u64, + params: InitializeParams, + product_client_id: String, + runtime: CodexRuntimeMetadata, + rpc_transport: AppServerRpcTransport, + ) { + self.connections.insert( + connection_id, + ConnectionState { + app_server_client: CodexAppServerClientMetadata { + product_client_id, + client_name: Some(params.client_info.name), + client_version: Some(params.client_info.version), + rpc_transport, + experimental_api_enabled: params + .capabilities + .map(|capabilities| capabilities.experimental_api), + }, + runtime, + }, + ); + } + + async fn ingest_skill_invoked( + &mut self, + input: SkillInvokedInput, + out: &mut Vec, + ) { + let SkillInvokedInput { + tracking, + invocations, + } = input; + for invocation in invocations { + let skill_scope = match invocation.skill_scope { + SkillScope::User => "user", + SkillScope::Repo => "repo", + SkillScope::System => "system", + SkillScope::Admin => "admin", + }; + let repo_root = get_git_repo_root(invocation.skill_path.as_path()); + let repo_url = if let Some(root) = repo_root.as_ref() { + collect_git_info(root) + .await + .and_then(|info| info.repository_url) + } else { + None + }; + let skill_id = skill_id_for_local_skill( + repo_url.as_deref(), + repo_root.as_deref(), + invocation.skill_path.as_path(), + invocation.skill_name.as_str(), + ); + out.push(TrackEventRequest::SkillInvocation( + SkillInvocationEventRequest { + event_type: "skill_invocation", + skill_id, + skill_name: invocation.skill_name.clone(), + event_params: SkillInvocationEventParams { + thread_id: Some(tracking.thread_id.clone()), + invoke_type: Some(invocation.invocation_type), + model_slug: Some(tracking.model_slug.clone()), + product_client_id: Some(originator().value), + repo_url, + skill_scope: Some(skill_scope.to_string()), + }, + }, + )); + } + } + + fn ingest_app_mentioned(&mut self, input: AppMentionedInput, out: &mut Vec) { + let AppMentionedInput { tracking, mentions } = input; + out.extend(mentions.into_iter().map(|mention| { + let event_params = codex_app_metadata(&tracking, mention); + TrackEventRequest::AppMentioned(CodexAppMentionedEventRequest { + event_type: "codex_app_mentioned", + event_params, + }) + })); + } + + fn ingest_app_used(&mut self, input: AppUsedInput, out: &mut Vec) { + let AppUsedInput { tracking, app } = input; + let event_params = codex_app_metadata(&tracking, app); + out.push(TrackEventRequest::AppUsed(CodexAppUsedEventRequest { + event_type: "codex_app_used", + event_params, + })); + } + + fn ingest_plugin_used(&mut self, input: PluginUsedInput, out: &mut Vec) { + let PluginUsedInput { tracking, plugin } = input; + out.push(TrackEventRequest::PluginUsed(CodexPluginUsedEventRequest { + event_type: "codex_plugin_used", + event_params: codex_plugin_used_metadata(&tracking, plugin), + })); + } + + fn ingest_plugin_state_changed( + &mut self, + input: PluginStateChangedInput, + out: &mut Vec, + ) { + let PluginStateChangedInput { plugin, state } = input; + let event = CodexPluginEventRequest { + event_type: plugin_state_event_type(state), + event_params: codex_plugin_metadata(plugin), + }; + out.push(match state { + PluginState::Installed => TrackEventRequest::PluginInstalled(event), + PluginState::Uninstalled => TrackEventRequest::PluginUninstalled(event), + PluginState::Enabled => TrackEventRequest::PluginEnabled(event), + PluginState::Disabled => TrackEventRequest::PluginDisabled(event), + }); + } + + fn ingest_response( + &mut self, + connection_id: u64, + response: ClientResponse, + out: &mut Vec, + ) { + let (thread, model, initialization_mode) = match response { + ClientResponse::ThreadStart { response, .. } => ( + response.thread, + response.model, + ThreadInitializationMode::New, + ), + ClientResponse::ThreadResume { response, .. } => ( + response.thread, + response.model, + ThreadInitializationMode::Resumed, + ), + ClientResponse::ThreadFork { response, .. } => ( + response.thread, + response.model, + ThreadInitializationMode::Forked, + ), + _ => return, + }; + let thread_source: SessionSource = thread.source.into(); + let Some(connection_state) = self.connections.get(&connection_id) else { + return; + }; + out.push(TrackEventRequest::ThreadInitialized( + ThreadInitializedEvent { + event_type: "codex_thread_initialized", + event_params: ThreadInitializedEventParams { + thread_id: thread.id, + app_server_client: connection_state.app_server_client.clone(), + runtime: connection_state.runtime.clone(), + model, + ephemeral: thread.ephemeral, + thread_source: thread_source_name(&thread_source), + initialization_mode, + subagent_source: None, + parent_thread_id: None, + created_at: u64::try_from(thread.created_at).unwrap_or_default(), + }, + }, + )); + } +} + +pub(crate) fn skill_id_for_local_skill( + repo_url: Option<&str>, + repo_root: Option<&Path>, + skill_path: &Path, + skill_name: &str, +) -> String { + let path = normalize_path_for_skill_id(repo_url, repo_root, skill_path); + let prefix = if let Some(url) = repo_url { + format!("repo_{url}") + } else { + "personal".to_string() + }; + let raw_id = format!("{prefix}_{path}_{skill_name}"); + let mut hasher = sha1::Sha1::new(); + sha1::Digest::update(&mut hasher, raw_id.as_bytes()); + format!("{:x}", sha1::Digest::finalize(hasher)) +} + +/// Returns a normalized path for skill ID construction. +/// +/// - Repo-scoped skills use a path relative to the repo root. +/// - User/admin/system skills use an absolute path. +pub(crate) fn normalize_path_for_skill_id( + repo_url: Option<&str>, + repo_root: Option<&Path>, + skill_path: &Path, +) -> String { + let resolved_path = + std::fs::canonicalize(skill_path).unwrap_or_else(|_| skill_path.to_path_buf()); + match (repo_url, repo_root) { + (Some(_), Some(root)) => { + let resolved_root = std::fs::canonicalize(root).unwrap_or_else(|_| root.to_path_buf()); + resolved_path + .strip_prefix(&resolved_root) + .unwrap_or(resolved_path.as_path()) + .to_string_lossy() + .replace('\\', "/") + } + _ => resolved_path.to_string_lossy().replace('\\', "/"), + } +} diff --git a/codex-rs/ansi-escape/Cargo.toml b/codex-rs/ansi-escape/Cargo.toml index a10dbf9134..3ebad2bdef 100644 --- a/codex-rs/ansi-escape/Cargo.toml +++ b/codex-rs/ansi-escape/Cargo.toml @@ -8,6 +8,9 @@ license.workspace = true name = "codex_ansi_escape" path = "src/lib.rs" +[lints] +workspace = true + [dependencies] ansi-to-tui = { workspace = true } ratatui = { workspace = true, features = [ diff --git a/codex-rs/app-server/src/codex_message_processor.rs b/codex-rs/app-server/src/codex_message_processor.rs index f82611df25..2952003c5a 100644 --- a/codex-rs/app-server/src/codex_message_processor.rs +++ b/codex-rs/app-server/src/codex_message_processor.rs @@ -32,6 +32,7 @@ use codex_app_server_protocol::CancelLoginAccountParams; use codex_app_server_protocol::CancelLoginAccountResponse; use codex_app_server_protocol::CancelLoginAccountStatus; use codex_app_server_protocol::ClientRequest; +use codex_app_server_protocol::ClientResponse; use codex_app_server_protocol::CodexErrorInfo as AppServerCodexErrorInfo; use codex_app_server_protocol::CollaborationModeListParams; use codex_app_server_protocol::CollaborationModeListResponse; @@ -179,6 +180,7 @@ use codex_arg0::Arg0DispatchPaths; use codex_backend_client::Client as BackendClient; use codex_chatgpt::connectors; use codex_cloud_requirements::cloud_requirements_loader; +use codex_core::AnalyticsEventsClient; use codex_core::AuthManager; use codex_core::CodexAuth; use codex_core::CodexThread; @@ -405,6 +407,7 @@ pub(crate) struct CodexMessageProcessor { auth_manager: Arc, thread_manager: Arc, outgoing: Arc, + analytics_events_client: AnalyticsEventsClient, arg0_paths: Arg0DispatchPaths, config: Arc, cli_overrides: Arc>>, @@ -435,6 +438,8 @@ struct ListenerTaskContext { thread_manager: Arc, thread_state_manager: ThreadStateManager, outgoing: Arc, + analytics_events_client: AnalyticsEventsClient, + general_analytics_enabled: bool, thread_watch_manager: ThreadWatchManager, fallback_model_provider: String, codex_home: PathBuf, @@ -457,6 +462,7 @@ pub(crate) struct CodexMessageProcessorArgs { pub(crate) auth_manager: Arc, pub(crate) thread_manager: Arc, pub(crate) outgoing: Arc, + pub(crate) analytics_events_client: AnalyticsEventsClient, pub(crate) arg0_paths: Arg0DispatchPaths, pub(crate) config: Arc, pub(crate) cli_overrides: Arc>>, @@ -521,6 +527,7 @@ impl CodexMessageProcessor { auth_manager, thread_manager, outgoing, + analytics_events_client, arg0_paths, config, cli_overrides, @@ -533,6 +540,7 @@ impl CodexMessageProcessor { auth_manager, thread_manager, outgoing: outgoing.clone(), + analytics_events_client, arg0_paths, config, cli_overrides, @@ -1017,7 +1025,7 @@ impl CodexMessageProcessor { &mut self, params: &LoginApiKeyParams, ) -> std::result::Result<(), JSONRPCErrorError> { - if self.auth_manager.is_external_auth_active() { + if self.auth_manager.is_external_chatgpt_auth_active() { return Err(self.external_auth_active_error()); } @@ -1096,7 +1104,7 @@ impl CodexMessageProcessor { ) -> std::result::Result { let config = self.config.as_ref(); - if self.auth_manager.is_external_auth_active() { + if self.auth_manager.is_external_chatgpt_auth_active() { return Err(self.external_auth_active_error()); } @@ -1533,7 +1541,7 @@ impl CodexMessageProcessor { } async fn refresh_token_if_requested(&self, do_refresh: bool) -> RefreshTokenRequestOutcome { - if self.auth_manager.is_external_auth_active() { + if self.auth_manager.is_external_chatgpt_auth_active() { return RefreshTokenRequestOutcome::NotAttemptedOrSucceeded; } if do_refresh && let Err(err) = self.auth_manager.refresh_token().await { @@ -2088,6 +2096,8 @@ impl CodexMessageProcessor { thread_manager: Arc::clone(&self.thread_manager), thread_state_manager: self.thread_state_manager.clone(), outgoing: Arc::clone(&self.outgoing), + analytics_events_client: self.analytics_events_client.clone(), + general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), thread_watch_manager: self.thread_watch_manager.clone(), fallback_model_provider: self.config.model_provider_id.clone(), codex_home: self.config.codex_home.clone(), @@ -2320,6 +2330,17 @@ impl CodexMessageProcessor { sandbox: config_snapshot.sandbox_policy.into(), reasoning_effort: config_snapshot.reasoning_effort, }; + if listener_task_context.general_analytics_enabled { + listener_task_context + .analytics_events_client + .track_response( + request_id.connection_id.0, + ClientResponse::ThreadStart { + request_id: request_id.request_id.clone(), + response: response.clone(), + }, + ); + } listener_task_context .outgoing @@ -3807,6 +3828,15 @@ impl CodexMessageProcessor { sandbox: session_configured.sandbox_policy.into(), reasoning_effort: session_configured.reasoning_effort, }; + if self.config.features.enabled(Feature::GeneralAnalytics) { + self.analytics_events_client.track_response( + request_id.connection_id.0, + ClientResponse::ThreadResume { + request_id: request_id.request_id.clone(), + response: response.clone(), + }, + ); + } self.outgoing.send_response(request_id, response).await; } @@ -4414,6 +4444,15 @@ impl CodexMessageProcessor { sandbox: session_configured.sandbox_policy.into(), reasoning_effort: session_configured.reasoning_effort, }; + if self.config.features.enabled(Feature::GeneralAnalytics) { + self.analytics_events_client.track_response( + request_id.connection_id.0, + ClientResponse::ThreadFork { + request_id: request_id.request_id.clone(), + response: response.clone(), + }, + ); + } self.outgoing.send_response(request_id, response).await; @@ -6995,6 +7034,8 @@ impl CodexMessageProcessor { thread_manager: Arc::clone(&self.thread_manager), thread_state_manager: self.thread_state_manager.clone(), outgoing: Arc::clone(&self.outgoing), + analytics_events_client: self.analytics_events_client.clone(), + general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), thread_watch_manager: self.thread_watch_manager.clone(), fallback_model_provider: self.config.model_provider_id.clone(), codex_home: self.config.codex_home.clone(), @@ -7082,6 +7123,8 @@ impl CodexMessageProcessor { thread_manager: Arc::clone(&self.thread_manager), thread_state_manager: self.thread_state_manager.clone(), outgoing: Arc::clone(&self.outgoing), + analytics_events_client: self.analytics_events_client.clone(), + general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), thread_watch_manager: self.thread_watch_manager.clone(), fallback_model_provider: self.config.model_provider_id.clone(), codex_home: self.config.codex_home.clone(), @@ -7113,6 +7156,8 @@ impl CodexMessageProcessor { outgoing, thread_manager, thread_state_manager, + analytics_events_client: _, + general_analytics_enabled: _, thread_watch_manager, fallback_model_provider, codex_home, diff --git a/codex-rs/app-server/src/in_process.rs b/codex-rs/app-server/src/in_process.rs index 6b96bb4e38..4f79b1445a 100644 --- a/codex-rs/app-server/src/in_process.rs +++ b/codex-rs/app-server/src/in_process.rs @@ -74,6 +74,7 @@ use codex_app_server_protocol::Result; use codex_app_server_protocol::ServerNotification; use codex_app_server_protocol::ServerRequest; use codex_arg0::Arg0DispatchPaths; +use codex_core::AppServerRpcTransport; use codex_core::config::Config; use codex_core::config_loader::CloudRequirementsLoader; use codex_core::config_loader::LoaderOverrides; @@ -393,6 +394,7 @@ fn start_uninitialized(args: InProcessStartArgs) -> InProcessClientHandle { config_warnings: args.config_warnings, session_source: args.session_source, enable_codex_api_key_env: args.enable_codex_api_key_env, + rpc_transport: AppServerRpcTransport::InProcess, }); let mut thread_created_rx = processor.thread_created_receiver(); let mut session = ConnectionSessionState::default(); diff --git a/codex-rs/app-server/src/lib.rs b/codex-rs/app-server/src/lib.rs index 3ecefa83fa..d2bae7ccd7 100644 --- a/codex-rs/app-server/src/lib.rs +++ b/codex-rs/app-server/src/lib.rs @@ -36,6 +36,7 @@ use codex_app_server_protocol::ConfigWarningNotification; use codex_app_server_protocol::JSONRPCMessage; use codex_app_server_protocol::TextPosition as AppTextPosition; use codex_app_server_protocol::TextRange as AppTextRange; +use codex_core::AppServerRpcTransport; use codex_core::ExecPolicyError; use codex_core::check_execpolicy_for_warnings; use codex_core::config_loader::ConfigLoadError; @@ -623,6 +624,7 @@ pub async fn run_main_with_transport( config_warnings, session_source, enable_codex_api_key_env: false, + rpc_transport: analytics_rpc_transport(transport), }); let mut thread_created_rx = processor.thread_created_receiver(); let mut running_turn_count_rx = processor.subscribe_running_assistant_turn_count(); @@ -846,6 +848,13 @@ pub async fn run_main_with_transport( Ok(()) } +fn analytics_rpc_transport(transport: AppServerTransport) -> AppServerRpcTransport { + match transport { + AppServerTransport::Stdio => AppServerRpcTransport::Stdio, + AppServerTransport::WebSocket { .. } => AppServerRpcTransport::Websocket, + } +} + #[cfg(test)] mod tests { use super::LogFormat; diff --git a/codex-rs/app-server/src/message_processor.rs b/codex-rs/app-server/src/message_processor.rs index 22c7279833..df7e3572e6 100644 --- a/codex-rs/app-server/src/message_processor.rs +++ b/codex-rs/app-server/src/message_processor.rs @@ -56,6 +56,7 @@ use codex_app_server_protocol::experimental_required_message; use codex_arg0::Arg0DispatchPaths; use codex_chatgpt::connectors; use codex_core::AnalyticsEventsClient; +use codex_core::AppServerRpcTransport; use codex_core::AuthManager; use codex_core::ThreadManager; use codex_core::config::Config; @@ -70,9 +71,10 @@ use codex_core::models_manager::collaboration_mode_presets::CollaborationModesCo use codex_exec_server::EnvironmentManager; use codex_features::Feature; use codex_feedback::CodexFeedback; +use codex_login::AuthMode as LoginAuthMode; +use codex_login::auth::ExternalAuth; use codex_login::auth::ExternalAuthRefreshContext; use codex_login::auth::ExternalAuthRefreshReason; -use codex_login::auth::ExternalAuthRefresher; use codex_login::auth::ExternalAuthTokens; use codex_protocol::ThreadId; use codex_protocol::protocol::SessionSource; @@ -102,7 +104,11 @@ impl ExternalAuthRefreshBridge { } #[async_trait] -impl ExternalAuthRefresher for ExternalAuthRefreshBridge { +impl ExternalAuth for ExternalAuthRefreshBridge { + fn auth_mode(&self) -> LoginAuthMode { + LoginAuthMode::Chatgpt + } + async fn refresh( &self, context: ExternalAuthRefreshContext, @@ -144,11 +150,11 @@ impl ExternalAuthRefresher for ExternalAuthRefreshBridge { let response: ChatgptAuthTokensRefreshResponse = serde_json::from_value(result).map_err(std::io::Error::other)?; - Ok(ExternalAuthTokens { - access_token: response.access_token, - chatgpt_account_id: response.chatgpt_account_id, - chatgpt_plan_type: response.chatgpt_plan_type, - }) + Ok(ExternalAuthTokens::chatgpt( + response.access_token, + response.chatgpt_account_id, + response.chatgpt_plan_type, + )) } } @@ -159,9 +165,11 @@ pub(crate) struct MessageProcessor { external_agent_config_api: ExternalAgentConfigApi, fs_api: FsApi, auth_manager: Arc, + analytics_events_client: AnalyticsEventsClient, fs_watch_manager: FsWatchManager, config: Arc, config_warnings: Arc>, + rpc_transport: AppServerRpcTransport, } #[derive(Clone, Debug, Default)] @@ -186,6 +194,7 @@ pub(crate) struct MessageProcessorArgs { pub(crate) config_warnings: Vec, pub(crate) session_source: SessionSource, pub(crate) enable_codex_api_key_env: bool, + pub(crate) rpc_transport: AppServerRpcTransport, } impl MessageProcessor { @@ -205,11 +214,15 @@ impl MessageProcessor { config_warnings, session_source, enable_codex_api_key_env, + rpc_transport, } = args; - let auth_manager = AuthManager::shared( + let auth_manager = AuthManager::shared_with_external_auth( config.codex_home.clone(), enable_codex_api_key_env, config.cli_auth_credentials_store_mode, + Arc::new(ExternalAuthRefreshBridge { + outgoing: outgoing.clone(), + }), ); let thread_manager = Arc::new(ThreadManager::new( config.as_ref(), @@ -225,9 +238,6 @@ impl MessageProcessor { environment_manager, )); auth_manager.set_forced_chatgpt_workspace_id(config.forced_chatgpt_workspace_id.clone()); - auth_manager.set_external_auth_refresher(Arc::new(ExternalAuthRefreshBridge { - outgoing: outgoing.clone(), - })); let analytics_events_client = AnalyticsEventsClient::new( Arc::clone(&auth_manager), config.chatgpt_base_url.trim_end_matches('/').to_string(), @@ -244,6 +254,7 @@ impl MessageProcessor { auth_manager: auth_manager.clone(), thread_manager: Arc::clone(&thread_manager), outgoing: outgoing.clone(), + analytics_events_client: analytics_events_client.clone(), arg0_paths, config: Arc::clone(&config), cli_overrides: cli_overrides.clone(), @@ -264,7 +275,7 @@ impl MessageProcessor { loader_overrides, cloud_requirements, thread_manager, - analytics_events_client, + analytics_events_client.clone(), ); let external_agent_config_api = ExternalAgentConfigApi::new(config.codex_home.clone()); let fs_api = FsApi::default(); @@ -277,14 +288,16 @@ impl MessageProcessor { external_agent_config_api, fs_api, auth_manager, + analytics_events_client, fs_watch_manager, config, config_warnings: Arc::new(config_warnings), + rpc_transport, } } pub(crate) fn clear_runtime_references(&self) { - self.auth_manager.clear_external_auth_refresher(); + self.auth_manager.clear_external_auth(); } pub(crate) async fn process_request( @@ -549,6 +562,7 @@ impl MessageProcessor { // shared thread when another connected client did not opt into // experimental API). Proposed direction is instance-global first-write-wins // with initialize-time mismatch rejection. + let analytics_initialize_params = params.clone(); let (experimental_api_enabled, opt_out_notification_methods) = match params.capabilities { Some(capabilities) => ( @@ -570,7 +584,7 @@ impl MessageProcessor { session.app_server_client_name = Some(name.clone()); session.client_version = Some(version.clone()); let originator = name.clone(); - if let Err(error) = set_default_originator(originator) { + if let Err(error) = set_default_originator(originator.clone()) { match error { SetOriginatorError::InvalidHeaderValue => { let error = JSONRPCErrorError { @@ -593,6 +607,14 @@ impl MessageProcessor { } } } + if self.config.features.enabled(Feature::GeneralAnalytics) { + self.analytics_events_client.track_initialize( + connection_id.0, + analytics_initialize_params, + originator, + self.rpc_transport, + ); + } set_default_client_residency_requirement(self.config.enforce_residency.value()); let user_agent_suffix = format!("{name}; {version}"); if let Ok(mut suffix) = USER_AGENT_SUFFIX.lock() { diff --git a/codex-rs/app-server/src/message_processor/tracing_tests.rs b/codex-rs/app-server/src/message_processor/tracing_tests.rs index f52b6affe6..44cbe21b60 100644 --- a/codex-rs/app-server/src/message_processor/tracing_tests.rs +++ b/codex-rs/app-server/src/message_processor/tracing_tests.rs @@ -20,6 +20,7 @@ use codex_app_server_protocol::TurnStartParams; use codex_app_server_protocol::TurnStartResponse; use codex_app_server_protocol::UserInput; use codex_arg0::Arg0DispatchPaths; +use codex_core::AppServerRpcTransport; use codex_core::config::Config; use codex_core::config::ConfigBuilder; use codex_core::config_loader::CloudRequirementsLoader; @@ -246,6 +247,7 @@ fn build_test_processor( config_warnings: Vec::new(), session_source: SessionSource::VSCode, enable_codex_api_key_env: false, + rpc_transport: AppServerRpcTransport::Stdio, }); (processor, outgoing_rx) } diff --git a/codex-rs/app-server/tests/common/Cargo.toml b/codex-rs/app-server/tests/common/Cargo.toml index 4eef03e969..c71e5797ad 100644 --- a/codex-rs/app-server/tests/common/Cargo.toml +++ b/codex-rs/app-server/tests/common/Cargo.toml @@ -7,6 +7,9 @@ license.workspace = true [lib] path = "lib.rs" +[lints] +workspace = true + [dependencies] anyhow = { workspace = true } base64 = { workspace = true } diff --git a/codex-rs/app-server/tests/common/models_cache.rs b/codex-rs/app-server/tests/common/models_cache.rs index 54fa0f8dc5..3fa8d7e9d9 100644 --- a/codex-rs/app-server/tests/common/models_cache.rs +++ b/codex-rs/app-server/tests/common/models_cache.rs @@ -28,7 +28,7 @@ fn preset_to_info(preset: &ModelPreset, priority: i32) -> ModelInfo { }, supported_in_api: preset.supported_in_api, priority, - upgrade: preset.upgrade.as_ref().map(|u| u.into()), + upgrade: preset.upgrade.as_ref().map(Into::into), base_instructions: "base instructions".to_string(), model_messages: None, supports_reasoning_summaries: false, diff --git a/codex-rs/app-server/tests/suite/v2/analytics.rs b/codex-rs/app-server/tests/suite/v2/analytics.rs index a389ffbe1a..bd9276ddcd 100644 --- a/codex-rs/app-server/tests/suite/v2/analytics.rs +++ b/codex-rs/app-server/tests/suite/v2/analytics.rs @@ -1,10 +1,23 @@ use anyhow::Result; +use app_test_support::ChatGptAuthFixture; +use app_test_support::DEFAULT_CLIENT_NAME; +use app_test_support::write_chatgpt_auth; +use codex_core::auth::AuthCredentialsStoreMode; use codex_core::config::ConfigBuilder; use codex_core::config::types::OtelExporterKind; use codex_core::config::types::OtelHttpProtocol; use pretty_assertions::assert_eq; +use serde_json::Value; use std::collections::HashMap; +use std::path::Path; +use std::time::Duration; use tempfile::TempDir; +use tokio::time::timeout; +use wiremock::Mock; +use wiremock::MockServer; +use wiremock::ResponseTemplate; +use wiremock::matchers::method; +use wiremock::matchers::path; const SERVICE_VERSION: &str = "0.0.0-test"; @@ -65,3 +78,91 @@ async fn app_server_default_analytics_enabled_with_flag() -> Result<()> { assert_eq!(has_metrics, true); Ok(()) } + +pub(crate) async fn enable_analytics_capture(server: &MockServer, codex_home: &Path) -> Result<()> { + Mock::given(method("POST")) + .and(path("/codex/analytics-events/events")) + .respond_with(ResponseTemplate::new(200)) + .mount(server) + .await; + + write_chatgpt_auth( + codex_home, + ChatGptAuthFixture::new("chatgpt-token") + .account_id("account-123") + .chatgpt_user_id("user-123") + .chatgpt_account_id("account-123"), + AuthCredentialsStoreMode::File, + )?; + + Ok(()) +} + +pub(crate) async fn wait_for_analytics_payload( + server: &MockServer, + read_timeout: Duration, +) -> Result { + let body = timeout(read_timeout, async { + loop { + let Some(requests) = server.received_requests().await else { + tokio::time::sleep(Duration::from_millis(25)).await; + continue; + }; + if let Some(request) = requests.iter().find(|request| { + request.method == "POST" && request.url.path() == "/codex/analytics-events/events" + }) { + break request.body.clone(); + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + }) + .await?; + serde_json::from_slice(&body).map_err(|err| anyhow::anyhow!("invalid analytics payload: {err}")) +} + +pub(crate) fn thread_initialized_event(payload: &Value) -> Result<&Value> { + let events = payload["events"] + .as_array() + .ok_or_else(|| anyhow::anyhow!("analytics payload missing events array"))?; + events + .iter() + .find(|event| event["event_type"] == "codex_thread_initialized") + .ok_or_else(|| anyhow::anyhow!("codex_thread_initialized event should be present")) +} + +pub(crate) fn assert_basic_thread_initialized_event( + event: &Value, + thread_id: &str, + expected_model: &str, + initialization_mode: &str, +) { + assert_eq!(event["event_params"]["thread_id"], thread_id); + assert_eq!( + event["event_params"]["app_server_client"]["product_client_id"], + DEFAULT_CLIENT_NAME + ); + assert_eq!( + event["event_params"]["app_server_client"]["client_name"], + DEFAULT_CLIENT_NAME + ); + assert_eq!( + event["event_params"]["app_server_client"]["rpc_transport"], + "stdio" + ); + assert_eq!(event["event_params"]["model"], expected_model); + assert_eq!(event["event_params"]["ephemeral"], false); + assert_eq!(event["event_params"]["thread_source"], "user"); + assert_eq!( + event["event_params"]["subagent_source"], + serde_json::Value::Null + ); + assert_eq!( + event["event_params"]["parent_thread_id"], + serde_json::Value::Null + ); + assert_eq!( + event["event_params"]["initialization_mode"], + initialization_mode + ); + assert!(event["event_params"]["created_at"].as_u64().is_some()); +} diff --git a/codex-rs/app-server/tests/suite/v2/thread_fork.rs b/codex-rs/app-server/tests/suite/v2/thread_fork.rs index e6be4e5a7c..dc68992fe4 100644 --- a/codex-rs/app-server/tests/suite/v2/thread_fork.rs +++ b/codex-rs/app-server/tests/suite/v2/thread_fork.rs @@ -38,6 +38,11 @@ use wiremock::ResponseTemplate; use wiremock::matchers::method; use wiremock::matchers::path; +use super::analytics::assert_basic_thread_initialized_event; +use super::analytics::enable_analytics_capture; +use super::analytics::thread_initialized_event; +use super::analytics::wait_for_analytics_payload; + const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); #[tokio::test] @@ -177,6 +182,50 @@ async fn thread_fork_creates_new_thread_and_emits_started() -> Result<()> { Ok(()) } +#[tokio::test] +async fn thread_fork_tracks_thread_initialized_analytics() -> Result<()> { + let server = create_mock_responses_server_repeating_assistant("Done").await; + + let codex_home = TempDir::new()?; + create_config_toml_with_chatgpt_base_url( + codex_home.path(), + &server.uri(), + &server.uri(), + /*general_analytics_enabled*/ true, + )?; + enable_analytics_capture(&server, codex_home.path()).await?; + + let conversation_id = create_fake_rollout( + codex_home.path(), + "2025-01-05T12-00-00", + "2025-01-05T12:00:00Z", + "Saved user message", + Some("mock_provider"), + /*git_info*/ None, + )?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let fork_id = mcp + .send_thread_fork_request(ThreadForkParams { + thread_id: conversation_id, + ..Default::default() + }) + .await?; + let fork_resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(fork_id)), + ) + .await??; + let ThreadForkResponse { thread, .. } = to_response::(fork_resp)?; + + let payload = wait_for_analytics_payload(&server, DEFAULT_READ_TIMEOUT).await?; + let event = thread_initialized_event(&payload)?; + assert_basic_thread_initialized_event(event, &thread.id, "mock-model", "forked"); + Ok(()) +} + #[tokio::test] async fn thread_fork_rejects_unmaterialized_thread() -> Result<()> { let server = create_mock_responses_server_repeating_assistant("Done").await; @@ -249,6 +298,7 @@ async fn thread_fork_surfaces_cloud_requirements_load_errors() -> Result<()> { codex_home.path(), &model_server.uri(), &chatgpt_base_url, + /*general_analytics_enabled*/ false, )?; write_chatgpt_auth( codex_home.path(), @@ -509,7 +559,13 @@ fn create_config_toml_with_chatgpt_base_url( codex_home: &Path, server_uri: &str, chatgpt_base_url: &str, + general_analytics_enabled: bool, ) -> std::io::Result<()> { + let general_analytics_toml = if general_analytics_enabled { + "\n[features]\ngeneral_analytics = true\n".to_string() + } else { + String::new() + }; let config_toml = codex_home.join("config.toml"); std::fs::write( config_toml, @@ -521,6 +577,7 @@ sandbox_mode = "read-only" chatgpt_base_url = "{chatgpt_base_url}" model_provider = "mock_provider" +{general_analytics_toml} [model_providers.mock_provider] name = "Mock provider for test" diff --git a/codex-rs/app-server/tests/suite/v2/thread_resume.rs b/codex-rs/app-server/tests/suite/v2/thread_resume.rs index a9c610f8b0..d7dd6f3f20 100644 --- a/codex-rs/app-server/tests/suite/v2/thread_resume.rs +++ b/codex-rs/app-server/tests/suite/v2/thread_resume.rs @@ -71,9 +71,11 @@ use wiremock::ResponseTemplate; use wiremock::matchers::method; use wiremock::matchers::path; -#[cfg(windows)] -const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(25); -#[cfg(not(windows))] +use super::analytics::assert_basic_thread_initialized_event; +use super::analytics::enable_analytics_capture; +use super::analytics::thread_initialized_event; +use super::analytics::wait_for_analytics_payload; + const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); const CODEX_5_2_INSTRUCTIONS_TEMPLATE_DEFAULT: &str = "You are Codex, a coding agent based on GPT-5. You and the user share the same workspace and collaborate to achieve the user's goals."; @@ -124,6 +126,51 @@ async fn thread_resume_rejects_unmaterialized_thread() -> Result<()> { Ok(()) } +#[tokio::test] +async fn thread_resume_tracks_thread_initialized_analytics() -> Result<()> { + let server = create_mock_responses_server_repeating_assistant("Done").await; + + let codex_home = TempDir::new()?; + create_config_toml_with_chatgpt_base_url( + codex_home.path(), + &server.uri(), + &server.uri(), + /*general_analytics_enabled*/ true, + )?; + enable_analytics_capture(&server, codex_home.path()).await?; + + let conversation_id = create_fake_rollout_with_text_elements( + codex_home.path(), + "2025-01-05T12-00-00", + "2025-01-05T12:00:00Z", + "Saved user message", + Vec::new(), + Some("mock_provider"), + /*git_info*/ None, + )?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let resume_id = mcp + .send_thread_resume_request(ThreadResumeParams { + thread_id: conversation_id, + ..Default::default() + }) + .await?; + let resume_resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(resume_id)), + ) + .await??; + let ThreadResumeResponse { thread, .. } = to_response::(resume_resp)?; + + let payload = wait_for_analytics_payload(&server, DEFAULT_READ_TIMEOUT).await?; + let event = thread_initialized_event(&payload)?; + assert_basic_thread_initialized_event(event, &thread.id, "gpt-5.2-codex", "resumed"); + Ok(()) +} + #[tokio::test] async fn thread_resume_returns_rollout_history() -> Result<()> { let server = create_mock_responses_server_repeating_assistant("Done").await; @@ -1514,6 +1561,7 @@ async fn thread_resume_surfaces_cloud_requirements_load_errors() -> Result<()> { codex_home.path(), &model_server.uri(), &chatgpt_base_url, + /*general_analytics_enabled*/ false, )?; write_chatgpt_auth( codex_home.path(), @@ -1892,6 +1940,7 @@ model_provider = "mock_provider" [features] personality = true +general_analytics = true [model_providers.mock_provider] name = "Mock provider for test" @@ -1908,7 +1957,13 @@ fn create_config_toml_with_chatgpt_base_url( codex_home: &std::path::Path, server_uri: &str, chatgpt_base_url: &str, + general_analytics_enabled: bool, ) -> std::io::Result<()> { + let general_analytics_toml = if general_analytics_enabled { + "\ngeneral_analytics = true".to_string() + } else { + String::new() + }; let config_toml = codex_home.join("config.toml"); std::fs::write( config_toml, @@ -1923,6 +1978,7 @@ model_provider = "mock_provider" [features] personality = true +{general_analytics_toml} [model_providers.mock_provider] name = "Mock provider for test" diff --git a/codex-rs/app-server/tests/suite/v2/thread_start.rs b/codex-rs/app-server/tests/suite/v2/thread_start.rs index e31a02ded6..f608d94f93 100644 --- a/codex-rs/app-server/tests/suite/v2/thread_start.rs +++ b/codex-rs/app-server/tests/suite/v2/thread_start.rs @@ -26,6 +26,7 @@ use pretty_assertions::assert_eq; use serde_json::Value; use serde_json::json; use std::path::Path; +use std::time::Duration; use tempfile::TempDir; use tokio::time::timeout; use wiremock::Mock; @@ -34,6 +35,11 @@ use wiremock::ResponseTemplate; use wiremock::matchers::method; use wiremock::matchers::path; +use super::analytics::assert_basic_thread_initialized_event; +use super::analytics::enable_analytics_capture; +use super::analytics::thread_initialized_event; +use super::analytics::wait_for_analytics_payload; + const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); #[tokio::test] @@ -153,6 +159,73 @@ async fn thread_start_creates_thread_and_emits_started() -> Result<()> { Ok(()) } +#[tokio::test] +async fn thread_start_tracks_thread_initialized_analytics() -> Result<()> { + let server = create_mock_responses_server_repeating_assistant("Done").await; + + let codex_home = TempDir::new()?; + create_config_toml_with_chatgpt_base_url( + codex_home.path(), + &server.uri(), + &server.uri(), + /*general_analytics_enabled*/ true, + )?; + enable_analytics_capture(&server, codex_home.path()).await?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let req_id = mcp + .send_thread_start_request(ThreadStartParams::default()) + .await?; + let resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(req_id)), + ) + .await??; + let ThreadStartResponse { thread, .. } = to_response::(resp)?; + + let payload = wait_for_analytics_payload(&server, DEFAULT_READ_TIMEOUT).await?; + assert_eq!(payload["events"].as_array().expect("events array").len(), 1); + let event = thread_initialized_event(&payload)?; + assert_basic_thread_initialized_event(event, &thread.id, "mock-model", "new"); + Ok(()) +} + +#[tokio::test] +async fn thread_start_does_not_track_thread_initialized_analytics_without_feature() -> Result<()> { + let server = create_mock_responses_server_repeating_assistant("Done").await; + + let codex_home = TempDir::new()?; + create_config_toml_with_chatgpt_base_url( + codex_home.path(), + &server.uri(), + &server.uri(), + /*general_analytics_enabled*/ false, + )?; + enable_analytics_capture(&server, codex_home.path()).await?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let req_id = mcp + .send_thread_start_request(ThreadStartParams::default()) + .await?; + let resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(req_id)), + ) + .await??; + let _ = to_response::(resp)?; + + let payload = wait_for_analytics_payload(&server, Duration::from_millis(250)).await; + assert!( + payload.is_err(), + "thread analytics should be gated off when general_analytics is disabled" + ); + Ok(()) +} + #[tokio::test] async fn thread_start_respects_project_config_from_cwd() -> Result<()> { let server = create_mock_responses_server_repeating_assistant("Done").await; @@ -460,6 +533,7 @@ async fn thread_start_surfaces_cloud_requirements_load_errors() -> Result<()> { codex_home.path(), &model_server.uri(), &chatgpt_base_url, + /*general_analytics_enabled*/ false, )?; write_chatgpt_auth( codex_home.path(), @@ -543,7 +617,13 @@ fn create_config_toml_with_chatgpt_base_url( codex_home: &Path, server_uri: &str, chatgpt_base_url: &str, + general_analytics_enabled: bool, ) -> std::io::Result<()> { + let general_analytics_toml = if general_analytics_enabled { + "\n[features]\ngeneral_analytics = true\n".to_string() + } else { + String::new() + }; let config_toml = codex_home.join("config.toml"); std::fs::write( config_toml, @@ -555,6 +635,7 @@ sandbox_mode = "read-only" chatgpt_base_url = "{chatgpt_base_url}" model_provider = "mock_provider" +{general_analytics_toml} [model_providers.mock_provider] name = "Mock provider for test" diff --git a/codex-rs/backend-client/Cargo.toml b/codex-rs/backend-client/Cargo.toml index 8279dba630..96e9d73599 100644 --- a/codex-rs/backend-client/Cargo.toml +++ b/codex-rs/backend-client/Cargo.toml @@ -8,6 +8,9 @@ publish = false [lib] path = "src/lib.rs" +[lints] +workspace = true + [dependencies] anyhow = "1" serde = { version = "1", features = ["derive"] } diff --git a/codex-rs/code-mode/src/description.rs b/codex-rs/code-mode/src/description.rs index 22639b3246..3a87fef092 100644 --- a/codex-rs/code-mode/src/description.rs +++ b/codex-rs/code-mode/src/description.rs @@ -7,15 +7,17 @@ use crate::PUBLIC_TOOL_NAME; const MAX_JS_SAFE_INTEGER: u64 = (1_u64 << 53) - 1; const CODE_MODE_ONLY_PREFACE: &str = "Use `exec/wait` tool to run all other tools, do not attempt to use any other tools directly"; -const EXEC_DESCRIPTION_TEMPLATE: &str = r#"## exec -- Runs raw JavaScript in an isolated context (no Node, no file system, or network access, no console). -- Send raw JavaScript source text, not JSON, quoted strings, or markdown code fences. +const EXEC_DESCRIPTION_TEMPLATE: &str = r#"Run JavaScript code to orchestrate/compose tool calls +- Evaluates the provided JavaScript code in a fresh V8 isolate as an async module. +- All nested tools are available on the global `tools` object, for example `await tools.exec_command(...)`. Tool names are exposed as normalized JavaScript identifiers, for example `await tools.mcp__ologs__get_profile(...)`. +- Nested tool methods take either a string or an object as their input argument. +- Nested tools return either an object or a string, based on the description. +- Runs raw JavaScript -- no Node, no file system, no network access, no console. +- Accepts raw JavaScript source text, not JSON, quoted strings, or markdown code fences. - You may optionally start the tool input with a first-line pragma like `// @exec: {"yield_time_ms": 10000, "max_output_tokens": 1000}`. - `yield_time_ms` asks `exec` to yield early after that many milliseconds if the script is still running. - `max_output_tokens` sets the token budget for direct `exec` results. By default the result is truncated to 10000 tokens. -- All nested tools are available on the global `tools` object, for example `await tools.exec_command(...)`. Tool names are exposed as normalized JavaScript identifiers, for example `await tools.mcp__ologs__get_profile(...)`. -- Tool methods take either string or object as parameter. -- They return either a structured value or a string based on the description above. +- When the JS code is fully evaluated, the isolate's lifetime ends and unawaited promises are silently discarded. - Global helpers: - `exit()`: Immediately ends the current script successfully (like an early return from the top level). diff --git a/codex-rs/codex-backend-openapi-models/Cargo.toml b/codex-rs/codex-backend-openapi-models/Cargo.toml index f9bad4a494..ed3a1043d6 100644 --- a/codex-rs/codex-backend-openapi-models/Cargo.toml +++ b/codex-rs/codex-backend-openapi-models/Cargo.toml @@ -8,6 +8,9 @@ license.workspace = true name = "codex_backend_openapi_models" path = "src/lib.rs" +[lints] +workspace = true + # Important: generated code often violates our workspace lints. # Allow unwrap/expect in this crate so the workspace builds cleanly # after models are regenerated. diff --git a/codex-rs/core/config.schema.json b/codex-rs/core/config.schema.json index 5c28366f19..e290ffb684 100644 --- a/codex-rs/core/config.schema.json +++ b/codex-rs/core/config.schema.json @@ -435,6 +435,9 @@ "fast_mode": { "type": "boolean" }, + "general_analytics": { + "type": "boolean" + }, "guardian_approval": { "type": "boolean" }, @@ -887,10 +890,62 @@ }, "type": "object" }, + "ModelProviderAuthInfo": { + "additionalProperties": false, + "description": "Configuration for obtaining a provider bearer token from a command.", + "properties": { + "args": { + "default": [], + "description": "Command arguments.", + "items": { + "type": "string" + }, + "type": "array" + }, + "command": { + "description": "Command to execute. Bare names are resolved via `PATH`; paths are resolved against `cwd`.", + "type": "string" + }, + "cwd": { + "allOf": [ + { + "$ref": "#/definitions/AbsolutePathBuf" + } + ], + "description": "Working directory used when running the token command." + }, + "refresh_interval_ms": { + "default": 300000, + "description": "Maximum age for the cached token before rerunning the command.", + "format": "uint64", + "minimum": 1.0, + "type": "integer" + }, + "timeout_ms": { + "default": 5000, + "description": "Maximum time to wait for the token command to exit successfully.", + "format": "uint64", + "minimum": 1.0, + "type": "integer" + } + }, + "required": [ + "command" + ], + "type": "object" + }, "ModelProviderInfo": { "additionalProperties": false, "description": "Serializable representation of a provider definition.", "properties": { + "auth": { + "allOf": [ + { + "$ref": "#/definitions/ModelProviderAuthInfo" + } + ], + "description": "Command-backed bearer-token configuration for this provider." + }, "base_url": { "description": "Base URL for the provider's OpenAI-compatible API.", "type": "string" @@ -2122,6 +2177,9 @@ "fast_mode": { "type": "boolean" }, + "general_analytics": { + "type": "boolean" + }, "guardian_approval": { "type": "boolean" }, diff --git a/codex-rs/core/src/agent/agent_resolver.rs b/codex-rs/core/src/agent/agent_resolver.rs index 3d1f75f57f..14f21c731c 100644 --- a/codex-rs/core/src/agent/agent_resolver.rs +++ b/codex-rs/core/src/agent/agent_resolver.rs @@ -28,25 +28,6 @@ pub(crate) async fn resolve_agent_target( }) } -/// Resolves multiple tool-facing agent targets to thread ids. -pub(crate) async fn resolve_agent_targets( - session: &Arc, - turn: &Arc, - targets: Vec, -) -> Result, FunctionCallError> { - if targets.is_empty() { - return Err(FunctionCallError::RespondToModel( - "agent targets must be non-empty".to_string(), - )); - } - - let mut resolved = Vec::with_capacity(targets.len()); - for target in &targets { - resolved.push(resolve_agent_target(session, turn, target).await?); - } - Ok(resolved) -} - fn register_session_root(session: &Arc, turn: &Arc) { session .services diff --git a/codex-rs/core/src/agent/control.rs b/codex-rs/core/src/agent/control.rs index 4b7bf32b80..23b81ea154 100644 --- a/codex-rs/core/src/agent/control.rs +++ b/codex-rs/core/src/agent/control.rs @@ -19,6 +19,7 @@ use crate::session_prefix::format_subagent_notification_message; use crate::shell_snapshot::ShellSnapshot; use crate::state_db; use crate::thread_manager::ThreadManagerState; +use crate::thread_rollout_truncation::truncate_rollout_to_last_n_fork_turns; use codex_features::Feature; use codex_protocol::AgentPath; use codex_protocol::ThreadId; @@ -54,10 +55,16 @@ const AGENT_NAMES: &str = include_str!("agent_names.txt"); const FORKED_SPAWN_AGENT_OUTPUT_MESSAGE: &str = "You are the newly spawned agent. The prior conversation history was forked from your parent agent. Treat the next user message as your new task, and use the forked history only as background context."; const ROOT_LAST_TASK_MESSAGE: &str = "Main thread"; +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum SpawnAgentForkMode { + FullHistory, + LastNTurns(usize), +} + #[derive(Clone, Debug, Default)] pub(crate) struct SpawnAgentOptions { pub(crate) fork_parent_spawn_call_id: Option, - pub(crate) post_fork_developer_message: Option, + pub(crate) fork_mode: Option, } #[derive(Clone, Debug)] @@ -241,107 +248,32 @@ impl AgentControl { let notification_source = session_source.clone(); // The same `AgentControl` is sent to spawn the thread. - let new_thread = match session_source { - Some(session_source) => { - if let Some(call_id) = options.fork_parent_spawn_call_id.as_ref() { - let SessionSource::SubAgent(SubAgentSource::ThreadSpawn { - parent_thread_id, - .. - }) = session_source.clone() - else { - return Err(CodexErr::Fatal( - "spawn_agent fork requires a thread-spawn session source".to_string(), - )); - }; - let parent_thread = state.get_thread(parent_thread_id).await.ok(); - if let Some(parent_thread) = parent_thread.as_ref() { - // `record_conversation_items` only queues rollout writes asynchronously. - // Flush/materialize the live parent before snapshotting JSONL for a fork. - parent_thread - .codex - .session - .ensure_rollout_materialized() - .await; - parent_thread.codex.session.flush_rollout().await; - } - let rollout_path = parent_thread - .as_ref() - .and_then(|parent_thread| parent_thread.rollout_path()) - .or(find_thread_path_by_id_str( - config.codex_home.as_path(), - &parent_thread_id.to_string(), - ) - .await?) - .ok_or_else(|| { - CodexErr::Fatal(format!( - "parent thread rollout unavailable for fork: {parent_thread_id}" - )) - })?; - let mut forked_rollout_items = RolloutRecorder::get_fork_history(&rollout_path) - .await? - .get_rollout_items(); - if forked_rollout_items - .iter() - .any(|item| matches!(item, RolloutItem::ForkReference(_))) - { - forked_rollout_items = - crate::rollout::truncation::materialize_rollout_items_for_replay( - config.codex_home.as_path(), - &forked_rollout_items, - ) - .await; - } - forked_rollout_items.push(RolloutItem::ForkReference(ForkReferenceItem { - rollout_path: rollout_path.clone(), - nth_user_message: usize::MAX, - })); - let mut output = FunctionCallOutputPayload::from_text( - FORKED_SPAWN_AGENT_OUTPUT_MESSAGE.to_string(), - ); - output.success = Some(true); - forked_rollout_items.push(RolloutItem::ResponseItem( - ResponseItem::FunctionCallOutput { - call_id: call_id.clone(), - output, - }, - )); - let post_fork_developer_message = build_post_fork_developer_message( - &config, - &session_source, - options.post_fork_developer_message.as_deref(), - ) - .await; - append_post_fork_developer_message( - &mut forked_rollout_items, - post_fork_developer_message, - ); - let initial_history = InitialHistory::Forked(forked_rollout_items); - state - .fork_thread_with_source( - config, - initial_history, - self.clone(), - session_source, - /*persist_extended_history*/ false, - inherited_shell_snapshot, - inherited_exec_policy, - ) - .await? - } else { - state - .spawn_new_thread_with_source( - config, - self.clone(), - session_source, - /*persist_extended_history*/ false, - /*metrics_service_name*/ None, - inherited_shell_snapshot, - inherited_exec_policy, - ) - .await? - } + let new_thread = match (session_source, options.fork_mode.as_ref()) { + (Some(session_source), Some(_)) => { + self.spawn_forked_thread( + &state, + config, + session_source, + &options, + inherited_shell_snapshot, + inherited_exec_policy, + ) + .await? } - None => state.spawn_new_thread(config, self.clone()).await?, + (Some(session_source), None) => { + state + .spawn_new_thread_with_source( + config, + self.clone(), + session_source, + /*persist_extended_history*/ false, + /*metrics_service_name*/ None, + inherited_shell_snapshot, + inherited_exec_policy, + ) + .await? + } + (None, _) => state.spawn_new_thread(config, self.clone()).await?, }; agent_metadata.agent_id = Some(new_thread.thread_id); reservation.commit(agent_metadata.clone()); @@ -360,17 +292,19 @@ impl AgentControl { self.send_input(new_thread.thread_id, initial_operation) .await?; - let child_reference = agent_metadata - .agent_path - .as_ref() - .map(ToString::to_string) - .unwrap_or_else(|| new_thread.thread_id.to_string()); - self.maybe_start_completion_watcher( - new_thread.thread_id, - notification_source, - child_reference, - agent_metadata.agent_path.clone(), - ); + if !new_thread.thread.enabled(Feature::MultiAgentV2) { + let child_reference = agent_metadata + .agent_path + .as_ref() + .map(ToString::to_string) + .unwrap_or_else(|| new_thread.thread_id.to_string()); + self.maybe_start_completion_watcher( + new_thread.thread_id, + notification_source, + child_reference, + agent_metadata.agent_path.clone(), + ); + } Ok(LiveAgent { thread_id: new_thread.thread_id, @@ -379,68 +313,39 @@ impl AgentControl { }) } - pub(crate) async fn spawn_agent_handle( + async fn spawn_forked_thread( &self, + state: &Arc, config: crate::config::Config, - session_source: Option, - ) -> CodexResult { - let state = self.upgrade()?; - let reservation = self - .reserve_spawn_slot_with_reconcile(&state, config.agent_max_threads) - .await?; - let inherited_shell_snapshot = self - .inherited_shell_snapshot_for_source(&state, session_source.as_ref()) - .await; - let inherited_exec_policy = self - .inherited_exec_policy_for_source(&state, session_source.as_ref(), &config) - .await; - - let new_thread = match session_source { - Some(session_source) => { - state - .spawn_new_thread_with_source( - config, - self.clone(), - session_source, - /*persist_extended_history*/ false, - /*metrics_service_name*/ None, - inherited_shell_snapshot, - inherited_exec_policy, - ) - .await? - } - None => state.spawn_new_thread(config, self.clone()).await?, - }; - let agent_metadata = AgentMetadata { - agent_id: Some(new_thread.thread_id), - ..AgentMetadata::default() - }; - reservation.commit(agent_metadata); - state.notify_thread_created(new_thread.thread_id); - Ok(new_thread.thread_id) - } - - pub(crate) async fn fork_agent( - &self, - config: crate::config::Config, - items: Vec, - parent_thread_id: ThreadId, - _nth_user_message: usize, session_source: SessionSource, - ) -> CodexResult { - let state = self.upgrade()?; - let reservation = self - .reserve_spawn_slot_with_reconcile(&state, config.agent_max_threads) - .await?; - let inherited_shell_snapshot = self - .inherited_shell_snapshot_for_source(&state, Some(&session_source)) - .await; - let inherited_exec_policy = self - .inherited_exec_policy_for_source(&state, Some(&session_source), &config) - .await; + options: &SpawnAgentOptions, + inherited_shell_snapshot: Option>, + inherited_exec_policy: Option>, + ) -> CodexResult { + let Some(call_id) = options.fork_parent_spawn_call_id.as_deref() else { + return Err(CodexErr::Fatal( + "spawn_agent fork requires a parent spawn call id".to_string(), + )); + }; + let Some(fork_mode) = options.fork_mode.as_ref() else { + return Err(CodexErr::Fatal( + "spawn_agent fork requires a fork mode".to_string(), + )); + }; + let SessionSource::SubAgent(SubAgentSource::ThreadSpawn { + parent_thread_id, .. + }) = &session_source + else { + return Err(CodexErr::Fatal( + "spawn_agent fork requires a thread-spawn session source".to_string(), + )); + }; + let parent_thread_id = *parent_thread_id; let parent_thread = state.get_thread(parent_thread_id).await.ok(); if let Some(parent_thread) = parent_thread.as_ref() { + // `record_conversation_items` only queues rollout writes asynchronously. + // Flush/materialize the live parent before snapshotting JSONL for a fork. parent_thread .codex .session @@ -448,52 +353,50 @@ impl AgentControl { .await; parent_thread.codex.session.flush_rollout().await; } + let rollout_path = parent_thread .as_ref() - .and_then(|thread| thread.rollout_path()) + .and_then(|parent_thread| parent_thread.rollout_path()) .or(find_thread_path_by_id_str( config.codex_home.as_path(), &parent_thread_id.to_string(), ) .await?) .ok_or_else(|| { - CodexErr::UnsupportedOperation(format!( - "rollout history unavailable for thread {parent_thread_id}" + CodexErr::Fatal(format!( + "parent thread rollout unavailable for fork: {parent_thread_id}" )) })?; - // Watchdog helpers must start as distinct child threads. Reusing the resume loader here - // preserves the parent conversation id and can cause the owner to resume itself. - let mut forked_rollout_items = RolloutRecorder::get_fork_history(&rollout_path) + + let mut forked_rollout_items = RolloutRecorder::get_rollout_history(&rollout_path) .await? .get_rollout_items(); - let post_fork_developer_message = build_post_fork_developer_message( - &config, - &session_source, - /*extra_message*/ None, - ) - .await; - append_post_fork_developer_message(&mut forked_rollout_items, post_fork_developer_message); - let initial_history = InitialHistory::Forked(forked_rollout_items); + if let SpawnAgentForkMode::LastNTurns(last_n_turns) = fork_mode { + forked_rollout_items = + truncate_rollout_to_last_n_fork_turns(&forked_rollout_items, *last_n_turns); + } - let new_thread = state + let mut output = + FunctionCallOutputPayload::from_text(FORKED_SPAWN_AGENT_OUTPUT_MESSAGE.to_string()); + output.success = Some(true); + forked_rollout_items.push(RolloutItem::ResponseItem( + ResponseItem::FunctionCallOutput { + call_id: call_id.to_string(), + output, + }, + )); + + state .fork_thread_with_source( config, - initial_history, + InitialHistory::Forked(forked_rollout_items), self.clone(), session_source, /*persist_extended_history*/ false, inherited_shell_snapshot, inherited_exec_policy, ) - .await?; - let agent_metadata = AgentMetadata { - agent_id: Some(new_thread.thread_id), - ..AgentMetadata::default() - }; - reservation.commit(agent_metadata); - state.notify_thread_created(new_thread.thread_id); - self.send_input(new_thread.thread_id, items.into()).await?; - Ok(new_thread.thread_id) + .await } /// Resume an existing agent thread from a recorded rollout file. @@ -649,17 +552,19 @@ impl AgentControl { // Resumed threads are re-registered in-memory and need the same listener // attachment path as freshly spawned threads. state.notify_thread_created(resumed_thread.thread_id); - let child_reference = agent_metadata - .agent_path - .as_ref() - .map(ToString::to_string) - .unwrap_or_else(|| resumed_thread.thread_id.to_string()); - self.maybe_start_completion_watcher( - resumed_thread.thread_id, - Some(notification_source.clone()), - child_reference, - agent_metadata.agent_path.clone(), - ); + if !resumed_thread.thread.enabled(Feature::MultiAgentV2) { + let child_reference = agent_metadata + .agent_path + .as_ref() + .map(ToString::to_string) + .unwrap_or_else(|| resumed_thread.thread_id.to_string()); + self.maybe_start_completion_watcher( + resumed_thread.thread_id, + Some(notification_source.clone()), + child_reference, + agent_metadata.agent_path.clone(), + ); + } self.persist_thread_spawn_edge_for_source( resumed_thread.thread.as_ref(), resumed_thread.thread_id, diff --git a/codex-rs/core/src/agent/control_tests.rs b/codex-rs/core/src/agent/control_tests.rs index 732838cd0d..3c423872e2 100644 --- a/codex-rs/core/src/agent/control_tests.rs +++ b/codex-rs/core/src/agent/control_tests.rs @@ -666,12 +666,7 @@ async fn send_inter_agent_communication_without_turn_queues_message_without_trig timeout(Duration::from_secs(5), async { loop { - if thread - .codex - .session - .has_queued_response_items_for_next_turn() - .await - { + if thread.codex.session.has_pending_input().await { break; } sleep(Duration::from_millis(10)).await; @@ -825,7 +820,7 @@ async fn spawn_agent_can_fork_parent_thread_history() { })), SpawnAgentOptions { fork_parent_spawn_call_id: Some(parent_spawn_call_id), - ..Default::default() + fork_mode: Some(SpawnAgentForkMode::FullHistory), }, ) .await @@ -911,7 +906,7 @@ async fn spawn_agent_fork_injects_output_for_parent_spawn_call() { })), SpawnAgentOptions { fork_parent_spawn_call_id: Some(parent_spawn_call_id.clone()), - ..Default::default() + fork_mode: Some(SpawnAgentForkMode::FullHistory), }, ) .await @@ -984,7 +979,7 @@ async fn spawn_agent_fork_flushes_parent_rollout_before_loading_history() { })), SpawnAgentOptions { fork_parent_spawn_call_id: Some(parent_spawn_call_id.clone()), - ..Default::default() + fork_mode: Some(SpawnAgentForkMode::FullHistory), }, ) .await @@ -1031,6 +1026,128 @@ async fn spawn_agent_fork_flushes_parent_rollout_before_loading_history() { .expect("parent shutdown should submit"); } +#[tokio::test] +async fn spawn_agent_fork_last_n_turns_keeps_only_recent_turns() { + let harness = AgentControlHarness::new().await; + let (parent_thread_id, parent_thread) = harness.start_thread().await; + + parent_thread + .inject_user_message_without_turn("old parent context".to_string()) + .await; + let queued_communication = InterAgentCommunication::new( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + Vec::new(), + "queued message".to_string(), + /*trigger_turn*/ false, + ); + let queued_turn_context = parent_thread.codex.session.new_default_turn().await; + parent_thread + .codex + .session + .record_conversation_items( + queued_turn_context.as_ref(), + &[queued_communication.to_response_input_item().into()], + ) + .await; + + let triggered_communication = InterAgentCommunication::new( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + Vec::new(), + "triggered context".to_string(), + /*trigger_turn*/ true, + ); + let triggered_turn_context = parent_thread.codex.session.new_default_turn().await; + parent_thread + .codex + .session + .record_conversation_items( + triggered_turn_context.as_ref(), + &[triggered_communication.to_response_input_item().into()], + ) + .await; + + parent_thread + .inject_user_message_without_turn("current parent task".to_string()) + .await; + let spawn_turn_context = parent_thread.codex.session.new_default_turn().await; + let parent_spawn_call_id = "spawn-call-last-n".to_string(); + let parent_spawn_call = ResponseItem::FunctionCall { + id: None, + name: "spawn_agent".to_string(), + namespace: None, + arguments: "{}".to_string(), + call_id: parent_spawn_call_id.clone(), + }; + parent_thread + .codex + .session + .record_conversation_items(spawn_turn_context.as_ref(), &[parent_spawn_call]) + .await; + parent_thread + .codex + .session + .ensure_rollout_materialized() + .await; + parent_thread.codex.session.flush_rollout().await; + + let child_thread_id = harness + .control + .spawn_agent_with_metadata( + harness.config.clone(), + text_input("child task"), + Some(SessionSource::SubAgent(SubAgentSource::ThreadSpawn { + parent_thread_id, + depth: 1, + agent_path: None, + agent_nickname: None, + agent_role: None, + })), + SpawnAgentOptions { + fork_parent_spawn_call_id: Some(parent_spawn_call_id), + fork_mode: Some(SpawnAgentForkMode::LastNTurns(2)), + }, + ) + .await + .expect("forked spawn should keep only the last two turns") + .thread_id; + + let child_thread = harness + .manager + .get_thread(child_thread_id) + .await + .expect("child thread should be registered"); + let history = child_thread.codex.session.clone_history().await; + + assert!(!history_contains_text( + history.raw_items(), + "old parent context" + )); + assert!(!history_contains_text( + history.raw_items(), + "queued message" + )); + assert!(history_contains_text( + history.raw_items(), + "triggered context" + )); + assert!(history_contains_text( + history.raw_items(), + "current parent task" + )); + + let _ = harness + .control + .shutdown_live_agent(child_thread_id) + .await + .expect("child shutdown should submit"); + let _ = parent_thread + .submit(Op::Shutdown {}) + .await + .expect("parent shutdown should submit"); +} + #[tokio::test] async fn spawn_agent_respects_max_threads_limit() { let max_threads = 1usize; diff --git a/codex-rs/core/src/agent/mailbox.rs b/codex-rs/core/src/agent/mailbox.rs new file mode 100644 index 0000000000..c328236475 --- /dev/null +++ b/codex-rs/core/src/agent/mailbox.rs @@ -0,0 +1,161 @@ +use codex_protocol::protocol::InterAgentCommunication; +use std::collections::VecDeque; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use tokio::sync::mpsc; +use tokio::sync::watch; + +#[cfg(test)] +use codex_protocol::AgentPath; + +pub(crate) struct Mailbox { + tx: mpsc::UnboundedSender, + next_seq: AtomicU64, + seq_tx: watch::Sender, +} + +pub(crate) struct MailboxReceiver { + rx: mpsc::UnboundedReceiver, + pending_mails: VecDeque, +} + +impl Mailbox { + pub(crate) fn new() -> (Self, MailboxReceiver) { + let (tx, rx) = mpsc::unbounded_channel(); + let (seq_tx, _) = watch::channel(0); + ( + Self { + tx, + next_seq: AtomicU64::new(0), + seq_tx, + }, + MailboxReceiver { + rx, + pending_mails: VecDeque::new(), + }, + ) + } + + pub(crate) fn subscribe(&self) -> watch::Receiver { + self.seq_tx.subscribe() + } + + pub(crate) fn send(&self, communication: InterAgentCommunication) -> u64 { + let seq = self.next_seq.fetch_add(1, Ordering::Relaxed) + 1; + let _ = self.tx.send(communication); + self.seq_tx.send_replace(seq); + seq + } +} + +impl MailboxReceiver { + fn sync_pending_mails(&mut self) { + while let Ok(mail) = self.rx.try_recv() { + self.pending_mails.push_back(mail); + } + } + + pub(crate) fn has_pending(&mut self) -> bool { + self.sync_pending_mails(); + !self.pending_mails.is_empty() + } + + pub(crate) fn has_pending_trigger_turn(&mut self) -> bool { + self.sync_pending_mails(); + self.pending_mails.iter().any(|mail| mail.trigger_turn) + } + + pub(crate) fn drain(&mut self) -> Vec { + self.sync_pending_mails(); + self.pending_mails.drain(..).collect() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + + fn make_mail( + author: AgentPath, + recipient: AgentPath, + content: &str, + trigger_turn: bool, + ) -> InterAgentCommunication { + InterAgentCommunication::new( + author, + recipient, + Vec::new(), + content.to_string(), + trigger_turn, + ) + } + + #[tokio::test] + async fn mailbox_assigns_monotonic_sequence_numbers() { + let (mailbox, _receiver) = Mailbox::new(); + let mut seq_rx = mailbox.subscribe(); + + let seq_a = mailbox.send(make_mail( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + "one", + /*trigger_turn*/ false, + )); + let seq_b = mailbox.send(make_mail( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + "two", + /*trigger_turn*/ false, + )); + + seq_rx.changed().await.expect("first seq update"); + assert_eq!(*seq_rx.borrow(), seq_b); + assert_eq!(seq_a, 1); + assert_eq!(seq_b, 2); + } + + #[tokio::test] + async fn mailbox_drains_in_delivery_order() { + let (mailbox, mut receiver) = Mailbox::new(); + let mail_one = make_mail( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + "one", + /*trigger_turn*/ false, + ); + let mail_two = make_mail( + AgentPath::try_from("/root/worker").expect("agent path"), + AgentPath::root(), + "two", + /*trigger_turn*/ false, + ); + + mailbox.send(mail_one.clone()); + mailbox.send(mail_two.clone()); + + assert_eq!(receiver.drain(), vec![mail_one, mail_two]); + assert!(!receiver.has_pending()); + } + + #[tokio::test] + async fn mailbox_tracks_pending_trigger_turn_mail() { + let (mailbox, mut receiver) = Mailbox::new(); + + mailbox.send(make_mail( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + "queued", + /*trigger_turn*/ false, + )); + assert!(!receiver.has_pending_trigger_turn()); + + mailbox.send(make_mail( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + "wake", + /*trigger_turn*/ true, + )); + assert!(receiver.has_pending_trigger_turn()); + } +} diff --git a/codex-rs/core/src/agent/mod.rs b/codex-rs/core/src/agent/mod.rs index 96c001b623..f1fe3c8ad5 100644 --- a/codex-rs/core/src/agent/mod.rs +++ b/codex-rs/core/src/agent/mod.rs @@ -1,5 +1,6 @@ pub(crate) mod agent_resolver; pub(crate) mod control; +pub(crate) mod mailbox; mod registry; pub(crate) mod role; pub(crate) mod status; @@ -7,8 +8,8 @@ mod watchdog; pub(crate) use codex_protocol::protocol::AgentStatus; pub(crate) use control::AgentControl; -pub(crate) use control::AgentListing; -pub(crate) use control::WatchdogParentCompactionResult; +pub(crate) use mailbox::Mailbox; +pub(crate) use mailbox::MailboxReceiver; pub(crate) use registry::exceeds_thread_spawn_depth_limit; pub(crate) use registry::next_thread_spawn_depth; pub(crate) use status::agent_status_from_event; diff --git a/codex-rs/core/src/auth_env_telemetry.rs b/codex-rs/core/src/auth_env_telemetry.rs index cc5ffa1207..583d79de16 100644 --- a/codex-rs/core/src/auth_env_telemetry.rs +++ b/codex-rs/core/src/auth_env_telemetry.rs @@ -64,6 +64,7 @@ mod tests { env_key: Some("sk-should-not-leak".to_string()), env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: crate::model_provider_info::WireApi::Responses, query_params: None, http_headers: None, diff --git a/codex-rs/core/src/client.rs b/codex-rs/core/src/client.rs index ddc4184f54..55d8ee51f5 100644 --- a/codex-rs/core/src/client.rs +++ b/codex-rs/core/src/client.rs @@ -104,6 +104,7 @@ use crate::error::Result; use crate::flags::CODEX_RS_SSE_FIXTURE; use crate::model_provider_info::ModelProviderInfo; use crate::model_provider_info::WireApi; +use crate::provider_auth::auth_manager_for_provider; use crate::response_debug_context::extract_response_debug_context; use crate::response_debug_context::extract_response_debug_context_from_api_error; use crate::response_debug_context::telemetry_api_error_message; @@ -261,6 +262,7 @@ impl ModelClient { include_timing_metrics: bool, beta_features_header: Option, ) -> Self { + let auth_manager = auth_manager_for_provider(auth_manager, &provider); let codex_api_key_env_enabled = auth_manager .as_ref() .is_some_and(|manager| manager.codex_api_key_env_enabled()); @@ -294,6 +296,10 @@ impl ModelClient { } } + pub(crate) fn auth_manager(&self) -> Option> { + self.state.auth_manager.clone() + } + fn take_cached_websocket_session(&self) -> WebsocketSession { let mut cached_websocket_session = self .state diff --git a/codex-rs/core/src/codex.rs b/codex-rs/core/src/codex.rs index 79e4c7d267..a80a47d5bb 100644 --- a/codex-rs/core/src/codex.rs +++ b/codex-rs/core/src/codex.rs @@ -13,7 +13,10 @@ use crate::CodexAuth; use crate::SandboxState; use crate::agent::AgentControl; use crate::agent::AgentStatus; +use crate::agent::Mailbox; +use crate::agent::MailboxReceiver; use crate::agent::agent_status_from_event; +use crate::agent::status::is_final; use crate::apps::render_apps_section; use crate::auth_env_telemetry::collect_auth_env_telemetry; use crate::commit_attribution::commit_message_trailer_instruction; @@ -39,6 +42,7 @@ use crate::realtime_conversation::handle_start as handle_realtime_conversation_s use crate::realtime_conversation::handle_text as handle_realtime_conversation_text; use crate::render_skills_section; use crate::rollout::session_index; +use crate::session_prefix::format_subagent_notification_message; use crate::skills_load_input_from_config; use crate::stream_events_utils::HandleOutputCtx; use crate::stream_events_utils::handle_non_tool_response_item; @@ -99,6 +103,7 @@ use codex_protocol::permissions::FileSystemSandboxPolicy; use codex_protocol::permissions::NetworkSandboxPolicy; use codex_protocol::protocol::FileChange; use codex_protocol::protocol::HasLegacyEvent; +use codex_protocol::protocol::InterAgentCommunication; use codex_protocol::protocol::ItemCompletedEvent; use codex_protocol::protocol::ItemStartedEvent; use codex_protocol::protocol::RawResponseItemEvent; @@ -120,6 +125,7 @@ use codex_protocol::request_user_input::RequestUserInputResponse; use codex_rmcp_client::ElicitationResponse; use codex_rmcp_client::OAuthCredentialsStoreMode; use codex_terminal_detection::user_agent; +use codex_tools::filter_tool_suggest_discoverable_tools_for_client; use codex_utils_output_truncation::TruncationPolicy; use codex_utils_stream_parser::AssistantTextChunk; use codex_utils_stream_parser::AssistantTextStreamParser; @@ -920,7 +926,9 @@ pub(crate) struct Session { pending_mcp_server_refresh_config: Mutex>, pub(crate) conversation: Arc, pub(crate) active_turn: Mutex>, - idle_pending_input: Mutex>, + mailbox: Mailbox, + mailbox_rx: Mutex, + idle_pending_input: Mutex>, // TODO (jif) merge with mailbox! pub(crate) guardian_review_session: GuardianReviewSessionManager, pub(crate) services: SessionServices, js_repl: Arc, @@ -2026,6 +2034,7 @@ impl Session { let (out_of_band_elicitation_paused, _out_of_band_elicitation_paused_rx) = watch::channel(false); + let (mailbox, mailbox_rx) = Mailbox::new(); let sess = Arc::new(Session { conversation_id, tx_event: tx_event.clone(), @@ -2036,6 +2045,8 @@ impl Session { pending_mcp_server_refresh_config: Mutex::new(None), conversation: Arc::new(RealtimeConversationManager::new()), active_turn: Mutex::new(None), + mailbox, + mailbox_rx: Mutex::new(mailbox_rx), idle_pending_input: Mutex::new(Vec::new()), guardian_review_session: GuardianReviewSessionManager::default(), services, @@ -2843,6 +2854,8 @@ impl Session { msg, }; self.send_event_raw(event).await; + self.maybe_notify_parent_of_terminal_turn(turn_context, &legacy_source) + .await; self.maybe_mirror_event_text_to_realtime(&legacy_source) .await; self.maybe_clear_realtime_handoff_for_event(&legacy_source) @@ -2858,6 +2871,73 @@ impl Session { } } + /// Forwards terminal turn events from spawned MultiAgentV2 children to their direct parent. + async fn maybe_notify_parent_of_terminal_turn( + &self, + turn_context: &TurnContext, + msg: &EventMsg, + ) { + if !self.enabled(Feature::MultiAgentV2) { + return; + } + + if !matches!(msg, EventMsg::TurnComplete(_) | EventMsg::TurnAborted(_)) { + return; + } + + let SessionSource::SubAgent(SubAgentSource::ThreadSpawn { + parent_thread_id, + agent_path: Some(child_agent_path), + .. + }) = &turn_context.session_source + else { + return; + }; + + let Some(status) = agent_status_from_event(msg) else { + return; + }; + if !is_final(&status) { + return; + } + + self.forward_child_completion_to_parent(*parent_thread_id, child_agent_path, status) + .await; + } + + /// Sends the standard completion envelope from a spawned MultiAgentV2 child to its parent. + async fn forward_child_completion_to_parent( + &self, + parent_thread_id: ThreadId, + child_agent_path: &codex_protocol::AgentPath, + status: AgentStatus, + ) { + let Some(parent_agent_path) = child_agent_path + .as_str() + .rsplit_once('/') + .and_then(|(parent, _)| codex_protocol::AgentPath::try_from(parent).ok()) + else { + return; + }; + + let message = format_subagent_notification_message(child_agent_path.as_str(), &status); + let communication = InterAgentCommunication::new( + child_agent_path.clone(), + parent_agent_path, + Vec::new(), + message, + /*trigger_turn*/ false, + ); + if let Err(err) = self + .services + .agent_control + .send_inter_agent_communication(parent_thread_id, communication) + .await + { + debug!("failed to notify parent thread {parent_thread_id}: {err}"); + } + } + async fn maybe_mirror_event_text_to_realtime(&self, msg: &EventMsg) { let Some(text) = realtime_text_for_event(msg) else { return; @@ -4174,6 +4254,18 @@ impl Session { } } + pub(crate) fn subscribe_mailbox_seq(&self) -> watch::Receiver { + self.mailbox.subscribe() + } + + pub(crate) fn enqueue_mailbox_communication(&self, communication: InterAgentCommunication) { + self.mailbox.send(communication); + } + + pub(crate) async fn has_trigger_turn_mailbox_items(&self) -> bool { + self.mailbox_rx.lock().await.has_pending_trigger_turn() + } + pub async fn prepend_pending_input(&self, input: Vec) -> Result<(), ()> { let mut active = self.active_turn.lock().await; match active.as_mut() { @@ -4187,17 +4279,37 @@ impl Session { } pub async fn get_pending_input(&self) -> Vec { - let mut active = self.active_turn.lock().await; - match active.as_mut() { - Some(at) => { - let mut ts = at.turn_state.lock().await; - ts.take_pending_input() + let pending_input = { + let mut active = self.active_turn.lock().await; + match active.as_mut() { + Some(at) => { + let mut ts = at.turn_state.lock().await; + ts.take_pending_input() + } + None => Vec::new(), } - None => Vec::with_capacity(0), + }; + let mailbox_items = { + let mut mailbox_rx = self.mailbox_rx.lock().await; + mailbox_rx + .drain() + .into_iter() + .map(|mail| mail.to_response_input_item()) + .collect::>() + }; + if pending_input.is_empty() { + mailbox_items + } else if mailbox_items.is_empty() { + pending_input + } else { + let mut pending_input = pending_input; + pending_input.extend(mailbox_items); + pending_input } } /// Queue response items to be injected into the next active turn created for this session. + #[cfg(test)] pub(crate) async fn queue_response_items_for_next_turn(&self, items: Vec) { if items.is_empty() { return; @@ -4216,6 +4328,9 @@ impl Session { } pub async fn has_pending_input(&self) -> bool { + if self.mailbox_rx.lock().await.has_pending() { + return true; + } let active = self.active_turn.lock().await; match active.as_ref() { Some(at) => { @@ -4881,25 +4996,17 @@ mod handlers { } } - /// Records an inter-agent assistant envelope and, when requested, wakes the recipient by - /// starting a regular turn if the session is currently idle. + /// Records an inter-agent assistant envelope, then lets the shared pending-work scheduler + /// decide whether an idle session should start a regular turn. pub async fn inter_agent_communication( sess: &Arc, sub_id: String, communication: InterAgentCommunication, ) { - let pending_item = communication.to_response_input_item(); - if let Ok(()) = sess.inject_response_items(vec![pending_item.clone()]).await { - return; - } - - sess.queue_response_items_for_next_turn(vec![pending_item]) - .await; - if communication.trigger_turn { - let turn_context = sess.new_default_turn_with_sub_id(sub_id).await; - sess.maybe_emit_unknown_model_warning_for_turn(turn_context.as_ref()) - .await; - sess.spawn_task(turn_context, Vec::new(), crate::tasks::RegularTask::new()) + let trigger_turn = communication.trigger_turn; + sess.enqueue_mailbox_communication(communication); + if trigger_turn { + sess.maybe_start_turn_for_pending_work_with_sub_id(sub_id) .await; } } @@ -6805,7 +6912,7 @@ pub(crate) async fn built_tools( ) .await .map(|discoverable_tools| { - crate::tools::discoverable::filter_tool_suggest_discoverable_tools_for_client( + filter_tool_suggest_discoverable_tools_for_client( discoverable_tools, turn_context.app_server_client_name.as_deref(), ) diff --git a/codex-rs/core/src/codex_tests.rs b/codex-rs/core/src/codex_tests.rs index 630657debf..f314f53fed 100644 --- a/codex-rs/core/src/codex_tests.rs +++ b/codex-rs/core/src/codex_tests.rs @@ -2743,6 +2743,7 @@ pub(crate) async fn make_session_and_context() -> (Session, TurnContext) { skills_outcome, ); + let (mailbox, mailbox_rx) = crate::agent::Mailbox::new(); let session = Session { conversation_id, tx_event, @@ -2753,6 +2754,8 @@ pub(crate) async fn make_session_and_context() -> (Session, TurnContext) { pending_mcp_server_refresh_config: Mutex::new(None), conversation: Arc::new(RealtimeConversationManager::new()), active_turn: Mutex::new(None), + mailbox, + mailbox_rx: Mutex::new(mailbox_rx), idle_pending_input: Mutex::new(Vec::new()), guardian_review_session: crate::guardian::GuardianReviewSessionManager::default(), services, @@ -3586,6 +3589,7 @@ pub(crate) async fn make_session_and_context_with_dynamic_tools_and_rx( skills_outcome, )); + let (mailbox, mailbox_rx) = crate::agent::Mailbox::new(); let session = Arc::new(Session { conversation_id, tx_event, @@ -3596,6 +3600,8 @@ pub(crate) async fn make_session_and_context_with_dynamic_tools_and_rx( pending_mcp_server_refresh_config: Mutex::new(None), conversation: Arc::new(RealtimeConversationManager::new()), active_turn: Mutex::new(None), + mailbox, + mailbox_rx: Mutex::new(mailbox_rx), idle_pending_input: Mutex::new(Vec::new()), guardian_review_session: crate::guardian::GuardianReviewSessionManager::default(), services, diff --git a/codex-rs/core/src/codex_thread.rs b/codex-rs/core/src/codex_thread.rs index 97bca0bba0..f3f9e2aea6 100644 --- a/codex-rs/core/src/codex_thread.rs +++ b/codex-rs/core/src/codex_thread.rs @@ -191,10 +191,7 @@ impl CodexThread { .session .queue_response_items_for_next_turn(items) .await; - self.codex - .session - .ensure_task_for_queued_response_items() - .await; + self.codex.session.maybe_start_turn_for_pending_work().await; } Ok(submission_id) diff --git a/codex-rs/core/src/config/config_tests.rs b/codex-rs/core/src/config/config_tests.rs index 4c4a2cf6a7..ba6598afc7 100644 --- a/codex-rs/core/src/config/config_tests.rs +++ b/codex-rs/core/src/config/config_tests.rs @@ -244,44 +244,22 @@ web_search = false } #[test] -fn config_toml_deserializes_custom_models() { - let custom_models = r#" -[[custom_models]] -name = "gpt-5.4 1m" -model = "gpt-5.4" -model_context_window = 1000000 -model_auto_compact_token_limit = 900000 -"#; - let custom_models_cfg = toml::from_str::(custom_models) - .expect("TOML deserialization should succeed for custom models"); +fn rejects_provider_auth_with_env_key() { + let err = toml::from_str::( + r#" +[model_providers.corp] +name = "Corp" +env_key = "CORP_TOKEN" - assert_eq!( - custom_models_cfg.custom_models, - vec![CustomModelToml { - name: "gpt-5.4 1m".to_string(), - model: "gpt-5.4".to_string(), - model_context_window: Some(1_000_000), - model_auto_compact_token_limit: Some(900_000), - }] - ); - - let config = Config::load_from_base_config_with_overrides( - custom_models_cfg, - ConfigOverrides::default(), - tempdir().expect("tempdir").path().to_path_buf(), +[model_providers.corp.auth] +command = "print-token" +"#, ) - .expect("load config from custom models settings"); + .unwrap_err(); - assert_eq!( - config.custom_models, - HashMap::from([( - "gpt-5.4 1m".to_string(), - CustomModelConfig { - model: "gpt-5.4".to_string(), - model_context_window: Some(1_000_000), - model_auto_compact_token_limit: Some(900_000), - }, - )]) + assert!( + err.to_string() + .contains("model_providers.corp: provider auth cannot be combined with env_key") ); } @@ -4399,6 +4377,7 @@ model_verbosity = "high" wire_api: crate::WireApi::Responses, env_key_instructions: None, experimental_bearer_token: None, + auth: None, query_params: None, http_headers: None, env_http_headers: None, diff --git a/codex-rs/core/src/config/mod.rs b/codex-rs/core/src/config/mod.rs index 92c4bfb06f..3f70089c74 100644 --- a/codex-rs/core/src/config/mod.rs +++ b/codex-rs/core/src/config/mod.rs @@ -1904,6 +1904,18 @@ Built-in providers cannot be overridden. Rename your custom provider (for exampl } } +fn validate_model_providers( + model_providers: &HashMap, +) -> Result<(), String> { + validate_reserved_model_provider_ids(model_providers)?; + for (key, provider) in model_providers { + provider + .validate() + .map_err(|message| format!("model_providers.{key}: {message}"))?; + } + Ok(()) +} + fn deserialize_model_providers<'de, D>( deserializer: D, ) -> Result, D::Error> @@ -1911,7 +1923,7 @@ where D: serde::Deserializer<'de>, { let model_providers = HashMap::::deserialize(deserializer)?; - validate_reserved_model_provider_ids(&model_providers).map_err(serde::de::Error::custom)?; + validate_model_providers(&model_providers).map_err(serde::de::Error::custom)?; Ok(model_providers) } @@ -2036,7 +2048,7 @@ impl Config { codex_home: PathBuf, config_layer_stack: ConfigLayerStack, ) -> std::io::Result { - validate_reserved_model_provider_ids(&cfg.model_providers) + validate_model_providers(&cfg.model_providers) .map_err(|message| std::io::Error::new(std::io::ErrorKind::InvalidInput, message))?; // Ensure that every field of ConfigRequirements is applied to the final // Config. diff --git a/codex-rs/core/src/connectors.rs b/codex-rs/core/src/connectors.rs index e4999b9b41..a15138f6fb 100644 --- a/codex-rs/core/src/connectors.rs +++ b/codex-rs/core/src/connectors.rs @@ -18,6 +18,7 @@ use codex_connectors::AllConnectorsCacheKey; use codex_connectors::DirectoryListResponse; use codex_login::token_data::TokenData; use codex_protocol::protocol::SandboxPolicy; +use codex_tools::DiscoverableTool; use rmcp::model::ToolAnnotations; use serde::Deserialize; use serde::de::DeserializeOwned; @@ -44,8 +45,6 @@ use crate::mcp_connection_manager::codex_apps_tools_cache_key; use crate::plugins::AppConnectorId; use crate::plugins::PluginsManager; use crate::plugins::list_tool_suggest_discoverable_plugins; -use crate::tools::discoverable::DiscoverablePluginInfo; -use crate::tools::discoverable::DiscoverableTool; use codex_features::Feature; pub use codex_connectors::CONNECTORS_CACHE_TTL; @@ -133,7 +132,6 @@ pub(crate) async fn list_tool_suggest_discoverable_tools_with_auth( .map(DiscoverableTool::from); let discoverable_plugins = list_tool_suggest_discoverable_plugins(config)? .into_iter() - .map(DiscoverablePluginInfo::from) .map(DiscoverableTool::from); Ok(discoverable_connectors .chain(discoverable_plugins) diff --git a/codex-rs/core/src/lib.rs b/codex-rs/core/src/lib.rs index 72263afcd7..f17099407f 100644 --- a/codex-rs/core/src/lib.rs +++ b/codex-rs/core/src/lib.rs @@ -65,6 +65,7 @@ pub mod utils; pub use utils::path_utils; pub mod personality_migration; pub mod plugins; +mod provider_auth; pub(crate) mod mentions { pub(crate) use crate::plugins::build_connector_slug_counts; pub(crate) use crate::plugins::build_skill_name_counts; @@ -104,6 +105,7 @@ mod text_encoding; mod unified_exec; pub mod windows_sandbox; pub use client::X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER; +pub use codex_protocol::config_types::ModelProviderAuthInfo; pub use model_provider_info::DEFAULT_LMSTUDIO_PORT; pub use model_provider_info::DEFAULT_OLLAMA_PORT; pub use model_provider_info::LMSTUDIO_OSS_PROVIDER_ID; @@ -133,6 +135,7 @@ pub type CodexConversation = CodexThread; pub use auth::AuthManager; pub use auth::CodexAuth; pub use codex_analytics::AnalyticsEventsClient; +pub use codex_analytics::AppServerRpcTransport; mod default_client_forwarding; /// Default Codex HTTP client headers and reqwest construction. diff --git a/codex-rs/core/src/model_provider_info.rs b/codex-rs/core/src/model_provider_info.rs index 737a47780d..a3df34b64d 100644 --- a/codex-rs/core/src/model_provider_info.rs +++ b/codex-rs/core/src/model_provider_info.rs @@ -9,6 +9,7 @@ use crate::auth::AuthMode; use crate::error::EnvVarError; use codex_api::Provider as ApiProvider; use codex_api::provider::RetryConfig as ApiRetryConfig; +use codex_protocol::config_types::ModelProviderAuthInfo; use http::HeaderMap; use http::header::HeaderName; use http::header::HeaderValue; @@ -86,6 +87,9 @@ pub struct ModelProviderInfo { /// this may be necessary when using this programmatically. pub experimental_bearer_token: Option, + /// Command-backed bearer-token configuration for this provider. + pub auth: Option, + /// Which wire protocol this provider expects. #[serde(default)] pub wire_api: WireApi, @@ -130,6 +134,36 @@ pub struct ModelProviderInfo { } impl ModelProviderInfo { + pub(crate) fn validate(&self) -> std::result::Result<(), String> { + let Some(auth) = self.auth.as_ref() else { + return Ok(()); + }; + + if auth.command.trim().is_empty() { + return Err("provider auth.command must not be empty".to_string()); + } + + let mut conflicts = Vec::new(); + if self.env_key.is_some() { + conflicts.push("env_key"); + } + if self.experimental_bearer_token.is_some() { + conflicts.push("experimental_bearer_token"); + } + if self.requires_openai_auth { + conflicts.push("requires_openai_auth"); + } + + if conflicts.is_empty() { + Ok(()) + } else { + Err(format!( + "provider auth cannot be combined with {}", + conflicts.join(", ") + )) + } + } + fn build_header_map(&self) -> crate::error::Result { let capacity = self.http_headers.as_ref().map_or(0, HashMap::len) + self.env_http_headers.as_ref().map_or(0, HashMap::len); @@ -246,6 +280,7 @@ impl ModelProviderInfo { env_key: None, env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: WireApi::Responses, query_params: None, http_headers: Some( @@ -277,6 +312,10 @@ impl ModelProviderInfo { pub fn is_openai(&self) -> bool { self.name == OPENAI_PROVIDER_NAME } + + pub(crate) fn has_command_auth(&self) -> bool { + self.auth.is_some() + } } pub const DEFAULT_LMSTUDIO_PORT: u16 = 1234; @@ -338,6 +377,7 @@ pub fn create_oss_provider_with_base_url(base_url: &str, wire_api: WireApi) -> M env_key: None, env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api, query_params: None, http_headers: None, diff --git a/codex-rs/core/src/model_provider_info_tests.rs b/codex-rs/core/src/model_provider_info_tests.rs index a5309117ae..676af4add3 100644 --- a/codex-rs/core/src/model_provider_info_tests.rs +++ b/codex-rs/core/src/model_provider_info_tests.rs @@ -1,5 +1,9 @@ use super::*; +use codex_utils_absolute_path::AbsolutePathBuf; +use codex_utils_absolute_path::AbsolutePathBufGuard; use pretty_assertions::assert_eq; +use std::num::NonZeroU64; +use tempfile::tempdir; #[test] fn test_deserialize_ollama_model_provider_toml() { @@ -13,6 +17,7 @@ base_url = "http://localhost:11434/v1" env_key: None, env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: WireApi::Responses, query_params: None, http_headers: None, @@ -43,6 +48,7 @@ query_params = { api-version = "2025-04-01-preview" } env_key: Some("AZURE_OPENAI_API_KEY".into()), env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: WireApi::Responses, query_params: Some(maplit::hashmap! { "api-version".to_string() => "2025-04-01-preview".to_string(), @@ -76,6 +82,7 @@ env_http_headers = { "X-Example-Env-Header" = "EXAMPLE_ENV_VAR" } env_key: Some("API_KEY".into()), env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: WireApi::Responses, query_params: None, http_headers: Some(maplit::hashmap! { @@ -121,3 +128,31 @@ supports_websockets = true let provider: ModelProviderInfo = toml::from_str(provider_toml).unwrap(); assert_eq!(provider.websocket_connect_timeout_ms, Some(15_000)); } + +#[test] +fn test_deserialize_provider_auth_config_defaults() { + let base_dir = tempdir().unwrap(); + let provider_toml = r#" +name = "Corp" + +[auth] +command = "./scripts/print-token" +args = ["--format=text"] + "#; + + let provider: ModelProviderInfo = { + let _guard = AbsolutePathBufGuard::new(base_dir.path()); + toml::from_str(provider_toml).unwrap() + }; + + assert_eq!( + provider.auth, + Some(ModelProviderAuthInfo { + command: "./scripts/print-token".to_string(), + args: vec!["--format=text".to_string()], + timeout_ms: NonZeroU64::new(5_000).unwrap(), + refresh_interval_ms: NonZeroU64::new(300_000).unwrap(), + cwd: AbsolutePathBuf::resolve_path_against_base(".", base_dir.path()).unwrap(), + }) + ); +} diff --git a/codex-rs/core/src/models_manager/manager.rs b/codex-rs/core/src/models_manager/manager.rs index dc6a856e3f..94b8111b6d 100644 --- a/codex-rs/core/src/models_manager/manager.rs +++ b/codex-rs/core/src/models_manager/manager.rs @@ -15,6 +15,7 @@ use crate::model_provider_info::ModelProviderInfo; use crate::models_manager::collaboration_mode_presets::CollaborationModesConfig; use crate::models_manager::collaboration_mode_presets::builtin_collaboration_mode_presets; use crate::models_manager::model_info; +use crate::provider_auth::required_auth_manager_for_provider; use crate::response_debug_context::extract_response_debug_context; use crate::response_debug_context::telemetry_transport_error_message; use crate::util::FeedbackRequestTags; @@ -219,6 +220,7 @@ impl ModelsManager { collaboration_modes_config: CollaborationModesConfig, provider: ModelProviderInfo, ) -> Self { + let auth_manager = required_auth_manager_for_provider(auth_manager, &provider); let cache_path = codex_home.join(MODEL_CACHE_FILE); let cache_manager = ModelsCacheManager::new(cache_path, DEFAULT_MODEL_CACHE_TTL); let catalog_mode = if model_catalog.is_some() { @@ -459,7 +461,9 @@ impl ModelsManager { return Ok(()); } - if self.auth_manager.auth_mode() != Some(AuthMode::Chatgpt) { + if self.auth_manager.auth_mode() != Some(AuthMode::Chatgpt) + && !self.provider.has_command_auth() + { if matches!( refresh_strategy, RefreshStrategy::Offline | RefreshStrategy::OnlineIfUncached diff --git a/codex-rs/core/src/models_manager/manager_tests.rs b/codex-rs/core/src/models_manager/manager_tests.rs index 68ff719f1a..9c3866cdc3 100644 --- a/codex-rs/core/src/models_manager/manager_tests.rs +++ b/codex-rs/core/src/models_manager/manager_tests.rs @@ -1,5 +1,7 @@ use super::*; +use crate::AuthManager; use crate::CodexAuth; +use crate::ModelProviderAuthInfo; use crate::auth::AuthCredentialsStoreMode; use crate::config::ConfigBuilder; use crate::model_provider_info::WireApi; @@ -13,8 +15,10 @@ use http::StatusCode; use pretty_assertions::assert_eq; use serde_json::json; use std::collections::BTreeMap; +use std::num::NonZeroU64; use std::sync::Arc; use std::sync::Mutex; +use tempfile::TempDir; use tempfile::tempdir; use tracing::Event; use tracing::Subscriber; @@ -24,7 +28,12 @@ use tracing_subscriber::layer::Context; use tracing_subscriber::layer::SubscriberExt; use tracing_subscriber::registry::LookupSpan; use tracing_subscriber::util::SubscriberInitExt; +use wiremock::Mock; use wiremock::MockServer; +use wiremock::ResponseTemplate; +use wiremock::matchers::header_regex; +use wiremock::matchers::method; +use wiremock::matchers::path; fn remote_model(slug: &str, display: &str, priority: i32) -> ModelInfo { remote_model_with_visibility(slug, display, priority, "list") @@ -79,6 +88,7 @@ fn provider_for(base_url: String) -> ModelProviderInfo { env_key: None, env_key_instructions: None, experimental_bearer_token: None, + auth: None, wire_api: WireApi::Responses, query_params: None, http_headers: None, @@ -92,6 +102,95 @@ fn provider_for(base_url: String) -> ModelProviderInfo { } } +struct ProviderAuthScript { + tempdir: TempDir, + command: String, + args: Vec, +} + +impl ProviderAuthScript { + fn new(tokens: &[&str]) -> std::io::Result { + let tempdir = tempfile::tempdir()?; + let tokens_file = tempdir.path().join("tokens.txt"); + let mut token_file_contents = String::new(); + for token in tokens { + token_file_contents.push_str(token); + token_file_contents.push('\n'); + } + std::fs::write(&tokens_file, token_file_contents)?; + + #[cfg(unix)] + let (command, args) = { + let script_path = tempdir.path().join("print-token.sh"); + std::fs::write( + &script_path, + r#"#!/bin/sh +first_line=$(sed -n '1p' tokens.txt) +printf '%s\n' "$first_line" +tail -n +2 tokens.txt > tokens.next +mv tokens.next tokens.txt +"#, + )?; + let mut permissions = std::fs::metadata(&script_path)?.permissions(); + { + use std::os::unix::fs::PermissionsExt; + permissions.set_mode(0o755); + } + std::fs::set_permissions(&script_path, permissions)?; + ("./print-token.sh".to_string(), Vec::new()) + }; + + #[cfg(windows)] + let (command, args) = { + let script_path = tempdir.path().join("print-token.ps1"); + std::fs::write( + &script_path, + r#"$lines = @(Get-Content -Path tokens.txt) +if ($lines.Count -eq 0) { exit 1 } +Write-Output $lines[0] +$lines | Select-Object -Skip 1 | Set-Content -Path tokens.txt +"#, + )?; + ( + "powershell".to_string(), + vec![ + "-NoProfile".to_string(), + "-ExecutionPolicy".to_string(), + "Bypass".to_string(), + "-File".to_string(), + ".\\print-token.ps1".to_string(), + ], + ) + }; + + Ok(Self { + tempdir, + command, + args, + }) + } + + fn auth_config(&self) -> ModelProviderAuthInfo { + ModelProviderAuthInfo { + command: self.command.clone(), + args: self.args.clone(), + timeout_ms: non_zero_u64(/*value*/ 1_000), + refresh_interval_ms: non_zero_u64(/*value*/ 60_000), + cwd: match codex_utils_absolute_path::AbsolutePathBuf::try_from(self.tempdir.path()) { + Ok(cwd) => cwd, + Err(err) => panic!("tempdir should be absolute: {err}"), + }, + } + } +} + +fn non_zero_u64(value: u64) -> NonZeroU64 { + match NonZeroU64::new(value) { + Some(value) => value, + None => panic!("expected non-zero value: {value}"), + } +} + #[derive(Default)] struct TagCollectorVisitor { tags: BTreeMap, @@ -314,6 +413,50 @@ async fn refresh_available_models_sorts_by_priority() { ); } +#[tokio::test] +async fn refresh_available_models_uses_provider_auth_token() { + let server = MockServer::start().await; + let auth_script = ProviderAuthScript::new(&["provider-token"]).unwrap(); + let remote_models = vec![remote_model( + "provider-model", + "Provider", + /*priority*/ 0, + )]; + + Mock::given(method("GET")) + .and(path("/models")) + .and(header_regex("Authorization", "Bearer provider-token")) + .respond_with( + ResponseTemplate::new(200) + .insert_header("content-type", "application/json") + .set_body_json(ModelsResponse { + models: remote_models.clone(), + }), + ) + .expect(1) + .mount(&server) + .await; + + let codex_home = tempdir().expect("temp dir"); + let auth_manager = AuthManager::from_auth_for_testing(CodexAuth::from_api_key("unused")); + let provider = ModelProviderInfo { + auth: Some(auth_script.auth_config()), + ..provider_for(server.uri()) + }; + let manager = ModelsManager::with_provider_for_tests( + codex_home.path().to_path_buf(), + auth_manager, + provider, + ); + + manager + .refresh_available_models(RefreshStrategy::Online) + .await + .expect("refresh succeeds"); + + assert_models_contain(&manager.get_remote_models().await, &remote_models); +} + #[tokio::test] async fn refresh_available_models_uses_cache_when_fresh() { let server = MockServer::start().await; diff --git a/codex-rs/core/src/plugins/discoverable.rs b/codex-rs/core/src/plugins/discoverable.rs index 427706235f..f7006185bf 100644 --- a/codex-rs/core/src/plugins/discoverable.rs +++ b/codex-rs/core/src/plugins/discoverable.rs @@ -9,6 +9,7 @@ use super::PluginsManager; use crate::config::Config; use crate::config::types::ToolSuggestDiscoverableType; use codex_features::Feature; +use codex_tools::DiscoverablePluginInfo; const TOOL_SUGGEST_DISCOVERABLE_PLUGIN_ALLOWLIST: &[&str] = &[ "github@openai-curated", @@ -23,7 +24,7 @@ const TOOL_SUGGEST_DISCOVERABLE_PLUGIN_ALLOWLIST: &[&str] = &[ pub(crate) fn list_tool_suggest_discoverable_plugins( config: &Config, -) -> anyhow::Result> { +) -> anyhow::Result> { if !config.features.enabled(Feature::Plugins) { return Ok(Vec::new()); } @@ -47,7 +48,7 @@ pub(crate) fn list_tool_suggest_discoverable_plugins( return Ok(Vec::new()); }; - let mut discoverable_plugins = Vec::::new(); + let mut discoverable_plugins = Vec::::new(); for plugin in curated_marketplace.plugins { if plugin.installed || (!TOOL_SUGGEST_DISCOVERABLE_PLUGIN_ALLOWLIST.contains(&plugin.id.as_str()) @@ -66,14 +67,28 @@ pub(crate) fn list_tool_suggest_discoverable_plugins( marketplace_path: curated_marketplace.path.clone(), }, ) { - Ok(plugin) => discoverable_plugins.push(plugin.plugin.into()), + Ok(plugin) => { + let plugin: PluginCapabilitySummary = plugin.plugin.into(); + discoverable_plugins.push(DiscoverablePluginInfo { + id: plugin.config_name, + name: plugin.display_name, + description: plugin.description, + has_skills: plugin.has_skills, + mcp_server_names: plugin.mcp_server_names, + app_connector_ids: plugin + .app_connector_ids + .into_iter() + .map(|connector_id| connector_id.0) + .collect(), + }); + } Err(err) => warn!("failed to load discoverable plugin suggestion {plugin_id}: {err:#}"), } } discoverable_plugins.sort_by(|left, right| { - left.display_name - .cmp(&right.display_name) - .then_with(|| left.config_name.cmp(&right.config_name)) + left.name + .cmp(&right.name) + .then_with(|| left.id.cmp(&right.id)) }); Ok(discoverable_plugins) } diff --git a/codex-rs/core/src/plugins/discoverable_tests.rs b/codex-rs/core/src/plugins/discoverable_tests.rs index fd9df9ddf6..70ac887cb6 100644 --- a/codex-rs/core/src/plugins/discoverable_tests.rs +++ b/codex-rs/core/src/plugins/discoverable_tests.rs @@ -5,7 +5,7 @@ use crate::plugins::test_support::write_curated_plugin_sha; use crate::plugins::test_support::write_file; use crate::plugins::test_support::write_openai_curated_marketplace; use crate::plugins::test_support::write_plugins_feature_config; -use crate::tools::discoverable::DiscoverablePluginInfo; +use codex_tools::DiscoverablePluginInfo; use codex_utils_absolute_path::AbsolutePathBuf; use pretty_assertions::assert_eq; use tempfile::tempdir; @@ -18,11 +18,7 @@ async fn list_tool_suggest_discoverable_plugins_returns_uninstalled_curated_plug write_plugins_feature_config(codex_home.path()); let config = load_plugins_config(codex_home.path()).await; - let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config) - .unwrap() - .into_iter() - .map(DiscoverablePluginInfo::from) - .collect::>(); + let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config).unwrap(); assert_eq!( discoverable_plugins, @@ -52,11 +48,7 @@ plugins = false ); let config = load_plugins_config(codex_home.path()).await; - let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config) - .unwrap() - .into_iter() - .map(DiscoverablePluginInfo::from) - .collect::>(); + let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config).unwrap(); assert_eq!(discoverable_plugins, Vec::::new()); } @@ -76,11 +68,7 @@ async fn list_tool_suggest_discoverable_plugins_normalizes_description() { ); let config = load_plugins_config(codex_home.path()).await; - let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config) - .unwrap() - .into_iter() - .map(DiscoverablePluginInfo::from) - .collect::>(); + let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config).unwrap(); assert_eq!( discoverable_plugins, @@ -115,11 +103,7 @@ async fn list_tool_suggest_discoverable_plugins_omits_installed_curated_plugins( .expect("plugin should install"); let refreshed_config = load_plugins_config(codex_home.path()).await; - let discoverable_plugins = list_tool_suggest_discoverable_plugins(&refreshed_config) - .unwrap() - .into_iter() - .map(DiscoverablePluginInfo::from) - .collect::>(); + let discoverable_plugins = list_tool_suggest_discoverable_plugins(&refreshed_config).unwrap(); assert_eq!(discoverable_plugins, Vec::::new()); } @@ -140,11 +124,7 @@ discoverables = [{ type = "plugin", id = "sample@openai-curated" }] ); let config = load_plugins_config(codex_home.path()).await; - let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config) - .unwrap() - .into_iter() - .map(DiscoverablePluginInfo::from) - .collect::>(); + let discoverable_plugins = list_tool_suggest_discoverable_plugins(&config).unwrap(); assert_eq!( discoverable_plugins, diff --git a/codex-rs/core/src/provider_auth.rs b/codex-rs/core/src/provider_auth.rs new file mode 100644 index 0000000000..4737ccc8bc --- /dev/null +++ b/codex-rs/core/src/provider_auth.rs @@ -0,0 +1,31 @@ +use std::sync::Arc; + +use crate::AuthManager; +use crate::model_provider_info::ModelProviderInfo; + +/// Returns the provider-scoped auth manager when this provider uses command-backed auth. +/// +/// Providers without custom auth continue using the caller-supplied base manager. +pub(crate) fn auth_manager_for_provider( + auth_manager: Option>, + provider: &ModelProviderInfo, +) -> Option> { + match provider.auth.clone() { + Some(config) => Some(AuthManager::external_bearer_only(config)), + None => auth_manager, + } +} + +/// Returns an auth manager for request paths that always require authentication. +/// +/// Providers with command-backed auth get a bearer-only manager; otherwise the caller's manager +/// is reused unchanged. +pub(crate) fn required_auth_manager_for_provider( + auth_manager: Arc, + provider: &ModelProviderInfo, +) -> Arc { + match provider.auth.clone() { + Some(config) => AuthManager::external_bearer_only(config), + None => auth_manager, + } +} diff --git a/codex-rs/core/src/realtime_conversation.rs b/codex-rs/core/src/realtime_conversation.rs index 1ddd72d0fd..1bfe1b5a88 100644 --- a/codex-rs/core/src/realtime_conversation.rs +++ b/codex-rs/core/src/realtime_conversation.rs @@ -452,7 +452,12 @@ async fn prepare_realtime_start( params: ConversationStartParams, ) -> CodexResult { let provider = sess.provider().await; - let auth = sess.services.auth_manager.auth().await; + let auth_manager = sess + .services + .model_client + .auth_manager() + .unwrap_or_else(|| Arc::clone(&sess.services.auth_manager)); + let auth = auth_manager.auth().await; let realtime_api_key = realtime_api_key(auth.as_ref(), &provider)?; let mut api_provider = provider.to_api_provider(Some(crate::auth::AuthMode::ApiKey))?; let config = sess.get_config().await; diff --git a/codex-rs/core/src/shell_snapshot_tests.rs b/codex-rs/core/src/shell_snapshot_tests.rs index 90288300ff..ff700ff7a6 100644 --- a/codex-rs/core/src/shell_snapshot_tests.rs +++ b/codex-rs/core/src/shell_snapshot_tests.rs @@ -313,9 +313,15 @@ async fn timed_out_snapshot_shell_is_terminated() -> Result<()> { shell_snapshot: crate::shell::empty_shell_snapshot_receiver(), }; - let err = run_script_with_timeout(&shell, &script, Duration::from_secs(1), true, dir.path()) - .await - .expect_err("snapshot shell should time out"); + let err = run_script_with_timeout( + &shell, + &script, + Duration::from_secs(1), + /*use_login_shell*/ true, + dir.path(), + ) + .await + .expect_err("snapshot shell should time out"); assert!( err.to_string().contains("timed out"), "expected timeout error, got {err:?}" diff --git a/codex-rs/core/src/tasks/mod.rs b/codex-rs/core/src/tasks/mod.rs index e558c8f9c1..86043bfc87 100644 --- a/codex-rs/core/src/tasks/mod.rs +++ b/codex-rs/core/src/tasks/mod.rs @@ -58,7 +58,7 @@ pub(crate) use user_shell::UserShellCommandTask; pub(crate) use user_shell::execute_user_shell_command; const GRACEFULL_INTERRUPTION_TIMEOUT_MS: u64 = 100; -const TURN_ABORTED_INTERRUPTED_GUIDANCE: &str = "The user interrupted the previous turn on purpose. Any running unified exec processes may still be running in the background. If any tools/commands were aborted, they may have partially executed; verify current state before retrying."; +const TURN_ABORTED_INTERRUPTED_GUIDANCE: &str = "The user interrupted the previous turn on purpose. Any running unified exec processes may still be running in the background. If any tools/commands were aborted, they may have partially executed."; /// Shared model-visible marker used by both the real interrupt path and /// interrupted fork snapshots. @@ -191,60 +191,68 @@ impl Session { let cancellation_token = CancellationToken::new(); let done = Arc::new(Notify::new()); - let done_clone = Arc::clone(&done); - let handle = { - let session_ctx = Arc::new(SessionTaskContext::new(Arc::clone(self))); - let ctx = Arc::clone(&turn_context); - let task_for_run = Arc::clone(&task); - let task_cancellation_token = cancellation_token.child_token(); - // Task-owned turn spans keep a core-owned span open for the - // full task lifecycle after the submission dispatch span ends. - let task_span = info_span!( - "turn", - otel.name = span_name, - thread.id = %self.conversation_id, - turn.id = %turn_context.sub_id, - model = %turn_context.model_info.slug, - ); - tokio::spawn( - async move { - let ctx_for_finish = Arc::clone(&ctx); - let last_agent_message = task_for_run - .run( - Arc::clone(&session_ctx), - ctx, - input, - task_cancellation_token.child_token(), - ) - .await; - let sess = session_ctx.clone_session(); - sess.flush_rollout().await; - if !task_cancellation_token.is_cancelled() { - // Emit completion uniformly from spawn site so all tasks share the same lifecycle. - sess.on_task_finished(Arc::clone(&ctx_for_finish), last_agent_message) - .await; - } - done_clone.notify_waiters(); - } - .instrument(task_span), - ) - }; - let queued_response_items = self.take_queued_response_items_for_next_turn().await; - let mut active = self.active_turn.lock().await; - let mut turn = ActiveTurn::default(); - let mut turn_state = turn.turn_state.lock().await; - turn_state.token_usage_at_turn_start = token_usage_at_turn_start; - for item in queued_response_items { - turn_state.push_pending_input(item); + let mailbox_items = self.get_pending_input().await; + let turn_state = { + let mut active = self.active_turn.lock().await; + let turn = active.get_or_insert_with(ActiveTurn::default); + debug_assert!(turn.tasks.is_empty()); + Arc::clone(&turn.turn_state) + }; + { + let mut turn_state = turn_state.lock().await; + turn_state.token_usage_at_turn_start = token_usage_at_turn_start; + for item in queued_response_items { + turn_state.push_pending_input(item); + } + for item in mailbox_items { + turn_state.push_pending_input(item); + } } - drop(turn_state); + let mut active = self.active_turn.lock().await; + let turn = active.get_or_insert_with(ActiveTurn::default); + debug_assert!(turn.tasks.is_empty()); + let done_clone = Arc::clone(&done); + let session_ctx = Arc::new(SessionTaskContext::new(Arc::clone(self))); + let ctx = Arc::clone(&turn_context); + let task_for_run = Arc::clone(&task); + let task_cancellation_token = cancellation_token.child_token(); + // Task-owned turn spans keep a core-owned span open for the + // full task lifecycle after the submission dispatch span ends. + let task_span = info_span!( + "turn", + otel.name = span_name, + thread.id = %self.conversation_id, + turn.id = %turn_context.sub_id, + model = %turn_context.model_info.slug, + ); + let handle = tokio::spawn( + async move { + let ctx_for_finish = Arc::clone(&ctx); + let last_agent_message = task_for_run + .run( + Arc::clone(&session_ctx), + ctx, + input, + task_cancellation_token.child_token(), + ) + .await; + let sess = session_ctx.clone_session(); + sess.flush_rollout().await; + if !task_cancellation_token.is_cancelled() { + // Emit completion uniformly from spawn site so all tasks share the same lifecycle. + sess.on_task_finished(Arc::clone(&ctx_for_finish), last_agent_message) + .await; + } + done_clone.notify_waiters(); + } + .instrument(task_span), + ); let timer = turn_context .session_telemetry .start_timer(TURN_E2E_DURATION_METRIC, &[]) .ok(); - let running_task = RunningTask { done, handle: Arc::new(AbortOnDropHandle::new(handle)), @@ -255,19 +263,44 @@ impl Session { _timer: timer, }; turn.add_task(running_task); - *active = Some(turn); } - pub(crate) async fn ensure_task_for_queued_response_items(self: &Arc) { - if !self.has_queued_response_items_for_next_turn().await { + /// Starts a regular turn when the session is idle and pending work is waiting. + /// + /// Pending work currently includes queued next-turn items and mailbox mail marked with + /// `trigger_turn`. + /// + /// This helper generates a fresh sub-id for the synthetic turn before delegating to the + /// explicit-sub-id variant. + pub(crate) async fn maybe_start_turn_for_pending_work(self: &Arc) { + self.maybe_start_turn_for_pending_work_with_sub_id(uuid::Uuid::new_v4().to_string()) + .await; + } + + /// Starts a regular turn with the provided sub-id when pending work should wake an idle + /// session. + /// + /// The turn is created only when there are queued next-turn items or mailbox mail marked with + /// `trigger_turn`, and only if the session is currently idle. + pub(crate) async fn maybe_start_turn_for_pending_work_with_sub_id( + self: &Arc, + sub_id: String, + ) { + if !self.has_queued_response_items_for_next_turn().await + && !self.has_trigger_turn_mailbox_items().await + { return; } - if self.active_turn.lock().await.is_some() { - return; + { + let mut active_turn = self.active_turn.lock().await; + if active_turn.is_some() { + return; + } + *active_turn = Some(ActiveTurn::default()); } - let turn_context = self.new_default_turn().await; + let turn_context = self.new_default_turn_with_sub_id(sub_id).await; self.maybe_emit_unknown_model_warning_for_turn(turn_context.as_ref()) .await; self.start_task(turn_context, Vec::new(), RegularTask::new()) @@ -284,7 +317,7 @@ impl Session { active_turn.clear_pending().await; } if reason == TurnAbortReason::Interrupted { - self.ensure_task_for_queued_response_items().await; + self.maybe_start_turn_for_pending_work().await; } } @@ -298,24 +331,31 @@ impl Session { .cancel_git_enrichment_task(); self.snapshot_agent_send_input_on_turn_complete(); - let mut active = self.active_turn.lock().await; let mut pending_input = Vec::::new(); let mut should_clear_active_turn = false; let mut token_usage_at_turn_start = None; let mut turn_tool_calls = 0_u64; - if let Some(at) = active.as_mut() - && at.remove_task(&turn_context.sub_id) - { - let mut ts = at.turn_state.lock().await; + let turn_state = { + let mut active = self.active_turn.lock().await; + if let Some(at) = active.as_mut() + && at.remove_task(&turn_context.sub_id) + { + should_clear_active_turn = true; + let turn_state = Arc::clone(&at.turn_state); + if should_clear_active_turn { + *active = None; + } + Some(turn_state) + } else { + None + } + }; + if let Some(turn_state) = turn_state { + let mut ts = turn_state.lock().await; pending_input = ts.take_pending_input(); turn_tool_calls = ts.tool_calls; token_usage_at_turn_start = Some(ts.token_usage_at_turn_start.clone()); - should_clear_active_turn = true; } - if should_clear_active_turn { - *active = None; - } - drop(active); if !pending_input.is_empty() { for pending_input_item in pending_input { match inspect_pending_input(self, &turn_context, pending_input_item).await { @@ -414,6 +454,15 @@ impl Session { last_agent_message, }); self.send_event(turn_context.as_ref(), event).await; + + if should_clear_active_turn { + let session = Arc::clone(self); + let _scheduler = tokio::task::spawn_blocking(move || { + tokio::runtime::Handle::current().block_on(async move { + session.maybe_start_turn_for_pending_work().await; + }); + }); + } } async fn take_active_turn(&self) -> Option { diff --git a/codex-rs/core/src/thread_rollout_truncation.rs b/codex-rs/core/src/thread_rollout_truncation.rs index e2d4b58188..c6af00c450 100644 --- a/codex-rs/core/src/thread_rollout_truncation.rs +++ b/codex-rs/core/src/thread_rollout_truncation.rs @@ -3,12 +3,14 @@ //! In core, "user turns" are detected by scanning `ResponseItem::Message` items and //! interpreting them via `event_mapping::parse_turn_item(...)`. +use crate::context_manager::is_user_turn_boundary; use crate::event_mapping; use crate::resolve_fork_reference_rollout_path; use crate::rollout::RolloutRecorder; use codex_protocol::items::TurnItem; use codex_protocol::models::ResponseItem; use codex_protocol::protocol::EventMsg; +use codex_protocol::protocol::InterAgentCommunication; use codex_protocol::protocol::RolloutItem; use std::path::Path; use tracing::warn; @@ -44,6 +46,52 @@ pub(crate) fn user_message_positions_in_rollout(items: &[RolloutItem]) -> Vec Vec { + let mut rollback_turn_positions = Vec::new(); + let mut fork_turn_positions = Vec::new(); + for (idx, item) in items.iter().enumerate() { + match item { + RolloutItem::ResponseItem(item) => { + if is_user_turn_boundary(item) { + rollback_turn_positions.push(idx); + } + if is_real_user_message_boundary(item) || is_trigger_turn_boundary(item) { + fork_turn_positions.push(idx); + } + } + RolloutItem::EventMsg(EventMsg::ThreadRolledBack(rollback)) => { + let num_turns = usize::try_from(rollback.num_turns).unwrap_or(usize::MAX); + if num_turns == 0 { + continue; + } + let Some(rollback_start_idx) = rollback_turn_positions + .len() + .checked_sub(num_turns) + .map(|rollback_start| rollback_turn_positions[rollback_start]) + .or_else(|| rollback_turn_positions.first().copied()) + else { + continue; + }; + let new_rollback_len = rollback_turn_positions.len().saturating_sub(num_turns); + rollback_turn_positions.truncate(new_rollback_len); + fork_turn_positions.retain(|position| *position < rollback_start_idx); + } + _ => {} + } + } + fork_turn_positions +} + /// Return a prefix of `items` obtained by cutting strictly before the nth user message. /// /// The boundary index is 0-based from the start of `items` (so `n_from_start = 0` returns @@ -72,75 +120,41 @@ pub(crate) fn truncate_rollout_before_nth_user_message_from_start( items[..cut_idx].to_vec() } -pub(crate) async fn materialize_rollout_items_for_replay( - codex_home: &Path, - rollout_items: &[RolloutItem], +/// Return a suffix of `items` that keeps the last `n_from_end` fork turns. +/// +/// If fewer than or equal to `n_from_end` fork turns exist, this returns the full rollout. +pub(crate) fn truncate_rollout_to_last_n_fork_turns( + items: &[RolloutItem], + n_from_end: usize, ) -> Vec { - const MAX_FORK_REFERENCE_DEPTH: usize = 8; - - let mut materialized = Vec::new(); - let mut stack: Vec<(Vec, usize, usize)> = vec![(rollout_items.to_vec(), 0, 0)]; - - while let Some((items, mut idx, depth)) = stack.pop() { - while idx < items.len() { - match &items[idx] { - RolloutItem::ForkReference(reference) => { - if depth >= MAX_FORK_REFERENCE_DEPTH { - warn!( - "skipping fork reference recursion at depth {} for {:?}", - depth, reference.rollout_path - ); - idx += 1; - continue; - } - - let resolved_rollout_path = match resolve_fork_reference_rollout_path( - codex_home, - &reference.rollout_path, - ) - .await - { - Ok(path) => path, - Err(err) => { - warn!( - "failed to resolve fork reference rollout {:?}: {err}", - reference.rollout_path - ); - idx += 1; - continue; - } - }; - let parent_history = match RolloutRecorder::get_rollout_history( - &resolved_rollout_path, - ) - .await - { - Ok(history) => history, - Err(err) => { - warn!( - "failed to load fork reference rollout {:?} (resolved from {:?}): {err}", - resolved_rollout_path, reference.rollout_path - ); - idx += 1; - continue; - } - }; - let parent_items = truncate_rollout_before_nth_user_message_from_start( - &parent_history.get_rollout_items(), - reference.nth_user_message, - ); - - stack.push((items, idx + 1, depth)); - stack.push((parent_items, 0, depth + 1)); - break; - } - item => materialized.push(item.clone()), - } - idx += 1; - } + if n_from_end == 0 { + return Vec::new(); } - materialized + let fork_turn_positions = fork_turn_positions_in_rollout(items); + if fork_turn_positions.len() <= n_from_end { + return items.to_vec(); + } + + let keep_idx = fork_turn_positions[fork_turn_positions.len() - n_from_end]; + items[keep_idx..].to_vec() +} + +fn is_real_user_message_boundary(item: &ResponseItem) -> bool { + matches!( + event_mapping::parse_turn_item(item), + Some(TurnItem::UserMessage(_)) + ) +} + +fn is_trigger_turn_boundary(item: &ResponseItem) -> bool { + let ResponseItem::Message { role, content, .. } = item else { + return false; + }; + + role == "assistant" + && InterAgentCommunication::from_message_content(content) + .is_some_and(|communication| communication.trigger_turn) } #[cfg(test)] diff --git a/codex-rs/core/src/thread_rollout_truncation_tests.rs b/codex-rs/core/src/thread_rollout_truncation_tests.rs index d251b722d9..d1380b7d45 100644 --- a/codex-rs/core/src/thread_rollout_truncation_tests.rs +++ b/codex-rs/core/src/thread_rollout_truncation_tests.rs @@ -1,7 +1,9 @@ use super::*; use crate::codex::make_session_and_context; +use codex_protocol::AgentPath; use codex_protocol::models::ContentItem; use codex_protocol::models::ReasoningItemReasoningSummary; +use codex_protocol::protocol::InterAgentCommunication; use codex_protocol::protocol::ThreadRolledBackEvent; use pretty_assertions::assert_eq; @@ -29,6 +31,17 @@ fn assistant_msg(text: &str) -> ResponseItem { } } +fn inter_agent_msg(text: &str, trigger_turn: bool) -> ResponseItem { + let communication = InterAgentCommunication::new( + AgentPath::root(), + AgentPath::try_from("/root/worker").expect("agent path"), + Vec::new(), + text.to_string(), + trigger_turn, + ); + communication.to_response_input_item().into() +} + #[test] fn truncates_rollout_from_start_before_nth_user_only() { let items = [ @@ -157,3 +170,149 @@ async fn ignores_session_prefix_messages_when_truncating_rollout_from_start() { serde_json::to_value(&expected).unwrap() ); } + +#[test] +fn truncates_rollout_to_last_n_fork_turns_counts_trigger_turn_messages() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(assistant_msg("a1")), + RolloutItem::ResponseItem(inter_agent_msg( + "queued message", + /*trigger_turn*/ false, + )), + RolloutItem::ResponseItem(assistant_msg("a2")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a3")), + RolloutItem::ResponseItem(user_msg("u2")), + RolloutItem::ResponseItem(assistant_msg("a4")), + ]; + + let truncated = truncate_rollout_to_last_n_fork_turns(&rollout, /*n_from_end*/ 2); + let expected = rollout[4..].to_vec(); + + assert_eq!( + serde_json::to_value(&truncated).unwrap(), + serde_json::to_value(&expected).unwrap() + ); +} + +#[test] +fn truncates_rollout_to_last_n_fork_turns_applies_thread_rollback_markers() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(assistant_msg("a1")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a2")), + RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent { + num_turns: 1, + })), + RolloutItem::ResponseItem(user_msg("u2")), + RolloutItem::ResponseItem(assistant_msg("a3")), + ]; + + let truncated = truncate_rollout_to_last_n_fork_turns(&rollout, /*n_from_end*/ 2); + + assert_eq!( + serde_json::to_value(&truncated).unwrap(), + serde_json::to_value(&rollout).unwrap() + ); +} + +#[test] +fn fork_turn_positions_ignore_zero_turn_rollback_markers() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task", + /*trigger_turn*/ true, + )), + RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent { + num_turns: 0, + })), + RolloutItem::ResponseItem(user_msg("u2")), + ]; + + assert_eq!(fork_turn_positions_in_rollout(&rollout), vec![0, 1, 3]); +} + +#[test] +fn truncates_rollout_to_last_n_fork_turns_discards_trigger_boundaries_in_rolled_back_suffix() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(user_msg("u2")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a1")), + RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent { + num_turns: 1, + })), + RolloutItem::ResponseItem(user_msg("u3")), + RolloutItem::ResponseItem(assistant_msg("a2")), + ]; + + let truncated = truncate_rollout_to_last_n_fork_turns(&rollout, /*n_from_end*/ 2); + + let expected = rollout[1..].to_vec(); + + assert_eq!( + serde_json::to_value(&truncated).unwrap(), + serde_json::to_value(&expected).unwrap() + ); +} + +#[test] +fn truncates_rollout_to_last_n_fork_turns_discards_rolled_back_assistant_instruction_turns() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(assistant_msg("a1")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task 1", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a2")), + RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent { + num_turns: 1, + })), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task 2", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a3")), + ]; + + let truncated = truncate_rollout_to_last_n_fork_turns(&rollout, /*n_from_end*/ 1); + let expected = rollout[5..].to_vec(); + + assert_eq!( + serde_json::to_value(&truncated).unwrap(), + serde_json::to_value(&expected).unwrap() + ); +} + +#[test] +fn truncates_rollout_to_last_n_fork_turns_keeps_full_rollout_when_n_is_large() { + let rollout = vec![ + RolloutItem::ResponseItem(user_msg("u1")), + RolloutItem::ResponseItem(assistant_msg("a1")), + RolloutItem::ResponseItem(inter_agent_msg( + "triggered task", + /*trigger_turn*/ true, + )), + RolloutItem::ResponseItem(assistant_msg("a2")), + ]; + + let truncated = truncate_rollout_to_last_n_fork_turns(&rollout, /*n_from_end*/ 10); + + assert_eq!( + serde_json::to_value(&truncated).unwrap(), + serde_json::to_value(&rollout).unwrap() + ); +} diff --git a/codex-rs/core/src/tools/discoverable.rs b/codex-rs/core/src/tools/discoverable.rs deleted file mode 100644 index e9403918e4..0000000000 --- a/codex-rs/core/src/tools/discoverable.rs +++ /dev/null @@ -1,134 +0,0 @@ -use crate::plugins::PluginCapabilitySummary; -use codex_app_server_protocol::AppInfo; -use serde::Deserialize; -use serde::Serialize; - -const TUI_CLIENT_NAME: &str = "codex-tui"; - -#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub(crate) enum DiscoverableToolType { - Connector, - Plugin, -} - -impl DiscoverableToolType { - pub(crate) fn as_str(self) -> &'static str { - match self { - Self::Connector => "connector", - Self::Plugin => "plugin", - } - } -} - -#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub(crate) enum DiscoverableToolAction { - Install, - Enable, -} - -impl DiscoverableToolAction { - pub(crate) fn as_str(self) -> &'static str { - match self { - Self::Install => "install", - Self::Enable => "enable", - } - } -} - -#[derive(Clone, Debug, PartialEq)] -pub(crate) enum DiscoverableTool { - Connector(Box), - Plugin(Box), -} - -impl DiscoverableTool { - pub(crate) fn tool_type(&self) -> DiscoverableToolType { - match self { - Self::Connector(_) => DiscoverableToolType::Connector, - Self::Plugin(_) => DiscoverableToolType::Plugin, - } - } - - pub(crate) fn id(&self) -> &str { - match self { - Self::Connector(connector) => connector.id.as_str(), - Self::Plugin(plugin) => plugin.id.as_str(), - } - } - - pub(crate) fn name(&self) -> &str { - match self { - Self::Connector(connector) => connector.name.as_str(), - Self::Plugin(plugin) => plugin.name.as_str(), - } - } - - pub(crate) fn description(&self) -> Option<&str> { - match self { - Self::Connector(connector) => connector.description.as_deref(), - Self::Plugin(plugin) => plugin.description.as_deref(), - } - } - - pub(crate) fn install_url(&self) -> Option<&str> { - match self { - Self::Connector(connector) => connector.install_url.as_deref(), - Self::Plugin(_) => None, - } - } -} - -impl From for DiscoverableTool { - fn from(value: AppInfo) -> Self { - Self::Connector(Box::new(value)) - } -} - -impl From for DiscoverableTool { - fn from(value: DiscoverablePluginInfo) -> Self { - Self::Plugin(Box::new(value)) - } -} - -pub(crate) fn filter_tool_suggest_discoverable_tools_for_client( - discoverable_tools: Vec, - app_server_client_name: Option<&str>, -) -> Vec { - if app_server_client_name != Some(TUI_CLIENT_NAME) { - return discoverable_tools; - } - - discoverable_tools - .into_iter() - .filter(|tool| !matches!(tool, DiscoverableTool::Plugin(_))) - .collect() -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub(crate) struct DiscoverablePluginInfo { - pub(crate) id: String, - pub(crate) name: String, - pub(crate) description: Option, - pub(crate) has_skills: bool, - pub(crate) mcp_server_names: Vec, - pub(crate) app_connector_ids: Vec, -} - -impl From for DiscoverablePluginInfo { - fn from(value: PluginCapabilitySummary) -> Self { - Self { - id: value.config_name, - name: value.display_name, - description: value.description, - has_skills: value.has_skills, - mcp_server_names: value.mcp_server_names, - app_connector_ids: value - .app_connector_ids - .into_iter() - .map(|connector_id| connector_id.0) - .collect(), - } - } -} diff --git a/codex-rs/core/src/tools/handlers/multi_agents/spawn.rs b/codex-rs/core/src/tools/handlers/multi_agents/spawn.rs index 14632b0165..185ff600fb 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents/spawn.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents/spawn.rs @@ -1,7 +1,5 @@ use super::*; -use crate::agent::RemovedWatchdog; -use crate::agent::WatchdogRegistration; -use crate::agent::control::LiveAgent; +use crate::agent::control::SpawnAgentForkMode; use crate::agent::control::SpawnAgentOptions; use crate::agent::control::render_input_preview; use crate::agent::next_thread_spawn_depth; @@ -102,31 +100,47 @@ impl ToolHandler for Handler { .await; let config = build_agent_spawn_config(&session.get_base_instructions().await, turn.as_ref())?; - let spawn_source = thread_spawn_source( - session.conversation_id, - &turn.session_source, - child_depth, - role_name, - /*task_name*/ None, - )?; - let candidates_to_try = if fork_context { - vec![SpawnAgentModelCandidate { - model: None, - reasoning_effort: None, - }] - } else { - let mut candidates = collect_spawn_agent_model_candidates( - args.model_fallback_list.as_ref(), - args.model.as_deref(), - args.reasoning_effort, - ); - if candidates.is_empty() { - candidates.push(SpawnAgentModelCandidate { - model: None, - reasoning_effort: None, - }); - } - candidates + apply_requested_spawn_agent_model_overrides( + &session, + turn.as_ref(), + &mut config, + args.model.as_deref(), + args.reasoning_effort, + ) + .await?; + apply_role_to_config(&mut config, role_name) + .await + .map_err(FunctionCallError::RespondToModel)?; + apply_spawn_agent_runtime_overrides(&mut config, turn.as_ref())?; + apply_spawn_agent_overrides(&mut config, child_depth); + + let result = session + .services + .agent_control + .spawn_agent_with_metadata( + config, + input_items, + Some(thread_spawn_source( + session.conversation_id, + &turn.session_source, + child_depth, + role_name, + /*task_name*/ None, + )?), + SpawnAgentOptions { + fork_parent_spawn_call_id: args.fork_context.then(|| call_id.clone()), + fork_mode: args.fork_context.then_some(SpawnAgentForkMode::FullHistory), + }, + ) + .await + .map_err(collab_spawn_error); + let (new_thread_id, new_agent_metadata, status) = match &result { + Ok(spawned_agent) => ( + Some(spawned_agent.thread_id), + Some(spawned_agent.metadata.clone()), + spawned_agent.status.clone(), + ), + Err(_) => (None, None, AgentStatus::NotFound), }; let mut spawn_result = None; diff --git a/codex-rs/core/src/tools/handlers/multi_agents_tests.rs b/codex-rs/core/src/tools/handlers/multi_agents_tests.rs index 6bb5c0f1e0..d51750aa96 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_tests.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_tests.rs @@ -16,7 +16,10 @@ use crate::protocol::Op; use crate::protocol::SandboxPolicy; use crate::protocol::SessionSource; use crate::protocol::SubAgentSource; +use crate::protocol::TurnAbortReason; +use crate::protocol::TurnAbortedEvent; use crate::protocol::TurnCompleteEvent; +use crate::session_prefix::format_subagent_notification_message; use crate::state::TaskKind; use crate::tasks::SessionTask; use crate::tasks::SessionTaskContext; @@ -500,8 +503,7 @@ async fn multi_agent_v2_spawn_requires_task_name() { Arc::new(turn), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", - "fork_context": false + "items": [{"type": "text", "text": "inspect this repo"}] })), ); let Err(err) = SpawnAgentHandlerV2.handle(invocation).await else { @@ -513,6 +515,42 @@ async fn multi_agent_v2_spawn_requires_task_name() { assert!(message.contains("missing field `task_name`")); } +#[tokio::test] +async fn multi_agent_v2_spawn_rejects_legacy_message_field() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + + let invocation = invocation( + Arc::new(session), + Arc::new(turn), + "spawn_agent", + function_payload(json!({ + "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], + "task_name": "worker" + })), + ); + let Err(err) = SpawnAgentHandlerV2.handle(invocation).await else { + panic!("legacy message field should be rejected"); + }; + let FunctionCallError::RespondToModel(message) = err else { + panic!("legacy message field should surface as a model-facing error"); + }; + assert!(message.contains("unknown field `message`")); +} + #[tokio::test] async fn spawn_agent_errors_when_manager_dropped() { let (session, turn) = make_session_and_context().await; @@ -562,7 +600,7 @@ async fn multi_agent_v2_spawn_returns_path_and_send_message_accepts_relative_pat turn.clone(), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "test_process" })), )) @@ -634,6 +672,123 @@ async fn multi_agent_v2_spawn_returns_path_and_send_message_accepts_relative_pat })); } +#[tokio::test] +async fn multi_agent_v2_spawn_rejects_legacy_fork_context() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + + let err = SpawnAgentHandlerV2 + .handle(invocation( + Arc::new(session), + Arc::new(turn), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "inspect this repo"}], + "task_name": "worker", + "fork_context": true + })), + )) + .await + .expect_err("legacy fork_context should be rejected"); + + assert_eq!( + err, + FunctionCallError::RespondToModel( + "fork_context is not supported in MultiAgentV2; use fork_turns instead".to_string() + ) + ); +} + +#[tokio::test] +async fn multi_agent_v2_spawn_rejects_invalid_fork_turns_string() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + + let err = SpawnAgentHandlerV2 + .handle(invocation( + Arc::new(session), + Arc::new(turn), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "inspect this repo"}], + "task_name": "worker", + "fork_turns": "banana" + })), + )) + .await + .expect_err("invalid fork_turns should be rejected"); + + assert_eq!( + err, + FunctionCallError::RespondToModel( + "fork_turns must be `none`, `all`, or a positive integer string".to_string() + ) + ); +} + +#[tokio::test] +async fn multi_agent_v2_spawn_rejects_zero_fork_turns() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + + let err = SpawnAgentHandlerV2 + .handle(invocation( + Arc::new(session), + Arc::new(turn), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "inspect this repo"}], + "task_name": "worker", + "fork_turns": "0" + })), + )) + .await + .expect_err("zero turn count should be rejected"); + + assert_eq!( + err, + FunctionCallError::RespondToModel( + "fork_turns must be `none`, `all`, or a positive integer string".to_string() + ) + ); +} + #[tokio::test] async fn multi_agent_v2_send_message_accepts_root_target_from_child() { let (mut session, mut turn) = make_session_and_context().await; @@ -732,7 +887,7 @@ async fn multi_agent_v2_list_agents_returns_completed_status_and_last_task_messa turn.clone(), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "worker" })), )) @@ -910,7 +1065,7 @@ async fn multi_agent_v2_list_agents_omits_closed_agents() { turn.clone(), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "worker" })), )) @@ -974,7 +1129,7 @@ async fn multi_agent_v2_send_message_rejects_structured_items() { turn.clone(), "spawn_agent", function_payload(json!({ - "message": "boot worker", + "items": [{"type": "text", "text": "boot worker"}], "task_name": "worker" })), )) @@ -1011,7 +1166,7 @@ async fn multi_agent_v2_send_message_rejects_structured_items() { } #[tokio::test] -async fn multi_agent_v2_send_message_interrupts_busy_child_without_triggering_turn() { +async fn multi_agent_v2_send_message_rejects_interrupt_parameter() { let (mut session, mut turn) = make_session_and_context().await; let manager = thread_manager(); let root = manager @@ -1032,7 +1187,7 @@ async fn multi_agent_v2_send_message_interrupts_busy_child_without_triggering_tu turn.clone(), "spawn_agent", function_payload(json!({ - "message": "boot worker", + "items": [{"type": "text", "text": "boot worker"}], "task_name": "worker" })), )) @@ -1044,112 +1199,43 @@ async fn multi_agent_v2_send_message_interrupts_busy_child_without_triggering_tu .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") .await .expect("worker should resolve"); - let thread = manager - .get_thread(agent_id) - .await - .expect("worker thread should exist"); - let active_turn = thread.codex.session.new_default_turn().await; - thread - .codex - .session - .spawn_task( - Arc::clone(&active_turn), - vec![UserInput::Text { - text: "working".to_string(), - text_elements: Vec::new(), - }], - NeverEndingTask, - ) - .await; + let invocation = invocation( + session, + turn, + "send_message", + function_payload(json!({ + "target": agent_id.to_string(), + "items": [{"type": "text", "text": "continue"}], + "interrupt": true + })), + ); - SendMessageHandlerV2 - .handle(invocation( - session.clone(), - turn.clone(), - "send_message", - function_payload(json!({ - "target": agent_id.to_string(), - "items": [{"type": "text", "text": "continue"}], - "interrupt": true - })), - )) - .await - .expect("interrupting v2 send_message should succeed"); + let Err(err) = SendMessageHandlerV2.handle(invocation).await else { + panic!("send_message interrupt parameter should be rejected"); + }; + let FunctionCallError::RespondToModel(message) = err else { + panic!("expected model-facing parse error"); + }; + assert!(message.starts_with( + "failed to parse function arguments: unknown field `interrupt`, expected `target` or `items`" + )); let ops = manager.captured_ops(); let ops_for_agent: Vec<&Op> = ops .iter() .filter_map(|(id, op)| (*id == agent_id).then_some(op)) .collect(); - assert!(ops_for_agent.iter().any(|op| matches!(op, Op::Interrupt))); - assert!(ops_for_agent.iter().any(|op| { - matches!( - op, - Op::InterAgentCommunication { communication } - if communication.author == AgentPath::root() - && communication.recipient.as_str() == "/root/worker" - && communication.other_recipients.is_empty() - && communication.content == "continue" - && !communication.trigger_turn - ) - })); - - timeout(Duration::from_secs(5), async { - loop { - if !thread - .codex - .session - .has_queued_response_items_for_next_turn() - .await - { - tokio::time::sleep(Duration::from_millis(10)).await; - continue; - } - let history_items = thread - .codex - .session - .clone_history() - .await - .raw_items() - .to_vec(); - let saw_envelope = history_contains_inter_agent_communication( - &history_items, - &InterAgentCommunication::new( - AgentPath::root(), - AgentPath::try_from("/root/worker").expect("agent path"), - Vec::new(), - "continue".to_string(), - /*trigger_turn*/ false, - ), - ); - let saw_user_message = history_items.iter().any(|item| { - matches!( - item, - ResponseItem::Message { role, content, .. } - if role == "user" - && content.iter().any(|content_item| matches!( - content_item, - ContentItem::InputText { text } if text == "continue" - )) - ) - }); - if saw_envelope && !saw_user_message { - panic!("send_message should not materialize the envelope into history"); - } - if !saw_user_message { - break; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .expect("interrupting v2 send_message should queue the redirected message without a turn"); - - let _ = thread - .submit(Op::Shutdown {}) - .await - .expect("shutdown should submit"); + assert!(!ops_for_agent.iter().any(|op| matches!(op, Op::Interrupt))); + assert!(!ops_for_agent.iter().any(|op| matches!( + op, + Op::InterAgentCommunication { communication } + if communication.author == AgentPath::root() + && communication.recipient.as_str() == "/root/worker" + && communication.other_recipients.is_empty() + && communication.content == "continue" + && !communication.trigger_turn + ))); } #[tokio::test] @@ -1174,7 +1260,7 @@ async fn multi_agent_v2_assign_task_interrupts_busy_child_without_losing_message turn.clone(), "spawn_agent", function_payload(json!({ - "message": "boot worker", + "items": [{"type": "text", "text": "boot worker"}], "task_name": "worker" })), )) @@ -1281,6 +1367,210 @@ async fn multi_agent_v2_assign_task_interrupts_busy_child_without_losing_message .expect("shutdown should submit"); } +#[tokio::test] +async fn multi_agent_v2_assign_task_completion_notifies_parent_on_every_turn() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = turn.config.as_ref().clone(); + let _ = config.features.enable(Feature::MultiAgentV2); + turn.config = Arc::new(config); + let session = Arc::new(session); + let turn = Arc::new(turn); + + SpawnAgentHandlerV2 + .handle(invocation( + session.clone(), + turn.clone(), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "boot worker"}], + "task_name": "worker" + })), + )) + .await + .expect("spawn worker"); + let agent_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") + .await + .expect("worker should resolve"); + let thread = manager + .get_thread(agent_id) + .await + .expect("worker thread should exist"); + let worker_path = AgentPath::try_from("/root/worker").expect("worker path"); + + let first_turn = thread.codex.session.new_default_turn().await; + thread + .codex + .session + .send_event( + first_turn.as_ref(), + EventMsg::TurnComplete(TurnCompleteEvent { + turn_id: first_turn.sub_id.clone(), + last_agent_message: Some("first done".to_string()), + }), + ) + .await; + + AssignTaskHandlerV2 + .handle(invocation( + session, + turn, + "assign_task", + function_payload(json!({ + "target": agent_id.to_string(), + "items": [{"type": "text", "text": "continue"}], + })), + )) + .await + .expect("assign_task should succeed"); + + let second_turn = thread.codex.session.new_default_turn().await; + thread + .codex + .session + .send_event( + second_turn.as_ref(), + EventMsg::TurnComplete(TurnCompleteEvent { + turn_id: second_turn.sub_id.clone(), + last_agent_message: Some("second done".to_string()), + }), + ) + .await; + + let first_notification = format_subagent_notification_message( + worker_path.as_str(), + &AgentStatus::Completed(Some("first done".to_string())), + ); + let second_notification = format_subagent_notification_message( + worker_path.as_str(), + &AgentStatus::Completed(Some("second done".to_string())), + ); + + let notifications = timeout(Duration::from_secs(5), async { + loop { + let notifications = manager + .captured_ops() + .into_iter() + .filter_map(|(id, op)| { + (id == root.thread_id) + .then_some(op) + .and_then(|op| match op { + Op::InterAgentCommunication { communication } + if communication.author == worker_path + && communication.recipient == AgentPath::root() + && communication.other_recipients.is_empty() + && !communication.trigger_turn => + { + Some(communication.content) + } + _ => None, + }) + }) + .collect::>(); + let first_count = notifications + .iter() + .filter(|message| **message == first_notification) + .count(); + let second_count = notifications + .iter() + .filter(|message| **message == second_notification) + .count(); + if first_count == 1 && second_count == 1 { + break notifications; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("parent should receive one completion notification per child turn"); + + assert_eq!(notifications.len(), 2); +} + +#[tokio::test] +async fn multi_agent_v2_interrupted_turn_does_not_notify_parent() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = turn.config.as_ref().clone(); + let _ = config.features.enable(Feature::MultiAgentV2); + turn.config = Arc::new(config); + let session = Arc::new(session); + let turn = Arc::new(turn); + + SpawnAgentHandlerV2 + .handle(invocation( + session.clone(), + turn.clone(), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "boot worker"}], + "task_name": "worker" + })), + )) + .await + .expect("spawn worker"); + let agent_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") + .await + .expect("worker should resolve"); + let thread = manager + .get_thread(agent_id) + .await + .expect("worker thread should exist"); + + let aborted_turn = thread.codex.session.new_default_turn().await; + thread + .codex + .session + .send_event( + aborted_turn.as_ref(), + EventMsg::TurnAborted(TurnAbortedEvent { + turn_id: Some(aborted_turn.sub_id.clone()), + reason: TurnAbortReason::Interrupted, + }), + ) + .await; + + let notifications = manager + .captured_ops() + .into_iter() + .filter_map(|(id, op)| { + (id == root.thread_id) + .then_some(op) + .and_then(|op| match op { + Op::InterAgentCommunication { communication } + if communication.author.as_str() == "/root/worker" + && communication.recipient == AgentPath::root() + && communication.other_recipients.is_empty() + && !communication.trigger_turn => + { + Some(communication.content) + } + _ => None, + }) + }) + .collect::>(); + + assert_eq!(notifications, Vec::::new()); +} + #[tokio::test] async fn multi_agent_v2_spawn_includes_agent_id_key_when_named() { let (mut session, mut turn) = make_session_and_context().await; @@ -1304,7 +1594,7 @@ async fn multi_agent_v2_spawn_includes_agent_id_key_when_named() { Arc::new(turn), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "test_process" })), )) @@ -1342,7 +1632,7 @@ async fn multi_agent_v2_spawn_surfaces_task_name_validation_errors() { Arc::new(turn), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "BadName" })), ); @@ -1959,27 +2249,78 @@ async fn wait_agent_rejects_empty_targets() { } #[tokio::test] -async fn multi_agent_v2_wait_agent_accepts_targets_argument() { +async fn multi_agent_v2_wait_agent_accepts_timeout_only_argument() { let (mut session, mut turn) = make_session_and_context().await; - let target = ThreadId::new().to_string(); let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; let mut config = (*turn.config).clone(); config .features .enable(Feature::MultiAgentV2) .expect("test config should allow feature update"); turn.config = Arc::new(config); - let invocation = invocation( - Arc::new(session), - Arc::new(turn), - "wait_agent", - function_payload(json!({"targets": [target.clone()]})), - ); - let output = WaitAgentHandlerV2 - .handle(invocation) + let session = Arc::new(session); + let turn = Arc::new(turn); + + SpawnAgentHandlerV2 + .handle(invocation( + session.clone(), + turn.clone(), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": "boot worker"}], + "task_name": "worker" + })), + )) .await - .expect("targets should be accepted in v2 mode"); + .expect("spawn worker"); + let agent_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") + .await + .expect("worker should resolve"); + let worker_path = session + .services + .agent_control + .get_agent_metadata(agent_id) + .expect("worker metadata") + .agent_path + .expect("worker path"); + + let wait_task = tokio::spawn({ + let session = session.clone(); + let turn = turn.clone(); + async move { + WaitAgentHandlerV2 + .handle(invocation( + session, + turn, + "wait_agent", + function_payload(json!({"timeout_ms": 1000})), + )) + .await + } + }); + tokio::task::yield_now().await; + + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_path, + AgentPath::root(), + Vec::new(), + "hello from worker".to_string(), + /*trigger_turn*/ false, + )); + + let output = wait_task + .await + .expect("wait task should join") + .expect("timeout-only args should be accepted in v2 mode"); let (content, success) = expect_text_output(output); let result: crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult = serde_json::from_str(&content).expect("wait_agent result should be json"); @@ -2154,7 +2495,7 @@ async fn wait_agent_returns_final_status_without_timeout() { } #[tokio::test] -async fn multi_agent_v2_wait_agent_returns_summary_for_named_targets() { +async fn multi_agent_v2_wait_agent_returns_summary_for_mailbox_activity() { let (mut session, mut turn) = make_session_and_context().await; let manager = thread_manager(); let root = manager @@ -2178,7 +2519,7 @@ async fn multi_agent_v2_wait_agent_returns_summary_for_named_targets() { turn.clone(), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "test_process" })), )) @@ -2196,37 +2537,229 @@ async fn multi_agent_v2_wait_agent_returns_summary_for_named_targets() { ) .await .expect("relative path should resolve"); - let mut status_rx = manager - .agent_control() - .subscribe_status(agent_id) - .await - .expect("subscribe should succeed"); + let worker_path = session + .services + .agent_control + .get_agent_metadata(agent_id) + .expect("worker metadata") + .agent_path + .expect("worker path"); + let wait_task = tokio::spawn({ + let session = session.clone(); + let turn = turn.clone(); + async move { + WaitAgentHandlerV2 + .handle(invocation( + session, + turn, + "wait_agent", + function_payload(json!({"timeout_ms": 1000})), + )) + .await + } + }); + tokio::task::yield_now().await; - let child_thread = manager - .get_thread(agent_id) - .await - .expect("child should exist"); - let _ = child_thread - .submit(Op::Shutdown {}) - .await - .expect("shutdown should submit"); - let _ = timeout(Duration::from_secs(1), status_rx.changed()) - .await - .expect("shutdown status should arrive"); + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_path, + AgentPath::root(), + Vec::new(), + "completed".to_string(), + /*trigger_turn*/ false, + )); - let wait_output = WaitAgentHandlerV2 + let wait_output = wait_task + .await + .expect("wait task should join") + .expect("wait_agent should succeed"); + let (content, success) = expect_text_output(wait_output); + let result: crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult = + serde_json::from_str(&content).expect("wait_agent result should be json"); + assert_eq!( + result, + crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult { + message: "Wait completed.".to_string(), + timed_out: false, + } + ); + assert_eq!(success, None); +} + +#[tokio::test] +async fn multi_agent_v2_wait_agent_waits_for_new_mail_after_start() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + let session = Arc::new(session); + let turn = Arc::new(turn); + + SpawnAgentHandlerV2 .handle(invocation( - session, - turn, - "wait_agent", + session.clone(), + turn.clone(), + "spawn_agent", function_payload(json!({ - "targets": ["test_process"], - "timeout_ms": 1000 + "items": [{"type": "text", "text": "boot worker"}], + "task_name": "worker" })), )) .await + .expect("spawn worker"); + let agent_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") + .await + .expect("worker should resolve"); + let worker_path = session + .services + .agent_control + .get_agent_metadata(agent_id) + .expect("worker metadata") + .agent_path + .expect("worker path"); + + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_path.clone(), + AgentPath::root(), + Vec::new(), + "already queued".to_string(), + /*trigger_turn*/ false, + )); + + let wait_task = tokio::spawn({ + let session = session.clone(); + let turn = turn.clone(); + async move { + WaitAgentHandlerV2 + .handle(invocation( + session, + turn, + "wait_agent", + function_payload(json!({"timeout_ms": 1000})), + )) + .await + } + }); + tokio::task::yield_now().await; + tokio::time::sleep(Duration::from_millis(50)).await; + assert!( + !wait_task.is_finished(), + "mail already queued before wait should not wake wait_agent" + ); + + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_path, + AgentPath::root(), + Vec::new(), + "new mail".to_string(), + /*trigger_turn*/ false, + )); + + let output = wait_task + .await + .expect("wait task should join") .expect("wait_agent should succeed"); - let (content, success) = expect_text_output(wait_output); + let (content, success) = expect_text_output(output); + let result: crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult = + serde_json::from_str(&content).expect("wait_agent result should be json"); + assert_eq!( + result, + crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult { + message: "Wait completed.".to_string(), + timed_out: false, + } + ); + assert_eq!(success, None); +} + +#[tokio::test] +async fn multi_agent_v2_wait_agent_wakes_on_any_mailbox_notification() { + let (mut session, mut turn) = make_session_and_context().await; + let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); + session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; + let mut config = (*turn.config).clone(); + config + .features + .enable(Feature::MultiAgentV2) + .expect("test config should allow feature update"); + turn.config = Arc::new(config); + let session = Arc::new(session); + let turn = Arc::new(turn); + + for task_name in ["worker_a", "worker_b"] { + SpawnAgentHandlerV2 + .handle(invocation( + session.clone(), + turn.clone(), + "spawn_agent", + function_payload(json!({ + "items": [{"type": "text", "text": format!("boot {task_name}")}], + "task_name": task_name + })), + )) + .await + .expect("spawn worker"); + } + let worker_b_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker_b") + .await + .expect("worker_b should resolve"); + let worker_b_path = session + .services + .agent_control + .get_agent_metadata(worker_b_id) + .expect("worker_b metadata") + .agent_path + .expect("worker_b path"); + + let wait_task = tokio::spawn({ + let session = session.clone(); + let turn = turn.clone(); + async move { + WaitAgentHandlerV2 + .handle(invocation( + session, + turn, + "wait_agent", + function_payload(json!({"timeout_ms": 1000})), + )) + .await + } + }); + tokio::task::yield_now().await; + + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_b_path, + AgentPath::root(), + Vec::new(), + "from worker b".to_string(), + /*trigger_turn*/ false, + )); + + let output = wait_task + .await + .expect("wait task should join") + .expect("wait_agent should succeed"); + let (content, success) = expect_text_output(output); let result: crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult = serde_json::from_str(&content).expect("wait_agent result should be json"); assert_eq!( @@ -2243,41 +2776,73 @@ async fn multi_agent_v2_wait_agent_returns_summary_for_named_targets() { async fn multi_agent_v2_wait_agent_does_not_return_completed_content() { let (mut session, mut turn) = make_session_and_context().await; let manager = thread_manager(); + let root = manager + .start_thread((*turn.config).clone()) + .await + .expect("root thread should start"); session.services.agent_control = manager.agent_control(); + session.conversation_id = root.thread_id; let mut config = (*turn.config).clone(); config .features .enable(Feature::MultiAgentV2) .expect("test config should allow feature update"); - turn.config = Arc::new(config.clone()); + turn.config = Arc::new(config); + let session = Arc::new(session); + let turn = Arc::new(turn); - let thread = manager.start_thread(config).await.expect("start thread"); - let agent_id = thread.thread_id; - let child_turn = thread.thread.codex.session.new_default_turn().await; - thread - .thread - .codex - .session - .send_event( - child_turn.as_ref(), - EventMsg::TurnComplete(TurnCompleteEvent { - turn_id: child_turn.sub_id.clone(), - last_agent_message: Some("sensitive child output".to_string()), - }), - ) - .await; - - let output = WaitAgentHandlerV2 + SpawnAgentHandlerV2 .handle(invocation( - Arc::new(session), - Arc::new(turn), - "wait_agent", + session.clone(), + turn.clone(), + "spawn_agent", function_payload(json!({ - "targets": [agent_id.to_string()], - "timeout_ms": 1000 + "items": [{"type": "text", "text": "boot worker"}], + "task_name": "worker" })), )) .await + .expect("spawn worker"); + let agent_id = session + .services + .agent_control + .resolve_agent_reference(session.conversation_id, &turn.session_source, "worker") + .await + .expect("worker should resolve"); + let worker_path = session + .services + .agent_control + .get_agent_metadata(agent_id) + .expect("worker metadata") + .agent_path + .expect("worker path"); + let wait_task = tokio::spawn({ + let session = session.clone(); + let turn = turn.clone(); + async move { + WaitAgentHandlerV2 + .handle(invocation( + session, + turn, + "wait_agent", + function_payload(json!({"timeout_ms": 1000})), + )) + .await + } + }); + tokio::task::yield_now().await; + + session.enqueue_mailbox_communication(InterAgentCommunication::new( + worker_path, + AgentPath::root(), + Vec::new(), + "sensitive child output".to_string(), + /*trigger_turn*/ false, + )); + + let output = wait_task + .await + .expect("wait task should join") .expect("wait_agent should succeed"); let (content, success) = expect_text_output(output); let result: crate::tools::handlers::multi_agents_v2::wait::WaitAgentResult = @@ -2318,7 +2883,7 @@ async fn multi_agent_v2_close_agent_accepts_task_name_target() { turn.clone(), "spawn_agent", function_payload(json!({ - "message": "inspect this repo", + "items": [{"type": "text", "text": "inspect this repo"}], "task_name": "worker" })), )) diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2.rs index cea303d600..d0b7dec8d9 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2.rs @@ -2,9 +2,7 @@ use crate::agent::AgentStatus; use crate::agent::agent_resolver::resolve_agent_target; -use crate::agent::agent_resolver::resolve_agent_targets; use crate::agent::exceeds_thread_spawn_depth_limit; -use crate::codex::Session; use crate::function_tool::FunctionCallError; use crate::tools::context::ToolInvocation; use crate::tools::context::ToolOutput; @@ -15,12 +13,10 @@ use crate::tools::registry::ToolHandler; use crate::tools::registry::ToolKind; use async_trait::async_trait; use codex_protocol::AgentPath; -use codex_protocol::ThreadId; use codex_protocol::models::ResponseInputItem; use codex_protocol::openai_models::ReasoningEffort; use codex_protocol::protocol::CollabAgentInteractionBeginEvent; use codex_protocol::protocol::CollabAgentInteractionEndEvent; -use codex_protocol::protocol::CollabAgentRef; use codex_protocol::protocol::CollabAgentSpawnBeginEvent; use codex_protocol::protocol::CollabAgentSpawnEndEvent; use codex_protocol::protocol::CollabCloseBeginEvent; diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2/assign_task.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2/assign_task.rs index e55b65b311..2aed22b3db 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2/assign_task.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2/assign_task.rs @@ -1,3 +1,4 @@ +use super::message_tool::AssignTaskArgs; use super::message_tool::MessageDeliveryMode; use super::message_tool::MessageToolResult; use super::message_tool::handle_message_tool; @@ -18,6 +19,15 @@ impl ToolHandler for Handler { } async fn handle(&self, invocation: ToolInvocation) -> Result { - handle_message_tool(invocation, MessageDeliveryMode::TriggerTurn).await + let arguments = function_arguments(invocation.payload.clone())?; + let args: AssignTaskArgs = parse_arguments(&arguments)?; + handle_message_tool( + invocation, + MessageDeliveryMode::TriggerTurn, + args.target, + args.items, + args.interrupt, + ) + .await } } diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2/message_tool.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2/message_tool.rs index a61c01f6a1..59eb49203b 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2/message_tool.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2/message_tool.rs @@ -38,8 +38,17 @@ impl MessageDeliveryMode { } #[derive(Debug, Deserialize)] -/// Input shared by the MultiAgentV2 `send_message` and `assign_task` tools. -pub(crate) struct MessageToolArgs { +#[serde(deny_unknown_fields)] +/// Input for the MultiAgentV2 `send_message` tool. +pub(crate) struct SendMessageArgs { + pub(crate) target: String, + pub(crate) items: Vec, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +/// Input for the MultiAgentV2 `assign_task` tool. +pub(crate) struct AssignTaskArgs { pub(crate) target: String, pub(crate) items: Vec, #[serde(default)] @@ -95,6 +104,9 @@ fn text_content( pub(crate) async fn handle_message_tool( invocation: ToolInvocation, mode: MessageDeliveryMode, + target: String, + items: Vec, + interrupt: bool, ) -> Result { let ToolInvocation { session, @@ -103,16 +115,15 @@ pub(crate) async fn handle_message_tool( call_id, .. } = invocation; - let arguments = function_arguments(payload)?; - let args: MessageToolArgs = parse_arguments(&arguments)?; - let receiver_thread_id = resolve_agent_target(&session, &turn, &args.target).await?; - let prompt = text_content(&args.items, mode)?; + let _ = payload; + let receiver_thread_id = resolve_agent_target(&session, &turn, &target).await?; + let prompt = text_content(&items, mode)?; let receiver_agent = session .services .agent_control .get_agent_metadata(receiver_thread_id) .unwrap_or_default(); - if args.interrupt { + if interrupt { session .services .agent_control diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2/send_message.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2/send_message.rs index ed10b124a6..e0e3ec9102 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2/send_message.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2/send_message.rs @@ -1,5 +1,6 @@ use super::message_tool::MessageDeliveryMode; use super::message_tool::MessageToolResult; +use super::message_tool::SendMessageArgs; use super::message_tool::handle_message_tool; use super::*; @@ -18,6 +19,15 @@ impl ToolHandler for Handler { } async fn handle(&self, invocation: ToolInvocation) -> Result { - handle_message_tool(invocation, MessageDeliveryMode::QueueOnly).await + let arguments = function_arguments(invocation.payload.clone())?; + let args: SendMessageArgs = parse_arguments(&arguments)?; + handle_message_tool( + invocation, + MessageDeliveryMode::QueueOnly, + args.target, + args.items, + /*interrupt*/ false, + ) + .await } } diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs index 3352bd136c..c5b230b997 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs @@ -1,4 +1,5 @@ use super::*; +use crate::agent::control::SpawnAgentForkMode; use crate::agent::control::SpawnAgentOptions; use crate::agent::control::render_input_preview; use crate::agent::next_thread_spawn_depth; @@ -34,13 +35,14 @@ impl ToolHandler for Handler { } = invocation; let arguments = function_arguments(payload)?; let args: SpawnAgentArgs = parse_arguments(&arguments)?; + let fork_mode = args.fork_mode()?; let role_name = args .agent_type .as_deref() .map(str::trim) .filter(|role| !role.is_empty()); - let initial_operation = parse_collab_input(args.message, args.items)?; + let initial_operation = parse_collab_input(/*message*/ None, Some(args.items))?; let prompt = render_input_preview(&initial_operation); let session_source = turn.session_source.clone(); @@ -94,25 +96,46 @@ impl ToolHandler for Handler { role_name, Some(args.task_name.clone()), )?; - let initial_agent_op = match (spawn_source.get_agent_path(), initial_operation) { - (Some(recipient), Op::UserInput { items, .. }) - if items - .iter() - .all(|item| matches!(item, UserInput::Text { .. })) => - { - Op::InterAgentCommunication { - communication: InterAgentCommunication::new( - turn.session_source - .get_agent_path() - .unwrap_or_else(AgentPath::root), - recipient, - Vec::new(), - prompt.clone(), - /*trigger_turn*/ true, - ), - } - } - (_, initial_operation) => initial_operation, + let result = session + .services + .agent_control + .spawn_agent_with_metadata( + config, + match (spawn_source.get_agent_path(), initial_operation) { + (Some(recipient), Op::UserInput { items, .. }) + if items + .iter() + .all(|item| matches!(item, UserInput::Text { .. })) => + { + Op::InterAgentCommunication { + communication: InterAgentCommunication::new( + turn.session_source + .get_agent_path() + .unwrap_or_else(AgentPath::root), + recipient, + Vec::new(), + prompt.clone(), + /*trigger_turn*/ true, + ), + } + } + (_, initial_operation) => initial_operation, + }, + Some(spawn_source), + SpawnAgentOptions { + fork_parent_spawn_call_id: fork_mode.as_ref().map(|_| call_id.clone()), + fork_mode, + }, + ) + .await + .map_err(collab_spawn_error); + let (new_thread_id, new_agent_metadata, status) = match &result { + Ok(spawned_agent) => ( + Some(spawned_agent.thread_id), + Some(spawned_agent.metadata.clone()), + spawned_agent.status.clone(), + ), + Err(_) => (None, None, AgentStatus::NotFound), }; let candidates_to_try = if fork_context { vec![SpawnAgentModelCandidate { @@ -264,17 +287,57 @@ impl ToolHandler for Handler { } #[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] struct SpawnAgentArgs { - message: Option, - items: Option>, + items: Vec, task_name: String, agent_type: Option, model: Option, model_fallback_list: Option>, reasoning_effort: Option, + fork_turns: Option, fork_context: Option, } +impl SpawnAgentArgs { + fn fork_mode(&self) -> Result, FunctionCallError> { + if self.fork_context.is_some() { + return Err(FunctionCallError::RespondToModel( + "fork_context is not supported in MultiAgentV2; use fork_turns instead".to_string(), + )); + } + + let Some(fork_turns) = self + .fork_turns + .as_deref() + .map(str::trim) + .filter(|fork_turns| !fork_turns.is_empty()) + else { + return Ok(None); + }; + + if fork_turns.eq_ignore_ascii_case("none") { + return Ok(None); + } + if fork_turns.eq_ignore_ascii_case("all") { + return Ok(Some(SpawnAgentForkMode::FullHistory)); + } + + let last_n_turns = fork_turns.parse::().map_err(|_| { + FunctionCallError::RespondToModel( + "fork_turns must be `none`, `all`, or a positive integer string".to_string(), + ) + })?; + if last_n_turns == 0 { + return Err(FunctionCallError::RespondToModel( + "fork_turns must be `none`, `all`, or a positive integer string".to_string(), + )); + } + + Ok(Some(SpawnAgentForkMode::LastNTurns(last_n_turns))) + } +} + #[derive(Debug, Serialize)] pub(crate) struct SpawnAgentResult { agent_id: Option, diff --git a/codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs b/codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs index 1b1dc90e42..2eb92842fe 100644 --- a/codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs +++ b/codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs @@ -1,11 +1,6 @@ use super::*; -use crate::agent::status::is_final; -use futures::FutureExt; -use futures::StreamExt; -use futures::stream::FuturesUnordered; use std::collections::HashMap; use std::time::Duration; -use tokio::sync::watch::Receiver; use tokio::time::Instant; use tokio::time::timeout_at; @@ -33,21 +28,6 @@ impl ToolHandler for Handler { } = invocation; let arguments = function_arguments(payload)?; let args: WaitArgs = parse_arguments(&arguments)?; - let receiver_thread_ids = resolve_agent_targets(&session, &turn, args.targets).await?; - let mut receiver_agents = Vec::with_capacity(receiver_thread_ids.len()); - for receiver_thread_id in &receiver_thread_ids { - let agent_metadata = session - .services - .agent_control - .get_agent_metadata(*receiver_thread_id) - .unwrap_or_default(); - receiver_agents.push(CollabAgentRef { - thread_id: *receiver_thread_id, - agent_nickname: agent_metadata.agent_nickname, - agent_role: agent_metadata.agent_role, - }); - } - let timeout_ms = args.timeout_ms.unwrap_or(DEFAULT_WAIT_TIMEOUT_MS); let timeout_ms = match timeout_ms { ms if ms <= 0 => { @@ -58,91 +38,23 @@ impl ToolHandler for Handler { ms => ms.clamp(MIN_WAIT_TIMEOUT_MS, MAX_WAIT_TIMEOUT_MS), }; + let mut mailbox_seq_rx = session.subscribe_mailbox_seq(); + session .send_event( &turn, CollabWaitingBeginEvent { sender_thread_id: session.conversation_id, - receiver_thread_ids: receiver_thread_ids.clone(), - receiver_agents: receiver_agents.clone(), + receiver_thread_ids: Vec::new(), + receiver_agents: Vec::new(), call_id: call_id.clone(), } .into(), ) .await; - let mut status_rxs = Vec::with_capacity(receiver_thread_ids.len()); - let mut initial_final_statuses = Vec::new(); - for id in &receiver_thread_ids { - match session.services.agent_control.subscribe_status(*id).await { - Ok(rx) => { - let status = rx.borrow().clone(); - if is_final(&status) { - initial_final_statuses.push((*id, status)); - } - status_rxs.push((*id, rx)); - } - Err(crate::error::CodexErr::ThreadNotFound(_)) => { - initial_final_statuses.push((*id, AgentStatus::NotFound)); - } - Err(err) => { - let mut statuses = HashMap::with_capacity(1); - statuses.insert(*id, session.services.agent_control.get_status(*id).await); - session - .send_event( - &turn, - CollabWaitingEndEvent { - sender_thread_id: session.conversation_id, - call_id: call_id.clone(), - agent_statuses: build_wait_agent_statuses( - &statuses, - &receiver_agents, - ), - statuses, - } - .into(), - ) - .await; - return Err(collab_agent_error(*id, err)); - } - } - } - - let statuses = if !initial_final_statuses.is_empty() { - initial_final_statuses - } else { - let mut futures = FuturesUnordered::new(); - for (id, rx) in status_rxs { - let session = session.clone(); - futures.push(wait_for_final_status(session, id, rx)); - } - let mut results = Vec::new(); - let deadline = Instant::now() + Duration::from_millis(timeout_ms as u64); - loop { - match timeout_at(deadline, futures.next()).await { - Ok(Some(Some(result))) => { - results.push(result); - break; - } - Ok(Some(None)) => continue, - Ok(None) | Err(_) => break, - } - } - if !results.is_empty() { - loop { - match futures.next().now_or_never() { - Some(Some(Some(result))) => results.push(result), - Some(Some(None)) => continue, - Some(None) | None => break, - } - } - } - results - }; - - let timed_out = statuses.is_empty(); - let statuses_by_id = statuses.clone().into_iter().collect::>(); - let agent_statuses = build_wait_agent_statuses(&statuses_by_id, &receiver_agents); + let deadline = Instant::now() + Duration::from_millis(timeout_ms as u64); + let timed_out = !wait_for_mailbox_change(&mut mailbox_seq_rx, deadline).await; let result = WaitAgentResult::from_timed_out(timed_out); session @@ -151,8 +63,8 @@ impl ToolHandler for Handler { CollabWaitingEndEvent { sender_thread_id: session.conversation_id, call_id, - agent_statuses, - statuses: statuses_by_id, + agent_statuses: Vec::new(), + statuses: HashMap::new(), } .into(), ) @@ -164,8 +76,6 @@ impl ToolHandler for Handler { #[derive(Debug, Deserialize)] struct WaitArgs { - #[serde(default)] - targets: Vec, timeout_ms: Option, } @@ -207,24 +117,12 @@ impl ToolOutput for WaitAgentResult { } } -async fn wait_for_final_status( - session: std::sync::Arc, - thread_id: ThreadId, - mut status_rx: Receiver, -) -> Option<(ThreadId, AgentStatus)> { - let mut status = status_rx.borrow().clone(); - if is_final(&status) { - return Some((thread_id, status)); - } - - loop { - if status_rx.changed().await.is_err() { - let latest = session.services.agent_control.get_status(thread_id).await; - return is_final(&latest).then_some((thread_id, latest)); - } - status = status_rx.borrow().clone(); - if is_final(&status) { - return Some((thread_id, status)); - } +async fn wait_for_mailbox_change( + mailbox_seq_rx: &mut tokio::sync::watch::Receiver, + deadline: Instant, +) -> bool { + match timeout_at(deadline, mailbox_seq_rx.changed()).await { + Ok(Ok(())) => true, + Ok(Err(_)) | Err(_) => false, } } diff --git a/codex-rs/core/src/tools/handlers/tool_suggest.rs b/codex-rs/core/src/tools/handlers/tool_suggest.rs index bdffbbbada..c68ac8589c 100644 --- a/codex-rs/core/src/tools/handlers/tool_suggest.rs +++ b/codex-rs/core/src/tools/handlers/tool_suggest.rs @@ -8,6 +8,10 @@ use codex_app_server_protocol::McpElicitationSchema; use codex_app_server_protocol::McpServerElicitationRequest; use codex_app_server_protocol::McpServerElicitationRequestParams; use codex_rmcp_client::ElicitationAction; +use codex_tools::DiscoverableTool; +use codex_tools::DiscoverableToolAction; +use codex_tools::DiscoverableToolType; +use codex_tools::filter_tool_suggest_discoverable_tools_for_client; use rmcp::model::RequestId; use serde::Deserialize; use serde::Serialize; @@ -20,10 +24,6 @@ use crate::mcp::CODEX_APPS_MCP_SERVER_NAME; use crate::tools::context::FunctionToolOutput; use crate::tools::context::ToolInvocation; use crate::tools::context::ToolPayload; -use crate::tools::discoverable::DiscoverableTool; -use crate::tools::discoverable::DiscoverableToolAction; -use crate::tools::discoverable::DiscoverableToolType; -use crate::tools::discoverable::filter_tool_suggest_discoverable_tools_for_client; use crate::tools::handlers::parse_arguments; use crate::tools::registry::ToolHandler; use crate::tools::registry::ToolKind; diff --git a/codex-rs/core/src/tools/handlers/tool_suggest_tests.rs b/codex-rs/core/src/tools/handlers/tool_suggest_tests.rs index 32d5061aed..56089a1aa9 100644 --- a/codex-rs/core/src/tools/handlers/tool_suggest_tests.rs +++ b/codex-rs/core/src/tools/handlers/tool_suggest_tests.rs @@ -5,9 +5,11 @@ use crate::plugins::test_support::load_plugins_config; use crate::plugins::test_support::write_curated_plugin_sha; use crate::plugins::test_support::write_openai_curated_marketplace; use crate::plugins::test_support::write_plugins_feature_config; -use crate::tools::discoverable::DiscoverablePluginInfo; -use crate::tools::discoverable::filter_tool_suggest_discoverable_tools_for_client; use codex_app_server_protocol::AppInfo; +use codex_tools::DiscoverablePluginInfo; +use codex_tools::DiscoverableTool; +use codex_tools::DiscoverableToolAction; +use codex_tools::DiscoverableToolType; use codex_utils_absolute_path::AbsolutePathBuf; use pretty_assertions::assert_eq; use serde_json::json; @@ -159,54 +161,6 @@ fn build_tool_suggestion_meta_uses_expected_shape() { ); } -#[test] -fn filter_tool_suggest_discoverable_tools_for_codex_tui_omits_plugins() { - let discoverable_tools = vec![ - DiscoverableTool::Connector(Box::new(AppInfo { - id: "connector_google_calendar".to_string(), - name: "Google Calendar".to_string(), - description: Some("Plan events and schedules.".to_string()), - logo_url: None, - logo_url_dark: None, - distribution_channel: None, - branding: None, - app_metadata: None, - labels: None, - install_url: Some("https://example.test/google-calendar".to_string()), - is_accessible: false, - is_enabled: true, - plugin_display_names: Vec::new(), - })), - DiscoverableTool::Plugin(Box::new(DiscoverablePluginInfo { - id: "slack@openai-curated".to_string(), - name: "Slack".to_string(), - description: Some("Search Slack messages".to_string()), - has_skills: true, - mcp_server_names: vec!["slack".to_string()], - app_connector_ids: vec!["connector_slack".to_string()], - })), - ]; - - assert_eq!( - filter_tool_suggest_discoverable_tools_for_client(discoverable_tools, Some("codex-tui"),), - vec![DiscoverableTool::Connector(Box::new(AppInfo { - id: "connector_google_calendar".to_string(), - name: "Google Calendar".to_string(), - description: Some("Plan events and schedules.".to_string()), - logo_url: None, - logo_url_dark: None, - distribution_channel: None, - branding: None, - app_metadata: None, - labels: None, - install_url: Some("https://example.test/google-calendar".to_string()), - is_accessible: false, - is_enabled: true, - plugin_display_names: Vec::new(), - }))] - ); -} - #[test] fn verified_connector_suggestion_completed_requires_accessible_connector() { let accessible_connectors = vec![AppInfo { diff --git a/codex-rs/core/src/tools/mod.rs b/codex-rs/core/src/tools/mod.rs index 57859bba42..e866d1f5f7 100644 --- a/codex-rs/core/src/tools/mod.rs +++ b/codex-rs/core/src/tools/mod.rs @@ -1,6 +1,5 @@ pub mod code_mode; pub mod context; -pub(crate) mod discoverable; pub mod events; pub(crate) mod handlers; pub mod js_repl; diff --git a/codex-rs/core/src/tools/router.rs b/codex-rs/core/src/tools/router.rs index 1145e3a8f2..646c1d1387 100644 --- a/codex-rs/core/src/tools/router.rs +++ b/codex-rs/core/src/tools/router.rs @@ -7,7 +7,6 @@ use crate::sandboxing::SandboxPermissions; use crate::tools::context::SharedTurnDiffTracker; use crate::tools::context::ToolInvocation; use crate::tools::context::ToolPayload; -use crate::tools::discoverable::DiscoverableTool; use crate::tools::registry::AnyToolResult; use crate::tools::registry::ToolRegistry; use crate::tools::spec::ToolsConfig; @@ -18,6 +17,7 @@ use codex_protocol::models::ResponseItem; use codex_protocol::models::SearchToolCallParams; use codex_protocol::models::ShellToolCallParams; use codex_tools::ConfiguredToolSpec; +use codex_tools::DiscoverableTool; use rmcp::model::Tool; use std::collections::HashMap; use std::sync::Arc; diff --git a/codex-rs/core/src/tools/spec.rs b/codex-rs/core/src/tools/spec.rs index 897ec2ca7e..1f970b4678 100644 --- a/codex-rs/core/src/tools/spec.rs +++ b/codex-rs/core/src/tools/spec.rs @@ -7,10 +7,6 @@ use crate::shell::Shell; use crate::shell::ShellType; use crate::tools::code_mode::PUBLIC_TOOL_NAME; use crate::tools::code_mode::WAIT_TOOL_NAME; -use crate::tools::discoverable::DiscoverablePluginInfo; -use crate::tools::discoverable::DiscoverableTool; -use crate::tools::discoverable::DiscoverableToolAction; -use crate::tools::discoverable::DiscoverableToolType; use crate::tools::handlers::PLAN_TOOL; use crate::tools::handlers::TOOL_SEARCH_DEFAULT_LIMIT; use crate::tools::handlers::TOOL_SEARCH_TOOL_NAME; @@ -41,11 +37,14 @@ use codex_protocol::protocol::SandboxPolicy; use codex_protocol::protocol::SessionSource; use codex_protocol::protocol::SubAgentSource; use codex_tools::CommandToolOptions; +use codex_tools::DiscoverableTool; +use codex_tools::DiscoverableToolType; use codex_tools::ResponsesApiNamespace; use codex_tools::ResponsesApiNamespaceTool; -use codex_tools::ResponsesApiTool; use codex_tools::ShellToolOptions; use codex_tools::SpawnAgentToolOptions; +use codex_tools::ToolSearchAppInfo; +use codex_tools::ToolSuggestEntry; use codex_tools::ViewImageToolOptions; use codex_tools::WaitAgentTimeoutOptions; use codex_tools::augment_tool_spec_for_code_mode; @@ -73,6 +72,8 @@ use codex_tools::create_spawn_agent_tool_v1; use codex_tools::create_spawn_agent_tool_v2; use codex_tools::create_spawn_agents_on_csv_tool; use codex_tools::create_test_sync_tool; +use codex_tools::create_tool_search_tool; +use codex_tools::create_tool_suggest_tool; use codex_tools::create_view_image_tool; use codex_tools::create_wait_agent_tool_v1; use codex_tools::create_wait_agent_tool_v2; @@ -83,37 +84,21 @@ use codex_tools::dynamic_tool_to_responses_api_tool; use codex_tools::mcp_tool_to_responses_api_tool; use codex_tools::tool_spec_to_code_mode_tool_definition; use codex_utils_absolute_path::AbsolutePathBuf; -use codex_utils_template::Template; use serde::Deserialize; use serde::Serialize; use std::collections::BTreeMap; use std::collections::HashMap; use std::path::PathBuf; use std::sync::Arc; -use std::sync::LazyLock; pub type JsonSchema = codex_tools::JsonSchema; #[cfg(test)] pub(crate) use codex_tools::mcp_call_tool_result_output_schema; -const TOOL_SEARCH_DESCRIPTION_TEMPLATE_SOURCE: &str = - include_str!("../../templates/search_tool/tool_description.md"); -const TOOL_SEARCH_DESCRIPTION_TEMPLATE_KEY: &str = "app_descriptions"; +const WEB_SEARCH_CONTENT_TYPES: [&str; 2] = ["text", "image"]; const AGENT_TOOLS_NAMESPACE: &str = "agents"; const AGENT_TOOLS_NAMESPACE_DESCRIPTION: &str = "Agent collaboration tools for spawning, messaging, waiting on, listing, and closing subagents."; -static TOOL_SEARCH_DESCRIPTION_TEMPLATE: LazyLock