diff --git a/.github/ISSUE_TEMPLATE/1-codex-app.yml b/.github/ISSUE_TEMPLATE/1-codex-app.yml new file mode 100644 index 0000000000..569094907f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/1-codex-app.yml @@ -0,0 +1,47 @@ +name: πŸ–₯️ Codex App Bug +description: Report an issue with the Codex App +labels: + - app +body: + - type: markdown + attributes: + value: | + Before submitting a new issue, please search for existing issues to see if your issue has already been reported. + If it has, please add a πŸ‘ reaction (no need to leave a comment) to the existing issue instead of creating a new one. + + - type: input + id: version + attributes: + label: What version of the Codex App are you using (From β€œAbout Codex” dialog)? + validations: + required: true + - type: input + id: plan + attributes: + label: What subscription do you have? + validations: + required: true + - type: textarea + id: actual + attributes: + label: What issue are you seeing? + description: Please include the full error messages and prompts with PII redacted. If possible, please provide text instead of a screenshot. + validations: + required: true + - type: textarea + id: steps + attributes: + label: What steps can reproduce the bug? + description: Explain the bug and provide a code snippet that can reproduce it. Please include session id, token limit usage, context window usage if applicable. + validations: + required: true + - type: textarea + id: expected + attributes: + label: What is the expected behavior? + description: If possible, please provide text instead of a screenshot. + - type: textarea + id: notes + attributes: + label: Additional information + description: Is there anything else you think we should know? diff --git a/.github/ISSUE_TEMPLATE/5-vs-code-extension.yml b/.github/ISSUE_TEMPLATE/2-extension.yml similarity index 83% rename from .github/ISSUE_TEMPLATE/5-vs-code-extension.yml rename to .github/ISSUE_TEMPLATE/2-extension.yml index 52da6a7cad..599bc08b42 100644 --- a/.github/ISSUE_TEMPLATE/5-vs-code-extension.yml +++ b/.github/ISSUE_TEMPLATE/2-extension.yml @@ -1,8 +1,7 @@ -name: πŸ§‘β€πŸ’» VS Code Extension -description: Report an issue with the VS Code extension +name: πŸ§‘β€πŸ’» IDE Extension Bug +description: Report an issue with the IDE extension labels: - extension - - needs triage body: - type: markdown attributes: @@ -13,7 +12,7 @@ body: - type: input id: version attributes: - label: What version of the VS Code extension are you using? + label: What version of the IDE extension are you using? validations: required: true - type: input @@ -34,20 +33,20 @@ body: attributes: label: What platform is your computer? description: | - For MacOS and Linux: copy the output of `uname -mprs` + For macOS and Linux: copy the output of `uname -mprs` For Windows: copy the output of `"$([Environment]::OSVersion | ForEach-Object VersionString) $(if ([Environment]::Is64BitOperatingSystem) { "x64" } else { "x86" })"` in the PowerShell console - type: textarea id: actual attributes: label: What issue are you seeing? - description: Please include the full error messages and prompts with PII redacted. If possible, please provide text instead of a screenshot. + description: Please include the full error messages and prompts with PII redacted. If possible, please provide text instead of a screenshot. validations: required: true - type: textarea id: steps attributes: label: What steps can reproduce the bug? - description: Explain the bug and provide a code snippet that can reproduce it. Please include session id, token limit usage, context window usage if applicable. + description: Explain the bug and provide a code snippet that can reproduce it. validations: required: true - type: textarea diff --git a/.github/ISSUE_TEMPLATE/2-bug-report.yml b/.github/ISSUE_TEMPLATE/3-cli.yml similarity index 69% rename from .github/ISSUE_TEMPLATE/2-bug-report.yml rename to .github/ISSUE_TEMPLATE/3-cli.yml index 2d5ec2ad3f..4aff813e5f 100644 --- a/.github/ISSUE_TEMPLATE/2-bug-report.yml +++ b/.github/ISSUE_TEMPLATE/3-cli.yml @@ -1,5 +1,5 @@ -name: πŸͺ² Bug Report -description: Report an issue that should be fixed +name: πŸ’» CLI Bug +description: Report an issue in the Codex CLI labels: - bug - needs triage @@ -7,19 +7,16 @@ body: - type: markdown attributes: value: | - Thank you for submitting a bug report! It helps make Codex better for everyone. - - If you need help or support using Codex, and are not reporting a bug, please post on [codex/discussions](https://github.com/openai/codex/discussions), where you can ask questions or engage with others on ideas for how to improve codex. + Before submitting a new issue, please search for existing issues to see if your issue has already been reported. + If it has, please add a πŸ‘ reaction (no need to leave a comment) to the existing issue instead of creating a new one. Make sure you are running the [latest](https://npmjs.com/package/@openai/codex) version of Codex CLI. The bug you are experiencing may already have been fixed. - Please try to include as much information as possible. - - type: input id: version attributes: - label: What version of Codex is running? - description: (App) look in "About Codex" dialog; (CLI) use `codex --version`; (IDE Extension) look in extensions panel. + label: What version of Codex CLI is running? + description: use `codex --version` validations: required: true - type: input @@ -32,13 +29,13 @@ body: id: model attributes: label: Which model were you using? - description: Like `gpt-4.1`, `o4-mini`, `o3`, etc. + description: Like `gpt-5.2`, `gpt-5.2-codex`, etc. - type: input id: platform attributes: label: What platform is your computer? description: | - For MacOS and Linux: copy the output of `uname -mprs` + For macOS and Linux: copy the output of `uname -mprs` For Windows: copy the output of `"$([Environment]::OSVersion | ForEach-Object VersionString) $(if ([Environment]::Is64BitOperatingSystem) { "x64" } else { "x86" })"` in the PowerShell console - type: input id: terminal @@ -58,7 +55,7 @@ body: id: steps attributes: label: What steps can reproduce the bug? - description: Explain the bug and provide a code snippet that can reproduce it. Please include session id, token limit usage, context window usage if applicable. + description: Explain the bug and provide a code snippet that can reproduce it. Please include thread id if applicable. validations: required: true - type: textarea diff --git a/.github/ISSUE_TEMPLATE/4-bug-report.yml b/.github/ISSUE_TEMPLATE/4-bug-report.yml new file mode 100644 index 0000000000..4de8841460 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/4-bug-report.yml @@ -0,0 +1,37 @@ +name: πŸͺ² Other Bug +description: Report an issue in Codex Web, integrations, or other Codex components +labels: + - bug +body: + - type: markdown + attributes: + value: | + Before submitting a new issue, please search for existing issues to see if your issue has already been reported. + If it has, please add a πŸ‘ reaction (no need to leave a comment) to the existing issue instead of creating a new one. + + If you need help or support using Codex and are not reporting a bug, please post on [codex/discussions](https://github.com/openai/codex/discussions), where you can ask questions or engage with others on ideas for how to improve codex. + + - type: textarea + id: actual + attributes: + label: What issue are you seeing? + description: Please include the full error messages and prompts with PII redacted. If possible, please provide text instead of a screenshot. + validations: + required: true + - type: textarea + id: steps + attributes: + label: What steps can reproduce the bug? + description: Explain the bug and provide a code snippet that can reproduce it. + validations: + required: true + - type: textarea + id: expected + attributes: + label: What is the expected behavior? + description: If possible, please provide text instead of a screenshot. + - type: textarea + id: notes + attributes: + label: Additional information + description: Is there anything else you think we should know? diff --git a/.github/ISSUE_TEMPLATE/4-feature-request.yml b/.github/ISSUE_TEMPLATE/5-feature-request.yml similarity index 94% rename from .github/ISSUE_TEMPLATE/4-feature-request.yml rename to .github/ISSUE_TEMPLATE/5-feature-request.yml index 780fbfab36..55ff9fbbcd 100644 --- a/.github/ISSUE_TEMPLATE/4-feature-request.yml +++ b/.github/ISSUE_TEMPLATE/5-feature-request.yml @@ -16,7 +16,7 @@ body: id: variant attributes: label: What variant of Codex are you using? - description: (e.g., CLI, App, IDE Extension, Web) + description: (e.g., App, IDE Extension, CLI, Web) validations: required: true - type: textarea diff --git a/.github/ISSUE_TEMPLATE/3-docs-issue.yml b/.github/ISSUE_TEMPLATE/6-docs-issue.yml similarity index 100% rename from .github/ISSUE_TEMPLATE/3-docs-issue.yml rename to .github/ISSUE_TEMPLATE/6-docs-issue.yml diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 55e05251d7..b5e91d7f22 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -100,6 +100,7 @@ jobs: - name: bazel test //... env: BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }} + CODEX_BWRAP_ENABLE_FFI: ${{ contains(matrix.target, 'unknown-linux') && '1' || '0' }} shell: bash run: | bazel $BAZEL_STARTUP_ARGS --bazelrc=.github/workflows/ci.bazelrc test //... \ diff --git a/.github/workflows/rust-ci.yml b/.github/workflows/rust-ci.yml index ec6319f8da..b6a3c50e18 100644 --- a/.github/workflows/rust-ci.yml +++ b/.github/workflows/rust-ci.yml @@ -99,6 +99,9 @@ jobs: USE_SCCACHE: ${{ startsWith(matrix.runner, 'windows') && 'false' || 'true' }} CARGO_INCREMENTAL: "0" SCCACHE_CACHE_SIZE: 10G + # Keep cargo-based CI independent of system bwrap build deps. + # The bwrap FFI path is validated in Bazel workflows. + CODEX_BWRAP_ENABLE_FFI: "0" strategy: fail-fast: false @@ -467,6 +470,9 @@ jobs: USE_SCCACHE: ${{ startsWith(matrix.runner, 'windows') && 'false' || 'true' }} CARGO_INCREMENTAL: "0" SCCACHE_CACHE_SIZE: 10G + # Keep cargo-based CI independent of system bwrap build deps. + # The bwrap FFI path is validated in Bazel workflows. + CODEX_BWRAP_ENABLE_FFI: "0" strategy: fail-fast: false @@ -502,7 +508,6 @@ jobs: steps: - uses: actions/checkout@v6 - # Some integration tests rely on DotSlash being installed. # See https://github.com/openai/codex/pull/7617. - name: Install DotSlash diff --git a/.github/workflows/rust-release.yml b/.github/workflows/rust-release.yml index 2f764df680..aa92693eee 100644 --- a/.github/workflows/rust-release.yml +++ b/.github/workflows/rust-release.yml @@ -65,6 +65,8 @@ jobs: defaults: run: working-directory: codex-rs + env: + CODEX_BWRAP_ENABLE_FFI: ${{ contains(matrix.target, 'unknown-linux') && '1' || '0' }} strategy: fail-fast: false @@ -89,6 +91,13 @@ jobs: steps: - uses: actions/checkout@v6 + - name: Install Linux bwrap build dependencies + if: ${{ runner.os == 'Linux' }} + shell: bash + run: | + set -euo pipefail + sudo apt-get update -y + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends pkg-config libcap-dev - name: Install UBSan runtime (musl) if: ${{ matrix.target == 'x86_64-unknown-linux-musl' || matrix.target == 'aarch64-unknown-linux-musl' }} shell: bash diff --git a/codex-rs/Cargo.lock b/codex-rs/Cargo.lock index 2596361529..ebc7ddd33c 100644 --- a/codex-rs/Cargo.lock +++ b/codex-rs/Cargo.lock @@ -398,6 +398,15 @@ dependencies = [ "wiremock", ] +[[package]] +name = "arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" +dependencies = [ + "derive_arbitrary", +] + [[package]] name = "arboard" version = "3.6.1" @@ -924,6 +933,25 @@ dependencies = [ "bytes", ] +[[package]] +name = "bzip2" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" +dependencies = [ + "bzip2-sys", +] + +[[package]] +name = "bzip2-sys" +version = "0.1.13+1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" +dependencies = [ + "cc", + "pkg-config", +] + [[package]] name = "cassowary" version = "0.3.0" @@ -1351,6 +1379,7 @@ dependencies = [ "clap_complete", "codex-app-server", "codex-app-server-protocol", + "codex-app-server-test-client", "codex-arg0", "codex-chatgpt", "codex-cloud-tasks", @@ -1535,6 +1564,7 @@ dependencies = [ "libc", "maplit", "multimap", + "notify", "once_cell", "openssl-sys", "opentelemetry_sdk", @@ -1578,6 +1608,7 @@ dependencies = [ "which", "wildmatch", "wiremock", + "zip", "zstd", ] @@ -1767,7 +1798,6 @@ dependencies = [ "serde_json", "tempfile", "tokio", - "which", ] [[package]] @@ -2353,6 +2383,12 @@ dependencies = [ "unicode-xid", ] +[[package]] +name = "constant_time_eq" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" + [[package]] name = "convert_case" version = "0.6.0" @@ -2771,6 +2807,12 @@ dependencies = [ "serde_json", ] +[[package]] +name = "deflate64" +version = "0.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26bf8fc351c5ed29b5c2f0cbbac1b209b74f60ecd62e675a998df72c49af5204" + [[package]] name = "der" version = "0.7.10" @@ -2803,6 +2845,17 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "derive_arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.114", +] + [[package]] name = "derive_more" version = "1.0.0" @@ -4929,6 +4982,27 @@ dependencies = [ "url", ] +[[package]] +name = "lzma-rs" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e" +dependencies = [ + "byteorder", + "crc", +] + +[[package]] +name = "lzma-sys" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + [[package]] name = "maplit" version = "1.0.2" @@ -10515,6 +10589,15 @@ dependencies = [ "windows-sys 0.59.0", ] +[[package]] +name = "xz2" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" +dependencies = [ + "lzma-sys", +] + [[package]] name = "yansi" version = "1.0.1" @@ -10701,12 +10784,54 @@ dependencies = [ "syn 2.0.114", ] +[[package]] +name = "zip" +version = "2.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50" +dependencies = [ + "aes", + "arbitrary", + "bzip2", + "constant_time_eq", + "crc32fast", + "crossbeam-utils", + "deflate64", + "displaydoc", + "flate2", + "getrandom 0.3.4", + "hmac", + "indexmap 2.13.0", + "lzma-rs", + "memchr", + "pbkdf2", + "sha1", + "thiserror 2.0.18", + "time", + "xz2", + "zeroize", + "zopfli", + "zstd", +] + [[package]] name = "zmij" version = "1.0.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3ff05f8caa9038894637571ae6b9e29466c1f4f829d26c9b28f869a29cbe3445" +[[package]] +name = "zopfli" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +dependencies = [ + "bumpalo", + "crc32fast", + "log", + "simd-adler32", +] + [[package]] name = "zstd" version = "0.13.3" diff --git a/codex-rs/Cargo.toml b/codex-rs/Cargo.toml index 22f53644f8..39ebe5967d 100644 --- a/codex-rs/Cargo.toml +++ b/codex-rs/Cargo.toml @@ -70,6 +70,7 @@ codex-ansi-escape = { path = "ansi-escape" } codex-api = { path = "codex-api" } codex-app-server = { path = "app-server" } codex-app-server-protocol = { path = "app-server-protocol" } +codex-app-server-test-client = { path = "app-server-test-client" } codex-apply-patch = { path = "apply-patch" } codex-arg0 = { path = "arg0" } codex-async-utils = { path = "async-utils" } @@ -249,6 +250,7 @@ walkdir = "2.5.0" webbrowser = "1.0" which = "8" wildmatch = "2.6.1" +zip = "2.4.2" wiremock = "0.6" zeroize = "1.8.2" diff --git a/codex-rs/app-server-protocol/schema/json/ClientRequest.json b/codex-rs/app-server-protocol/schema/json/ClientRequest.json index 3461572779..54041b40f2 100644 --- a/codex-rs/app-server-protocol/schema/json/ClientRequest.json +++ b/codex-rs/app-server-protocol/schema/json/ClientRequest.json @@ -480,6 +480,19 @@ }, "type": "object" }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -526,19 +539,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -548,7 +552,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -1171,6 +1175,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], @@ -2183,6 +2188,24 @@ }, "type": "object" }, + "SkillsRemoteReadParams": { + "type": "object" + }, + "SkillsRemoteWriteParams": { + "properties": { + "hazelnutId": { + "type": "string" + }, + "isPreload": { + "type": "boolean" + } + }, + "required": [ + "hazelnutId", + "isPreload" + ], + "type": "object" + }, "TextElement": { "properties": { "byteRange": { @@ -2217,6 +2240,17 @@ ], "type": "object" }, + "ThreadCompactStartParams": { + "properties": { + "threadId": { + "type": "string" + } + }, + "required": [ + "threadId" + ], + "type": "object" + }, "ThreadForkParams": { "description": "There are two ways to fork a thread: 1. By thread_id: load the thread from disk by thread_id and fork it into a new thread. 2. By path: load the thread from disk by path and fork it into a new thread.\n\nIf using path, the thread_id param will be ignored.\n\nPrefer using thread_id whenever possible.", "properties": { @@ -3192,6 +3226,30 @@ "title": "Thread/unarchiveRequest", "type": "object" }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "thread/compact/start" + ], + "title": "Thread/compact/startRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/ThreadCompactStartParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Thread/compact/startRequest", + "type": "object" + }, { "properties": { "id": { @@ -3312,6 +3370,54 @@ "title": "Skills/listRequest", "type": "object" }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "skills/remote/read" + ], + "title": "Skills/remote/readRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/SkillsRemoteReadParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Skills/remote/readRequest", + "type": "object" + }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "skills/remote/write" + ], + "title": "Skills/remote/writeRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/SkillsRemoteWriteParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Skills/remote/writeRequest", + "type": "object" + }, { "properties": { "id": { diff --git a/codex-rs/app-server-protocol/schema/json/DynamicToolCallResponse.json b/codex-rs/app-server-protocol/schema/json/DynamicToolCallResponse.json index 662d3bda4f..e0e29641d2 100644 --- a/codex-rs/app-server-protocol/schema/json/DynamicToolCallResponse.json +++ b/codex-rs/app-server-protocol/schema/json/DynamicToolCallResponse.json @@ -1,15 +1,64 @@ { "$schema": "http://json-schema.org/draft-07/schema#", + "definitions": { + "DynamicToolCallOutputContentItem": { + "oneOf": [ + { + "properties": { + "text": { + "type": "string" + }, + "type": { + "enum": [ + "inputText" + ], + "title": "InputTextDynamicToolCallOutputContentItemType", + "type": "string" + } + }, + "required": [ + "text", + "type" + ], + "title": "InputTextDynamicToolCallOutputContentItem", + "type": "object" + }, + { + "properties": { + "imageUrl": { + "type": "string" + }, + "type": { + "enum": [ + "inputImage" + ], + "title": "InputImageDynamicToolCallOutputContentItemType", + "type": "string" + } + }, + "required": [ + "imageUrl", + "type" + ], + "title": "InputImageDynamicToolCallOutputContentItem", + "type": "object" + } + ] + } + }, "properties": { - "output": { - "type": "string" + "contentItems": { + "items": { + "$ref": "#/definitions/DynamicToolCallOutputContentItem" + }, + "type": "array" }, "success": { "type": "boolean" } }, "required": [ - "output", + "contentItems", "success" ], "title": "DynamicToolCallResponse", diff --git a/codex-rs/app-server-protocol/schema/json/EventMsg.json b/codex-rs/app-server-protocol/schema/json/EventMsg.json index 1274e1cd7f..f399912e03 100644 --- a/codex-rs/app-server-protocol/schema/json/EventMsg.json +++ b/codex-rs/app-server-protocol/schema/json/EventMsg.json @@ -2012,6 +2012,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -2811,6 +2864,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -2857,19 +2923,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -2879,7 +2936,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -3429,6 +3486,25 @@ } ] }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RequestId": { "anyOf": [ { @@ -6536,6 +6612,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { diff --git a/codex-rs/app-server-protocol/schema/json/ServerNotification.json b/codex-rs/app-server-protocol/schema/json/ServerNotification.json index f89bcf24e4..0e2f94f4d8 100644 --- a/codex-rs/app-server-protocol/schema/json/ServerNotification.json +++ b/codex-rs/app-server-protocol/schema/json/ServerNotification.json @@ -2590,6 +2590,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -3431,6 +3484,19 @@ ], "type": "object" }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -3477,19 +3543,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -3499,7 +3556,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -4469,6 +4526,25 @@ ], "type": "object" }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RequestId": { "anyOf": [ { diff --git a/codex-rs/app-server-protocol/schema/json/codex_app_server_protocol.schemas.json b/codex-rs/app-server-protocol/schema/json/codex_app_server_protocol.schemas.json index bc72ceb21f..9b22ff2207 100644 --- a/codex-rs/app-server-protocol/schema/json/codex_app_server_protocol.schemas.json +++ b/codex-rs/app-server-protocol/schema/json/codex_app_server_protocol.schemas.json @@ -598,6 +598,30 @@ "title": "Thread/unarchiveRequest", "type": "object" }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "thread/compact/start" + ], + "title": "Thread/compact/startRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/v2/ThreadCompactStartParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Thread/compact/startRequest", + "type": "object" + }, { "properties": { "id": { @@ -718,6 +742,54 @@ "title": "Skills/listRequest", "type": "object" }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "skills/remote/read" + ], + "title": "Skills/remote/readRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/v2/SkillsRemoteReadParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Skills/remote/readRequest", + "type": "object" + }, + { + "properties": { + "id": { + "$ref": "#/definitions/RequestId" + }, + "method": { + "enum": [ + "skills/remote/write" + ], + "title": "Skills/remote/writeRequestMethod", + "type": "string" + }, + "params": { + "$ref": "#/definitions/v2/SkillsRemoteWriteParams" + } + }, + "required": [ + "id", + "method", + "params" + ], + "title": "Skills/remote/writeRequest", + "type": "object" + }, { "properties": { "id": { @@ -2240,6 +2312,50 @@ ], "type": "object" }, + "DynamicToolCallOutputContentItem": { + "oneOf": [ + { + "properties": { + "text": { + "type": "string" + }, + "type": { + "enum": [ + "inputText" + ], + "title": "InputTextDynamicToolCallOutputContentItemType", + "type": "string" + } + }, + "required": [ + "text", + "type" + ], + "title": "InputTextDynamicToolCallOutputContentItem", + "type": "object" + }, + { + "properties": { + "imageUrl": { + "type": "string" + }, + "type": { + "enum": [ + "inputImage" + ], + "title": "InputImageDynamicToolCallOutputContentItemType", + "type": "string" + } + }, + "required": [ + "imageUrl", + "type" + ], + "title": "InputImageDynamicToolCallOutputContentItem", + "type": "object" + } + ] + }, "DynamicToolCallParams": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { @@ -2270,15 +2386,18 @@ "DynamicToolCallResponse": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { - "output": { - "type": "string" + "contentItems": { + "items": { + "$ref": "#/definitions/DynamicToolCallOutputContentItem" + }, + "type": "array" }, "success": { "type": "boolean" } }, "required": [ - "output", + "contentItems", "success" ], "title": "DynamicToolCallResponse", @@ -3852,6 +3971,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/v2/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -4919,6 +5091,19 @@ "title": "ForkConversationResponse", "type": "object" }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -4965,19 +5150,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -4987,7 +5163,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -6351,6 +6527,25 @@ } ] }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RemoveConversationListenerParams": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { @@ -11102,6 +11297,19 @@ ], "type": "string" }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/v2/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -11148,19 +11356,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/v2/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/v2/FunctionCallOutputBody" }, "success": { "type": [ @@ -11170,7 +11369,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -11818,6 +12017,12 @@ "supportsPersonality": { "default": false, "type": "boolean" + }, + "upgrade": { + "type": [ + "string", + "null" + ] } }, "required": [ @@ -11970,6 +12175,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], @@ -12382,6 +12588,25 @@ "title": "ReasoningTextDeltaNotification", "type": "object" }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "ResidencyRequirement": { "enum": [ "us" @@ -13417,6 +13642,65 @@ "title": "SkillsListResponse", "type": "object" }, + "SkillsRemoteReadParams": { + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "SkillsRemoteReadParams", + "type": "object" + }, + "SkillsRemoteReadResponse": { + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "data": { + "items": { + "$ref": "#/definitions/v2/RemoteSkillSummary" + }, + "type": "array" + } + }, + "required": [ + "data" + ], + "title": "SkillsRemoteReadResponse", + "type": "object" + }, + "SkillsRemoteWriteParams": { + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "hazelnutId": { + "type": "string" + }, + "isPreload": { + "type": "boolean" + } + }, + "required": [ + "hazelnutId", + "isPreload" + ], + "title": "SkillsRemoteWriteParams", + "type": "object" + }, + "SkillsRemoteWriteResponse": { + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "path" + ], + "title": "SkillsRemoteWriteResponse", + "type": "object" + }, "SubAgentSource": { "oneOf": [ { @@ -13651,6 +13935,24 @@ "title": "ThreadArchiveResponse", "type": "object" }, + "ThreadCompactStartParams": { + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "threadId": { + "type": "string" + } + }, + "required": [ + "threadId" + ], + "title": "ThreadCompactStartParams", + "type": "object" + }, + "ThreadCompactStartResponse": { + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "ThreadCompactStartResponse", + "type": "object" + }, "ThreadForkParams": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "There are two ways to fork a thread: 1. By thread_id: load the thread from disk by thread_id and fork it into a new thread. 2. By path: load the thread from disk by path and fork it into a new thread.\n\nIf using path, the thread_id param will be ignored.\n\nPrefer using thread_id whenever possible.", diff --git a/codex-rs/app-server-protocol/schema/json/v1/ForkConversationResponse.json b/codex-rs/app-server-protocol/schema/json/v1/ForkConversationResponse.json index 9894c47c1e..a5838e89e7 100644 --- a/codex-rs/app-server-protocol/schema/json/v1/ForkConversationResponse.json +++ b/codex-rs/app-server-protocol/schema/json/v1/ForkConversationResponse.json @@ -2012,6 +2012,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -2811,6 +2864,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -2857,19 +2923,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -2879,7 +2936,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -3429,6 +3486,25 @@ } ] }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RequestId": { "anyOf": [ { diff --git a/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationParams.json b/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationParams.json index 1261306e61..9ce52963d9 100644 --- a/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationParams.json +++ b/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationParams.json @@ -98,6 +98,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -144,19 +157,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -166,7 +170,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, diff --git a/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationResponse.json b/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationResponse.json index fe4037a40f..718b17aa28 100644 --- a/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationResponse.json +++ b/codex-rs/app-server-protocol/schema/json/v1/ResumeConversationResponse.json @@ -2012,6 +2012,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -2811,6 +2864,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -2857,19 +2923,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -2879,7 +2936,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -3429,6 +3486,25 @@ } ] }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RequestId": { "anyOf": [ { diff --git a/codex-rs/app-server-protocol/schema/json/v1/SessionConfiguredNotification.json b/codex-rs/app-server-protocol/schema/json/v1/SessionConfiguredNotification.json index 8e4c361979..a85b78281b 100644 --- a/codex-rs/app-server-protocol/schema/json/v1/SessionConfiguredNotification.json +++ b/codex-rs/app-server-protocol/schema/json/v1/SessionConfiguredNotification.json @@ -2012,6 +2012,59 @@ "title": "ListSkillsResponseEventMsg", "type": "object" }, + { + "description": "List of remote skills available to the agent.", + "properties": { + "skills": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + }, + "type": { + "enum": [ + "list_remote_skills_response" + ], + "title": "ListRemoteSkillsResponseEventMsgType", + "type": "string" + } + }, + "required": [ + "skills", + "type" + ], + "title": "ListRemoteSkillsResponseEventMsg", + "type": "object" + }, + { + "description": "Remote skill downloaded to local cache.", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + }, + "type": { + "enum": [ + "remote_skill_downloaded" + ], + "title": "RemoteSkillDownloadedEventMsgType", + "type": "string" + } + }, + "required": [ + "id", + "name", + "path", + "type" + ], + "title": "RemoteSkillDownloadedEventMsg", + "type": "object" + }, { "description": "Notification that skill data may have been updated and clients may want to reload.", "properties": { @@ -2811,6 +2864,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -2857,19 +2923,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -2879,7 +2936,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -3429,6 +3486,25 @@ } ] }, + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + }, "RequestId": { "anyOf": [ { diff --git a/codex-rs/app-server-protocol/schema/json/v2/ModelListResponse.json b/codex-rs/app-server-protocol/schema/json/v2/ModelListResponse.json index 8c8126bb6b..a1b3f7114c 100644 --- a/codex-rs/app-server-protocol/schema/json/v2/ModelListResponse.json +++ b/codex-rs/app-server-protocol/schema/json/v2/ModelListResponse.json @@ -59,6 +59,12 @@ "supportsPersonality": { "default": false, "type": "boolean" + }, + "upgrade": { + "type": [ + "string", + "null" + ] } }, "required": [ diff --git a/codex-rs/app-server-protocol/schema/json/v2/RawResponseItemCompletedNotification.json b/codex-rs/app-server-protocol/schema/json/v2/RawResponseItemCompletedNotification.json index 1b307c9b89..c1e36ad8e8 100644 --- a/codex-rs/app-server-protocol/schema/json/v2/RawResponseItemCompletedNotification.json +++ b/codex-rs/app-server-protocol/schema/json/v2/RawResponseItemCompletedNotification.json @@ -65,6 +65,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -111,19 +124,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -133,7 +137,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, diff --git a/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadParams.json b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadParams.json new file mode 100644 index 0000000000..ace2fb5ba7 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadParams.json @@ -0,0 +1,5 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "SkillsRemoteReadParams", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadResponse.json b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadResponse.json new file mode 100644 index 0000000000..a8e19c65bb --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteReadResponse.json @@ -0,0 +1,37 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "definitions": { + "RemoteSkillSummary": { + "properties": { + "description": { + "type": "string" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": [ + "description", + "id", + "name" + ], + "type": "object" + } + }, + "properties": { + "data": { + "items": { + "$ref": "#/definitions/RemoteSkillSummary" + }, + "type": "array" + } + }, + "required": [ + "data" + ], + "title": "SkillsRemoteReadResponse", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteParams.json b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteParams.json new file mode 100644 index 0000000000..871a5a428a --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteParams.json @@ -0,0 +1,17 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "hazelnutId": { + "type": "string" + }, + "isPreload": { + "type": "boolean" + } + }, + "required": [ + "hazelnutId", + "isPreload" + ], + "title": "SkillsRemoteWriteParams", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteResponse.json b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteResponse.json new file mode 100644 index 0000000000..1a9473d054 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/SkillsRemoteWriteResponse.json @@ -0,0 +1,21 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "path": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "path" + ], + "title": "SkillsRemoteWriteResponse", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartParams.json b/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartParams.json new file mode 100644 index 0000000000..a174ff95d9 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartParams.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "threadId": { + "type": "string" + } + }, + "required": [ + "threadId" + ], + "title": "ThreadCompactStartParams", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartResponse.json b/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartResponse.json new file mode 100644 index 0000000000..bb372b6ddd --- /dev/null +++ b/codex-rs/app-server-protocol/schema/json/v2/ThreadCompactStartResponse.json @@ -0,0 +1,5 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "ThreadCompactStartResponse", + "type": "object" +} \ No newline at end of file diff --git a/codex-rs/app-server-protocol/schema/json/v2/ThreadResumeParams.json b/codex-rs/app-server-protocol/schema/json/v2/ThreadResumeParams.json index cc05de490a..ed1d5274e0 100644 --- a/codex-rs/app-server-protocol/schema/json/v2/ThreadResumeParams.json +++ b/codex-rs/app-server-protocol/schema/json/v2/ThreadResumeParams.json @@ -74,6 +74,19 @@ } ] }, + "FunctionCallOutputBody": { + "anyOf": [ + { + "type": "string" + }, + { + "items": { + "$ref": "#/definitions/FunctionCallOutputContentItem" + }, + "type": "array" + } + ] + }, "FunctionCallOutputContentItem": { "description": "Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs.", "oneOf": [ @@ -120,19 +133,10 @@ ] }, "FunctionCallOutputPayload": { - "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`content` preserves the historical plain-string payload so downstream integrations (tests, logging, etc.) can keep treating tool output as `String`. When an MCP server returns richer data we additionally populate `content_items` with the structured form that the Responses API understands.", + "description": "The payload we send back to OpenAI when reporting a tool call result.\n\n`body` serializes directly as the wire value for `function_call_output.output`. `success` remains internal metadata for downstream handling.", "properties": { - "content": { - "type": "string" - }, - "content_items": { - "items": { - "$ref": "#/definitions/FunctionCallOutputContentItem" - }, - "type": [ - "array", - "null" - ] + "body": { + "$ref": "#/definitions/FunctionCallOutputBody" }, "success": { "type": [ @@ -142,7 +146,7 @@ } }, "required": [ - "content" + "body" ], "type": "object" }, @@ -251,6 +255,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], diff --git a/codex-rs/app-server-protocol/schema/json/v2/ThreadStartParams.json b/codex-rs/app-server-protocol/schema/json/v2/ThreadStartParams.json index 03bed7f93c..f2b0052e6d 100644 --- a/codex-rs/app-server-protocol/schema/json/v2/ThreadStartParams.json +++ b/codex-rs/app-server-protocol/schema/json/v2/ThreadStartParams.json @@ -29,6 +29,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], diff --git a/codex-rs/app-server-protocol/schema/json/v2/TurnStartParams.json b/codex-rs/app-server-protocol/schema/json/v2/TurnStartParams.json index 6a739b31ac..be351a48ea 100644 --- a/codex-rs/app-server-protocol/schema/json/v2/TurnStartParams.json +++ b/codex-rs/app-server-protocol/schema/json/v2/TurnStartParams.json @@ -66,6 +66,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], diff --git a/codex-rs/app-server-protocol/schema/typescript/ClientRequest.ts b/codex-rs/app-server-protocol/schema/typescript/ClientRequest.ts index b65d9ee931..176f86be56 100644 --- a/codex-rs/app-server-protocol/schema/typescript/ClientRequest.ts +++ b/codex-rs/app-server-protocol/schema/typescript/ClientRequest.ts @@ -36,7 +36,10 @@ import type { ModelListParams } from "./v2/ModelListParams"; import type { ReviewStartParams } from "./v2/ReviewStartParams"; import type { SkillsConfigWriteParams } from "./v2/SkillsConfigWriteParams"; import type { SkillsListParams } from "./v2/SkillsListParams"; +import type { SkillsRemoteReadParams } from "./v2/SkillsRemoteReadParams"; +import type { SkillsRemoteWriteParams } from "./v2/SkillsRemoteWriteParams"; import type { ThreadArchiveParams } from "./v2/ThreadArchiveParams"; +import type { ThreadCompactStartParams } from "./v2/ThreadCompactStartParams"; import type { ThreadForkParams } from "./v2/ThreadForkParams"; import type { ThreadListParams } from "./v2/ThreadListParams"; import type { ThreadLoadedListParams } from "./v2/ThreadLoadedListParams"; @@ -52,4 +55,4 @@ import type { TurnStartParams } from "./v2/TurnStartParams"; /** * Request from the client to the server. */ -export type ClientRequest ={ "method": "initialize", id: RequestId, params: InitializeParams, } | { "method": "thread/start", id: RequestId, params: ThreadStartParams, } | { "method": "thread/resume", id: RequestId, params: ThreadResumeParams, } | { "method": "thread/fork", id: RequestId, params: ThreadForkParams, } | { "method": "thread/archive", id: RequestId, params: ThreadArchiveParams, } | { "method": "thread/name/set", id: RequestId, params: ThreadSetNameParams, } | { "method": "thread/unarchive", id: RequestId, params: ThreadUnarchiveParams, } | { "method": "thread/rollback", id: RequestId, params: ThreadRollbackParams, } | { "method": "thread/list", id: RequestId, params: ThreadListParams, } | { "method": "thread/loaded/list", id: RequestId, params: ThreadLoadedListParams, } | { "method": "thread/read", id: RequestId, params: ThreadReadParams, } | { "method": "skills/list", id: RequestId, params: SkillsListParams, } | { "method": "app/list", id: RequestId, params: AppsListParams, } | { "method": "skills/config/write", id: RequestId, params: SkillsConfigWriteParams, } | { "method": "turn/start", id: RequestId, params: TurnStartParams, } | { "method": "turn/interrupt", id: RequestId, params: TurnInterruptParams, } | { "method": "review/start", id: RequestId, params: ReviewStartParams, } | { "method": "model/list", id: RequestId, params: ModelListParams, } | { "method": "mcpServer/oauth/login", id: RequestId, params: McpServerOauthLoginParams, } | { "method": "config/mcpServer/reload", id: RequestId, params: undefined, } | { "method": "mcpServerStatus/list", id: RequestId, params: ListMcpServerStatusParams, } | { "method": "account/login/start", id: RequestId, params: LoginAccountParams, } | { "method": "account/login/cancel", id: RequestId, params: CancelLoginAccountParams, } | { "method": "account/logout", id: RequestId, params: undefined, } | { "method": "account/rateLimits/read", id: RequestId, params: undefined, } | { "method": "feedback/upload", id: RequestId, params: FeedbackUploadParams, } | { "method": "command/exec", id: RequestId, params: CommandExecParams, } | { "method": "config/read", id: RequestId, params: ConfigReadParams, } | { "method": "config/value/write", id: RequestId, params: ConfigValueWriteParams, } | { "method": "config/batchWrite", id: RequestId, params: ConfigBatchWriteParams, } | { "method": "configRequirements/read", id: RequestId, params: undefined, } | { "method": "account/read", id: RequestId, params: GetAccountParams, } | { "method": "newConversation", id: RequestId, params: NewConversationParams, } | { "method": "getConversationSummary", id: RequestId, params: GetConversationSummaryParams, } | { "method": "listConversations", id: RequestId, params: ListConversationsParams, } | { "method": "resumeConversation", id: RequestId, params: ResumeConversationParams, } | { "method": "forkConversation", id: RequestId, params: ForkConversationParams, } | { "method": "archiveConversation", id: RequestId, params: ArchiveConversationParams, } | { "method": "sendUserMessage", id: RequestId, params: SendUserMessageParams, } | { "method": "sendUserTurn", id: RequestId, params: SendUserTurnParams, } | { "method": "interruptConversation", id: RequestId, params: InterruptConversationParams, } | { "method": "addConversationListener", id: RequestId, params: AddConversationListenerParams, } | { "method": "removeConversationListener", id: RequestId, params: RemoveConversationListenerParams, } | { "method": "gitDiffToRemote", id: RequestId, params: GitDiffToRemoteParams, } | { "method": "loginApiKey", id: RequestId, params: LoginApiKeyParams, } | { "method": "loginChatGpt", id: RequestId, params: undefined, } | { "method": "cancelLoginChatGpt", id: RequestId, params: CancelLoginChatGptParams, } | { "method": "logoutChatGpt", id: RequestId, params: undefined, } | { "method": "getAuthStatus", id: RequestId, params: GetAuthStatusParams, } | { "method": "getUserSavedConfig", id: RequestId, params: undefined, } | { "method": "setDefaultModel", id: RequestId, params: SetDefaultModelParams, } | { "method": "getUserAgent", id: RequestId, params: undefined, } | { "method": "userInfo", id: RequestId, params: undefined, } | { "method": "fuzzyFileSearch", id: RequestId, params: FuzzyFileSearchParams, } | { "method": "execOneOffCommand", id: RequestId, params: ExecOneOffCommandParams, }; +export type ClientRequest ={ "method": "initialize", id: RequestId, params: InitializeParams, } | { "method": "thread/start", id: RequestId, params: ThreadStartParams, } | { "method": "thread/resume", id: RequestId, params: ThreadResumeParams, } | { "method": "thread/fork", id: RequestId, params: ThreadForkParams, } | { "method": "thread/archive", id: RequestId, params: ThreadArchiveParams, } | { "method": "thread/name/set", id: RequestId, params: ThreadSetNameParams, } | { "method": "thread/unarchive", id: RequestId, params: ThreadUnarchiveParams, } | { "method": "thread/compact/start", id: RequestId, params: ThreadCompactStartParams, } | { "method": "thread/rollback", id: RequestId, params: ThreadRollbackParams, } | { "method": "thread/list", id: RequestId, params: ThreadListParams, } | { "method": "thread/loaded/list", id: RequestId, params: ThreadLoadedListParams, } | { "method": "thread/read", id: RequestId, params: ThreadReadParams, } | { "method": "skills/list", id: RequestId, params: SkillsListParams, } | { "method": "skills/remote/read", id: RequestId, params: SkillsRemoteReadParams, } | { "method": "skills/remote/write", id: RequestId, params: SkillsRemoteWriteParams, } | { "method": "app/list", id: RequestId, params: AppsListParams, } | { "method": "skills/config/write", id: RequestId, params: SkillsConfigWriteParams, } | { "method": "turn/start", id: RequestId, params: TurnStartParams, } | { "method": "turn/interrupt", id: RequestId, params: TurnInterruptParams, } | { "method": "review/start", id: RequestId, params: ReviewStartParams, } | { "method": "model/list", id: RequestId, params: ModelListParams, } | { "method": "mcpServer/oauth/login", id: RequestId, params: McpServerOauthLoginParams, } | { "method": "config/mcpServer/reload", id: RequestId, params: undefined, } | { "method": "mcpServerStatus/list", id: RequestId, params: ListMcpServerStatusParams, } | { "method": "account/login/start", id: RequestId, params: LoginAccountParams, } | { "method": "account/login/cancel", id: RequestId, params: CancelLoginAccountParams, } | { "method": "account/logout", id: RequestId, params: undefined, } | { "method": "account/rateLimits/read", id: RequestId, params: undefined, } | { "method": "feedback/upload", id: RequestId, params: FeedbackUploadParams, } | { "method": "command/exec", id: RequestId, params: CommandExecParams, } | { "method": "config/read", id: RequestId, params: ConfigReadParams, } | { "method": "config/value/write", id: RequestId, params: ConfigValueWriteParams, } | { "method": "config/batchWrite", id: RequestId, params: ConfigBatchWriteParams, } | { "method": "configRequirements/read", id: RequestId, params: undefined, } | { "method": "account/read", id: RequestId, params: GetAccountParams, } | { "method": "newConversation", id: RequestId, params: NewConversationParams, } | { "method": "getConversationSummary", id: RequestId, params: GetConversationSummaryParams, } | { "method": "listConversations", id: RequestId, params: ListConversationsParams, } | { "method": "resumeConversation", id: RequestId, params: ResumeConversationParams, } | { "method": "forkConversation", id: RequestId, params: ForkConversationParams, } | { "method": "archiveConversation", id: RequestId, params: ArchiveConversationParams, } | { "method": "sendUserMessage", id: RequestId, params: SendUserMessageParams, } | { "method": "sendUserTurn", id: RequestId, params: SendUserTurnParams, } | { "method": "interruptConversation", id: RequestId, params: InterruptConversationParams, } | { "method": "addConversationListener", id: RequestId, params: AddConversationListenerParams, } | { "method": "removeConversationListener", id: RequestId, params: RemoveConversationListenerParams, } | { "method": "gitDiffToRemote", id: RequestId, params: GitDiffToRemoteParams, } | { "method": "loginApiKey", id: RequestId, params: LoginApiKeyParams, } | { "method": "loginChatGpt", id: RequestId, params: undefined, } | { "method": "cancelLoginChatGpt", id: RequestId, params: CancelLoginChatGptParams, } | { "method": "logoutChatGpt", id: RequestId, params: undefined, } | { "method": "getAuthStatus", id: RequestId, params: GetAuthStatusParams, } | { "method": "getUserSavedConfig", id: RequestId, params: undefined, } | { "method": "setDefaultModel", id: RequestId, params: SetDefaultModelParams, } | { "method": "getUserAgent", id: RequestId, params: undefined, } | { "method": "userInfo", id: RequestId, params: undefined, } | { "method": "fuzzyFileSearch", id: RequestId, params: FuzzyFileSearchParams, } | { "method": "execOneOffCommand", id: RequestId, params: ExecOneOffCommandParams, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/EventMsg.ts b/codex-rs/app-server-protocol/schema/typescript/EventMsg.ts index 5f158de2cb..c18088eaaf 100644 --- a/codex-rs/app-server-protocol/schema/typescript/EventMsg.ts +++ b/codex-rs/app-server-protocol/schema/typescript/EventMsg.ts @@ -33,6 +33,7 @@ import type { GetHistoryEntryResponseEvent } from "./GetHistoryEntryResponseEven import type { ItemCompletedEvent } from "./ItemCompletedEvent"; import type { ItemStartedEvent } from "./ItemStartedEvent"; import type { ListCustomPromptsResponseEvent } from "./ListCustomPromptsResponseEvent"; +import type { ListRemoteSkillsResponseEvent } from "./ListRemoteSkillsResponseEvent"; import type { ListSkillsResponseEvent } from "./ListSkillsResponseEvent"; import type { McpListToolsResponseEvent } from "./McpListToolsResponseEvent"; import type { McpStartupCompleteEvent } from "./McpStartupCompleteEvent"; @@ -45,6 +46,7 @@ import type { PlanDeltaEvent } from "./PlanDeltaEvent"; import type { RawResponseItemEvent } from "./RawResponseItemEvent"; import type { ReasoningContentDeltaEvent } from "./ReasoningContentDeltaEvent"; import type { ReasoningRawContentDeltaEvent } from "./ReasoningRawContentDeltaEvent"; +import type { RemoteSkillDownloadedEvent } from "./RemoteSkillDownloadedEvent"; import type { RequestUserInputEvent } from "./RequestUserInputEvent"; import type { ReviewRequest } from "./ReviewRequest"; import type { SessionConfiguredEvent } from "./SessionConfiguredEvent"; @@ -70,4 +72,4 @@ import type { WebSearchEndEvent } from "./WebSearchEndEvent"; * Response event from the agent * NOTE: Make sure none of these values have optional types, as it will mess up the extension code-gen. */ -export type EventMsg = { "type": "error" } & ErrorEvent | { "type": "warning" } & WarningEvent | { "type": "context_compacted" } & ContextCompactedEvent | { "type": "thread_rolled_back" } & ThreadRolledBackEvent | { "type": "task_started" } & TurnStartedEvent | { "type": "task_complete" } & TurnCompleteEvent | { "type": "token_count" } & TokenCountEvent | { "type": "agent_message" } & AgentMessageEvent | { "type": "user_message" } & UserMessageEvent | { "type": "agent_message_delta" } & AgentMessageDeltaEvent | { "type": "agent_reasoning" } & AgentReasoningEvent | { "type": "agent_reasoning_delta" } & AgentReasoningDeltaEvent | { "type": "agent_reasoning_raw_content" } & AgentReasoningRawContentEvent | { "type": "agent_reasoning_raw_content_delta" } & AgentReasoningRawContentDeltaEvent | { "type": "agent_reasoning_section_break" } & AgentReasoningSectionBreakEvent | { "type": "session_configured" } & SessionConfiguredEvent | { "type": "thread_name_updated" } & ThreadNameUpdatedEvent | { "type": "mcp_startup_update" } & McpStartupUpdateEvent | { "type": "mcp_startup_complete" } & McpStartupCompleteEvent | { "type": "mcp_tool_call_begin" } & McpToolCallBeginEvent | { "type": "mcp_tool_call_end" } & McpToolCallEndEvent | { "type": "web_search_begin" } & WebSearchBeginEvent | { "type": "web_search_end" } & WebSearchEndEvent | { "type": "exec_command_begin" } & ExecCommandBeginEvent | { "type": "exec_command_output_delta" } & ExecCommandOutputDeltaEvent | { "type": "terminal_interaction" } & TerminalInteractionEvent | { "type": "exec_command_end" } & ExecCommandEndEvent | { "type": "view_image_tool_call" } & ViewImageToolCallEvent | { "type": "exec_approval_request" } & ExecApprovalRequestEvent | { "type": "request_user_input" } & RequestUserInputEvent | { "type": "dynamic_tool_call_request" } & DynamicToolCallRequest | { "type": "elicitation_request" } & ElicitationRequestEvent | { "type": "apply_patch_approval_request" } & ApplyPatchApprovalRequestEvent | { "type": "deprecation_notice" } & DeprecationNoticeEvent | { "type": "background_event" } & BackgroundEventEvent | { "type": "undo_started" } & UndoStartedEvent | { "type": "undo_completed" } & UndoCompletedEvent | { "type": "stream_error" } & StreamErrorEvent | { "type": "patch_apply_begin" } & PatchApplyBeginEvent | { "type": "patch_apply_end" } & PatchApplyEndEvent | { "type": "turn_diff" } & TurnDiffEvent | { "type": "get_history_entry_response" } & GetHistoryEntryResponseEvent | { "type": "mcp_list_tools_response" } & McpListToolsResponseEvent | { "type": "list_custom_prompts_response" } & ListCustomPromptsResponseEvent | { "type": "list_skills_response" } & ListSkillsResponseEvent | { "type": "skills_update_available" } | { "type": "plan_update" } & UpdatePlanArgs | { "type": "turn_aborted" } & TurnAbortedEvent | { "type": "shutdown_complete" } | { "type": "entered_review_mode" } & ReviewRequest | { "type": "exited_review_mode" } & ExitedReviewModeEvent | { "type": "raw_response_item" } & RawResponseItemEvent | { "type": "item_started" } & ItemStartedEvent | { "type": "item_completed" } & ItemCompletedEvent | { "type": "agent_message_content_delta" } & AgentMessageContentDeltaEvent | { "type": "plan_delta" } & PlanDeltaEvent | { "type": "reasoning_content_delta" } & ReasoningContentDeltaEvent | { "type": "reasoning_raw_content_delta" } & ReasoningRawContentDeltaEvent | { "type": "collab_agent_spawn_begin" } & CollabAgentSpawnBeginEvent | { "type": "collab_agent_spawn_end" } & CollabAgentSpawnEndEvent | { "type": "collab_agent_interaction_begin" } & CollabAgentInteractionBeginEvent | { "type": "collab_agent_interaction_end" } & CollabAgentInteractionEndEvent | { "type": "collab_waiting_begin" } & CollabWaitingBeginEvent | { "type": "collab_waiting_end" } & CollabWaitingEndEvent | { "type": "collab_close_begin" } & CollabCloseBeginEvent | { "type": "collab_close_end" } & CollabCloseEndEvent; +export type EventMsg = { "type": "error" } & ErrorEvent | { "type": "warning" } & WarningEvent | { "type": "context_compacted" } & ContextCompactedEvent | { "type": "thread_rolled_back" } & ThreadRolledBackEvent | { "type": "task_started" } & TurnStartedEvent | { "type": "task_complete" } & TurnCompleteEvent | { "type": "token_count" } & TokenCountEvent | { "type": "agent_message" } & AgentMessageEvent | { "type": "user_message" } & UserMessageEvent | { "type": "agent_message_delta" } & AgentMessageDeltaEvent | { "type": "agent_reasoning" } & AgentReasoningEvent | { "type": "agent_reasoning_delta" } & AgentReasoningDeltaEvent | { "type": "agent_reasoning_raw_content" } & AgentReasoningRawContentEvent | { "type": "agent_reasoning_raw_content_delta" } & AgentReasoningRawContentDeltaEvent | { "type": "agent_reasoning_section_break" } & AgentReasoningSectionBreakEvent | { "type": "session_configured" } & SessionConfiguredEvent | { "type": "thread_name_updated" } & ThreadNameUpdatedEvent | { "type": "mcp_startup_update" } & McpStartupUpdateEvent | { "type": "mcp_startup_complete" } & McpStartupCompleteEvent | { "type": "mcp_tool_call_begin" } & McpToolCallBeginEvent | { "type": "mcp_tool_call_end" } & McpToolCallEndEvent | { "type": "web_search_begin" } & WebSearchBeginEvent | { "type": "web_search_end" } & WebSearchEndEvent | { "type": "exec_command_begin" } & ExecCommandBeginEvent | { "type": "exec_command_output_delta" } & ExecCommandOutputDeltaEvent | { "type": "terminal_interaction" } & TerminalInteractionEvent | { "type": "exec_command_end" } & ExecCommandEndEvent | { "type": "view_image_tool_call" } & ViewImageToolCallEvent | { "type": "exec_approval_request" } & ExecApprovalRequestEvent | { "type": "request_user_input" } & RequestUserInputEvent | { "type": "dynamic_tool_call_request" } & DynamicToolCallRequest | { "type": "elicitation_request" } & ElicitationRequestEvent | { "type": "apply_patch_approval_request" } & ApplyPatchApprovalRequestEvent | { "type": "deprecation_notice" } & DeprecationNoticeEvent | { "type": "background_event" } & BackgroundEventEvent | { "type": "undo_started" } & UndoStartedEvent | { "type": "undo_completed" } & UndoCompletedEvent | { "type": "stream_error" } & StreamErrorEvent | { "type": "patch_apply_begin" } & PatchApplyBeginEvent | { "type": "patch_apply_end" } & PatchApplyEndEvent | { "type": "turn_diff" } & TurnDiffEvent | { "type": "get_history_entry_response" } & GetHistoryEntryResponseEvent | { "type": "mcp_list_tools_response" } & McpListToolsResponseEvent | { "type": "list_custom_prompts_response" } & ListCustomPromptsResponseEvent | { "type": "list_skills_response" } & ListSkillsResponseEvent | { "type": "list_remote_skills_response" } & ListRemoteSkillsResponseEvent | { "type": "remote_skill_downloaded" } & RemoteSkillDownloadedEvent | { "type": "skills_update_available" } | { "type": "plan_update" } & UpdatePlanArgs | { "type": "turn_aborted" } & TurnAbortedEvent | { "type": "shutdown_complete" } | { "type": "entered_review_mode" } & ReviewRequest | { "type": "exited_review_mode" } & ExitedReviewModeEvent | { "type": "raw_response_item" } & RawResponseItemEvent | { "type": "item_started" } & ItemStartedEvent | { "type": "item_completed" } & ItemCompletedEvent | { "type": "agent_message_content_delta" } & AgentMessageContentDeltaEvent | { "type": "plan_delta" } & PlanDeltaEvent | { "type": "reasoning_content_delta" } & ReasoningContentDeltaEvent | { "type": "reasoning_raw_content_delta" } & ReasoningRawContentDeltaEvent | { "type": "collab_agent_spawn_begin" } & CollabAgentSpawnBeginEvent | { "type": "collab_agent_spawn_end" } & CollabAgentSpawnEndEvent | { "type": "collab_agent_interaction_begin" } & CollabAgentInteractionBeginEvent | { "type": "collab_agent_interaction_end" } & CollabAgentInteractionEndEvent | { "type": "collab_waiting_begin" } & CollabWaitingBeginEvent | { "type": "collab_waiting_end" } & CollabWaitingEndEvent | { "type": "collab_close_begin" } & CollabCloseBeginEvent | { "type": "collab_close_end" } & CollabCloseEndEvent; diff --git a/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputBody.ts b/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputBody.ts new file mode 100644 index 0000000000..6bcb7e25d6 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputBody.ts @@ -0,0 +1,6 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { FunctionCallOutputContentItem } from "./FunctionCallOutputContentItem"; + +export type FunctionCallOutputBody = string | Array; diff --git a/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputPayload.ts b/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputPayload.ts index 94370f582d..6376c5b8eb 100644 --- a/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputPayload.ts +++ b/codex-rs/app-server-protocol/schema/typescript/FunctionCallOutputPayload.ts @@ -1,14 +1,12 @@ // GENERATED CODE! DO NOT MODIFY BY HAND! // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -import type { FunctionCallOutputContentItem } from "./FunctionCallOutputContentItem"; +import type { FunctionCallOutputBody } from "./FunctionCallOutputBody"; /** * The payload we send back to OpenAI when reporting a tool call result. * - * `content` preserves the historical plain-string payload so downstream - * integrations (tests, logging, etc.) can keep treating tool output as - * `String`. When an MCP server returns richer data we additionally populate - * `content_items` with the structured form that the Responses API understands. + * `body` serializes directly as the wire value for `function_call_output.output`. + * `success` remains internal metadata for downstream handling. */ -export type FunctionCallOutputPayload = { content: string, content_items: Array | null, success: boolean | null, }; +export type FunctionCallOutputPayload = { body: FunctionCallOutputBody, success: boolean | null, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/ListRemoteSkillsResponseEvent.ts b/codex-rs/app-server-protocol/schema/typescript/ListRemoteSkillsResponseEvent.ts new file mode 100644 index 0000000000..e3b277f4d6 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/ListRemoteSkillsResponseEvent.ts @@ -0,0 +1,9 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { RemoteSkillSummary } from "./RemoteSkillSummary"; + +/** + * Response payload for `Op::ListRemoteSkills`. + */ +export type ListRemoteSkillsResponseEvent = { skills: Array, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/Personality.ts b/codex-rs/app-server-protocol/schema/typescript/Personality.ts index b9ccad4dc2..45165f4e33 100644 --- a/codex-rs/app-server-protocol/schema/typescript/Personality.ts +++ b/codex-rs/app-server-protocol/schema/typescript/Personality.ts @@ -2,4 +2,4 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -export type Personality = "friendly" | "pragmatic"; +export type Personality = "none" | "friendly" | "pragmatic"; diff --git a/codex-rs/app-server-protocol/schema/typescript/RemoteSkillDownloadedEvent.ts b/codex-rs/app-server-protocol/schema/typescript/RemoteSkillDownloadedEvent.ts new file mode 100644 index 0000000000..83082f2a57 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/RemoteSkillDownloadedEvent.ts @@ -0,0 +1,8 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * Response payload for `Op::DownloadRemoteSkill`. + */ +export type RemoteSkillDownloadedEvent = { id: string, name: string, path: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/RemoteSkillSummary.ts b/codex-rs/app-server-protocol/schema/typescript/RemoteSkillSummary.ts new file mode 100644 index 0000000000..7bf57b3b09 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/RemoteSkillSummary.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type RemoteSkillSummary = { id: string, name: string, description: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/index.ts b/codex-rs/app-server-protocol/schema/typescript/index.ts index 91da0708e1..a6ff6fbaf1 100644 --- a/codex-rs/app-server-protocol/schema/typescript/index.ts +++ b/codex-rs/app-server-protocol/schema/typescript/index.ts @@ -69,6 +69,7 @@ export type { FileChange } from "./FileChange"; export type { ForcedLoginMethod } from "./ForcedLoginMethod"; export type { ForkConversationParams } from "./ForkConversationParams"; export type { ForkConversationResponse } from "./ForkConversationResponse"; +export type { FunctionCallOutputBody } from "./FunctionCallOutputBody"; export type { FunctionCallOutputContentItem } from "./FunctionCallOutputContentItem"; export type { FunctionCallOutputPayload } from "./FunctionCallOutputPayload"; export type { FuzzyFileSearchParams } from "./FuzzyFileSearchParams"; @@ -98,6 +99,7 @@ export type { ItemStartedEvent } from "./ItemStartedEvent"; export type { ListConversationsParams } from "./ListConversationsParams"; export type { ListConversationsResponse } from "./ListConversationsResponse"; export type { ListCustomPromptsResponseEvent } from "./ListCustomPromptsResponseEvent"; +export type { ListRemoteSkillsResponseEvent } from "./ListRemoteSkillsResponseEvent"; export type { ListSkillsResponseEvent } from "./ListSkillsResponseEvent"; export type { LocalShellAction } from "./LocalShellAction"; export type { LocalShellExecAction } from "./LocalShellExecAction"; @@ -140,6 +142,8 @@ export type { ReasoningItemContent } from "./ReasoningItemContent"; export type { ReasoningItemReasoningSummary } from "./ReasoningItemReasoningSummary"; export type { ReasoningRawContentDeltaEvent } from "./ReasoningRawContentDeltaEvent"; export type { ReasoningSummary } from "./ReasoningSummary"; +export type { RemoteSkillDownloadedEvent } from "./RemoteSkillDownloadedEvent"; +export type { RemoteSkillSummary } from "./RemoteSkillSummary"; export type { RemoveConversationListenerParams } from "./RemoveConversationListenerParams"; export type { RemoveConversationSubscriptionResponse } from "./RemoveConversationSubscriptionResponse"; export type { RequestId } from "./RequestId"; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallOutputContentItem.ts b/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallOutputContentItem.ts new file mode 100644 index 0000000000..8f432109d1 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallOutputContentItem.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type DynamicToolCallOutputContentItem = { "type": "inputText", text: string, } | { "type": "inputImage", imageUrl: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallResponse.ts b/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallResponse.ts index a35b9b394a..788e6242dc 100644 --- a/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallResponse.ts +++ b/codex-rs/app-server-protocol/schema/typescript/v2/DynamicToolCallResponse.ts @@ -1,5 +1,6 @@ // GENERATED CODE! DO NOT MODIFY BY HAND! // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { DynamicToolCallOutputContentItem } from "./DynamicToolCallOutputContentItem"; -export type DynamicToolCallResponse = { output: string, success: boolean, }; +export type DynamicToolCallResponse = { contentItems: Array, success: boolean, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/Model.ts b/codex-rs/app-server-protocol/schema/typescript/v2/Model.ts index 72daa93edd..7528a8fad3 100644 --- a/codex-rs/app-server-protocol/schema/typescript/v2/Model.ts +++ b/codex-rs/app-server-protocol/schema/typescript/v2/Model.ts @@ -5,4 +5,4 @@ import type { InputModality } from "../InputModality"; import type { ReasoningEffort } from "../ReasoningEffort"; import type { ReasoningEffortOption } from "./ReasoningEffortOption"; -export type Model = { id: string, model: string, displayName: string, description: string, supportedReasoningEfforts: Array, defaultReasoningEffort: ReasoningEffort, inputModalities: Array, supportsPersonality: boolean, isDefault: boolean, }; +export type Model = { id: string, model: string, upgrade: string | null, displayName: string, description: string, supportedReasoningEfforts: Array, defaultReasoningEffort: ReasoningEffort, inputModalities: Array, supportsPersonality: boolean, isDefault: boolean, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/RemoteSkillSummary.ts b/codex-rs/app-server-protocol/schema/typescript/v2/RemoteSkillSummary.ts new file mode 100644 index 0000000000..7bf57b3b09 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/RemoteSkillSummary.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type RemoteSkillSummary = { id: string, name: string, description: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadParams.ts b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadParams.ts new file mode 100644 index 0000000000..9f91787686 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadParams.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SkillsRemoteReadParams = Record; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadResponse.ts b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadResponse.ts new file mode 100644 index 0000000000..c1c7b1cc70 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteReadResponse.ts @@ -0,0 +1,6 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { RemoteSkillSummary } from "./RemoteSkillSummary"; + +export type SkillsRemoteReadResponse = { data: Array, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteParams.ts b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteParams.ts new file mode 100644 index 0000000000..857b609ef1 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteParams.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SkillsRemoteWriteParams = { hazelnutId: string, isPreload: boolean, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteResponse.ts b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteResponse.ts new file mode 100644 index 0000000000..cf1665ab97 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/SkillsRemoteWriteResponse.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SkillsRemoteWriteResponse = { id: string, name: string, path: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartParams.ts b/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartParams.ts new file mode 100644 index 0000000000..a60b2c2812 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartParams.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type ThreadCompactStartParams = { threadId: string, }; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartResponse.ts b/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartResponse.ts new file mode 100644 index 0000000000..3794feb270 --- /dev/null +++ b/codex-rs/app-server-protocol/schema/typescript/v2/ThreadCompactStartResponse.ts @@ -0,0 +1,5 @@ +// GENERATED CODE! DO NOT MODIFY BY HAND! + +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type ThreadCompactStartResponse = Record; diff --git a/codex-rs/app-server-protocol/schema/typescript/v2/index.ts b/codex-rs/app-server-protocol/schema/typescript/v2/index.ts index 0c50231a80..ec5d54cba1 100644 --- a/codex-rs/app-server-protocol/schema/typescript/v2/index.ts +++ b/codex-rs/app-server-protocol/schema/typescript/v2/index.ts @@ -46,6 +46,7 @@ export type { ConfigWriteResponse } from "./ConfigWriteResponse"; export type { ContextCompactedNotification } from "./ContextCompactedNotification"; export type { CreditsSnapshot } from "./CreditsSnapshot"; export type { DeprecationNoticeNotification } from "./DeprecationNoticeNotification"; +export type { DynamicToolCallOutputContentItem } from "./DynamicToolCallOutputContentItem"; export type { DynamicToolCallParams } from "./DynamicToolCallParams"; export type { DynamicToolCallResponse } from "./DynamicToolCallResponse"; export type { DynamicToolSpec } from "./DynamicToolSpec"; @@ -96,6 +97,7 @@ export type { ReasoningEffortOption } from "./ReasoningEffortOption"; export type { ReasoningSummaryPartAddedNotification } from "./ReasoningSummaryPartAddedNotification"; export type { ReasoningSummaryTextDeltaNotification } from "./ReasoningSummaryTextDeltaNotification"; export type { ReasoningTextDeltaNotification } from "./ReasoningTextDeltaNotification"; +export type { RemoteSkillSummary } from "./RemoteSkillSummary"; export type { ResidencyRequirement } from "./ResidencyRequirement"; export type { ReviewDelivery } from "./ReviewDelivery"; export type { ReviewStartParams } from "./ReviewStartParams"; @@ -116,6 +118,10 @@ export type { SkillsConfigWriteResponse } from "./SkillsConfigWriteResponse"; export type { SkillsListEntry } from "./SkillsListEntry"; export type { SkillsListParams } from "./SkillsListParams"; export type { SkillsListResponse } from "./SkillsListResponse"; +export type { SkillsRemoteReadParams } from "./SkillsRemoteReadParams"; +export type { SkillsRemoteReadResponse } from "./SkillsRemoteReadResponse"; +export type { SkillsRemoteWriteParams } from "./SkillsRemoteWriteParams"; +export type { SkillsRemoteWriteResponse } from "./SkillsRemoteWriteResponse"; export type { TerminalInteractionNotification } from "./TerminalInteractionNotification"; export type { TextElement } from "./TextElement"; export type { TextPosition } from "./TextPosition"; @@ -123,6 +129,8 @@ export type { TextRange } from "./TextRange"; export type { Thread } from "./Thread"; export type { ThreadArchiveParams } from "./ThreadArchiveParams"; export type { ThreadArchiveResponse } from "./ThreadArchiveResponse"; +export type { ThreadCompactStartParams } from "./ThreadCompactStartParams"; +export type { ThreadCompactStartResponse } from "./ThreadCompactStartResponse"; export type { ThreadForkParams } from "./ThreadForkParams"; export type { ThreadForkResponse } from "./ThreadForkResponse"; export type { ThreadItem } from "./ThreadItem"; diff --git a/codex-rs/app-server-protocol/src/protocol/common.rs b/codex-rs/app-server-protocol/src/protocol/common.rs index d20f23a88a..c84d28ae89 100644 --- a/codex-rs/app-server-protocol/src/protocol/common.rs +++ b/codex-rs/app-server-protocol/src/protocol/common.rs @@ -208,6 +208,10 @@ client_request_definitions! { params: v2::ThreadUnarchiveParams, response: v2::ThreadUnarchiveResponse, }, + ThreadCompactStart => "thread/compact/start" { + params: v2::ThreadCompactStartParams, + response: v2::ThreadCompactStartResponse, + }, ThreadRollback => "thread/rollback" { params: v2::ThreadRollbackParams, response: v2::ThreadRollbackResponse, @@ -228,6 +232,14 @@ client_request_definitions! { params: v2::SkillsListParams, response: v2::SkillsListResponse, }, + SkillsRemoteRead => "skills/remote/read" { + params: v2::SkillsRemoteReadParams, + response: v2::SkillsRemoteReadResponse, + }, + SkillsRemoteWrite => "skills/remote/write" { + params: v2::SkillsRemoteWriteParams, + response: v2::SkillsRemoteWriteResponse, + }, AppsList => "app/list" { params: v2::AppsListParams, response: v2::AppsListResponse, diff --git a/codex-rs/app-server-protocol/src/protocol/v2.rs b/codex-rs/app-server-protocol/src/protocol/v2.rs index 7ce8cc2ed3..baafa84677 100644 --- a/codex-rs/app-server-protocol/src/protocol/v2.rs +++ b/codex-rs/app-server-protocol/src/protocol/v2.rs @@ -998,6 +998,7 @@ pub struct ModelListParams { pub struct Model { pub id: String, pub model: String, + pub upgrade: Option, pub display_name: String, pub description: String, pub supported_reasoning_efforts: Vec, @@ -1407,6 +1408,18 @@ pub struct ThreadUnarchiveResponse { pub thread: Thread, } +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct ThreadCompactStartParams { + pub thread_id: String, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct ThreadCompactStartResponse {} + #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] #[serde(rename_all = "camelCase")] #[ts(export_to = "v2/")] @@ -1554,6 +1567,44 @@ pub struct SkillsListResponse { pub data: Vec, } +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct SkillsRemoteReadParams {} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct RemoteSkillSummary { + pub id: String, + pub name: String, + pub description: String, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct SkillsRemoteReadResponse { + pub data: Vec, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct SkillsRemoteWriteParams { + pub hazelnut_id: String, + pub is_preload: bool, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(rename_all = "camelCase")] +#[ts(export_to = "v2/")] +pub struct SkillsRemoteWriteResponse { + pub id: String, + pub name: String, + pub path: PathBuf, +} + #[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, JsonSchema, TS)] #[serde(rename_all = "snake_case")] #[ts(rename_all = "snake_case")] @@ -2775,10 +2826,34 @@ pub struct DynamicToolCallParams { #[serde(rename_all = "camelCase")] #[ts(export_to = "v2/")] pub struct DynamicToolCallResponse { - pub output: String, + pub content_items: Vec, pub success: bool, } +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] +#[serde(tag = "type", rename_all = "camelCase")] +#[ts(tag = "type")] +#[ts(export_to = "v2/")] +pub enum DynamicToolCallOutputContentItem { + #[serde(rename_all = "camelCase")] + InputText { text: String }, + #[serde(rename_all = "camelCase")] + InputImage { image_url: String }, +} + +impl From + for codex_protocol::dynamic_tools::DynamicToolCallOutputContentItem +{ + fn from(item: DynamicToolCallOutputContentItem) -> Self { + match item { + DynamicToolCallOutputContentItem::InputText { text } => Self::InputText { text }, + DynamicToolCallOutputContentItem::InputImage { image_url } => { + Self::InputImage { image_url } + } + } + } +} + #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)] #[serde(rename_all = "camelCase")] #[ts(export_to = "v2/")] @@ -3138,4 +3213,61 @@ mod tests { }) ); } + + #[test] + fn dynamic_tool_response_serializes_content_items() { + let value = serde_json::to_value(DynamicToolCallResponse { + content_items: vec![DynamicToolCallOutputContentItem::InputText { + text: "dynamic-ok".to_string(), + }], + success: true, + }) + .unwrap(); + + assert_eq!( + value, + json!({ + "contentItems": [ + { + "type": "inputText", + "text": "dynamic-ok" + } + ], + "success": true, + }) + ); + } + + #[test] + fn dynamic_tool_response_serializes_text_and_image_content_items() { + let value = serde_json::to_value(DynamicToolCallResponse { + content_items: vec![ + DynamicToolCallOutputContentItem::InputText { + text: "dynamic-ok".to_string(), + }, + DynamicToolCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + ], + success: true, + }) + .unwrap(); + + assert_eq!( + value, + json!({ + "contentItems": [ + { + "type": "inputText", + "text": "dynamic-ok" + }, + { + "type": "inputImage", + "imageUrl": "data:image/png;base64,AAA" + } + ], + "success": true, + }) + ); + } } diff --git a/codex-rs/app-server-test-client/BUILD.bazel b/codex-rs/app-server-test-client/BUILD.bazel index e3610747cd..3a1686a04e 100644 --- a/codex-rs/app-server-test-client/BUILD.bazel +++ b/codex-rs/app-server-test-client/BUILD.bazel @@ -1,6 +1,6 @@ load("//:defs.bzl", "codex_rust_crate") codex_rust_crate( - name = "codex-app-server-test-client", + name = "app-server-test-client", crate_name = "codex_app_server_test_client", ) diff --git a/codex-rs/app-server-test-client/src/lib.rs b/codex-rs/app-server-test-client/src/lib.rs new file mode 100644 index 0000000000..90f6adf572 --- /dev/null +++ b/codex-rs/app-server-test-client/src/lib.rs @@ -0,0 +1,1057 @@ +use std::collections::VecDeque; +use std::fs; +use std::io::BufRead; +use std::io::BufReader; +use std::io::Write; +use std::path::Path; +use std::path::PathBuf; +use std::process::Child; +use std::process::ChildStdin; +use std::process::ChildStdout; +use std::process::Command; +use std::process::Stdio; +use std::thread; +use std::time::Duration; + +use anyhow::Context; +use anyhow::Result; +use anyhow::bail; +use clap::ArgAction; +use clap::Parser; +use clap::Subcommand; +use codex_app_server_protocol::AddConversationListenerParams; +use codex_app_server_protocol::AddConversationSubscriptionResponse; +use codex_app_server_protocol::AskForApproval; +use codex_app_server_protocol::ClientInfo; +use codex_app_server_protocol::ClientRequest; +use codex_app_server_protocol::CommandExecutionApprovalDecision; +use codex_app_server_protocol::CommandExecutionRequestApprovalParams; +use codex_app_server_protocol::CommandExecutionRequestApprovalResponse; +use codex_app_server_protocol::DynamicToolSpec; +use codex_app_server_protocol::FileChangeApprovalDecision; +use codex_app_server_protocol::FileChangeRequestApprovalParams; +use codex_app_server_protocol::FileChangeRequestApprovalResponse; +use codex_app_server_protocol::GetAccountRateLimitsResponse; +use codex_app_server_protocol::InitializeCapabilities; +use codex_app_server_protocol::InitializeParams; +use codex_app_server_protocol::InitializeResponse; +use codex_app_server_protocol::InputItem; +use codex_app_server_protocol::JSONRPCMessage; +use codex_app_server_protocol::JSONRPCNotification; +use codex_app_server_protocol::JSONRPCRequest; +use codex_app_server_protocol::JSONRPCResponse; +use codex_app_server_protocol::LoginChatGptCompleteNotification; +use codex_app_server_protocol::LoginChatGptResponse; +use codex_app_server_protocol::ModelListParams; +use codex_app_server_protocol::ModelListResponse; +use codex_app_server_protocol::NewConversationParams; +use codex_app_server_protocol::NewConversationResponse; +use codex_app_server_protocol::RequestId; +use codex_app_server_protocol::SandboxPolicy; +use codex_app_server_protocol::SendUserMessageParams; +use codex_app_server_protocol::SendUserMessageResponse; +use codex_app_server_protocol::ServerNotification; +use codex_app_server_protocol::ServerRequest; +use codex_app_server_protocol::ThreadStartParams; +use codex_app_server_protocol::ThreadStartResponse; +use codex_app_server_protocol::TurnStartParams; +use codex_app_server_protocol::TurnStartResponse; +use codex_app_server_protocol::TurnStatus; +use codex_app_server_protocol::UserInput as V2UserInput; +use codex_protocol::ThreadId; +use codex_protocol::protocol::Event; +use codex_protocol::protocol::EventMsg; +use serde::Serialize; +use serde::de::DeserializeOwned; +use serde_json::Value; +use uuid::Uuid; + +/// Minimal launcher that initializes the Codex app-server and logs the handshake. +#[derive(Parser)] +#[command(author = "Codex", version, about = "Bootstrap Codex app-server", long_about = None)] +struct Cli { + /// Path to the `codex` CLI binary. + #[arg(long, env = "CODEX_BIN", default_value = "codex")] + codex_bin: PathBuf, + + /// Forwarded to the `codex` CLI as `--config key=value`. Repeatable. + /// + /// Example: + /// `--config 'model_providers.mock.base_url="http://localhost:4010/v2"'` + #[arg( + short = 'c', + long = "config", + value_name = "key=value", + action = ArgAction::Append, + global = true + )] + config_overrides: Vec, + + /// JSON array of dynamic tool specs or a single tool object. + /// Prefix a filename with '@' to read from a file. + /// + /// Example: + /// --dynamic-tools '[{"name":"demo","description":"Demo","inputSchema":{"type":"object"}}]' + /// --dynamic-tools @/path/to/tools.json + #[arg(long, value_name = "json-or-@file", global = true)] + dynamic_tools: Option, + + #[command(subcommand)] + command: CliCommand, +} + +#[derive(Subcommand)] +enum CliCommand { + /// Send a user message through the Codex app-server. + SendMessage { + /// User message to send to Codex. + user_message: String, + }, + /// Send a user message through the app-server V2 thread/turn APIs. + SendMessageV2 { + /// User message to send to Codex. + user_message: String, + }, + /// Start a V2 turn that elicits an ExecCommand approval. + #[command(name = "trigger-cmd-approval")] + TriggerCmdApproval { + /// Optional prompt; defaults to a simple python command. + user_message: Option, + }, + /// Start a V2 turn that elicits an ApplyPatch approval. + #[command(name = "trigger-patch-approval")] + TriggerPatchApproval { + /// Optional prompt; defaults to creating a file via apply_patch. + user_message: Option, + }, + /// Start a V2 turn that should not elicit an ExecCommand approval. + #[command(name = "no-trigger-cmd-approval")] + NoTriggerCmdApproval, + /// Send two sequential V2 turns in the same thread to test follow-up behavior. + SendFollowUpV2 { + /// Initial user message for the first turn. + first_message: String, + /// Follow-up user message for the second turn. + follow_up_message: String, + }, + /// Trigger the ChatGPT login flow and wait for completion. + TestLogin, + /// Fetch the current account rate limits from the Codex app-server. + GetAccountRateLimits, + /// List the available models from the Codex app-server. + #[command(name = "model-list")] + ModelList, +} + +pub fn run() -> Result<()> { + let Cli { + codex_bin, + config_overrides, + dynamic_tools, + command, + } = Cli::parse(); + + let dynamic_tools = parse_dynamic_tools_arg(&dynamic_tools)?; + + match command { + CliCommand::SendMessage { user_message } => { + ensure_dynamic_tools_unused(&dynamic_tools, "send-message")?; + send_message(&codex_bin, &config_overrides, user_message) + } + CliCommand::SendMessageV2 { user_message } => { + send_message_v2(&codex_bin, &config_overrides, user_message, &dynamic_tools) + } + CliCommand::TriggerCmdApproval { user_message } => { + trigger_cmd_approval(&codex_bin, &config_overrides, user_message, &dynamic_tools) + } + CliCommand::TriggerPatchApproval { user_message } => { + trigger_patch_approval(&codex_bin, &config_overrides, user_message, &dynamic_tools) + } + CliCommand::NoTriggerCmdApproval => { + no_trigger_cmd_approval(&codex_bin, &config_overrides, &dynamic_tools) + } + CliCommand::SendFollowUpV2 { + first_message, + follow_up_message, + } => send_follow_up_v2( + &codex_bin, + &config_overrides, + first_message, + follow_up_message, + &dynamic_tools, + ), + CliCommand::TestLogin => { + ensure_dynamic_tools_unused(&dynamic_tools, "test-login")?; + test_login(&codex_bin, &config_overrides) + } + CliCommand::GetAccountRateLimits => { + ensure_dynamic_tools_unused(&dynamic_tools, "get-account-rate-limits")?; + get_account_rate_limits(&codex_bin, &config_overrides) + } + CliCommand::ModelList => { + ensure_dynamic_tools_unused(&dynamic_tools, "model-list")?; + model_list(&codex_bin, &config_overrides) + } + } +} + +fn send_message(codex_bin: &Path, config_overrides: &[String], user_message: String) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let conversation = client.start_thread()?; + println!("< newConversation response: {conversation:?}"); + + let subscription = client.add_conversation_listener(&conversation.conversation_id)?; + println!("< addConversationListener response: {subscription:?}"); + + let send_response = client.send_user_message(&conversation.conversation_id, &user_message)?; + println!("< sendUserMessage response: {send_response:?}"); + + client.stream_conversation(&conversation.conversation_id)?; + + client.remove_thread_listener(subscription.subscription_id)?; + + Ok(()) +} + +pub fn send_message_v2( + codex_bin: &Path, + config_overrides: &[String], + user_message: String, + dynamic_tools: &Option>, +) -> Result<()> { + send_message_v2_with_policies( + codex_bin, + config_overrides, + user_message, + None, + None, + dynamic_tools, + ) +} + +fn trigger_cmd_approval( + codex_bin: &Path, + config_overrides: &[String], + user_message: Option, + dynamic_tools: &Option>, +) -> Result<()> { + let default_prompt = + "Run `touch /tmp/should-trigger-approval` so I can confirm the file exists."; + let message = user_message.unwrap_or_else(|| default_prompt.to_string()); + send_message_v2_with_policies( + codex_bin, + config_overrides, + message, + Some(AskForApproval::OnRequest), + Some(SandboxPolicy::ReadOnly), + dynamic_tools, + ) +} + +fn trigger_patch_approval( + codex_bin: &Path, + config_overrides: &[String], + user_message: Option, + dynamic_tools: &Option>, +) -> Result<()> { + let default_prompt = + "Create a file named APPROVAL_DEMO.txt containing a short hello message using apply_patch."; + let message = user_message.unwrap_or_else(|| default_prompt.to_string()); + send_message_v2_with_policies( + codex_bin, + config_overrides, + message, + Some(AskForApproval::OnRequest), + Some(SandboxPolicy::ReadOnly), + dynamic_tools, + ) +} + +fn no_trigger_cmd_approval( + codex_bin: &Path, + config_overrides: &[String], + dynamic_tools: &Option>, +) -> Result<()> { + let prompt = "Run `touch should_not_trigger_approval.txt`"; + send_message_v2_with_policies( + codex_bin, + config_overrides, + prompt.to_string(), + None, + None, + dynamic_tools, + ) +} + +fn send_message_v2_with_policies( + codex_bin: &Path, + config_overrides: &[String], + user_message: String, + approval_policy: Option, + sandbox_policy: Option, + dynamic_tools: &Option>, +) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let thread_response = client.thread_start(ThreadStartParams { + dynamic_tools: dynamic_tools.clone(), + ..Default::default() + })?; + println!("< thread/start response: {thread_response:?}"); + let mut turn_params = TurnStartParams { + thread_id: thread_response.thread.id.clone(), + input: vec![V2UserInput::Text { + text: user_message, + // Test client sends plain text without UI element ranges. + text_elements: Vec::new(), + }], + ..Default::default() + }; + turn_params.approval_policy = approval_policy; + turn_params.sandbox_policy = sandbox_policy; + + let turn_response = client.turn_start(turn_params)?; + println!("< turn/start response: {turn_response:?}"); + + client.stream_turn(&thread_response.thread.id, &turn_response.turn.id)?; + + Ok(()) +} + +fn send_follow_up_v2( + codex_bin: &Path, + config_overrides: &[String], + first_message: String, + follow_up_message: String, + dynamic_tools: &Option>, +) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let thread_response = client.thread_start(ThreadStartParams { + dynamic_tools: dynamic_tools.clone(), + ..Default::default() + })?; + println!("< thread/start response: {thread_response:?}"); + + let first_turn_params = TurnStartParams { + thread_id: thread_response.thread.id.clone(), + input: vec![V2UserInput::Text { + text: first_message, + // Test client sends plain text without UI element ranges. + text_elements: Vec::new(), + }], + ..Default::default() + }; + let first_turn_response = client.turn_start(first_turn_params)?; + println!("< turn/start response (initial): {first_turn_response:?}"); + client.stream_turn(&thread_response.thread.id, &first_turn_response.turn.id)?; + + let follow_up_params = TurnStartParams { + thread_id: thread_response.thread.id.clone(), + input: vec![V2UserInput::Text { + text: follow_up_message, + // Test client sends plain text without UI element ranges. + text_elements: Vec::new(), + }], + ..Default::default() + }; + let follow_up_response = client.turn_start(follow_up_params)?; + println!("< turn/start response (follow-up): {follow_up_response:?}"); + client.stream_turn(&thread_response.thread.id, &follow_up_response.turn.id)?; + + Ok(()) +} + +fn test_login(codex_bin: &Path, config_overrides: &[String]) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let login_response = client.login_chat_gpt()?; + println!("< loginChatGpt response: {login_response:?}"); + println!( + "Open the following URL in your browser to continue:\n{}", + login_response.auth_url + ); + + let completion = client.wait_for_login_completion(&login_response.login_id)?; + println!("< loginChatGptComplete notification: {completion:?}"); + + if completion.success { + println!("Login succeeded."); + Ok(()) + } else { + bail!( + "login failed: {}", + completion + .error + .as_deref() + .unwrap_or("unknown error from loginChatGptComplete") + ); + } +} + +fn get_account_rate_limits(codex_bin: &Path, config_overrides: &[String]) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let response = client.get_account_rate_limits()?; + println!("< account/rateLimits/read response: {response:?}"); + + Ok(()) +} + +fn model_list(codex_bin: &Path, config_overrides: &[String]) -> Result<()> { + let mut client = CodexClient::spawn(codex_bin, config_overrides)?; + + let initialize = client.initialize()?; + println!("< initialize response: {initialize:?}"); + + let response = client.model_list(ModelListParams::default())?; + println!("< model/list response: {response:?}"); + + Ok(()) +} + +fn ensure_dynamic_tools_unused( + dynamic_tools: &Option>, + command: &str, +) -> Result<()> { + if dynamic_tools.is_some() { + bail!( + "dynamic tools are only supported for v2 thread/start; remove --dynamic-tools for {command} or use send-message-v2" + ); + } + Ok(()) +} + +fn parse_dynamic_tools_arg(dynamic_tools: &Option) -> Result>> { + let Some(raw_arg) = dynamic_tools.as_deref() else { + return Ok(None); + }; + + let raw_json = if let Some(path) = raw_arg.strip_prefix('@') { + fs::read_to_string(Path::new(path)) + .with_context(|| format!("read dynamic tools file {path}"))? + } else { + raw_arg.to_string() + }; + + let value: Value = serde_json::from_str(&raw_json).context("parse dynamic tools JSON")?; + let tools = match value { + Value::Array(_) => serde_json::from_value(value).context("decode dynamic tools array")?, + Value::Object(_) => vec![serde_json::from_value(value).context("decode dynamic tool")?], + _ => bail!("dynamic tools JSON must be an object or array"), + }; + + Ok(Some(tools)) +} + +struct CodexClient { + child: Child, + stdin: Option, + stdout: BufReader, + pending_notifications: VecDeque, +} + +impl CodexClient { + fn spawn(codex_bin: &Path, config_overrides: &[String]) -> Result { + let codex_bin_display = codex_bin.display(); + let mut cmd = Command::new(codex_bin); + for override_kv in config_overrides { + cmd.arg("--config").arg(override_kv); + } + let mut codex_app_server = cmd + .arg("app-server") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()) + .spawn() + .with_context(|| format!("failed to start `{codex_bin_display}` app-server"))?; + + let stdin = codex_app_server + .stdin + .take() + .context("codex app-server stdin unavailable")?; + let stdout = codex_app_server + .stdout + .take() + .context("codex app-server stdout unavailable")?; + + Ok(Self { + child: codex_app_server, + stdin: Some(stdin), + stdout: BufReader::new(stdout), + pending_notifications: VecDeque::new(), + }) + } + + fn initialize(&mut self) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::Initialize { + request_id: request_id.clone(), + params: InitializeParams { + client_info: ClientInfo { + name: "codex-toy-app-server".to_string(), + title: Some("Codex Toy App Server".to_string()), + version: env!("CARGO_PKG_VERSION").to_string(), + }, + capabilities: Some(InitializeCapabilities { + experimental_api: true, + }), + }, + }; + + self.send_request(request, request_id, "initialize") + } + + fn start_thread(&mut self) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::NewConversation { + request_id: request_id.clone(), + params: NewConversationParams::default(), + }; + + self.send_request(request, request_id, "newConversation") + } + + fn add_conversation_listener( + &mut self, + conversation_id: &ThreadId, + ) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::AddConversationListener { + request_id: request_id.clone(), + params: AddConversationListenerParams { + conversation_id: *conversation_id, + experimental_raw_events: false, + }, + }; + + self.send_request(request, request_id, "addConversationListener") + } + + fn remove_thread_listener(&mut self, subscription_id: Uuid) -> Result<()> { + let request_id = self.request_id(); + let request = ClientRequest::RemoveConversationListener { + request_id: request_id.clone(), + params: codex_app_server_protocol::RemoveConversationListenerParams { subscription_id }, + }; + + self.send_request::( + request, + request_id, + "removeConversationListener", + )?; + + Ok(()) + } + + fn send_user_message( + &mut self, + conversation_id: &ThreadId, + message: &str, + ) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::SendUserMessage { + request_id: request_id.clone(), + params: SendUserMessageParams { + conversation_id: *conversation_id, + items: vec![InputItem::Text { + text: message.to_string(), + // Test client sends plain text without UI element ranges. + text_elements: Vec::new(), + }], + }, + }; + + self.send_request(request, request_id, "sendUserMessage") + } + + fn thread_start(&mut self, params: ThreadStartParams) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::ThreadStart { + request_id: request_id.clone(), + params, + }; + + self.send_request(request, request_id, "thread/start") + } + + fn turn_start(&mut self, params: TurnStartParams) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::TurnStart { + request_id: request_id.clone(), + params, + }; + + self.send_request(request, request_id, "turn/start") + } + + fn login_chat_gpt(&mut self) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::LoginChatGpt { + request_id: request_id.clone(), + params: None, + }; + + self.send_request(request, request_id, "loginChatGpt") + } + + fn get_account_rate_limits(&mut self) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::GetAccountRateLimits { + request_id: request_id.clone(), + params: None, + }; + + self.send_request(request, request_id, "account/rateLimits/read") + } + + fn model_list(&mut self, params: ModelListParams) -> Result { + let request_id = self.request_id(); + let request = ClientRequest::ModelList { + request_id: request_id.clone(), + params, + }; + + self.send_request(request, request_id, "model/list") + } + + fn stream_conversation(&mut self, conversation_id: &ThreadId) -> Result<()> { + loop { + let notification = self.next_notification()?; + + if !notification.method.starts_with("codex/event/") { + continue; + } + + if let Some(event) = self.extract_event(notification, conversation_id)? { + match &event.msg { + EventMsg::AgentMessage(event) => { + println!("{}", event.message); + } + EventMsg::AgentMessageDelta(event) => { + print!("{}", event.delta); + std::io::stdout().flush().ok(); + } + EventMsg::TurnComplete(event) => { + println!("\n[task complete: {event:?}]"); + break; + } + EventMsg::TurnAborted(event) => { + println!("\n[turn aborted: {:?}]", event.reason); + break; + } + EventMsg::Error(event) => { + println!("[error] {event:?}"); + } + _ => { + println!("[UNKNOWN EVENT] {:?}", event.msg); + } + } + } + } + + Ok(()) + } + + fn wait_for_login_completion( + &mut self, + expected_login_id: &Uuid, + ) -> Result { + loop { + let notification = self.next_notification()?; + + if let Ok(server_notification) = ServerNotification::try_from(notification) { + match server_notification { + ServerNotification::LoginChatGptComplete(completion) => { + if &completion.login_id == expected_login_id { + return Ok(completion); + } + + println!( + "[ignoring loginChatGptComplete for unexpected login_id: {}]", + completion.login_id + ); + } + ServerNotification::AuthStatusChange(status) => { + println!("< authStatusChange notification: {status:?}"); + } + ServerNotification::AccountRateLimitsUpdated(snapshot) => { + println!("< accountRateLimitsUpdated notification: {snapshot:?}"); + } + ServerNotification::SessionConfigured(_) => { + // SessionConfigured notifications are unrelated to login; skip. + } + _ => {} + } + } + + // Not a server notification (likely a conversation event); keep waiting. + } + } + + fn stream_turn(&mut self, thread_id: &str, turn_id: &str) -> Result<()> { + loop { + let notification = self.next_notification()?; + + let Ok(server_notification) = ServerNotification::try_from(notification) else { + continue; + }; + + match server_notification { + ServerNotification::ThreadStarted(payload) => { + if payload.thread.id == thread_id { + println!("< thread/started notification: {:?}", payload.thread); + } + } + ServerNotification::TurnStarted(payload) => { + if payload.turn.id == turn_id { + println!("< turn/started notification: {:?}", payload.turn.status); + } + } + ServerNotification::AgentMessageDelta(delta) => { + print!("{}", delta.delta); + std::io::stdout().flush().ok(); + } + ServerNotification::CommandExecutionOutputDelta(delta) => { + print!("{}", delta.delta); + std::io::stdout().flush().ok(); + } + ServerNotification::TerminalInteraction(delta) => { + println!("[stdin sent: {}]", delta.stdin); + std::io::stdout().flush().ok(); + } + ServerNotification::ItemStarted(payload) => { + println!("\n< item started: {:?}", payload.item); + } + ServerNotification::ItemCompleted(payload) => { + println!("< item completed: {:?}", payload.item); + } + ServerNotification::TurnCompleted(payload) => { + if payload.turn.id == turn_id { + println!("\n< turn/completed notification: {:?}", payload.turn.status); + if payload.turn.status == TurnStatus::Failed + && let Some(error) = payload.turn.error + { + println!("[turn error] {}", error.message); + } + break; + } + } + ServerNotification::McpToolCallProgress(payload) => { + println!("< MCP tool progress: {}", payload.message); + } + _ => { + println!("[UNKNOWN SERVER NOTIFICATION] {server_notification:?}"); + } + } + } + + Ok(()) + } + + fn extract_event( + &self, + notification: JSONRPCNotification, + conversation_id: &ThreadId, + ) -> Result> { + let params = notification + .params + .context("event notification missing params")?; + + let mut map = match params { + Value::Object(map) => map, + other => bail!("unexpected params shape: {other:?}"), + }; + + let conversation_value = map + .remove("conversationId") + .context("event missing conversationId")?; + let notification_conversation: ThreadId = serde_json::from_value(conversation_value) + .context("conversationId was not a valid UUID")?; + + if ¬ification_conversation != conversation_id { + return Ok(None); + } + + let event_value = Value::Object(map); + let event: Event = + serde_json::from_value(event_value).context("failed to decode event payload")?; + Ok(Some(event)) + } + + fn send_request( + &mut self, + request: ClientRequest, + request_id: RequestId, + method: &str, + ) -> Result + where + T: DeserializeOwned, + { + self.write_request(&request)?; + self.wait_for_response(request_id, method) + } + + fn write_request(&mut self, request: &ClientRequest) -> Result<()> { + let request_json = serde_json::to_string(request)?; + let request_pretty = serde_json::to_string_pretty(request)?; + print_multiline_with_prefix("> ", &request_pretty); + + if let Some(stdin) = self.stdin.as_mut() { + writeln!(stdin, "{request_json}")?; + stdin + .flush() + .context("failed to flush request to codex app-server")?; + } else { + bail!("codex app-server stdin closed"); + } + + Ok(()) + } + + fn wait_for_response(&mut self, request_id: RequestId, method: &str) -> Result + where + T: DeserializeOwned, + { + loop { + let message = self.read_jsonrpc_message()?; + + match message { + JSONRPCMessage::Response(JSONRPCResponse { id, result }) => { + if id == request_id { + return serde_json::from_value(result) + .with_context(|| format!("{method} response missing payload")); + } + } + JSONRPCMessage::Error(err) => { + if err.id == request_id { + bail!("{method} failed: {err:?}"); + } + } + JSONRPCMessage::Notification(notification) => { + self.pending_notifications.push_back(notification); + } + JSONRPCMessage::Request(request) => { + self.handle_server_request(request)?; + } + } + } + } + + fn next_notification(&mut self) -> Result { + if let Some(notification) = self.pending_notifications.pop_front() { + return Ok(notification); + } + + loop { + let message = self.read_jsonrpc_message()?; + + match message { + JSONRPCMessage::Notification(notification) => return Ok(notification), + JSONRPCMessage::Response(_) | JSONRPCMessage::Error(_) => { + // No outstanding requests, so ignore stray responses/errors for now. + continue; + } + JSONRPCMessage::Request(request) => { + self.handle_server_request(request)?; + } + } + } + } + + fn read_jsonrpc_message(&mut self) -> Result { + loop { + let mut response_line = String::new(); + let bytes = self + .stdout + .read_line(&mut response_line) + .context("failed to read from codex app-server")?; + + if bytes == 0 { + bail!("codex app-server closed stdout"); + } + + let trimmed = response_line.trim(); + if trimmed.is_empty() { + continue; + } + + let parsed: Value = + serde_json::from_str(trimmed).context("response was not valid JSON-RPC")?; + let pretty = serde_json::to_string_pretty(&parsed)?; + print_multiline_with_prefix("< ", &pretty); + let message: JSONRPCMessage = serde_json::from_value(parsed) + .context("response was not a valid JSON-RPC message")?; + return Ok(message); + } + } + + fn request_id(&self) -> RequestId { + RequestId::String(Uuid::new_v4().to_string()) + } + + fn handle_server_request(&mut self, request: JSONRPCRequest) -> Result<()> { + let server_request = ServerRequest::try_from(request) + .context("failed to deserialize ServerRequest from JSONRPCRequest")?; + + match server_request { + ServerRequest::CommandExecutionRequestApproval { request_id, params } => { + self.handle_command_execution_request_approval(request_id, params)?; + } + ServerRequest::FileChangeRequestApproval { request_id, params } => { + self.approve_file_change_request(request_id, params)?; + } + other => { + bail!("received unsupported server request: {other:?}"); + } + } + + Ok(()) + } + + fn handle_command_execution_request_approval( + &mut self, + request_id: RequestId, + params: CommandExecutionRequestApprovalParams, + ) -> Result<()> { + let CommandExecutionRequestApprovalParams { + thread_id, + turn_id, + item_id, + reason, + command, + cwd, + command_actions, + proposed_execpolicy_amendment, + } = params; + + println!( + "\n< commandExecution approval requested for thread {thread_id}, turn {turn_id}, item {item_id}" + ); + if let Some(reason) = reason.as_deref() { + println!("< reason: {reason}"); + } + if let Some(command) = command.as_deref() { + println!("< command: {command}"); + } + if let Some(cwd) = cwd.as_ref() { + println!("< cwd: {}", cwd.display()); + } + if let Some(command_actions) = command_actions.as_ref() + && !command_actions.is_empty() + { + println!("< command actions: {command_actions:?}"); + } + if let Some(execpolicy_amendment) = proposed_execpolicy_amendment.as_ref() { + println!("< proposed execpolicy amendment: {execpolicy_amendment:?}"); + } + + let response = CommandExecutionRequestApprovalResponse { + decision: CommandExecutionApprovalDecision::Accept, + }; + self.send_server_request_response(request_id, &response)?; + println!("< approved commandExecution request for item {item_id}"); + Ok(()) + } + + fn approve_file_change_request( + &mut self, + request_id: RequestId, + params: FileChangeRequestApprovalParams, + ) -> Result<()> { + let FileChangeRequestApprovalParams { + thread_id, + turn_id, + item_id, + reason, + grant_root, + } = params; + + println!( + "\n< fileChange approval requested for thread {thread_id}, turn {turn_id}, item {item_id}" + ); + if let Some(reason) = reason.as_deref() { + println!("< reason: {reason}"); + } + if let Some(grant_root) = grant_root.as_deref() { + println!("< grant root: {}", grant_root.display()); + } + + let response = FileChangeRequestApprovalResponse { + decision: FileChangeApprovalDecision::Accept, + }; + self.send_server_request_response(request_id, &response)?; + println!("< approved fileChange request for item {item_id}"); + Ok(()) + } + + fn send_server_request_response(&mut self, request_id: RequestId, response: &T) -> Result<()> + where + T: Serialize, + { + let message = JSONRPCMessage::Response(JSONRPCResponse { + id: request_id, + result: serde_json::to_value(response)?, + }); + self.write_jsonrpc_message(message) + } + + fn write_jsonrpc_message(&mut self, message: JSONRPCMessage) -> Result<()> { + let payload = serde_json::to_string(&message)?; + let pretty = serde_json::to_string_pretty(&message)?; + print_multiline_with_prefix("> ", &pretty); + + if let Some(stdin) = self.stdin.as_mut() { + writeln!(stdin, "{payload}")?; + stdin + .flush() + .context("failed to flush response to codex app-server")?; + return Ok(()); + } + + bail!("codex app-server stdin closed") + } +} + +fn print_multiline_with_prefix(prefix: &str, payload: &str) { + for line in payload.lines() { + println!("{prefix}{line}"); + } +} + +impl Drop for CodexClient { + fn drop(&mut self) { + let _ = self.stdin.take(); + + if let Ok(Some(status)) = self.child.try_wait() { + println!("[codex app-server exited: {status}]"); + return; + } + + thread::sleep(Duration::from_millis(100)); + + if let Ok(Some(status)) = self.child.try_wait() { + println!("[codex app-server exited: {status}]"); + return; + } + + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} diff --git a/codex-rs/app-server-test-client/src/main.rs b/codex-rs/app-server-test-client/src/main.rs index 6d3fc06d81..a4da2e4026 100644 --- a/codex-rs/app-server-test-client/src/main.rs +++ b/codex-rs/app-server-test-client/src/main.rs @@ -1,1061 +1,5 @@ -use std::collections::VecDeque; -use std::fs; -use std::io::BufRead; -use std::io::BufReader; -use std::io::Write; -use std::path::Path; -use std::process::Child; -use std::process::ChildStdin; -use std::process::ChildStdout; -use std::process::Command; -use std::process::Stdio; -use std::thread; -use std::time::Duration; - -use anyhow::Context; use anyhow::Result; -use anyhow::bail; -use clap::ArgAction; -use clap::Parser; -use clap::Subcommand; -use codex_app_server_protocol::AddConversationListenerParams; -use codex_app_server_protocol::AddConversationSubscriptionResponse; -use codex_app_server_protocol::AskForApproval; -use codex_app_server_protocol::ClientInfo; -use codex_app_server_protocol::ClientRequest; -use codex_app_server_protocol::CommandExecutionApprovalDecision; -use codex_app_server_protocol::CommandExecutionRequestApprovalParams; -use codex_app_server_protocol::CommandExecutionRequestApprovalResponse; -use codex_app_server_protocol::DynamicToolSpec; -use codex_app_server_protocol::FileChangeApprovalDecision; -use codex_app_server_protocol::FileChangeRequestApprovalParams; -use codex_app_server_protocol::FileChangeRequestApprovalResponse; -use codex_app_server_protocol::GetAccountRateLimitsResponse; -use codex_app_server_protocol::InitializeCapabilities; -use codex_app_server_protocol::InitializeParams; -use codex_app_server_protocol::InitializeResponse; -use codex_app_server_protocol::InputItem; -use codex_app_server_protocol::JSONRPCMessage; -use codex_app_server_protocol::JSONRPCNotification; -use codex_app_server_protocol::JSONRPCRequest; -use codex_app_server_protocol::JSONRPCResponse; -use codex_app_server_protocol::LoginChatGptCompleteNotification; -use codex_app_server_protocol::LoginChatGptResponse; -use codex_app_server_protocol::ModelListParams; -use codex_app_server_protocol::ModelListResponse; -use codex_app_server_protocol::NewConversationParams; -use codex_app_server_protocol::NewConversationResponse; -use codex_app_server_protocol::RequestId; -use codex_app_server_protocol::SandboxPolicy; -use codex_app_server_protocol::SendUserMessageParams; -use codex_app_server_protocol::SendUserMessageResponse; -use codex_app_server_protocol::ServerNotification; -use codex_app_server_protocol::ServerRequest; -use codex_app_server_protocol::ThreadStartParams; -use codex_app_server_protocol::ThreadStartResponse; -use codex_app_server_protocol::TurnStartParams; -use codex_app_server_protocol::TurnStartResponse; -use codex_app_server_protocol::TurnStatus; -use codex_app_server_protocol::UserInput as V2UserInput; -use codex_protocol::ThreadId; -use codex_protocol::protocol::Event; -use codex_protocol::protocol::EventMsg; -use serde::Serialize; -use serde::de::DeserializeOwned; -use serde_json::Value; -use uuid::Uuid; - -/// Minimal launcher that initializes the Codex app-server and logs the handshake. -#[derive(Parser)] -#[command(author = "Codex", version, about = "Bootstrap Codex app-server", long_about = None)] -struct Cli { - /// Path to the `codex` CLI binary. - #[arg(long, env = "CODEX_BIN", default_value = "codex")] - codex_bin: String, - - /// Forwarded to the `codex` CLI as `--config key=value`. Repeatable. - /// - /// Example: - /// `--config 'model_providers.mock.base_url="http://localhost:4010/v2"'` - #[arg( - short = 'c', - long = "config", - value_name = "key=value", - action = ArgAction::Append, - global = true - )] - config_overrides: Vec, - - /// JSON array of dynamic tool specs or a single tool object. - /// Prefix a filename with '@' to read from a file. - /// - /// Example: - /// --dynamic-tools '[{"name":"demo","description":"Demo","inputSchema":{"type":"object"}}]' - /// --dynamic-tools @/path/to/tools.json - #[arg(long, value_name = "json-or-@file", global = true)] - dynamic_tools: Option, - - #[command(subcommand)] - command: CliCommand, -} - -#[derive(Subcommand)] -enum CliCommand { - /// Send a user message through the Codex app-server. - SendMessage { - /// User message to send to Codex. - #[arg()] - user_message: String, - }, - /// Send a user message through the app-server V2 thread/turn APIs. - SendMessageV2 { - /// User message to send to Codex. - #[arg()] - user_message: String, - }, - /// Start a V2 turn that elicits an ExecCommand approval. - #[command(name = "trigger-cmd-approval")] - TriggerCmdApproval { - /// Optional prompt; defaults to a simple python command. - #[arg()] - user_message: Option, - }, - /// Start a V2 turn that elicits an ApplyPatch approval. - #[command(name = "trigger-patch-approval")] - TriggerPatchApproval { - /// Optional prompt; defaults to creating a file via apply_patch. - #[arg()] - user_message: Option, - }, - /// Start a V2 turn that should not elicit an ExecCommand approval. - #[command(name = "no-trigger-cmd-approval")] - NoTriggerCmdApproval, - /// Send two sequential V2 turns in the same thread to test follow-up behavior. - SendFollowUpV2 { - /// Initial user message for the first turn. - #[arg()] - first_message: String, - /// Follow-up user message for the second turn. - #[arg()] - follow_up_message: String, - }, - /// Trigger the ChatGPT login flow and wait for completion. - TestLogin, - /// Fetch the current account rate limits from the Codex app-server. - GetAccountRateLimits, - /// List the available models from the Codex app-server. - #[command(name = "model-list")] - ModelList, -} fn main() -> Result<()> { - let Cli { - codex_bin, - config_overrides, - dynamic_tools, - command, - } = Cli::parse(); - - let dynamic_tools = parse_dynamic_tools_arg(&dynamic_tools)?; - - match command { - CliCommand::SendMessage { user_message } => { - ensure_dynamic_tools_unused(&dynamic_tools, "send-message")?; - send_message(&codex_bin, &config_overrides, user_message) - } - CliCommand::SendMessageV2 { user_message } => { - send_message_v2(&codex_bin, &config_overrides, user_message, &dynamic_tools) - } - CliCommand::TriggerCmdApproval { user_message } => { - trigger_cmd_approval(&codex_bin, &config_overrides, user_message, &dynamic_tools) - } - CliCommand::TriggerPatchApproval { user_message } => { - trigger_patch_approval(&codex_bin, &config_overrides, user_message, &dynamic_tools) - } - CliCommand::NoTriggerCmdApproval => { - no_trigger_cmd_approval(&codex_bin, &config_overrides, &dynamic_tools) - } - CliCommand::SendFollowUpV2 { - first_message, - follow_up_message, - } => send_follow_up_v2( - &codex_bin, - &config_overrides, - first_message, - follow_up_message, - &dynamic_tools, - ), - CliCommand::TestLogin => { - ensure_dynamic_tools_unused(&dynamic_tools, "test-login")?; - test_login(&codex_bin, &config_overrides) - } - CliCommand::GetAccountRateLimits => { - ensure_dynamic_tools_unused(&dynamic_tools, "get-account-rate-limits")?; - get_account_rate_limits(&codex_bin, &config_overrides) - } - CliCommand::ModelList => { - ensure_dynamic_tools_unused(&dynamic_tools, "model-list")?; - model_list(&codex_bin, &config_overrides) - } - } -} - -fn send_message(codex_bin: &str, config_overrides: &[String], user_message: String) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let conversation = client.start_thread()?; - println!("< newConversation response: {conversation:?}"); - - let subscription = client.add_conversation_listener(&conversation.conversation_id)?; - println!("< addConversationListener response: {subscription:?}"); - - let send_response = client.send_user_message(&conversation.conversation_id, &user_message)?; - println!("< sendUserMessage response: {send_response:?}"); - - client.stream_conversation(&conversation.conversation_id)?; - - client.remove_thread_listener(subscription.subscription_id)?; - - Ok(()) -} - -fn send_message_v2( - codex_bin: &str, - config_overrides: &[String], - user_message: String, - dynamic_tools: &Option>, -) -> Result<()> { - send_message_v2_with_policies( - codex_bin, - config_overrides, - user_message, - None, - None, - dynamic_tools, - ) -} - -fn trigger_cmd_approval( - codex_bin: &str, - config_overrides: &[String], - user_message: Option, - dynamic_tools: &Option>, -) -> Result<()> { - let default_prompt = - "Run `touch /tmp/should-trigger-approval` so I can confirm the file exists."; - let message = user_message.unwrap_or_else(|| default_prompt.to_string()); - send_message_v2_with_policies( - codex_bin, - config_overrides, - message, - Some(AskForApproval::OnRequest), - Some(SandboxPolicy::ReadOnly), - dynamic_tools, - ) -} - -fn trigger_patch_approval( - codex_bin: &str, - config_overrides: &[String], - user_message: Option, - dynamic_tools: &Option>, -) -> Result<()> { - let default_prompt = - "Create a file named APPROVAL_DEMO.txt containing a short hello message using apply_patch."; - let message = user_message.unwrap_or_else(|| default_prompt.to_string()); - send_message_v2_with_policies( - codex_bin, - config_overrides, - message, - Some(AskForApproval::OnRequest), - Some(SandboxPolicy::ReadOnly), - dynamic_tools, - ) -} - -fn no_trigger_cmd_approval( - codex_bin: &str, - config_overrides: &[String], - dynamic_tools: &Option>, -) -> Result<()> { - let prompt = "Run `touch should_not_trigger_approval.txt`"; - send_message_v2_with_policies( - codex_bin, - config_overrides, - prompt.to_string(), - None, - None, - dynamic_tools, - ) -} - -fn send_message_v2_with_policies( - codex_bin: &str, - config_overrides: &[String], - user_message: String, - approval_policy: Option, - sandbox_policy: Option, - dynamic_tools: &Option>, -) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let thread_response = client.thread_start(ThreadStartParams { - dynamic_tools: dynamic_tools.clone(), - ..Default::default() - })?; - println!("< thread/start response: {thread_response:?}"); - let mut turn_params = TurnStartParams { - thread_id: thread_response.thread.id.clone(), - input: vec![V2UserInput::Text { - text: user_message, - // Test client sends plain text without UI element ranges. - text_elements: Vec::new(), - }], - ..Default::default() - }; - turn_params.approval_policy = approval_policy; - turn_params.sandbox_policy = sandbox_policy; - - let turn_response = client.turn_start(turn_params)?; - println!("< turn/start response: {turn_response:?}"); - - client.stream_turn(&thread_response.thread.id, &turn_response.turn.id)?; - - Ok(()) -} - -fn send_follow_up_v2( - codex_bin: &str, - config_overrides: &[String], - first_message: String, - follow_up_message: String, - dynamic_tools: &Option>, -) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let thread_response = client.thread_start(ThreadStartParams { - dynamic_tools: dynamic_tools.clone(), - ..Default::default() - })?; - println!("< thread/start response: {thread_response:?}"); - - let first_turn_params = TurnStartParams { - thread_id: thread_response.thread.id.clone(), - input: vec![V2UserInput::Text { - text: first_message, - // Test client sends plain text without UI element ranges. - text_elements: Vec::new(), - }], - ..Default::default() - }; - let first_turn_response = client.turn_start(first_turn_params)?; - println!("< turn/start response (initial): {first_turn_response:?}"); - client.stream_turn(&thread_response.thread.id, &first_turn_response.turn.id)?; - - let follow_up_params = TurnStartParams { - thread_id: thread_response.thread.id.clone(), - input: vec![V2UserInput::Text { - text: follow_up_message, - // Test client sends plain text without UI element ranges. - text_elements: Vec::new(), - }], - ..Default::default() - }; - let follow_up_response = client.turn_start(follow_up_params)?; - println!("< turn/start response (follow-up): {follow_up_response:?}"); - client.stream_turn(&thread_response.thread.id, &follow_up_response.turn.id)?; - - Ok(()) -} - -fn test_login(codex_bin: &str, config_overrides: &[String]) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let login_response = client.login_chat_gpt()?; - println!("< loginChatGpt response: {login_response:?}"); - println!( - "Open the following URL in your browser to continue:\n{}", - login_response.auth_url - ); - - let completion = client.wait_for_login_completion(&login_response.login_id)?; - println!("< loginChatGptComplete notification: {completion:?}"); - - if completion.success { - println!("Login succeeded."); - Ok(()) - } else { - bail!( - "login failed: {}", - completion - .error - .as_deref() - .unwrap_or("unknown error from loginChatGptComplete") - ); - } -} - -fn get_account_rate_limits(codex_bin: &str, config_overrides: &[String]) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let response = client.get_account_rate_limits()?; - println!("< account/rateLimits/read response: {response:?}"); - - Ok(()) -} - -fn model_list(codex_bin: &str, config_overrides: &[String]) -> Result<()> { - let mut client = CodexClient::spawn(codex_bin, config_overrides)?; - - let initialize = client.initialize()?; - println!("< initialize response: {initialize:?}"); - - let response = client.model_list(ModelListParams::default())?; - println!("< model/list response: {response:?}"); - - Ok(()) -} - -fn ensure_dynamic_tools_unused( - dynamic_tools: &Option>, - command: &str, -) -> Result<()> { - if dynamic_tools.is_some() { - bail!( - "dynamic tools are only supported for v2 thread/start; remove --dynamic-tools for {command} or use send-message-v2" - ); - } - Ok(()) -} - -fn parse_dynamic_tools_arg(dynamic_tools: &Option) -> Result>> { - let Some(raw_arg) = dynamic_tools.as_deref() else { - return Ok(None); - }; - - let raw_json = if let Some(path) = raw_arg.strip_prefix('@') { - fs::read_to_string(Path::new(path)) - .with_context(|| format!("read dynamic tools file {path}"))? - } else { - raw_arg.to_string() - }; - - let value: Value = serde_json::from_str(&raw_json).context("parse dynamic tools JSON")?; - let tools = match value { - Value::Array(_) => serde_json::from_value(value).context("decode dynamic tools array")?, - Value::Object(_) => vec![serde_json::from_value(value).context("decode dynamic tool")?], - _ => bail!("dynamic tools JSON must be an object or array"), - }; - - Ok(Some(tools)) -} - -struct CodexClient { - child: Child, - stdin: Option, - stdout: BufReader, - pending_notifications: VecDeque, -} - -impl CodexClient { - fn spawn(codex_bin: &str, config_overrides: &[String]) -> Result { - let mut cmd = Command::new(codex_bin); - for override_kv in config_overrides { - cmd.arg("--config").arg(override_kv); - } - let mut codex_app_server = cmd - .arg("app-server") - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::inherit()) - .spawn() - .with_context(|| format!("failed to start `{codex_bin}` app-server"))?; - - let stdin = codex_app_server - .stdin - .take() - .context("codex app-server stdin unavailable")?; - let stdout = codex_app_server - .stdout - .take() - .context("codex app-server stdout unavailable")?; - - Ok(Self { - child: codex_app_server, - stdin: Some(stdin), - stdout: BufReader::new(stdout), - pending_notifications: VecDeque::new(), - }) - } - - fn initialize(&mut self) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::Initialize { - request_id: request_id.clone(), - params: InitializeParams { - client_info: ClientInfo { - name: "codex-toy-app-server".to_string(), - title: Some("Codex Toy App Server".to_string()), - version: env!("CARGO_PKG_VERSION").to_string(), - }, - capabilities: Some(InitializeCapabilities { - experimental_api: true, - }), - }, - }; - - self.send_request(request, request_id, "initialize") - } - - fn start_thread(&mut self) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::NewConversation { - request_id: request_id.clone(), - params: NewConversationParams::default(), - }; - - self.send_request(request, request_id, "newConversation") - } - - fn add_conversation_listener( - &mut self, - conversation_id: &ThreadId, - ) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::AddConversationListener { - request_id: request_id.clone(), - params: AddConversationListenerParams { - conversation_id: *conversation_id, - experimental_raw_events: false, - }, - }; - - self.send_request(request, request_id, "addConversationListener") - } - - fn remove_thread_listener(&mut self, subscription_id: Uuid) -> Result<()> { - let request_id = self.request_id(); - let request = ClientRequest::RemoveConversationListener { - request_id: request_id.clone(), - params: codex_app_server_protocol::RemoveConversationListenerParams { subscription_id }, - }; - - self.send_request::( - request, - request_id, - "removeConversationListener", - )?; - - Ok(()) - } - - fn send_user_message( - &mut self, - conversation_id: &ThreadId, - message: &str, - ) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::SendUserMessage { - request_id: request_id.clone(), - params: SendUserMessageParams { - conversation_id: *conversation_id, - items: vec![InputItem::Text { - text: message.to_string(), - // Test client sends plain text without UI element ranges. - text_elements: Vec::new(), - }], - }, - }; - - self.send_request(request, request_id, "sendUserMessage") - } - - fn thread_start(&mut self, params: ThreadStartParams) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::ThreadStart { - request_id: request_id.clone(), - params, - }; - - self.send_request(request, request_id, "thread/start") - } - - fn turn_start(&mut self, params: TurnStartParams) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::TurnStart { - request_id: request_id.clone(), - params, - }; - - self.send_request(request, request_id, "turn/start") - } - - fn login_chat_gpt(&mut self) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::LoginChatGpt { - request_id: request_id.clone(), - params: None, - }; - - self.send_request(request, request_id, "loginChatGpt") - } - - fn get_account_rate_limits(&mut self) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::GetAccountRateLimits { - request_id: request_id.clone(), - params: None, - }; - - self.send_request(request, request_id, "account/rateLimits/read") - } - - fn model_list(&mut self, params: ModelListParams) -> Result { - let request_id = self.request_id(); - let request = ClientRequest::ModelList { - request_id: request_id.clone(), - params, - }; - - self.send_request(request, request_id, "model/list") - } - - fn stream_conversation(&mut self, conversation_id: &ThreadId) -> Result<()> { - loop { - let notification = self.next_notification()?; - - if !notification.method.starts_with("codex/event/") { - continue; - } - - if let Some(event) = self.extract_event(notification, conversation_id)? { - match &event.msg { - EventMsg::AgentMessage(event) => { - println!("{}", event.message); - } - EventMsg::AgentMessageDelta(event) => { - print!("{}", event.delta); - std::io::stdout().flush().ok(); - } - EventMsg::TurnComplete(event) => { - println!("\n[task complete: {event:?}]"); - break; - } - EventMsg::TurnAborted(event) => { - println!("\n[turn aborted: {:?}]", event.reason); - break; - } - EventMsg::Error(event) => { - println!("[error] {event:?}"); - } - _ => { - println!("[UNKNOWN EVENT] {:?}", event.msg); - } - } - } - } - - Ok(()) - } - - fn wait_for_login_completion( - &mut self, - expected_login_id: &Uuid, - ) -> Result { - loop { - let notification = self.next_notification()?; - - if let Ok(server_notification) = ServerNotification::try_from(notification) { - match server_notification { - ServerNotification::LoginChatGptComplete(completion) => { - if &completion.login_id == expected_login_id { - return Ok(completion); - } - - println!( - "[ignoring loginChatGptComplete for unexpected login_id: {}]", - completion.login_id - ); - } - ServerNotification::AuthStatusChange(status) => { - println!("< authStatusChange notification: {status:?}"); - } - ServerNotification::AccountRateLimitsUpdated(snapshot) => { - println!("< accountRateLimitsUpdated notification: {snapshot:?}"); - } - ServerNotification::SessionConfigured(_) => { - // SessionConfigured notifications are unrelated to login; skip. - } - _ => {} - } - } - - // Not a server notification (likely a conversation event); keep waiting. - } - } - - fn stream_turn(&mut self, thread_id: &str, turn_id: &str) -> Result<()> { - loop { - let notification = self.next_notification()?; - - let Ok(server_notification) = ServerNotification::try_from(notification) else { - continue; - }; - - match server_notification { - ServerNotification::ThreadStarted(payload) => { - if payload.thread.id == thread_id { - println!("< thread/started notification: {:?}", payload.thread); - } - } - ServerNotification::TurnStarted(payload) => { - if payload.turn.id == turn_id { - println!("< turn/started notification: {:?}", payload.turn.status); - } - } - ServerNotification::AgentMessageDelta(delta) => { - print!("{}", delta.delta); - std::io::stdout().flush().ok(); - } - ServerNotification::CommandExecutionOutputDelta(delta) => { - print!("{}", delta.delta); - std::io::stdout().flush().ok(); - } - ServerNotification::TerminalInteraction(delta) => { - println!("[stdin sent: {}]", delta.stdin); - std::io::stdout().flush().ok(); - } - ServerNotification::ItemStarted(payload) => { - println!("\n< item started: {:?}", payload.item); - } - ServerNotification::ItemCompleted(payload) => { - println!("< item completed: {:?}", payload.item); - } - ServerNotification::TurnCompleted(payload) => { - if payload.turn.id == turn_id { - println!("\n< turn/completed notification: {:?}", payload.turn.status); - if payload.turn.status == TurnStatus::Failed - && let Some(error) = payload.turn.error - { - println!("[turn error] {}", error.message); - } - break; - } - } - ServerNotification::McpToolCallProgress(payload) => { - println!("< MCP tool progress: {}", payload.message); - } - _ => { - println!("[UNKNOWN SERVER NOTIFICATION] {server_notification:?}"); - } - } - } - - Ok(()) - } - - fn extract_event( - &self, - notification: JSONRPCNotification, - conversation_id: &ThreadId, - ) -> Result> { - let params = notification - .params - .context("event notification missing params")?; - - let mut map = match params { - Value::Object(map) => map, - other => bail!("unexpected params shape: {other:?}"), - }; - - let conversation_value = map - .remove("conversationId") - .context("event missing conversationId")?; - let notification_conversation: ThreadId = serde_json::from_value(conversation_value) - .context("conversationId was not a valid UUID")?; - - if ¬ification_conversation != conversation_id { - return Ok(None); - } - - let event_value = Value::Object(map); - let event: Event = - serde_json::from_value(event_value).context("failed to decode event payload")?; - Ok(Some(event)) - } - - fn send_request( - &mut self, - request: ClientRequest, - request_id: RequestId, - method: &str, - ) -> Result - where - T: DeserializeOwned, - { - self.write_request(&request)?; - self.wait_for_response(request_id, method) - } - - fn write_request(&mut self, request: &ClientRequest) -> Result<()> { - let request_json = serde_json::to_string(request)?; - let request_pretty = serde_json::to_string_pretty(request)?; - print_multiline_with_prefix("> ", &request_pretty); - - if let Some(stdin) = self.stdin.as_mut() { - writeln!(stdin, "{request_json}")?; - stdin - .flush() - .context("failed to flush request to codex app-server")?; - } else { - bail!("codex app-server stdin closed"); - } - - Ok(()) - } - - fn wait_for_response(&mut self, request_id: RequestId, method: &str) -> Result - where - T: DeserializeOwned, - { - loop { - let message = self.read_jsonrpc_message()?; - - match message { - JSONRPCMessage::Response(JSONRPCResponse { id, result }) => { - if id == request_id { - return serde_json::from_value(result) - .with_context(|| format!("{method} response missing payload")); - } - } - JSONRPCMessage::Error(err) => { - if err.id == request_id { - bail!("{method} failed: {err:?}"); - } - } - JSONRPCMessage::Notification(notification) => { - self.pending_notifications.push_back(notification); - } - JSONRPCMessage::Request(request) => { - self.handle_server_request(request)?; - } - } - } - } - - fn next_notification(&mut self) -> Result { - if let Some(notification) = self.pending_notifications.pop_front() { - return Ok(notification); - } - - loop { - let message = self.read_jsonrpc_message()?; - - match message { - JSONRPCMessage::Notification(notification) => return Ok(notification), - JSONRPCMessage::Response(_) | JSONRPCMessage::Error(_) => { - // No outstanding requests, so ignore stray responses/errors for now. - continue; - } - JSONRPCMessage::Request(request) => { - self.handle_server_request(request)?; - } - } - } - } - - fn read_jsonrpc_message(&mut self) -> Result { - loop { - let mut response_line = String::new(); - let bytes = self - .stdout - .read_line(&mut response_line) - .context("failed to read from codex app-server")?; - - if bytes == 0 { - bail!("codex app-server closed stdout"); - } - - let trimmed = response_line.trim(); - if trimmed.is_empty() { - continue; - } - - let parsed: Value = - serde_json::from_str(trimmed).context("response was not valid JSON-RPC")?; - let pretty = serde_json::to_string_pretty(&parsed)?; - print_multiline_with_prefix("< ", &pretty); - let message: JSONRPCMessage = serde_json::from_value(parsed) - .context("response was not a valid JSON-RPC message")?; - return Ok(message); - } - } - - fn request_id(&self) -> RequestId { - RequestId::String(Uuid::new_v4().to_string()) - } - - fn handle_server_request(&mut self, request: JSONRPCRequest) -> Result<()> { - let server_request = ServerRequest::try_from(request) - .context("failed to deserialize ServerRequest from JSONRPCRequest")?; - - match server_request { - ServerRequest::CommandExecutionRequestApproval { request_id, params } => { - self.handle_command_execution_request_approval(request_id, params)?; - } - ServerRequest::FileChangeRequestApproval { request_id, params } => { - self.approve_file_change_request(request_id, params)?; - } - other => { - bail!("received unsupported server request: {other:?}"); - } - } - - Ok(()) - } - - fn handle_command_execution_request_approval( - &mut self, - request_id: RequestId, - params: CommandExecutionRequestApprovalParams, - ) -> Result<()> { - let CommandExecutionRequestApprovalParams { - thread_id, - turn_id, - item_id, - reason, - command, - cwd, - command_actions, - proposed_execpolicy_amendment, - } = params; - - println!( - "\n< commandExecution approval requested for thread {thread_id}, turn {turn_id}, item {item_id}" - ); - if let Some(reason) = reason.as_deref() { - println!("< reason: {reason}"); - } - if let Some(command) = command.as_deref() { - println!("< command: {command}"); - } - if let Some(cwd) = cwd.as_ref() { - println!("< cwd: {}", cwd.display()); - } - if let Some(command_actions) = command_actions.as_ref() - && !command_actions.is_empty() - { - println!("< command actions: {command_actions:?}"); - } - if let Some(execpolicy_amendment) = proposed_execpolicy_amendment.as_ref() { - println!("< proposed execpolicy amendment: {execpolicy_amendment:?}"); - } - - let response = CommandExecutionRequestApprovalResponse { - decision: CommandExecutionApprovalDecision::Accept, - }; - self.send_server_request_response(request_id, &response)?; - println!("< approved commandExecution request for item {item_id}"); - Ok(()) - } - - fn approve_file_change_request( - &mut self, - request_id: RequestId, - params: FileChangeRequestApprovalParams, - ) -> Result<()> { - let FileChangeRequestApprovalParams { - thread_id, - turn_id, - item_id, - reason, - grant_root, - } = params; - - println!( - "\n< fileChange approval requested for thread {thread_id}, turn {turn_id}, item {item_id}" - ); - if let Some(reason) = reason.as_deref() { - println!("< reason: {reason}"); - } - if let Some(grant_root) = grant_root.as_deref() { - println!("< grant root: {}", grant_root.display()); - } - - let response = FileChangeRequestApprovalResponse { - decision: FileChangeApprovalDecision::Accept, - }; - self.send_server_request_response(request_id, &response)?; - println!("< approved fileChange request for item {item_id}"); - Ok(()) - } - - fn send_server_request_response(&mut self, request_id: RequestId, response: &T) -> Result<()> - where - T: Serialize, - { - let message = JSONRPCMessage::Response(JSONRPCResponse { - id: request_id, - result: serde_json::to_value(response)?, - }); - self.write_jsonrpc_message(message) - } - - fn write_jsonrpc_message(&mut self, message: JSONRPCMessage) -> Result<()> { - let payload = serde_json::to_string(&message)?; - let pretty = serde_json::to_string_pretty(&message)?; - print_multiline_with_prefix("> ", &pretty); - - if let Some(stdin) = self.stdin.as_mut() { - writeln!(stdin, "{payload}")?; - stdin - .flush() - .context("failed to flush response to codex app-server")?; - return Ok(()); - } - - bail!("codex app-server stdin closed") - } -} - -fn print_multiline_with_prefix(prefix: &str, payload: &str) { - for line in payload.lines() { - println!("{prefix}{line}"); - } -} - -impl Drop for CodexClient { - fn drop(&mut self) { - let _ = self.stdin.take(); - - if let Ok(Some(status)) = self.child.try_wait() { - println!("[codex app-server exited: {status}]"); - return; - } - - thread::sleep(Duration::from_millis(100)); - - if let Ok(Some(status)) = self.child.try_wait() { - println!("[codex app-server exited: {status}]"); - return; - } - - let _ = self.child.kill(); - let _ = self.child.wait(); - } + codex_app_server_test_client::run() } diff --git a/codex-rs/app-server/README.md b/codex-rs/app-server/README.md index 1d5b6d15fd..4df8399e88 100644 --- a/codex-rs/app-server/README.md +++ b/codex-rs/app-server/README.md @@ -15,7 +15,7 @@ - [Skills](#skills) - [Apps](#apps) - [Auth endpoints](#auth-endpoints) -- [Adding an experimental field](#adding-an-experimental-field) +- [Experimental API Opt-in](#experimental-api-opt-in) ## Protocol @@ -85,14 +85,17 @@ Example (from OpenAI's official VSCode extension): - `thread/archive` β€” move a thread’s rollout file into the archived directory; returns `{}` on success. - `thread/name/set` β€” set or update a thread’s user-facing name; returns `{}` on success. Thread names are not required to be unique; name lookups resolve to the most recently updated thread. - `thread/unarchive` β€” move an archived rollout file back into the sessions directory; returns the restored `thread` on success. +- `thread/compact/start` β€” trigger conversation history compaction for a thread; returns `{}` immediately while progress streams through standard turn/item notifications. - `thread/rollback` β€” drop the last N turns from the agent’s in-memory context and persist a rollback marker in the rollout so future resumes see the pruned history; returns the updated `thread` (with `turns` populated) on success. - `turn/start` β€” add user input to a thread and begin Codex generation; responds with the initial `turn` object and streams `turn/started`, `item/*`, and `turn/completed` notifications. - `turn/interrupt` β€” request cancellation of an in-flight turn by `(thread_id, turn_id)`; success is an empty `{}` response and the turn finishes with `status: "interrupted"`. - `review/start` β€” kick off Codex’s automated reviewer for a thread; responds like `turn/start` and emits `item/started`/`item/completed` notifications with `enteredReviewMode` and `exitedReviewMode` items, plus a final assistant `agentMessage` containing the review. - `command/exec` β€” run a single command under the server sandbox without starting a thread/turn (handy for utilities and validation). -- `model/list` β€” list available models (with reasoning effort options). +- `model/list` β€” list available models (with reasoning effort options and optional `upgrade` model ids). - `collaborationMode/list` β€” list available collaboration mode presets (experimental, no pagination). - `skills/list` β€” list skills for one or more `cwd` values (optional `forceReload`). +- `skills/remote/read` β€” list public remote skills (**under development; do not call from production clients yet**). +- `skills/remote/write` β€” download a public remote skill by `hazelnutId`; `isPreload=true` writes to `.codex/vendor_imports/skills` under `codex_home` (**under development; do not call from production clients yet**). - `app/list` β€” list available apps. - `skills/config/write` β€” write user-level skill config by path. - `mcpServer/oauth/login` β€” start an OAuth login for a configured MCP server; returns an `authorization_url` and later emits `mcpServer/oauthLogin/completed` once the browser flow finishes. @@ -119,6 +122,7 @@ Start a fresh thread when you need a new Codex conversation. "approvalPolicy": "never", "sandbox": "workspaceWrite", "personality": "friendly", + // Experimental: requires opt-in "dynamicTools": [ { "name": "lookup_ticket", @@ -144,6 +148,8 @@ Start a fresh thread when you need a new Codex conversation. { "method": "thread/started", "params": { "thread": { … } } } ``` +Valid `personality` values are `"friendly"`, `"pragmatic"`, and `"none"`. When `"none"` is selected, the personality placeholder is replaced with an empty string. + To continue a stored session, call `thread/resume` with the `thread.id` you previously recorded. The response shape matches `thread/start`, and no additional notifications are emitted. You can also pass the same configuration overrides supported by `thread/start`, such as `personality`: ```json @@ -237,6 +243,22 @@ Use `thread/unarchive` to move an archived rollout back into the sessions direct { "id": 24, "result": { "thread": { "id": "thr_b" } } } ``` +### Example: Trigger thread compaction + +Use `thread/compact/start` to trigger manual history compaction for a thread. The request returns immediately with `{}`. + +Progress is emitted as standard `turn/*` and `item/*` notifications on the same `threadId`. Clients should expect a single compaction item: + +- `item/started` with `item: { "type": "contextCompaction", ... }` +- `item/completed` with the same `contextCompaction` item id + +While compaction is running, the thread is effectively in a turn so clients should surface progress UI based on the notifications. + +```json +{ "method": "thread/compact/start", "id": 25, "params": { "threadId": "thr_b" } } +{ "id": 25, "result": {} } +``` + ### Example: Start a turn (send user input) Turns attach user input (text or images) to a thread and trigger Codex generation. The `input` field is a list of discriminated unions: @@ -535,6 +557,41 @@ Order of messages: UI guidance for IDEs: surface an approval dialog as soon as the request arrives. The turn will proceed after the server receives a response to the approval request. The terminal `item/completed` notification will be sent with the appropriate status. +### Dynamic tool calls (experimental) + +`dynamicTools` on `thread/start` and the corresponding `item/tool/call` request/response flow are experimental APIs. To enable them, set `initialize.params.capabilities.experimentalApi = true`. + +When a dynamic tool is invoked during a turn, the server sends an `item/tool/call` JSON-RPC request to the client: + +```json +{ + "method": "item/tool/call", + "id": 60, + "params": { + "threadId": "thr_123", + "turnId": "turn_123", + "callId": "call_123", + "tool": "lookup_ticket", + "arguments": { "id": "ABC-123" } + } +} +``` + +The client must respond with content items. Use `inputText` for text and `inputImage` for image URLs/data URLs: + +```json +{ + "id": 60, + "result": { + "contentItems": [ + { "type": "inputText", "text": "Ticket ABC-123 is open." }, + { "type": "inputImage", "imageUrl": "data:image/png;base64,AAA" } + ], + "success": true + } +} +``` + ## Skills Invoke a skill by including `$` in the text input. Add a `skill` input item (recommended) so the backend injects full skill instructions instead of relying on the model to resolve the name. @@ -770,7 +827,67 @@ Field notes: - `windowDurationMins` is the quota window length. - `resetsAt` is a Unix timestamp (seconds) for the next reset. -## Adding an experimental field +## Experimental API Opt-in + +Some app-server methods and fields are intentionally gated behind an experimental capability with no backwards-compatible guarantees. This lets clients choose between: + +- Stable surface only (default): no opt-in, no experimental methods/fields exposed. +- Experimental surface: opt in during `initialize`. + +### Generating stable vs experimental client schemas + +`codex app-server` schema generation defaults to the stable API surface (experimental fields and methods filtered out). Pass `--experimental` to include experimental methods/fields in generated TypeScript or JSON schema: + +```bash +# Stable-only output (default) +codex app-server generate-ts --out DIR +codex app-server generate-json-schema --out DIR + +# Include experimental API surface +codex app-server generate-ts --out DIR --experimental +codex app-server generate-json-schema --out DIR --experimental +``` + +### How clients opt in at runtime + +Set `capabilities.experimentalApi` to `true` in your single `initialize` request: + +```json +{ + "method": "initialize", + "id": 1, + "params": { + "clientInfo": { + "name": "my_client", + "title": "My Client", + "version": "0.1.0" + }, + "capabilities": { + "experimentalApi": true + } + } +} +``` + +Then send the standard `initialized` notification and proceed normally. + +Notes: + +- If `capabilities` is omitted, `experimentalApi` is treated as `false`. +- This setting is negotiated once at initialization time for the process lifetime (re-initializing is rejected with `"Already initialized"`). + +### What happens without opt-in + +If a request uses an experimental method or sets an experimental field without opting in, app-server rejects it with a JSON-RPC error. The message is: + +` requires experimentalApi capability` + +Examples of descriptor strings: + +- `mock/experimentalMethod` (method-level gate) +- `thread/start.mockExperimentalField` (field-level gate) + +### For maintainers: Adding experimental fields and methods Use this checklist when introducing a field/method that should only be available when the client opts into experimental APIs. At runtime, clients must send `initialize` with `capabilities.experimentalApi = true` to use experimental methods or fields. diff --git a/codex-rs/app-server/src/bespoke_event_handling.rs b/codex-rs/app-server/src/bespoke_event_handling.rs index 55b185e5a8..fd8db9cb62 100644 --- a/codex-rs/app-server/src/bespoke_event_handling.rs +++ b/codex-rs/app-server/src/bespoke_event_handling.rs @@ -88,6 +88,7 @@ use codex_core::protocol::TurnDiffEvent; use codex_core::review_format::format_review_findings_block; use codex_core::review_prompts; use codex_protocol::ThreadId; +use codex_protocol::dynamic_tools::DynamicToolCallOutputContentItem as CoreDynamicToolCallOutputContentItem; use codex_protocol::dynamic_tools::DynamicToolResponse as CoreDynamicToolResponse; use codex_protocol::plan_tool::UpdatePlanArgs; use codex_protocol::protocol::ReviewOutputEvent; @@ -351,8 +352,9 @@ pub(crate) async fn apply_bespoke_event_handling( .submit(Op::DynamicToolResponse { id: call_id.clone(), response: CoreDynamicToolResponse { - call_id, - output: "dynamic tool calls require api v2".to_string(), + content_items: vec![CoreDynamicToolCallOutputContentItem::InputText { + text: "dynamic tool calls require api v2".to_string(), + }], success: false, }, }) diff --git a/codex-rs/app-server/src/codex_message_processor.rs b/codex-rs/app-server/src/codex_message_processor.rs index 2a1c534056..ef7e175b10 100644 --- a/codex-rs/app-server/src/codex_message_processor.rs +++ b/codex-rs/app-server/src/codex_message_processor.rs @@ -97,9 +97,15 @@ use codex_app_server_protocol::SkillsConfigWriteParams; use codex_app_server_protocol::SkillsConfigWriteResponse; use codex_app_server_protocol::SkillsListParams; use codex_app_server_protocol::SkillsListResponse; +use codex_app_server_protocol::SkillsRemoteReadParams; +use codex_app_server_protocol::SkillsRemoteReadResponse; +use codex_app_server_protocol::SkillsRemoteWriteParams; +use codex_app_server_protocol::SkillsRemoteWriteResponse; use codex_app_server_protocol::Thread; use codex_app_server_protocol::ThreadArchiveParams; use codex_app_server_protocol::ThreadArchiveResponse; +use codex_app_server_protocol::ThreadCompactStartParams; +use codex_app_server_protocol::ThreadCompactStartResponse; use codex_app_server_protocol::ThreadForkParams; use codex_app_server_protocol::ThreadForkResponse; use codex_app_server_protocol::ThreadItem; @@ -176,6 +182,8 @@ use codex_core::read_head_for_summary; use codex_core::read_session_meta_line; use codex_core::rollout_date_parts; use codex_core::sandboxing::SandboxPermissions; +use codex_core::skills::remote::download_remote_skill; +use codex_core::skills::remote::list_remote_skills; use codex_core::state_db::get_state_db; use codex_core::token_data::parse_id_token; use codex_core::windows_sandbox::WindowsSandboxLevelExt; @@ -449,6 +457,9 @@ impl CodexMessageProcessor { ClientRequest::ThreadUnarchive { request_id, params } => { self.thread_unarchive(request_id, params).await; } + ClientRequest::ThreadCompactStart { request_id, params } => { + self.thread_compact_start(request_id, params).await; + } ClientRequest::ThreadRollback { request_id, params } => { self.thread_rollback(request_id, params).await; } @@ -464,6 +475,12 @@ impl CodexMessageProcessor { ClientRequest::SkillsList { request_id, params } => { self.skills_list(request_id, params).await; } + ClientRequest::SkillsRemoteRead { request_id, params } => { + self.skills_remote_read(request_id, params).await; + } + ClientRequest::SkillsRemoteWrite { request_id, params } => { + self.skills_remote_write(request_id, params).await; + } ClientRequest::AppsList { request_id, params } => { self.apps_list(request_id, params).await; } @@ -1467,6 +1484,7 @@ impl CodexMessageProcessor { let outgoing = self.outgoing.clone(); let req_id = request_id; let sandbox_cwd = self.config.cwd.clone(); + let use_linux_sandbox_bwrap = self.config.features.enabled(Feature::UseLinuxSandboxBwrap); tokio::spawn(async move { match codex_core::exec::process_exec_tool_call( @@ -1474,6 +1492,7 @@ impl CodexMessageProcessor { &effective_policy, sandbox_cwd.as_path(), &codex_linux_sandbox_exe, + use_linux_sandbox_bwrap, None, ) .await @@ -2082,6 +2101,30 @@ impl CodexMessageProcessor { } } + async fn thread_compact_start(&self, request_id: RequestId, params: ThreadCompactStartParams) { + let ThreadCompactStartParams { thread_id } = params; + + let (_, thread) = match self.load_thread(&thread_id).await { + Ok(v) => v, + Err(error) => { + self.outgoing.send_error(request_id, error).await; + return; + } + }; + + match thread.submit(Op::Compact).await { + Ok(_) => { + self.outgoing + .send_response(request_id, ThreadCompactStartResponse {}) + .await; + } + Err(err) => { + self.send_internal_error(request_id, format!("failed to start compaction: {err}")) + .await; + } + } + } + async fn thread_list(&self, request_id: RequestId, params: ThreadListParams) { let ThreadListParams { cursor, @@ -4053,6 +4096,61 @@ impl CodexMessageProcessor { .await; } + async fn skills_remote_read(&self, request_id: RequestId, _params: SkillsRemoteReadParams) { + match list_remote_skills(&self.config).await { + Ok(skills) => { + let data = skills + .into_iter() + .map(|skill| codex_app_server_protocol::RemoteSkillSummary { + id: skill.id, + name: skill.name, + description: skill.description, + }) + .collect(); + self.outgoing + .send_response(request_id, SkillsRemoteReadResponse { data }) + .await; + } + Err(err) => { + self.send_internal_error( + request_id, + format!("failed to read remote skills: {err}"), + ) + .await; + } + } + } + + async fn skills_remote_write(&self, request_id: RequestId, params: SkillsRemoteWriteParams) { + let SkillsRemoteWriteParams { + hazelnut_id, + is_preload, + } = params; + let response = download_remote_skill(&self.config, hazelnut_id.as_str(), is_preload).await; + + match response { + Ok(downloaded) => { + self.outgoing + .send_response( + request_id, + SkillsRemoteWriteResponse { + id: downloaded.id, + name: downloaded.name, + path: downloaded.path, + }, + ) + .await; + } + Err(err) => { + self.send_internal_error( + request_id, + format!("failed to download remote skill: {err}"), + ) + .await; + } + } + } + async fn skills_config_write(&self, request_id: RequestId, params: SkillsConfigWriteParams) { let SkillsConfigWriteParams { path, enabled } = params; let edits = vec![ConfigEdit::SetSkillConfig { path, enabled }]; diff --git a/codex-rs/app-server/src/dynamic_tools.rs b/codex-rs/app-server/src/dynamic_tools.rs index a1b424d0ee..ed284452b4 100644 --- a/codex-rs/app-server/src/dynamic_tools.rs +++ b/codex-rs/app-server/src/dynamic_tools.rs @@ -1,5 +1,6 @@ use codex_app_server_protocol::DynamicToolCallResponse; use codex_core::CodexThread; +use codex_protocol::dynamic_tools::DynamicToolCallOutputContentItem as CoreDynamicToolCallOutputContentItem; use codex_protocol::dynamic_tools::DynamicToolResponse as CoreDynamicToolResponse; use codex_protocol::protocol::Op; use std::sync::Arc; @@ -17,8 +18,9 @@ pub(crate) async fn on_call_response( Err(err) => { error!("request failed: {err:?}"); let fallback = CoreDynamicToolResponse { - call_id: call_id.clone(), - output: "dynamic tool request failed".to_string(), + content_items: vec![CoreDynamicToolCallOutputContentItem::InputText { + text: "dynamic tool request failed".to_string(), + }], success: false, }; if let Err(err) = conversation @@ -37,14 +39,25 @@ pub(crate) async fn on_call_response( let response = serde_json::from_value::(value).unwrap_or_else(|err| { error!("failed to deserialize DynamicToolCallResponse: {err}"); DynamicToolCallResponse { - output: "dynamic tool response was invalid".to_string(), + content_items: vec![ + codex_app_server_protocol::DynamicToolCallOutputContentItem::InputText { + text: "dynamic tool response was invalid".to_string(), + }, + ], success: false, } }); + + let DynamicToolCallResponse { + content_items, + success, + } = response; let response = CoreDynamicToolResponse { - call_id: call_id.clone(), - output: response.output, - success: response.success, + content_items: content_items + .into_iter() + .map(CoreDynamicToolCallOutputContentItem::from) + .collect(), + success, }; if let Err(err) = conversation .submit(Op::DynamicToolResponse { diff --git a/codex-rs/app-server/src/models.rs b/codex-rs/app-server/src/models.rs index 133dc73cc0..350b86f92a 100644 --- a/codex-rs/app-server/src/models.rs +++ b/codex-rs/app-server/src/models.rs @@ -22,6 +22,7 @@ fn model_from_preset(preset: ModelPreset) -> Model { Model { id: preset.id.to_string(), model: preset.model.to_string(), + upgrade: preset.upgrade.map(|upgrade| upgrade.id), display_name: preset.display_name.to_string(), description: preset.description.to_string(), supported_reasoning_efforts: reasoning_efforts_from_preset( diff --git a/codex-rs/app-server/tests/common/mcp_process.rs b/codex-rs/app-server/tests/common/mcp_process.rs index 4804a8cd37..ba6dc058e6 100644 --- a/codex-rs/app-server/tests/common/mcp_process.rs +++ b/codex-rs/app-server/tests/common/mcp_process.rs @@ -50,6 +50,7 @@ use codex_app_server_protocol::SendUserTurnParams; use codex_app_server_protocol::ServerRequest; use codex_app_server_protocol::SetDefaultModelParams; use codex_app_server_protocol::ThreadArchiveParams; +use codex_app_server_protocol::ThreadCompactStartParams; use codex_app_server_protocol::ThreadForkParams; use codex_app_server_protocol::ThreadListParams; use codex_app_server_protocol::ThreadLoadedListParams; @@ -418,6 +419,15 @@ impl McpProcess { self.send_request("thread/unarchive", params).await } + /// Send a `thread/compact/start` JSON-RPC request. + pub async fn send_thread_compact_start_request( + &mut self, + params: ThreadCompactStartParams, + ) -> anyhow::Result { + let params = Some(serde_json::to_value(params)?); + self.send_request("thread/compact/start", params).await + } + /// Send a `thread/rollback` JSON-RPC request. pub async fn send_thread_rollback_request( &mut self, diff --git a/codex-rs/app-server/tests/suite/codex_message_processor_flow.rs b/codex-rs/app-server/tests/suite/codex_message_processor_flow.rs index d905c3c1b5..a041b8b0ba 100644 --- a/codex-rs/app-server/tests/suite/codex_message_processor_flow.rs +++ b/codex-rs/app-server/tests/suite/codex_message_processor_flow.rs @@ -36,7 +36,7 @@ use std::path::Path; use tempfile::TempDir; use tokio::time::timeout; -const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); +const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20); #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn test_codex_jsonrpc_conversation_flow() -> Result<()> { @@ -76,6 +76,7 @@ async fn test_codex_jsonrpc_conversation_flow() -> Result<()> { let new_conv_id = mcp .send_new_conversation_request(NewConversationParams { cwd: Some(working_directory.to_string_lossy().into_owned()), + sandbox: Some(SandboxMode::DangerFullAccess), ..Default::default() }) .await?; diff --git a/codex-rs/app-server/tests/suite/interrupt.rs b/codex-rs/app-server/tests/suite/interrupt.rs index f8dc2a7e8e..2270afce28 100644 --- a/codex-rs/app-server/tests/suite/interrupt.rs +++ b/codex-rs/app-server/tests/suite/interrupt.rs @@ -147,7 +147,7 @@ fn create_config_toml(codex_home: &Path, server_uri: String) -> std::io::Result< r#" model = "mock-model" approval_policy = "never" -sandbox_mode = "read-only" +sandbox_mode = "danger-full-access" model_provider = "mock_provider" diff --git a/codex-rs/app-server/tests/suite/v2/compaction.rs b/codex-rs/app-server/tests/suite/v2/compaction.rs index eeedcb286b..4730d920c8 100644 --- a/codex-rs/app-server/tests/suite/v2/compaction.rs +++ b/codex-rs/app-server/tests/suite/v2/compaction.rs @@ -15,9 +15,12 @@ use app_test_support::write_chatgpt_auth; use app_test_support::write_mock_responses_config_toml; use codex_app_server_protocol::ItemCompletedNotification; use codex_app_server_protocol::ItemStartedNotification; +use codex_app_server_protocol::JSONRPCError; use codex_app_server_protocol::JSONRPCNotification; use codex_app_server_protocol::JSONRPCResponse; use codex_app_server_protocol::RequestId; +use codex_app_server_protocol::ThreadCompactStartParams; +use codex_app_server_protocol::ThreadCompactStartResponse; use codex_app_server_protocol::ThreadItem; use codex_app_server_protocol::ThreadStartParams; use codex_app_server_protocol::ThreadStartResponse; @@ -39,6 +42,7 @@ use tokio::time::timeout; const DEFAULT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); const AUTO_COMPACT_LIMIT: i64 = 1_000; const COMPACT_PROMPT: &str = "Summarize the conversation."; +const INVALID_REQUEST_ERROR_CODE: i64 = -32600; #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn auto_compaction_local_emits_started_and_completed_items() -> Result<()> { @@ -196,6 +200,134 @@ async fn auto_compaction_remote_emits_started_and_completed_items() -> Result<() Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn thread_compact_start_triggers_compaction_and_returns_empty_response() -> Result<()> { + skip_if_no_network!(Ok(())); + + let server = responses::start_mock_server().await; + let sse = responses::sse(vec![ + responses::ev_assistant_message("m1", "MANUAL_COMPACT_SUMMARY"), + responses::ev_completed_with_tokens("r1", 200), + ]); + responses::mount_sse_sequence(&server, vec![sse]).await; + + let codex_home = TempDir::new()?; + write_mock_responses_config_toml( + codex_home.path(), + &server.uri(), + &BTreeMap::default(), + AUTO_COMPACT_LIMIT, + None, + "mock_provider", + COMPACT_PROMPT, + )?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let thread_id = start_thread(&mut mcp).await?; + let compact_id = mcp + .send_thread_compact_start_request(ThreadCompactStartParams { + thread_id: thread_id.clone(), + }) + .await?; + let compact_resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(compact_id)), + ) + .await??; + let _compact: ThreadCompactStartResponse = + to_response::(compact_resp)?; + + let started = wait_for_context_compaction_started(&mut mcp).await?; + let completed = wait_for_context_compaction_completed(&mut mcp).await?; + + let ThreadItem::ContextCompaction { id: started_id } = started.item else { + unreachable!("started item should be context compaction"); + }; + let ThreadItem::ContextCompaction { id: completed_id } = completed.item else { + unreachable!("completed item should be context compaction"); + }; + + assert_eq!(started.thread_id, thread_id); + assert_eq!(completed.thread_id, thread_id); + assert_eq!(started_id, completed_id); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn thread_compact_start_rejects_invalid_thread_id() -> Result<()> { + skip_if_no_network!(Ok(())); + + let server = responses::start_mock_server().await; + let codex_home = TempDir::new()?; + write_mock_responses_config_toml( + codex_home.path(), + &server.uri(), + &BTreeMap::default(), + AUTO_COMPACT_LIMIT, + None, + "mock_provider", + COMPACT_PROMPT, + )?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let request_id = mcp + .send_thread_compact_start_request(ThreadCompactStartParams { + thread_id: "not-a-thread-id".to_string(), + }) + .await?; + let error: JSONRPCError = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_error_message(RequestId::Integer(request_id)), + ) + .await??; + + assert_eq!(error.error.code, INVALID_REQUEST_ERROR_CODE); + assert!(error.error.message.contains("invalid thread id")); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn thread_compact_start_rejects_unknown_thread_id() -> Result<()> { + skip_if_no_network!(Ok(())); + + let server = responses::start_mock_server().await; + let codex_home = TempDir::new()?; + write_mock_responses_config_toml( + codex_home.path(), + &server.uri(), + &BTreeMap::default(), + AUTO_COMPACT_LIMIT, + None, + "mock_provider", + COMPACT_PROMPT, + )?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let request_id = mcp + .send_thread_compact_start_request(ThreadCompactStartParams { + thread_id: "67e55044-10b1-426f-9247-bb680e5fe0c8".to_string(), + }) + .await?; + let error: JSONRPCError = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_error_message(RequestId::Integer(request_id)), + ) + .await??; + + assert_eq!(error.error.code, INVALID_REQUEST_ERROR_CODE); + assert!(error.error.message.contains("thread not found")); + + Ok(()) +} + async fn start_thread(mcp: &mut McpProcess) -> Result { let thread_id = mcp .send_thread_start_request(ThreadStartParams { diff --git a/codex-rs/app-server/tests/suite/v2/dynamic_tools.rs b/codex-rs/app-server/tests/suite/v2/dynamic_tools.rs index dc985ac49f..f4cd85a61d 100644 --- a/codex-rs/app-server/tests/suite/v2/dynamic_tools.rs +++ b/codex-rs/app-server/tests/suite/v2/dynamic_tools.rs @@ -4,6 +4,7 @@ use app_test_support::McpProcess; use app_test_support::create_final_assistant_message_sse_response; use app_test_support::create_mock_responses_server_sequence_unchecked; use app_test_support::to_response; +use codex_app_server_protocol::DynamicToolCallOutputContentItem; use codex_app_server_protocol::DynamicToolCallParams; use codex_app_server_protocol::DynamicToolCallResponse; use codex_app_server_protocol::DynamicToolSpec; @@ -15,6 +16,9 @@ use codex_app_server_protocol::ThreadStartResponse; use codex_app_server_protocol::TurnStartParams; use codex_app_server_protocol::TurnStartResponse; use codex_app_server_protocol::UserInput as V2UserInput; +use codex_protocol::models::FunctionCallOutputBody; +use codex_protocol::models::FunctionCallOutputContentItem; +use codex_protocol::models::FunctionCallOutputPayload; use core_test_support::responses; use pretty_assertions::assert_eq; use serde_json::Value; @@ -111,7 +115,7 @@ async fn thread_start_injects_dynamic_tools_into_model_requests() -> Result<()> /// Exercises the full dynamic tool call path (server request, client response, model output). #[tokio::test] -async fn dynamic_tool_call_round_trip_sends_output_to_model() -> Result<()> { +async fn dynamic_tool_call_round_trip_sends_text_content_items_to_model() -> Result<()> { let call_id = "dyn-call-1"; let tool_name = "demo_tool"; let tool_args = json!({ "city": "Paris" }); @@ -200,7 +204,9 @@ async fn dynamic_tool_call_round_trip_sends_output_to_model() -> Result<()> { // Respond to the tool call so the model receives a function_call_output. let response = DynamicToolCallResponse { - output: "dynamic-ok".to_string(), + content_items: vec![DynamicToolCallOutputContentItem::InputText { + text: "dynamic-ok".to_string(), + }], success: true, }; mcp.send_response(request_id, serde_json::to_value(response)?) @@ -213,11 +219,171 @@ async fn dynamic_tool_call_round_trip_sends_output_to_model() -> Result<()> { .await??; let bodies = responses_bodies(&server).await?; - let output = bodies + let payload = bodies .iter() - .find_map(|body| function_call_output_text(body, call_id)) + .find_map(|body| function_call_output_payload(body, call_id)) .context("expected function_call_output in follow-up request")?; - assert_eq!(output, "dynamic-ok"); + let expected_payload = FunctionCallOutputPayload::from_content_items(vec![ + FunctionCallOutputContentItem::InputText { + text: "dynamic-ok".to_string(), + }, + ]); + assert_eq!(payload, expected_payload); + + Ok(()) +} + +/// Ensures dynamic tool call responses can include structured content items. +#[tokio::test] +async fn dynamic_tool_call_round_trip_sends_content_items_to_model() -> Result<()> { + let call_id = "dyn-call-items-1"; + let tool_name = "demo_tool"; + let tool_args = json!({ "city": "Paris" }); + let tool_call_arguments = serde_json::to_string(&tool_args)?; + + let responses = vec![ + responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_function_call(call_id, tool_name, &tool_call_arguments), + responses::ev_completed("resp-1"), + ]), + create_final_assistant_message_sse_response("Done")?, + ]; + let server = create_mock_responses_server_sequence_unchecked(responses).await; + + let codex_home = TempDir::new()?; + create_config_toml(codex_home.path(), &server.uri())?; + + let mut mcp = McpProcess::new(codex_home.path()).await?; + timeout(DEFAULT_READ_TIMEOUT, mcp.initialize()).await??; + + let dynamic_tool = DynamicToolSpec { + name: tool_name.to_string(), + description: "Demo dynamic tool".to_string(), + input_schema: json!({ + "type": "object", + "properties": { + "city": { "type": "string" } + }, + "required": ["city"], + "additionalProperties": false, + }), + }; + + let thread_req = mcp + .send_thread_start_request(ThreadStartParams { + dynamic_tools: Some(vec![dynamic_tool]), + ..Default::default() + }) + .await?; + let thread_resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(thread_req)), + ) + .await??; + let ThreadStartResponse { thread, .. } = to_response::(thread_resp)?; + + let turn_req = mcp + .send_turn_start_request(TurnStartParams { + thread_id: thread.id.clone(), + input: vec![V2UserInput::Text { + text: "Run the tool".to_string(), + text_elements: Vec::new(), + }], + ..Default::default() + }) + .await?; + let turn_resp: JSONRPCResponse = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_response_message(RequestId::Integer(turn_req)), + ) + .await??; + let TurnStartResponse { turn } = to_response::(turn_resp)?; + + let request = timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_request_message(), + ) + .await??; + let (request_id, params) = match request { + ServerRequest::DynamicToolCall { request_id, params } => (request_id, params), + other => panic!("expected DynamicToolCall request, got {other:?}"), + }; + + let expected = DynamicToolCallParams { + thread_id: thread.id, + turn_id: turn.id, + call_id: call_id.to_string(), + tool: tool_name.to_string(), + arguments: tool_args, + }; + assert_eq!(params, expected); + + let response_content_items = vec![ + DynamicToolCallOutputContentItem::InputText { + text: "dynamic-ok".to_string(), + }, + DynamicToolCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + ]; + let content_items = response_content_items + .clone() + .into_iter() + .map(|item| match item { + DynamicToolCallOutputContentItem::InputText { text } => { + FunctionCallOutputContentItem::InputText { text } + } + DynamicToolCallOutputContentItem::InputImage { image_url } => { + FunctionCallOutputContentItem::InputImage { image_url } + } + }) + .collect::>(); + let response = DynamicToolCallResponse { + content_items: response_content_items, + success: true, + }; + mcp.send_response(request_id, serde_json::to_value(response)?) + .await?; + + timeout( + DEFAULT_READ_TIMEOUT, + mcp.read_stream_until_notification_message("turn/completed"), + ) + .await??; + + let bodies = responses_bodies(&server).await?; + let output_value = bodies + .iter() + .find_map(|body| function_call_output_raw_output(body, call_id)) + .context("expected function_call_output output in follow-up request")?; + assert_eq!( + output_value, + json!([ + { + "type": "input_text", + "text": "dynamic-ok" + }, + { + "type": "input_image", + "image_url": "data:image/png;base64,AAA" + } + ]) + ); + + let payload = bodies + .iter() + .find_map(|body| function_call_output_payload(body, call_id)) + .context("expected function_call_output in follow-up request")?; + assert_eq!( + payload.body, + FunctionCallOutputBody::ContentItems(content_items.clone()) + ); + assert_eq!(payload.success, None); + assert_eq!( + serde_json::to_string(&payload)?, + serde_json::to_string(&content_items)? + ); Ok(()) } @@ -248,7 +414,12 @@ fn find_tool<'a>(body: &'a Value, name: &str) -> Option<&'a Value> { }) } -fn function_call_output_text(body: &Value, call_id: &str) -> Option { +fn function_call_output_payload(body: &Value, call_id: &str) -> Option { + function_call_output_raw_output(body, call_id) + .and_then(|output| serde_json::from_value(output).ok()) +} + +fn function_call_output_raw_output(body: &Value, call_id: &str) -> Option { body.get("input") .and_then(Value::as_array) .and_then(|items| { @@ -258,8 +429,7 @@ fn function_call_output_text(body: &Value, call_id: &str) -> Option { }) }) .and_then(|item| item.get("output")) - .and_then(Value::as_str) - .map(str::to_string) + .cloned() } fn create_config_toml(codex_home: &Path, server_uri: &str) -> std::io::Result<()> { diff --git a/codex-rs/app-server/tests/suite/v2/model_list.rs b/codex-rs/app-server/tests/suite/v2/model_list.rs index c3b4ec7088..c2431d24d0 100644 --- a/codex-rs/app-server/tests/suite/v2/model_list.rs +++ b/codex-rs/app-server/tests/suite/v2/model_list.rs @@ -51,6 +51,7 @@ async fn list_models_returns_all_models_with_large_limit() -> Result<()> { Model { id: "gpt-5.2-codex".to_string(), model: "gpt-5.2-codex".to_string(), + upgrade: None, display_name: "gpt-5.2-codex".to_string(), description: "Latest frontier agentic coding model.".to_string(), supported_reasoning_efforts: vec![ @@ -80,6 +81,7 @@ async fn list_models_returns_all_models_with_large_limit() -> Result<()> { Model { id: "gpt-5.1-codex-max".to_string(), model: "gpt-5.1-codex-max".to_string(), + upgrade: Some("gpt-5.2-codex".to_string()), display_name: "gpt-5.1-codex-max".to_string(), description: "Codex-optimized flagship for deep and fast reasoning.".to_string(), supported_reasoning_efforts: vec![ @@ -109,6 +111,7 @@ async fn list_models_returns_all_models_with_large_limit() -> Result<()> { Model { id: "gpt-5.1-codex-mini".to_string(), model: "gpt-5.1-codex-mini".to_string(), + upgrade: Some("gpt-5.2-codex".to_string()), display_name: "gpt-5.1-codex-mini".to_string(), description: "Optimized for codex. Cheaper, faster, but less capable.".to_string(), supported_reasoning_efforts: vec![ @@ -130,6 +133,7 @@ async fn list_models_returns_all_models_with_large_limit() -> Result<()> { Model { id: "gpt-5.2".to_string(), model: "gpt-5.2".to_string(), + upgrade: Some("gpt-5.2-codex".to_string()), display_name: "gpt-5.2".to_string(), description: "Latest frontier model with improvements across knowledge, reasoning and coding" diff --git a/codex-rs/app-server/tests/suite/v2/turn_interrupt.rs b/codex-rs/app-server/tests/suite/v2/turn_interrupt.rs index 9c804aa666..486e915f6f 100644 --- a/codex-rs/app-server/tests/suite/v2/turn_interrupt.rs +++ b/codex-rs/app-server/tests/suite/v2/turn_interrupt.rs @@ -129,7 +129,7 @@ fn create_config_toml(codex_home: &std::path::Path, server_uri: &str) -> std::io r#" model = "mock-model" approval_policy = "never" -sandbox_mode = "workspace-write" +sandbox_mode = "danger-full-access" model_provider = "mock_provider" diff --git a/codex-rs/app-server/tests/suite/v2/turn_start.rs b/codex-rs/app-server/tests/suite/v2/turn_start.rs index 67a147c39e..2fc68e0beb 100644 --- a/codex-rs/app-server/tests/suite/v2/turn_start.rs +++ b/codex-rs/app-server/tests/suite/v2/turn_start.rs @@ -1652,6 +1652,7 @@ async fn command_execution_notifications_include_process_id() -> Result<()> { text: "run a command".to_string(), text_elements: Vec::new(), }], + sandbox_policy: Some(codex_app_server_protocol::SandboxPolicy::DangerFullAccess), ..Default::default() }) .await?; diff --git a/codex-rs/arg0/src/lib.rs b/codex-rs/arg0/src/lib.rs index bf2f7afb7c..9c455ddbba 100644 --- a/codex-rs/arg0/src/lib.rs +++ b/codex-rs/arg0/src/lib.rs @@ -1,3 +1,4 @@ +use std::fs::File; use std::future::Future; use std::path::Path; use std::path::PathBuf; @@ -10,8 +11,24 @@ use tempfile::TempDir; const LINUX_SANDBOX_ARG0: &str = "codex-linux-sandbox"; const APPLY_PATCH_ARG0: &str = "apply_patch"; const MISSPELLED_APPLY_PATCH_ARG0: &str = "applypatch"; +const LOCK_FILENAME: &str = ".lock"; -pub fn arg0_dispatch() -> Option { +/// Keeps the per-session PATH entry alive and locked for the process lifetime. +pub struct Arg0PathEntryGuard { + _temp_dir: TempDir, + _lock_file: File, +} + +impl Arg0PathEntryGuard { + fn new(temp_dir: TempDir, lock_file: File) -> Self { + Self { + _temp_dir: temp_dir, + _lock_file: lock_file, + } + } +} + +pub fn arg0_dispatch() -> Option { // Determine if we were invoked via the special alias. let mut args = std::env::args_os(); let argv0 = args.next().unwrap_or_default(); @@ -149,7 +166,7 @@ where /// /// IMPORTANT: This function modifies the PATH environment variable, so it MUST /// be called before multiple threads are spawned. -pub fn prepend_path_entry_for_codex_aliases() -> std::io::Result { +pub fn prepend_path_entry_for_codex_aliases() -> std::io::Result { let codex_home = codex_core::config::find_codex_home()?; #[cfg(not(debug_assertions))] { @@ -167,7 +184,7 @@ pub fn prepend_path_entry_for_codex_aliases() -> std::io::Result { std::fs::create_dir_all(&codex_home)?; // Use a CODEX_HOME-scoped temp root to avoid cluttering the top-level directory. - let temp_root = codex_home.join("tmp").join("path"); + let temp_root = codex_home.join("tmp").join("arg0"); std::fs::create_dir_all(&temp_root)?; #[cfg(unix)] { @@ -177,11 +194,25 @@ pub fn prepend_path_entry_for_codex_aliases() -> std::io::Result { std::fs::set_permissions(&temp_root, std::fs::Permissions::from_mode(0o700))?; } + // Best-effort cleanup of stale per-session dirs. Ignore failures so startup proceeds. + if let Err(err) = janitor_cleanup(&temp_root) { + eprintln!("WARNING: failed to clean up stale arg0 temp dirs: {err}"); + } + let temp_dir = tempfile::Builder::new() .prefix("codex-arg0") .tempdir_in(&temp_root)?; let path = temp_dir.path(); + let lock_path = path.join(LOCK_FILENAME); + let lock_file = File::options() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&lock_path)?; + lock_file.try_lock()?; + for filename in &[ APPLY_PATCH_ARG0, MISSPELLED_APPLY_PATCH_ARG0, @@ -231,5 +262,107 @@ pub fn prepend_path_entry_for_codex_aliases() -> std::io::Result { std::env::set_var("PATH", updated_path_env_var); } - Ok(temp_dir) + Ok(Arg0PathEntryGuard::new(temp_dir, lock_file)) +} + +fn janitor_cleanup(temp_root: &Path) -> std::io::Result<()> { + let entries = match std::fs::read_dir(temp_root) { + Ok(entries) => entries, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(err) => return Err(err), + }; + + for entry in entries.flatten() { + let path = entry.path(); + if !path.is_dir() { + continue; + } + + // Skip the directory if locking fails or the lock is currently held. + let Some(_lock_file) = try_lock_dir(&path)? else { + continue; + }; + + match std::fs::remove_dir_all(&path) { + Ok(()) => {} + // Expected TOCTOU race: directory can disappear after read_dir/lock checks. + Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue, + Err(err) => return Err(err), + } + } + + Ok(()) +} + +fn try_lock_dir(dir: &Path) -> std::io::Result> { + let lock_path = dir.join(LOCK_FILENAME); + let lock_file = match File::options().read(true).write(true).open(&lock_path) { + Ok(file) => file, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(err) => return Err(err), + }; + + match lock_file.try_lock() { + Ok(()) => Ok(Some(lock_file)), + Err(std::fs::TryLockError::WouldBlock) => Ok(None), + Err(err) => Err(err.into()), + } +} + +#[cfg(test)] +mod tests { + use super::LOCK_FILENAME; + use super::janitor_cleanup; + use std::fs; + use std::fs::File; + use std::path::Path; + + fn create_lock(dir: &Path) -> std::io::Result { + let lock_path = dir.join(LOCK_FILENAME); + File::options() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(lock_path) + } + + #[test] + fn janitor_skips_dirs_without_lock_file() -> std::io::Result<()> { + let root = tempfile::tempdir()?; + let dir = root.path().join("no-lock"); + fs::create_dir(&dir)?; + + janitor_cleanup(root.path())?; + + assert!(dir.exists()); + Ok(()) + } + + #[test] + fn janitor_skips_dirs_with_held_lock() -> std::io::Result<()> { + let root = tempfile::tempdir()?; + let dir = root.path().join("locked"); + fs::create_dir(&dir)?; + let lock_file = create_lock(&dir)?; + lock_file.try_lock()?; + + janitor_cleanup(root.path())?; + + assert!(dir.exists()); + Ok(()) + } + + #[test] + fn janitor_removes_dirs_with_unlocked_lock() -> std::io::Result<()> { + let root = tempfile::tempdir()?; + let dir = root.path().join("stale"); + fs::create_dir(&dir)?; + create_lock(&dir)?; + + janitor_cleanup(root.path())?; + + assert!(!dir.exists()); + Ok(()) + } } diff --git a/codex-rs/cli/Cargo.toml b/codex-rs/cli/Cargo.toml index cfcfce88f9..77344df477 100644 --- a/codex-rs/cli/Cargo.toml +++ b/codex-rs/cli/Cargo.toml @@ -21,6 +21,7 @@ clap = { workspace = true, features = ["derive"] } clap_complete = { workspace = true } codex-app-server = { workspace = true } codex-app-server-protocol = { workspace = true } +codex-app-server-test-client = { workspace = true } codex-arg0 = { workspace = true } codex-chatgpt = { workspace = true } codex-cloud-tasks = { path = "../cloud-tasks" } diff --git a/codex-rs/cli/src/debug_sandbox.rs b/codex-rs/cli/src/debug_sandbox.rs index 5b165f9778..30352cbb83 100644 --- a/codex-rs/cli/src/debug_sandbox.rs +++ b/codex-rs/cli/src/debug_sandbox.rs @@ -227,16 +227,19 @@ async fn run_command_under_sandbox( .await? } SandboxType::Landlock => { + use codex_core::features::Feature; #[expect(clippy::expect_used)] let codex_linux_sandbox_exe = config .codex_linux_sandbox_exe .expect("codex-linux-sandbox executable not found"); + let use_bwrap_sandbox = config.features.enabled(Feature::UseLinuxSandboxBwrap); spawn_command_under_linux_sandbox( codex_linux_sandbox_exe, command, cwd, config.sandbox_policy.get(), sandbox_policy_cwd.as_path(), + use_bwrap_sandbox, stdio_policy, env, ) diff --git a/codex-rs/cli/src/main.rs b/codex-rs/cli/src/main.rs index 022d6bbdf5..defc063eb6 100644 --- a/codex-rs/cli/src/main.rs +++ b/codex-rs/cli/src/main.rs @@ -110,9 +110,11 @@ enum Subcommand { Completion(CompletionCommand), /// Run commands within a Codex-provided sandbox. - #[clap(visible_alias = "debug")] Sandbox(SandboxArgs), + /// Debugging tools. + Debug(DebugCommand), + /// Execpolicy tooling. #[clap(hide = true)] Execpolicy(ExecpolicyCommand), @@ -150,6 +152,36 @@ struct CompletionCommand { shell: Shell, } +#[derive(Debug, Parser)] +struct DebugCommand { + #[command(subcommand)] + subcommand: DebugSubcommand, +} + +#[derive(Debug, clap::Subcommand)] +enum DebugSubcommand { + /// Tooling: helps debug the app server. + AppServer(DebugAppServerCommand), +} + +#[derive(Debug, Parser)] +struct DebugAppServerCommand { + #[command(subcommand)] + subcommand: DebugAppServerSubcommand, +} + +#[derive(Debug, clap::Subcommand)] +enum DebugAppServerSubcommand { + // Send message to app server V2. + SendMessageV2(DebugAppServerSendMessageV2Command), +} + +#[derive(Debug, Parser)] +struct DebugAppServerSendMessageV2Command { + #[arg(value_name = "USER_MESSAGE", required = true)] + user_message: String, +} + #[derive(Debug, Parser)] struct ResumeCommand { /// Conversation/session id (UUID) or thread name. UUIDs take precedence if it parses. @@ -425,6 +457,15 @@ fn run_execpolicycheck(cmd: ExecPolicyCheckCommand) -> anyhow::Result<()> { cmd.run() } +fn run_debug_app_server_command(cmd: DebugAppServerCommand) -> anyhow::Result<()> { + match cmd.subcommand { + DebugAppServerSubcommand::SendMessageV2(cmd) => { + let codex_bin = std::env::current_exe()?; + codex_app_server_test_client::send_message_v2(&codex_bin, &[], cmd.user_message, &None) + } + } +} + #[derive(Debug, Default, Parser, Clone)] struct FeatureToggles { /// Enable a feature (repeatable). Equivalent to `-c features.=true`. @@ -693,6 +734,11 @@ async fn cli_main(codex_linux_sandbox_exe: Option) -> anyhow::Result<() .await?; } }, + Some(Subcommand::Debug(DebugCommand { subcommand })) => match subcommand { + DebugSubcommand::AppServer(cmd) => { + run_debug_app_server_command(cmd)?; + } + }, Some(Subcommand::Execpolicy(ExecpolicyCommand { sub })) => match sub { ExecpolicySubcommand::Check(cmd) => run_execpolicycheck(cmd)?, }, diff --git a/codex-rs/cloud-requirements/src/lib.rs b/codex-rs/cloud-requirements/src/lib.rs index 535fef3876..95f5180a83 100644 --- a/codex-rs/cloud-requirements/src/lib.rs +++ b/codex-rs/cloud-requirements/src/lib.rs @@ -20,8 +20,7 @@ use std::time::Duration; use std::time::Instant; use tokio::time::timeout; -/// This blocks codex startup, so must be short. -const CLOUD_REQUIREMENTS_TIMEOUT: Duration = Duration::from_secs(5); +const CLOUD_REQUIREMENTS_TIMEOUT: Duration = Duration::from_secs(15); #[async_trait] trait RequirementsFetcher: Send + Sync { diff --git a/codex-rs/codex-api/README.md b/codex-rs/codex-api/README.md index c1f7d230c0..0570cf7570 100644 --- a/codex-rs/codex-api/README.md +++ b/codex-rs/codex-api/README.md @@ -29,4 +29,13 @@ The public interface of this crate is intentionally small and uniform: - Output: `Vec`. - `CompactClient::compact_input(&CompactionInput, extra_headers)` wraps the JSON encoding and retry/telemetry wiring. +- **Memory trace summarize endpoint** + - Input: `MemoryTraceSummarizeInput` (re-exported as `codex_api::MemoryTraceSummarizeInput`): + - `model: String`. + - `traces: Vec`. + - `MemoryTrace` includes `id`, `metadata.source_path`, and normalized `items`. + - `reasoning: Option`. + - Output: `Vec`. + - `MemoriesClient::trace_summarize_input(&MemoryTraceSummarizeInput, extra_headers)` wraps JSON encoding and retry/telemetry wiring. + All HTTP details (URLs, headers, retry/backoff policies, SSE framing) are encapsulated in `codex-api` and `codex-client`. Callers construct prompts/inputs using protocol types and work with typed streams of `ResponseEvent` or compacted `ResponseItem` values. diff --git a/codex-rs/codex-api/src/common.rs b/codex-rs/codex-api/src/common.rs index a9127644f1..f1a996f12c 100644 --- a/codex-rs/codex-api/src/common.rs +++ b/codex-rs/codex-api/src/common.rs @@ -6,6 +6,7 @@ use codex_protocol::openai_models::ReasoningEffort as ReasoningEffortConfig; use codex_protocol::protocol::RateLimitSnapshot; use codex_protocol::protocol::TokenUsage; use futures::Stream; +use serde::Deserialize; use serde::Serialize; use serde_json::Value; use std::pin::Pin; @@ -37,6 +38,33 @@ pub struct CompactionInput<'a> { pub instructions: &'a str, } +/// Canonical input payload for the memory trace summarize endpoint. +#[derive(Debug, Clone, Serialize)] +pub struct MemoryTraceSummarizeInput { + pub model: String, + pub traces: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub reasoning: Option, +} + +#[derive(Debug, Clone, Serialize)] +pub struct MemoryTrace { + pub id: String, + pub metadata: MemoryTraceMetadata, + pub items: Vec, +} + +#[derive(Debug, Clone, Serialize)] +pub struct MemoryTraceMetadata { + pub source_path: String, +} + +#[derive(Debug, Clone, Deserialize, PartialEq, Eq)] +pub struct MemoryTraceSummaryOutput { + pub trace_summary: String, + pub memory_summary: String, +} + #[derive(Debug)] pub enum ResponseEvent { Created, diff --git a/codex-rs/codex-api/src/endpoint/memories.rs b/codex-rs/codex-api/src/endpoint/memories.rs new file mode 100644 index 0000000000..c8f35d7e16 --- /dev/null +++ b/codex-rs/codex-api/src/endpoint/memories.rs @@ -0,0 +1,108 @@ +use crate::auth::AuthProvider; +use crate::common::MemoryTraceSummarizeInput; +use crate::common::MemoryTraceSummaryOutput; +use crate::endpoint::session::EndpointSession; +use crate::error::ApiError; +use crate::provider::Provider; +use codex_client::HttpTransport; +use codex_client::RequestTelemetry; +use http::HeaderMap; +use http::Method; +use serde::Deserialize; +use serde_json::to_value; +use std::sync::Arc; + +pub struct MemoriesClient { + session: EndpointSession, +} + +impl MemoriesClient { + pub fn new(transport: T, provider: Provider, auth: A) -> Self { + Self { + session: EndpointSession::new(transport, provider, auth), + } + } + + pub fn with_telemetry(self, request: Option>) -> Self { + Self { + session: self.session.with_request_telemetry(request), + } + } + + fn path() -> &'static str { + "memories/trace_summarize" + } + + pub async fn trace_summarize( + &self, + body: serde_json::Value, + extra_headers: HeaderMap, + ) -> Result, ApiError> { + let resp = self + .session + .execute(Method::POST, Self::path(), extra_headers, Some(body)) + .await?; + let parsed: TraceSummarizeResponse = + serde_json::from_slice(&resp.body).map_err(|e| ApiError::Stream(e.to_string()))?; + Ok(parsed.output) + } + + pub async fn trace_summarize_input( + &self, + input: &MemoryTraceSummarizeInput, + extra_headers: HeaderMap, + ) -> Result, ApiError> { + let body = to_value(input).map_err(|e| { + ApiError::Stream(format!( + "failed to encode memory trace summarize input: {e}" + )) + })?; + self.trace_summarize(body, extra_headers).await + } +} + +#[derive(Debug, Deserialize)] +struct TraceSummarizeResponse { + output: Vec, +} + +#[cfg(test)] +mod tests { + use super::*; + use async_trait::async_trait; + use codex_client::Request; + use codex_client::Response; + use codex_client::StreamResponse; + use codex_client::TransportError; + + #[derive(Clone, Default)] + struct DummyTransport; + + #[async_trait] + impl HttpTransport for DummyTransport { + async fn execute(&self, _req: Request) -> Result { + Err(TransportError::Build("execute should not run".to_string())) + } + + async fn stream(&self, _req: Request) -> Result { + Err(TransportError::Build("stream should not run".to_string())) + } + } + + #[derive(Clone, Default)] + struct DummyAuth; + + impl AuthProvider for DummyAuth { + fn bearer_token(&self) -> Option { + None + } + } + + #[test] + fn path_is_memories_trace_summarize() { + assert_eq!( + MemoriesClient::::path(), + "memories/trace_summarize" + ); + } +} diff --git a/codex-rs/codex-api/src/endpoint/mod.rs b/codex-rs/codex-api/src/endpoint/mod.rs index 23579ffcf1..0dede138e8 100644 --- a/codex-rs/codex-api/src/endpoint/mod.rs +++ b/codex-rs/codex-api/src/endpoint/mod.rs @@ -1,5 +1,6 @@ pub mod aggregate; pub mod compact; +pub mod memories; pub mod models; pub mod responses; pub mod responses_websocket; diff --git a/codex-rs/codex-api/src/endpoint/responses_websocket.rs b/codex-rs/codex-api/src/endpoint/responses_websocket.rs index cac686dd02..20c6067b17 100644 --- a/codex-rs/codex-api/src/endpoint/responses_websocket.rs +++ b/codex-rs/codex-api/src/endpoint/responses_websocket.rs @@ -5,6 +5,7 @@ use crate::common::ResponseStream; use crate::common::ResponsesWsRequest; use crate::error::ApiError; use crate::provider::Provider; +use crate::rate_limits::parse_rate_limit_event; use crate::sse::responses::ResponsesStreamEvent; use crate::sse::responses::process_responses_event; use crate::telemetry::WebsocketTelemetry; @@ -33,6 +34,7 @@ use url::Url; type WsStream = WebSocketStream>; const X_CODEX_TURN_STATE_HEADER: &str = "x-codex-turn-state"; +const X_MODELS_ETAG_HEADER: &str = "x-models-etag"; const X_REASONING_INCLUDED_HEADER: &str = "x-reasoning-included"; pub struct ResponsesWebsocketConnection { @@ -40,6 +42,7 @@ pub struct ResponsesWebsocketConnection { // TODO (pakrym): is this the right place for timeout? idle_timeout: Duration, server_reasoning_included: bool, + models_etag: Option, telemetry: Option>, } @@ -48,12 +51,14 @@ impl ResponsesWebsocketConnection { stream: WsStream, idle_timeout: Duration, server_reasoning_included: bool, + models_etag: Option, telemetry: Option>, ) -> Self { Self { stream: Arc::new(Mutex::new(Some(stream))), idle_timeout, server_reasoning_included, + models_etag, telemetry, } } @@ -71,12 +76,16 @@ impl ResponsesWebsocketConnection { let stream = Arc::clone(&self.stream); let idle_timeout = self.idle_timeout; let server_reasoning_included = self.server_reasoning_included; + let models_etag = self.models_etag.clone(); let telemetry = self.telemetry.clone(); let request_body = serde_json::to_value(&request).map_err(|err| { ApiError::Stream(format!("failed to encode websocket request: {err}")) })?; tokio::spawn(async move { + if let Some(etag) = models_etag { + let _ = tx_event.send(Ok(ResponseEvent::ModelsEtag(etag))).await; + } if server_reasoning_included { let _ = tx_event .send(Ok(ResponseEvent::ServerReasoningIncluded(true))) @@ -136,12 +145,13 @@ impl ResponsesWebsocketClient { headers.extend(extra_headers); add_auth_headers_to_header_map(&self.auth, &mut headers); - let (stream, server_reasoning_included) = - connect_websocket(ws_url, headers, turn_state).await?; + let (stream, server_reasoning_included, models_etag) = + connect_websocket(ws_url, headers, turn_state.clone()).await?; Ok(ResponsesWebsocketConnection::new( stream, self.provider.stream_idle_timeout, server_reasoning_included, + models_etag, telemetry, )) } @@ -151,7 +161,7 @@ async fn connect_websocket( url: Url, headers: HeaderMap, turn_state: Option>>, -) -> Result<(WsStream, bool), ApiError> { +) -> Result<(WsStream, bool, Option), ApiError> { info!("connecting to websocket: {url}"); let mut request = url @@ -177,6 +187,11 @@ async fn connect_websocket( }; let reasoning_included = response.headers().contains_key(X_REASONING_INCLUDED_HEADER); + let models_etag = response + .headers() + .get(X_MODELS_ETAG_HEADER) + .and_then(|value| value.to_str().ok()) + .map(ToString::to_string); if let Some(turn_state) = turn_state && let Some(header_value) = response .headers() @@ -185,7 +200,7 @@ async fn connect_websocket( { let _ = turn_state.set(header_value.to_string()); } - Ok((stream, reasoning_included)) + Ok((stream, reasoning_included, models_etag)) } fn map_ws_error(err: WsError, url: &Url) -> ApiError { @@ -273,6 +288,12 @@ async fn run_websocket_response_stream( continue; } }; + if event.kind() == "codex.rate_limits" { + if let Some(snapshot) = parse_rate_limit_event(&text) { + let _ = tx_event.send(Ok(ResponseEvent::RateLimits(snapshot))).await; + } + continue; + } match process_responses_event(event) { Ok(Some(event)) => { let is_completed = matches!(event, ResponseEvent::Completed { .. }); diff --git a/codex-rs/codex-api/src/lib.rs b/codex-rs/codex-api/src/lib.rs index b0c70084d4..70652d2d78 100644 --- a/codex-rs/codex-api/src/lib.rs +++ b/codex-rs/codex-api/src/lib.rs @@ -15,6 +15,10 @@ pub use codex_client::TransportError; pub use crate::auth::AuthProvider; pub use crate::common::CompactionInput; +pub use crate::common::MemoryTrace; +pub use crate::common::MemoryTraceMetadata; +pub use crate::common::MemoryTraceSummarizeInput; +pub use crate::common::MemoryTraceSummaryOutput; pub use crate::common::Prompt; pub use crate::common::ResponseAppendWsRequest; pub use crate::common::ResponseCreateWsRequest; @@ -24,6 +28,7 @@ pub use crate::common::ResponsesApiRequest; pub use crate::common::create_text_param_for_request; pub use crate::endpoint::aggregate::AggregateStreamExt; pub use crate::endpoint::compact::CompactClient; +pub use crate::endpoint::memories::MemoriesClient; pub use crate::endpoint::models::ModelsClient; pub use crate::endpoint::responses::ResponsesClient; pub use crate::endpoint::responses::ResponsesOptions; diff --git a/codex-rs/codex-api/src/rate_limits.rs b/codex-rs/codex-api/src/rate_limits.rs index c29aab21f8..047c5934bd 100644 --- a/codex-rs/codex-api/src/rate_limits.rs +++ b/codex-rs/codex-api/src/rate_limits.rs @@ -1,7 +1,9 @@ +use codex_protocol::account::PlanType; use codex_protocol::protocol::CreditsSnapshot; use codex_protocol::protocol::RateLimitSnapshot; use codex_protocol::protocol::RateLimitWindow; use http::HeaderMap; +use serde::Deserialize; use std::fmt::Display; #[derive(Debug)] @@ -41,6 +43,70 @@ pub fn parse_rate_limit(headers: &HeaderMap) -> Option { }) } +#[derive(Debug, Deserialize)] +struct RateLimitEventWindow { + used_percent: f64, + window_minutes: Option, + reset_at: Option, +} + +#[derive(Debug, Deserialize)] +struct RateLimitEventDetails { + primary: Option, + secondary: Option, +} + +#[derive(Debug, Deserialize)] +struct RateLimitEventCredits { + has_credits: bool, + unlimited: bool, + balance: Option, +} + +#[derive(Debug, Deserialize)] +struct RateLimitEvent { + #[serde(rename = "type")] + kind: String, + plan_type: Option, + rate_limits: Option, + credits: Option, +} + +pub fn parse_rate_limit_event(payload: &str) -> Option { + let event: RateLimitEvent = serde_json::from_str(payload).ok()?; + if event.kind != "codex.rate_limits" { + return None; + } + let (primary, secondary) = if let Some(details) = event.rate_limits.as_ref() { + ( + map_event_window(details.primary.as_ref()), + map_event_window(details.secondary.as_ref()), + ) + } else { + (None, None) + }; + let credits = event.credits.map(|credits| CreditsSnapshot { + has_credits: credits.has_credits, + unlimited: credits.unlimited, + balance: credits.balance, + }); + Some(RateLimitSnapshot { + primary, + secondary, + credits, + plan_type: event.plan_type, + }) +} + +fn map_event_window(window: Option<&RateLimitEventWindow>) -> Option { + let window = window?; + Some(RateLimitWindow { + used_percent: window.used_percent, + window_minutes: window.window_minutes, + resets_at: window.reset_at, + }) +} + /// Parses the bespoke Codex rate-limit headers into a `RateLimitSnapshot`. pub fn parse_promo_message(headers: &HeaderMap) -> Option { parse_header_str(headers, "x-codex-promo-message") diff --git a/codex-rs/codex-api/src/sse/responses.rs b/codex-rs/codex-api/src/sse/responses.rs index 2c91104727..95bda35fb2 100644 --- a/codex-rs/codex-api/src/sse/responses.rs +++ b/codex-rs/codex-api/src/sse/responses.rs @@ -165,6 +165,12 @@ pub struct ResponsesStreamEvent { content_index: Option, } +impl ResponsesStreamEvent { + pub fn kind(&self) -> &str { + &self.kind + } +} + #[derive(Debug)] pub enum ResponsesEventError { Api(ApiError), diff --git a/codex-rs/common/src/config_override.rs b/codex-rs/common/src/config_override.rs index 59dde92a22..9bbec2b6f2 100644 --- a/codex-rs/common/src/config_override.rs +++ b/codex-rs/common/src/config_override.rs @@ -71,7 +71,7 @@ impl CliConfigOverrides { } }; - Ok((key.to_string(), value)) + Ok((canonicalize_override_key(key), value)) }) .collect() } @@ -88,6 +88,14 @@ impl CliConfigOverrides { } } +fn canonicalize_override_key(key: &str) -> String { + if key == "use_linux_sandbox_bwrap" { + "features.use_linux_sandbox_bwrap".to_string() + } else { + key.to_string() + } +} + /// Apply a single override onto `root`, creating intermediate objects as /// necessary. fn apply_single_override(root: &mut Value, path: &str, value: Value) { @@ -172,6 +180,16 @@ mod tests { assert_eq!(arr.len(), 3); } + #[test] + fn canonicalizes_use_linux_sandbox_bwrap_alias() { + let overrides = CliConfigOverrides { + raw_overrides: vec!["use_linux_sandbox_bwrap=true".to_string()], + }; + let parsed = overrides.parse_overrides().expect("parse_overrides"); + assert_eq!(parsed[0].0.as_str(), "features.use_linux_sandbox_bwrap"); + assert_eq!(parsed[0].1.as_bool(), Some(true)); + } + #[test] fn parses_inline_table() { let v = parse_toml_value("{a = 1, b = 2}").expect("parse"); diff --git a/codex-rs/core/Cargo.toml b/codex-rs/core/Cargo.toml index 38da38b929..d686343ae7 100644 --- a/codex-rs/core/Cargo.toml +++ b/codex-rs/core/Cargo.toml @@ -58,6 +58,7 @@ indoc = { workspace = true } keyring = { workspace = true, features = ["crypto-rust"] } libc = { workspace = true } multimap = { workspace = true } +notify = { workspace = true } once_cell = { workspace = true } os_info = { workspace = true } rand = { workspace = true } @@ -107,6 +108,7 @@ url = { workspace = true } uuid = { workspace = true, features = ["serde", "v4", "v5"] } which = { workspace = true } wildmatch = { workspace = true } +zip = { workspace = true } [features] deterministic_process_ids = [] diff --git a/codex-rs/core/config.schema.json b/codex-rs/core/config.schema.json index bddc9d8a0b..8e9550c092 100644 --- a/codex-rs/core/config.schema.json +++ b/codex-rs/core/config.schema.json @@ -235,6 +235,9 @@ "unified_exec": { "type": "boolean" }, + "use_linux_sandbox_bwrap": { + "type": "boolean" + }, "web_search": { "type": "boolean" }, @@ -705,6 +708,7 @@ }, "Personality": { "enum": [ + "none", "friendly", "pragmatic" ], @@ -1253,6 +1257,9 @@ "unified_exec": { "type": "boolean" }, + "use_linux_sandbox_bwrap": { + "type": "boolean" + }, "web_search": { "type": "boolean" }, diff --git a/codex-rs/core/src/client.rs b/codex-rs/core/src/client.rs index fcc308f380..570cdc830b 100644 --- a/codex-rs/core/src/client.rs +++ b/codex-rs/core/src/client.rs @@ -1,15 +1,32 @@ -use std::path::PathBuf; +//! Session- and turn-scoped helpers for talking to model provider APIs. +//! +//! `ModelClient` is intended to live for the lifetime of a Codex session and holds the stable +//! configuration and state needed to talk to a provider (auth, provider selection, conversation id, +//! and feature-gated request behavior). +//! +//! Per-turn settings (model selection, reasoning controls, telemetry context, web search +//! eligibility, and turn metadata) are passed explicitly to streaming and unary methods so that the +//! turn lifetime is visible at the call site. +//! +//! A [`ModelClientSession`] is created per turn and is used to stream one or more Responses API +//! requests during that turn. It caches a Responses WebSocket connection (opened lazily) and +//! stores per-turn state such as the `x-codex-turn-state` token used for sticky routing. + use std::sync::Arc; use std::sync::OnceLock; -use std::sync::RwLock; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; use crate::api_bridge::CoreAuthProvider; use crate::api_bridge::auth_provider_from_auth; use crate::api_bridge::map_api_error; use crate::auth::UnauthorizedRecovery; -use crate::turn_metadata::build_turn_metadata_header; use codex_api::CompactClient as ApiCompactClient; use codex_api::CompactionInput as ApiCompactionInput; +use codex_api::MemoriesClient as ApiMemoriesClient; +use codex_api::MemoryTrace as ApiMemoryTrace; +use codex_api::MemoryTraceSummarizeInput as ApiMemoryTraceSummarizeInput; +use codex_api::MemoryTraceSummaryOutput as ApiMemoryTraceSummaryOutput; use codex_api::Prompt as ApiPrompt; use codex_api::RequestTelemetry; use codex_api::ReqwestTransport; @@ -32,7 +49,7 @@ use codex_otel::OtelManager; use codex_protocol::ThreadId; use codex_protocol::config_types::ReasoningSummary as ReasoningSummaryConfig; -use codex_protocol::config_types::WebSearchMode; +use codex_protocol::config_types::Verbosity as VerbosityConfig; use codex_protocol::models::ResponseItem; use codex_protocol::openai_models::ModelInfo; use codex_protocol::openai_models::ReasoningEffort as ReasoningEffortConfig; @@ -57,53 +74,74 @@ use crate::auth::RefreshTokenError; use crate::client_common::Prompt; use crate::client_common::ResponseEvent; use crate::client_common::ResponseStream; -use crate::config::Config; use crate::default_client::build_reqwest_client; use crate::error::CodexErr; use crate::error::Result; -use crate::features::FEATURES; -use crate::features::Feature; use crate::flags::CODEX_RS_SSE_FIXTURE; use crate::model_provider_info::ModelProviderInfo; use crate::model_provider_info::WireApi; use crate::tools::spec::create_tools_json_for_responses_api; -use crate::transport_manager::TransportManager; pub const WEB_SEARCH_ELIGIBLE_HEADER: &str = "x-oai-web-search-eligible"; pub const X_CODEX_TURN_STATE_HEADER: &str = "x-codex-turn-state"; pub const X_CODEX_TURN_METADATA_HEADER: &str = "x-codex-turn-metadata"; +pub const X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER: &str = + "x-responsesapi-include-timing-metrics"; -#[derive(Debug, Default)] -struct TurnMetadataCache { - cwd: Option, - header: Option, -} - +/// Session-scoped state shared by all [`ModelClient`] clones. +/// +/// This is intentionally kept minimal so `ModelClient` does not need to hold a full `Config`. Most +/// configuration is per turn and is passed explicitly to streaming/unary methods. #[derive(Debug)] struct ModelClientState { - config: Arc, auth_manager: Option>, - model_info: ModelInfo, - otel_manager: OtelManager, - provider: ModelProviderInfo, conversation_id: ThreadId, - effort: Option, - summary: ReasoningSummaryConfig, + provider: ModelProviderInfo, session_source: SessionSource, - transport_manager: TransportManager, - turn_metadata_cache: Arc>, + model_verbosity: Option, + enable_responses_websockets: bool, + enable_request_compression: bool, + include_timing_metrics: bool, + beta_features_header: Option, + disable_websockets: AtomicBool, } +/// A session-scoped client for model-provider API calls. +/// +/// This holds configuration and state that should be shared across turns within a Codex session +/// (auth, provider selection, conversation id, feature-gated request behavior, and transport +/// fallback state). +/// +/// WebSocket fallback is session-scoped: once a turn activates the HTTP fallback, subsequent turns +/// will also use HTTP for the remainder of the session. +/// +/// Turn-scoped settings (model selection, reasoning controls, telemetry context, web search +/// eligibility, and turn metadata) are passed explicitly to the relevant methods to keep turn +/// lifetime visible at the call site. +/// +/// This type is cheap to clone. #[derive(Debug, Clone)] pub struct ModelClient { state: Arc, } +/// A turn-scoped streaming session created from a [`ModelClient`]. +/// +/// The session lazily establishes a Responses WebSocket connection (and reuses it across multiple +/// requests) and caches per-turn state: +/// +/// - The last request's input items, so subsequent calls can use `response.append` when the input +/// is an incremental extension of the previous request. +/// - The `x-codex-turn-state` sticky-routing token, which must be replayed for all requests within +/// the same turn. +/// +/// Create a fresh `ModelClientSession` for each Codex turn. Reusing it across turns would replay +/// the previous turn's sticky-routing token into the next turn, which violates the client/server +/// contract and can cause routing bugs. pub struct ModelClientSession { - state: Arc, + client: ModelClient, connection: Option, websocket_last_items: Vec, - transport_manager: TransportManager, /// Turn state for sticky routing. /// /// This is an `OnceLock` that stores the turn state value received from the server @@ -117,142 +155,65 @@ pub struct ModelClientSession { turn_state: Arc>, } -#[allow(clippy::too_many_arguments)] impl ModelClient { + #[allow(clippy::too_many_arguments)] + /// Creates a new session-scoped `ModelClient`. + /// + /// All arguments are expected to be stable for the lifetime of a Codex session. Per-turn values + /// are passed to [`ModelClientSession::stream`] (and other turn-scoped methods) explicitly. pub fn new( - config: Arc, auth_manager: Option>, - model_info: ModelInfo, - otel_manager: OtelManager, - provider: ModelProviderInfo, - effort: Option, - summary: ReasoningSummaryConfig, conversation_id: ThreadId, + provider: ModelProviderInfo, session_source: SessionSource, - transport_manager: TransportManager, + model_verbosity: Option, + enable_responses_websockets: bool, + enable_request_compression: bool, + include_timing_metrics: bool, + beta_features_header: Option, ) -> Self { Self { state: Arc::new(ModelClientState { - config, auth_manager, - model_info, - otel_manager, - provider, conversation_id, - effort, - summary, + provider, session_source, - transport_manager, - turn_metadata_cache: Arc::new(RwLock::new(TurnMetadataCache::default())), + model_verbosity, + enable_responses_websockets, + enable_request_compression, + include_timing_metrics, + beta_features_header, + disable_websockets: AtomicBool::new(false), }), } } - pub fn new_session(&self, turn_metadata_cwd: Option) -> ModelClientSession { - self.prewarm_turn_metadata_header(turn_metadata_cwd); + /// Creates a fresh turn-scoped streaming session. + /// + /// This does not open any network connections; the WebSocket connection is established lazily + /// when the first WebSocket stream request is issued. + pub fn new_session(&self) -> ModelClientSession { ModelClientSession { - state: Arc::clone(&self.state), + client: self.clone(), connection: None, websocket_last_items: Vec::new(), - transport_manager: self.state.transport_manager.clone(), turn_state: Arc::new(OnceLock::new()), } } - /// Refresh turn metadata in the background and update a cached header that request - /// builders can read without blocking. - fn prewarm_turn_metadata_header(&self, turn_metadata_cwd: Option) { - let turn_metadata_cwd = - turn_metadata_cwd.map(|cwd| std::fs::canonicalize(&cwd).unwrap_or(cwd)); - - if let Ok(mut cache) = self.state.turn_metadata_cache.write() - && cache.cwd != turn_metadata_cwd - { - cache.cwd = turn_metadata_cwd.clone(); - cache.header = None; - } - - let Some(cwd) = turn_metadata_cwd else { - return; - }; - let turn_metadata_cache = Arc::clone(&self.state.turn_metadata_cache); - if let Ok(handle) = tokio::runtime::Handle::try_current() { - let _task = handle.spawn(async move { - let header = build_turn_metadata_header(cwd.as_path()) - .await - .and_then(|value| HeaderValue::from_str(value.as_str()).ok()); - - if let Ok(mut cache) = turn_metadata_cache.write() - && cache.cwd.as_ref() == Some(&cwd) - { - cache.header = header; - } - }); - } - } -} - -impl ModelClient { - pub fn get_model_context_window(&self) -> Option { - let model_info = &self.state.model_info; - let effective_context_window_percent = model_info.effective_context_window_percent; - model_info.context_window.map(|context_window| { - context_window.saturating_mul(effective_context_window_percent) / 100 - }) - } - - pub fn config(&self) -> Arc { - Arc::clone(&self.state.config) - } - - pub fn provider(&self) -> &ModelProviderInfo { - &self.state.provider - } - - pub fn get_provider(&self) -> ModelProviderInfo { - self.state.provider.clone() - } - - pub fn get_otel_manager(&self) -> OtelManager { - self.state.otel_manager.clone() - } - - pub fn get_session_source(&self) -> SessionSource { - self.state.session_source.clone() - } - - pub(crate) fn transport_manager(&self) -> TransportManager { - self.state.transport_manager.clone() - } - - /// Returns the currently configured model slug. - pub fn get_model(&self) -> String { - self.state.model_info.slug.clone() - } - - pub fn get_model_info(&self) -> ModelInfo { - self.state.model_info.clone() - } - - /// Returns the current reasoning effort setting. - pub fn get_reasoning_effort(&self) -> Option { - self.state.effort - } - - /// Returns the current reasoning summary setting. - pub fn get_reasoning_summary(&self) -> ReasoningSummaryConfig { - self.state.summary - } - - pub fn get_auth_manager(&self) -> Option> { - self.state.auth_manager.clone() - } - /// Compacts the current conversation history using the Compact endpoint. /// /// This is a unary call (no streaming) that returns a new list of /// `ResponseItem`s representing the compacted transcript. - pub async fn compact_conversation_history(&self, prompt: &Prompt) -> Result> { + /// + /// The model selection and telemetry context are passed explicitly to keep `ModelClient` + /// session-scoped. + pub async fn compact_conversation_history( + &self, + prompt: &Prompt, + model_info: &ModelInfo, + otel_manager: &OtelManager, + ) -> Result> { if prompt.input.is_empty() { return Ok(Vec::new()); } @@ -267,17 +228,72 @@ impl ModelClient { .to_api_provider(auth.as_ref().map(CodexAuth::internal_auth_mode))?; let api_auth = auth_provider_from_auth(auth.clone(), &self.state.provider)?; let transport = ReqwestTransport::new(build_reqwest_client()); - let request_telemetry = self.build_request_telemetry(); + let request_telemetry = Self::build_request_telemetry(otel_manager); let client = ApiCompactClient::new(transport, api_provider, api_auth) .with_telemetry(Some(request_telemetry)); let instructions = prompt.base_instructions.text.clone(); let payload = ApiCompactionInput { - model: &self.state.model_info.slug, + model: &model_info.slug, input: &prompt.input, instructions: &instructions, }; + let extra_headers = self.build_subagent_headers(); + client + .compact_input(&payload, extra_headers) + .await + .map_err(map_api_error) + } + + /// Builds memory summaries for each provided normalized trace. + /// + /// This is a unary call (no streaming) to `/v1/memories/trace_summarize`. + /// + /// The model selection, reasoning effort, and telemetry context are passed explicitly to keep + /// `ModelClient` session-scoped. + pub async fn summarize_memory_traces( + &self, + traces: Vec, + model_info: &ModelInfo, + effort: Option, + otel_manager: &OtelManager, + ) -> Result> { + if traces.is_empty() { + return Ok(Vec::new()); + } + + let auth_manager = self.state.auth_manager.clone(); + let auth = match auth_manager.as_ref() { + Some(manager) => manager.auth().await, + None => None, + }; + let api_provider = self + .state + .provider + .to_api_provider(auth.as_ref().map(CodexAuth::internal_auth_mode))?; + let api_auth = auth_provider_from_auth(auth, &self.state.provider)?; + let transport = ReqwestTransport::new(build_reqwest_client()); + let request_telemetry = Self::build_request_telemetry(otel_manager); + let client = ApiMemoriesClient::new(transport, api_provider, api_auth) + .with_telemetry(Some(request_telemetry)); + + let payload = ApiMemoryTraceSummarizeInput { + model: model_info.slug.clone(), + traces, + reasoning: effort.map(|effort| Reasoning { + effort: Some(effort), + summary: None, + }), + }; + + client + .trace_summarize_input(&payload, self.build_subagent_headers()) + .await + .map_err(map_api_error) + } + + fn build_subagent_headers(&self) -> ApiHeaderMap { let mut extra_headers = ApiHeaderMap::new(); if let SessionSource::SubAgent(sub) = &self.state.session_source { let subagent = match sub { @@ -290,89 +306,64 @@ impl ModelClient { extra_headers.insert("x-openai-subagent", val); } } - client - .compact_input(&payload, extra_headers) - .await - .map_err(map_api_error) + extra_headers + } + + /// Builds request telemetry for unary API calls (e.g., Compact endpoint). + fn build_request_telemetry(otel_manager: &OtelManager) -> Arc { + let telemetry = Arc::new(ApiTelemetry::new(otel_manager.clone())); + let request_telemetry: Arc = telemetry; + request_telemetry } } impl ModelClientSession { - fn turn_metadata_header(&self) -> Option { - self.state - .turn_metadata_cache - .try_read() - .ok() - .and_then(|cache| cache.header.clone()) + fn disable_websockets(&self) -> bool { + self.client.state.disable_websockets.load(Ordering::Relaxed) } - /// Streams a single model turn using the configured Responses transport. - pub async fn stream(&mut self, prompt: &Prompt) -> Result { - let wire_api = self.state.provider.wire_api; - match wire_api { - WireApi::Responses => { - let websocket_enabled = self.responses_websocket_enabled() - && !self.transport_manager.disable_websockets(); - - if websocket_enabled { - self.stream_responses_websocket(prompt).await - } else { - self.stream_responses_api(prompt).await - } - } - } - } - - pub(crate) fn try_switch_fallback_transport(&mut self) -> bool { - let websocket_enabled = self.responses_websocket_enabled(); - let activated = self - .transport_manager - .activate_http_fallback(websocket_enabled); - if activated { - warn!("falling back to HTTP"); - self.state.otel_manager.counter( - "codex.transport.fallback_to_http", - 1, - &[("from_wire_api", "responses_websocket")], - ); - - self.connection = None; - self.websocket_last_items.clear(); - } - activated + fn activate_http_fallback(&self, websocket_enabled: bool) -> bool { + websocket_enabled + && !self + .client + .state + .disable_websockets + .swap(true, Ordering::Relaxed) } fn responses_websocket_enabled(&self) -> bool { - self.state.provider.supports_websockets - && self - .state - .config - .features - .enabled(Feature::ResponsesWebsockets) + self.client.state.provider.supports_websockets + && self.client.state.enable_responses_websockets } - fn build_responses_request(&self, prompt: &Prompt) -> Result { + fn build_responses_request(prompt: &Prompt) -> Result { let instructions = prompt.base_instructions.text.clone(); let tools_json: Vec = create_tools_json_for_responses_api(&prompt.tools)?; Ok(build_api_prompt(prompt, instructions, tools_json)) } + #[allow(clippy::too_many_arguments)] fn build_responses_options( &self, prompt: &Prompt, + model_info: &ModelInfo, + effort: Option, + summary: ReasoningSummaryConfig, + web_search_eligible: bool, + turn_metadata_header: Option<&str>, compression: Compression, ) -> ApiResponsesOptions { - let turn_metadata_header = self.turn_metadata_header(); - let model_info = &self.state.model_info; + let turn_metadata_header = + turn_metadata_header.and_then(|value| HeaderValue::from_str(value).ok()); let default_reasoning_effort = model_info.default_reasoning_level; let reasoning = if model_info.supports_reasoning_summaries { Some(Reasoning { - effort: self.state.effort.or(default_reasoning_effort), - summary: if self.state.summary == ReasoningSummaryConfig::None { + effort: effort.or(default_reasoning_effort), + summary: if summary == ReasoningSummaryConfig::None { None } else { - Some(self.state.summary) + Some(summary) }, }) } else { @@ -386,12 +377,12 @@ impl ModelClientSession { }; let verbosity = if model_info.support_verbosity { - self.state - .config + self.client + .state .model_verbosity .or(model_info.default_verbosity) } else { - if self.state.config.model_verbosity.is_some() { + if self.client.state.model_verbosity.is_some() { warn!( "model_verbosity is set but ignored as the model does not support verbosity: {}", model_info.slug @@ -401,7 +392,7 @@ impl ModelClientSession { }; let text = create_text_param_for_request(verbosity, &prompt.output_schema); - let conversation_id = self.state.conversation_id.to_string(); + let conversation_id = self.client.state.conversation_id.to_string(); ApiResponsesOptions { reasoning, @@ -410,9 +401,10 @@ impl ModelClientSession { text, store_override: None, conversation_id: Some(conversation_id), - session_source: Some(self.state.session_source.clone()), + session_source: Some(self.client.state.session_source.clone()), extra_headers: build_responses_headers( - &self.state.config, + self.client.state.beta_features_header.as_deref(), + web_search_eligible, Some(&self.turn_state), turn_metadata_header.as_ref(), ), @@ -438,6 +430,7 @@ impl ModelClientSession { fn prepare_websocket_request( &self, + model_slug: &str, api_prompt: &ApiPrompt, options: &ApiResponsesOptions, ) -> ResponsesWsRequest { @@ -458,7 +451,7 @@ impl ModelClientSession { let store = store_override.unwrap_or(false); let payload = ResponseCreateWsRequest { - model: self.state.model_info.slug.clone(), + model: model_slug.to_string(), instructions: api_prompt.instructions.clone(), input: api_prompt.input.clone(), tools: api_prompt.tools.clone(), @@ -477,6 +470,7 @@ impl ModelClientSession { async fn websocket_connection( &mut self, + otel_manager: &OtelManager, api_provider: codex_api::Provider, api_auth: CoreAuthProvider, options: &ApiResponsesOptions, @@ -489,7 +483,13 @@ impl ModelClientSession { if needs_new { let mut headers = options.extra_headers.clone(); headers.extend(build_conversation_headers(options.conversation_id.clone())); - let websocket_telemetry = self.build_websocket_telemetry(); + if self.client.state.include_timing_metrics { + headers.insert( + X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER, + HeaderValue::from_static("true"), + ); + } + let websocket_telemetry = Self::build_websocket_telemetry(otel_manager); let new_conn: ApiWebSocketConnection = ApiWebSocketResponsesClient::new(api_provider, api_auth) .connect( @@ -507,13 +507,9 @@ impl ModelClientSession { } fn responses_request_compression(&self, auth: Option<&crate::auth::CodexAuth>) -> Compression { - if self - .state - .config - .features - .enabled(Feature::EnableRequestCompression) + if self.client.state.enable_request_compression && auth.is_some_and(CodexAuth::is_chatgpt_auth) - && self.state.provider.is_openai() + && self.client.state.provider.is_openai() { Compression::Zstd } else { @@ -525,17 +521,29 @@ impl ModelClientSession { /// /// Handles SSE fixtures, reasoning summaries, verbosity, and the /// `text` controls used for output schemas. - async fn stream_responses_api(&self, prompt: &Prompt) -> Result { + #[allow(clippy::too_many_arguments)] + async fn stream_responses_api( + &self, + prompt: &Prompt, + model_info: &ModelInfo, + otel_manager: &OtelManager, + effort: Option, + summary: ReasoningSummaryConfig, + web_search_eligible: bool, + turn_metadata_header: Option<&str>, + ) -> Result { if let Some(path) = &*CODEX_RS_SSE_FIXTURE { warn!(path, "Streaming from fixture"); - let stream = - codex_api::stream_from_fixture(path, self.state.provider.stream_idle_timeout()) - .map_err(map_api_error)?; - return Ok(map_response_stream(stream, self.state.otel_manager.clone())); + let stream = codex_api::stream_from_fixture( + path, + self.client.state.provider.stream_idle_timeout(), + ) + .map_err(map_api_error)?; + return Ok(map_response_stream(stream, otel_manager.clone())); } - let auth_manager = self.state.auth_manager.clone(); - let api_prompt = self.build_responses_request(prompt)?; + let auth_manager = self.client.state.auth_manager.clone(); + let api_prompt = Self::build_responses_request(prompt)?; let mut auth_recovery = auth_manager .as_ref() @@ -546,26 +554,35 @@ impl ModelClientSession { None => None, }; let api_provider = self + .client .state .provider .to_api_provider(auth.as_ref().map(CodexAuth::internal_auth_mode))?; - let api_auth = auth_provider_from_auth(auth.clone(), &self.state.provider)?; + let api_auth = auth_provider_from_auth(auth.clone(), &self.client.state.provider)?; let transport = ReqwestTransport::new(build_reqwest_client()); - let (request_telemetry, sse_telemetry) = self.build_streaming_telemetry(); + let (request_telemetry, sse_telemetry) = Self::build_streaming_telemetry(otel_manager); let compression = self.responses_request_compression(auth.as_ref()); let client = ApiResponsesClient::new(transport, api_provider, api_auth) .with_telemetry(Some(request_telemetry), Some(sse_telemetry)); - let options = self.build_responses_options(prompt, compression); + let options = self.build_responses_options( + prompt, + model_info, + effort, + summary, + web_search_eligible, + turn_metadata_header, + compression, + ); let stream_result = client - .stream_prompt(&self.state.model_info.slug, &api_prompt, options) + .stream_prompt(&model_info.slug, &api_prompt, options) .await; match stream_result { Ok(stream) => { - return Ok(map_response_stream(stream, self.state.otel_manager.clone())); + return Ok(map_response_stream(stream, otel_manager.clone())); } Err(ApiError::Transport( unauthorized_transport @ TransportError::Http { status, .. }, @@ -579,9 +596,19 @@ impl ModelClientSession { } /// Streams a turn via the Responses API over WebSocket transport. - async fn stream_responses_websocket(&mut self, prompt: &Prompt) -> Result { - let auth_manager = self.state.auth_manager.clone(); - let api_prompt = self.build_responses_request(prompt)?; + #[allow(clippy::too_many_arguments)] + async fn stream_responses_websocket( + &mut self, + prompt: &Prompt, + model_info: &ModelInfo, + otel_manager: &OtelManager, + effort: Option, + summary: ReasoningSummaryConfig, + web_search_eligible: bool, + turn_metadata_header: Option<&str>, + ) -> Result { + let auth_manager = self.client.state.auth_manager.clone(); + let api_prompt = Self::build_responses_request(prompt)?; let mut auth_recovery = auth_manager .as_ref() @@ -592,17 +619,31 @@ impl ModelClientSession { None => None, }; let api_provider = self + .client .state .provider .to_api_provider(auth.as_ref().map(CodexAuth::internal_auth_mode))?; - let api_auth = auth_provider_from_auth(auth.clone(), &self.state.provider)?; + let api_auth = auth_provider_from_auth(auth.clone(), &self.client.state.provider)?; let compression = self.responses_request_compression(auth.as_ref()); - let options = self.build_responses_options(prompt, compression); - let request = self.prepare_websocket_request(&api_prompt, &options); + let options = self.build_responses_options( + prompt, + model_info, + effort, + summary, + web_search_eligible, + turn_metadata_header, + compression, + ); + let request = self.prepare_websocket_request(&model_info.slug, &api_prompt, &options); let connection = match self - .websocket_connection(api_provider.clone(), api_auth.clone(), &options) + .websocket_connection( + otel_manager, + api_provider.clone(), + api_auth.clone(), + &options, + ) .await { Ok(connection) => connection, @@ -621,35 +662,97 @@ impl ModelClientSession { .map_err(map_api_error)?; self.websocket_last_items = api_prompt.input.clone(); - return Ok(map_response_stream( - stream_result, - self.state.otel_manager.clone(), - )); + return Ok(map_response_stream(stream_result, otel_manager.clone())); } } /// Builds request and SSE telemetry for streaming API calls. - fn build_streaming_telemetry(&self) -> (Arc, Arc) { - let telemetry = Arc::new(ApiTelemetry::new(self.state.otel_manager.clone())); + fn build_streaming_telemetry( + otel_manager: &OtelManager, + ) -> (Arc, Arc) { + let telemetry = Arc::new(ApiTelemetry::new(otel_manager.clone())); let request_telemetry: Arc = telemetry.clone(); let sse_telemetry: Arc = telemetry; (request_telemetry, sse_telemetry) } /// Builds telemetry for the Responses API WebSocket transport. - fn build_websocket_telemetry(&self) -> Arc { - let telemetry = Arc::new(ApiTelemetry::new(self.state.otel_manager.clone())); + fn build_websocket_telemetry(otel_manager: &OtelManager) -> Arc { + let telemetry = Arc::new(ApiTelemetry::new(otel_manager.clone())); let websocket_telemetry: Arc = telemetry; websocket_telemetry } -} -impl ModelClient { - /// Builds request telemetry for unary API calls (e.g., Compact endpoint). - fn build_request_telemetry(&self) -> Arc { - let telemetry = Arc::new(ApiTelemetry::new(self.state.otel_manager.clone())); - let request_telemetry: Arc = telemetry; - request_telemetry + #[allow(clippy::too_many_arguments)] + /// Streams a single model request within the current turn. + /// + /// The caller is responsible for passing per-turn settings explicitly (model selection, + /// reasoning settings, telemetry context, web search eligibility, and turn metadata). This + /// method will prefer the Responses WebSocket transport when enabled and healthy, and will + /// fall back to the HTTP Responses API transport otherwise. + pub async fn stream( + &mut self, + prompt: &Prompt, + model_info: &ModelInfo, + otel_manager: &OtelManager, + effort: Option, + summary: ReasoningSummaryConfig, + web_search_eligible: bool, + turn_metadata_header: Option<&str>, + ) -> Result { + let wire_api = self.client.state.provider.wire_api; + match wire_api { + WireApi::Responses => { + let websocket_enabled = + self.responses_websocket_enabled() && !self.disable_websockets(); + + if websocket_enabled { + self.stream_responses_websocket( + prompt, + model_info, + otel_manager, + effort, + summary, + web_search_eligible, + turn_metadata_header, + ) + .await + } else { + self.stream_responses_api( + prompt, + model_info, + otel_manager, + effort, + summary, + web_search_eligible, + turn_metadata_header, + ) + .await + } + } + } + } + + /// Permanently disables WebSockets for this Codex session and resets WebSocket state. + /// + /// This is used after exhausting the provider retry budget, to force subsequent requests onto + /// the HTTP transport. Returns `true` if this call activated fallback, or `false` if fallback + /// was already active. + pub(crate) fn try_switch_fallback_transport(&mut self, otel_manager: &OtelManager) -> bool { + let websocket_enabled = self.responses_websocket_enabled(); + let activated = self.activate_http_fallback(websocket_enabled); + if activated { + warn!("falling back to HTTP"); + otel_manager.counter( + "codex.transport.fallback_to_http", + 1, + &[("from_wire_api", "responses_websocket")], + ); + + self.connection = None; + self.websocket_last_items.clear(); + } + activated } } @@ -664,44 +767,30 @@ fn build_api_prompt(prompt: &Prompt, instructions: String, tools_json: Vec ApiHeaderMap { - let enabled = FEATURES - .iter() - .filter_map(|spec| { - if spec.stage.experimental_menu_description().is_some() - && config.features.enabled(spec.id) - { - Some(spec.key) - } else { - None - } - }) - .collect::>(); - let value = enabled.join(","); - let mut headers = ApiHeaderMap::new(); - if !value.is_empty() - && let Ok(header_value) = HeaderValue::from_str(value.as_str()) - { - headers.insert("x-codex-beta-features", header_value); - } - headers -} - +/// Builds the extra headers attached to Responses API requests. +/// +/// These headers implement Codex-specific conventions: +/// +/// - `x-codex-beta-features`: comma-separated beta feature keys enabled for the session. +/// - `x-oai-web-search-eligible`: whether this turn is allowed to use web search. +/// - `x-codex-turn-state`: sticky routing token captured earlier in the turn. +/// - `x-codex-turn-metadata`: optional per-turn metadata for observability. fn build_responses_headers( - config: &Config, + beta_features_header: Option<&str>, + web_search_eligible: bool, turn_state: Option<&Arc>>, turn_metadata_header: Option<&HeaderValue>, ) -> ApiHeaderMap { - let mut headers = experimental_feature_headers(config); + let mut headers = ApiHeaderMap::new(); + if let Some(value) = beta_features_header + && !value.is_empty() + && let Ok(header_value) = HeaderValue::from_str(value) + { + headers.insert("x-codex-beta-features", header_value); + } headers.insert( WEB_SEARCH_ELIGIBLE_HEADER, - HeaderValue::from_static( - if matches!(config.web_search_mode, Some(WebSearchMode::Disabled)) { - "false" - } else { - "true" - }, - ), + HeaderValue::from_static(if web_search_eligible { "true" } else { "false" }), ); if let Some(turn_state) = turn_state && let Some(state) = turn_state.get() diff --git a/codex-rs/core/src/client_common.rs b/codex-rs/core/src/client_common.rs index 2614ce83ef..5a4eea8836 100644 --- a/codex-rs/core/src/client_common.rs +++ b/codex-rs/core/src/client_common.rs @@ -3,6 +3,7 @@ use crate::config::types::Personality; use crate::error::Result; pub use codex_api::common::ResponseEvent; use codex_protocol::models::BaseInstructions; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::ResponseItem; use futures::Stream; use serde::Deserialize; @@ -97,9 +98,11 @@ fn reserialize_shell_outputs(items: &mut [ResponseItem]) { } ResponseItem::FunctionCallOutput { call_id, output } => { if shell_call_ids.remove(call_id) - && let Some(structured) = parse_structured_shell_output(&output.content) + && let Some(structured) = output + .text_content() + .and_then(parse_structured_shell_output) { - output.content = structured + output.body = FunctionCallOutputBody::Text(structured); } } _ => {} diff --git a/codex-rs/core/src/codex.rs b/codex-rs/core/src/codex.rs index ba34af5ed8..cef00f64eb 100644 --- a/codex-rs/core/src/codex.rs +++ b/codex-rs/core/src/codex.rs @@ -20,6 +20,7 @@ use crate::compact::should_use_remote_compact_task; use crate::compact_remote::run_inline_remote_auto_compact_task; use crate::connectors; use crate::exec_policy::ExecPolicyManager; +use crate::features::FEATURES; use crate::features::Feature; use crate::features::Features; use crate::features::maybe_push_unstable_features_warning; @@ -32,8 +33,8 @@ use crate::stream_events_utils::handle_non_tool_response_item; use crate::stream_events_utils::handle_output_item_done; use crate::stream_events_utils::last_assistant_message_from_item; use crate::terminal; -use crate::transport_manager::TransportManager; use crate::truncate::TruncationPolicy; +use crate::turn_metadata::build_turn_metadata_header; use crate::user_notification::UserNotifier; use crate::util::error_or_panic; use async_channel::Receiver; @@ -80,6 +81,7 @@ use rmcp::model::RequestId; use serde_json; use serde_json::Value; use tokio::sync::Mutex; +use tokio::sync::OnceCell; use tokio::sync::RwLock; use tokio::sync::oneshot; use tokio_util::sync::CancellationToken; @@ -90,6 +92,7 @@ use tracing::field; use tracing::info; use tracing::info_span; use tracing::instrument; +use tracing::trace; use tracing::trace_span; use tracing::warn; @@ -115,6 +118,8 @@ use crate::error::Result as CodexResult; use crate::exec::StreamOutput; use crate::exec_policy::ExecPolicyUpdateError; use crate::feedback_tags; +use crate::file_watcher::FileWatcher; +use crate::file_watcher::FileWatcherEvent; use crate::git_info::get_git_repo_root; use crate::instructions::UserInstructions; use crate::mcp::CODEX_APPS_MCP_SERVER_NAME; @@ -209,6 +214,7 @@ use codex_protocol::models::ContentItem; use codex_protocol::models::DeveloperInstructions; use codex_protocol::models::ResponseInputItem; use codex_protocol::models::ResponseItem; +use codex_protocol::openai_models::ReasoningEffort as ReasoningEffortConfig; use codex_protocol::protocol::CodexErrorInfo; use codex_protocol::protocol::InitialHistory; use codex_protocol::user_input::UserInput; @@ -248,6 +254,7 @@ impl Codex { auth_manager: Arc, models_manager: Arc, skills_manager: Arc, + file_watcher: Arc, conversation_history: InitialHistory, session_source: SessionSource, agent_control: AgentControl, @@ -384,6 +391,7 @@ impl Codex { conversation_history, session_source_clone, skills_manager, + file_watcher, agent_control, ) .instrument(session_init_span) @@ -480,7 +488,14 @@ pub(crate) struct Session { #[derive(Debug)] pub(crate) struct TurnContext { pub(crate) sub_id: String, - pub(crate) client: ModelClient, + pub(crate) config: Arc, + pub(crate) auth_manager: Option>, + pub(crate) model_info: ModelInfo, + pub(crate) otel_manager: OtelManager, + pub(crate) provider: ModelProviderInfo, + pub(crate) reasoning_effort: Option, + pub(crate) reasoning_summary: ReasoningSummaryConfig, + pub(crate) session_source: SessionSource, /// The session's current working directory. All relative paths provided by /// the model as well as sandbox policies are resolved against this path /// instead of `std::env::current_dir()`. @@ -495,14 +510,23 @@ pub(crate) struct TurnContext { pub(crate) windows_sandbox_level: WindowsSandboxLevel, pub(crate) shell_environment_policy: ShellEnvironmentPolicy, pub(crate) tools_config: ToolsConfig, + pub(crate) features: Features, pub(crate) ghost_snapshot: GhostSnapshotConfig, pub(crate) final_output_json_schema: Option, pub(crate) codex_linux_sandbox_exe: Option, pub(crate) tool_call_gate: Arc, pub(crate) truncation_policy: TruncationPolicy, pub(crate) dynamic_tools: Vec, + turn_metadata_header: OnceCell>, } impl TurnContext { + pub(crate) fn model_context_window(&self) -> Option { + let effective_context_window_percent = self.model_info.effective_context_window_percent; + self.model_info.context_window.map(|context_window| { + context_window.saturating_mul(effective_context_window_percent) / 100 + }) + } + pub(crate) fn resolve_path(&self, path: Option) -> PathBuf { path.as_ref() .map(PathBuf::from) @@ -514,6 +538,38 @@ impl TurnContext { .as_deref() .unwrap_or(compact::SUMMARIZATION_PROMPT) } + + async fn build_turn_metadata_header(&self) -> Option { + self.turn_metadata_header + .get_or_init(|| async { build_turn_metadata_header(self.cwd.as_path()).await }) + .await + .clone() + } + + pub async fn resolve_turn_metadata_header(&self) -> Option { + const TURN_METADATA_HEADER_TIMEOUT_MS: u64 = 250; + match tokio::time::timeout( + std::time::Duration::from_millis(TURN_METADATA_HEADER_TIMEOUT_MS), + self.build_turn_metadata_header(), + ) + .await + { + Ok(header) => header, + Err(_) => { + warn!("timed out after 250ms while building turn metadata header"); + self.turn_metadata_header.get().cloned().flatten() + } + } + } + + pub fn spawn_turn_metadata_header_task(self: &Arc) { + let context = Arc::clone(self); + tokio::spawn(async move { + trace!("Spawning turn metadata calculation task"); + context.build_turn_metadata_header().await; + trace!("Turn metadata calculation task completed"); + }); + } } #[derive(Clone)] @@ -623,6 +679,33 @@ pub(crate) struct SessionSettingsUpdate { } impl Session { + /// Builds the `x-codex-beta-features` header value for this session. + /// + /// `ModelClient` is session-scoped and intentionally does not depend on the full `Config`, so + /// we precompute the comma-separated list of enabled experimental feature keys at session + /// creation time and thread it into the client. + fn build_model_client_beta_features_header(config: &Config) -> Option { + let beta_features_header = FEATURES + .iter() + .filter_map(|spec| { + if spec.stage.experimental_menu_description().is_some() + && config.features.enabled(spec.id) + { + Some(spec.key) + } else { + None + } + }) + .collect::>() + .join(","); + + if beta_features_header.is_empty() { + None + } else { + Some(beta_features_header) + } + } + /// Don't expand the number of mutated arguments on config. We are in the process of getting rid of it. pub(crate) fn build_per_turn_config(session_configuration: &SessionConfiguration) -> Config { // todo(aibrahim): store this state somewhere else so we don't need to mut config @@ -646,6 +729,29 @@ impl Session { state.session_configuration.codex_home().clone() } + fn start_file_watcher_listener(self: &Arc) { + let mut rx = self.services.file_watcher.subscribe(); + let weak_sess = Arc::downgrade(self); + tokio::spawn(async move { + loop { + match rx.recv().await { + Ok(FileWatcherEvent::SkillsChanged { .. }) => { + let Some(sess) = weak_sess.upgrade() else { + break; + }; + let event = Event { + id: sess.next_internal_sub_id(), + msg: EventMsg::SkillsUpdateAvailable, + }; + sess.send_event_raw(event).await; + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => break, + Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => continue, + } + } + }); + } + #[allow(clippy::too_many_arguments)] fn make_turn_context( auth_manager: Option>, @@ -654,27 +760,19 @@ impl Session { session_configuration: &SessionConfiguration, per_turn_config: Config, model_info: ModelInfo, - conversation_id: ThreadId, sub_id: String, - transport_manager: TransportManager, ) -> TurnContext { + let reasoning_effort = session_configuration.collaboration_mode.reasoning_effort(); + let reasoning_summary = session_configuration.model_reasoning_summary; let otel_manager = otel_manager.clone().with_model( session_configuration.collaboration_mode.model(), model_info.slug.as_str(), ); + let session_source = session_configuration.session_source.clone(); + let auth_manager_for_context = auth_manager; + let provider_for_context = provider; + let otel_manager_for_context = otel_manager; let per_turn_config = Arc::new(per_turn_config); - let client = ModelClient::new( - per_turn_config.clone(), - auth_manager, - model_info.clone(), - otel_manager, - provider, - session_configuration.collaboration_mode.reasoning_effort(), - session_configuration.model_reasoning_summary, - conversation_id, - session_configuration.session_source.clone(), - transport_manager, - ); let tools_config = ToolsConfig::new(&ToolsConfigParams { model_info: &model_info, @@ -682,10 +780,18 @@ impl Session { web_search_mode: per_turn_config.web_search_mode, }); + let cwd = session_configuration.cwd.clone(); TurnContext { sub_id, - client, - cwd: session_configuration.cwd.clone(), + config: per_turn_config.clone(), + auth_manager: auth_manager_for_context, + model_info: model_info.clone(), + otel_manager: otel_manager_for_context, + provider: provider_for_context, + reasoning_effort, + reasoning_summary, + session_source, + cwd, developer_instructions: session_configuration.developer_instructions.clone(), compact_prompt: session_configuration.compact_prompt.clone(), user_instructions: session_configuration.user_instructions.clone(), @@ -696,12 +802,14 @@ impl Session { windows_sandbox_level: session_configuration.windows_sandbox_level, shell_environment_policy: per_turn_config.shell_environment_policy.clone(), tools_config, + features: per_turn_config.features.clone(), ghost_snapshot: per_turn_config.ghost_snapshot.clone(), final_output_json_schema: None, codex_linux_sandbox_exe: per_turn_config.codex_linux_sandbox_exe.clone(), tool_call_gate: Arc::new(ReadinessFlag::new()), truncation_policy: model_info.truncation_policy.into(), dynamic_tools: session_configuration.dynamic_tools.clone(), + turn_metadata_header: OnceCell::new(), } } @@ -717,6 +825,7 @@ impl Session { initial_history: InitialHistory, session_source: SessionSource, skills_manager: Arc, + file_watcher: Arc, agent_control: AgentControl, ) -> anyhow::Result> { debug!( @@ -916,9 +1025,20 @@ impl Session { models_manager: Arc::clone(&models_manager), tool_approvals: Mutex::new(ApprovalStore::default()), skills_manager, + file_watcher, agent_control, state_db: state_db_ctx.clone(), - transport_manager: TransportManager::new(), + model_client: ModelClient::new( + Some(Arc::clone(&auth_manager)), + conversation_id, + session_configuration.provider.clone(), + session_configuration.session_source.clone(), + config.model_verbosity, + config.features.enabled(Feature::ResponsesWebsockets), + config.features.enabled(Feature::EnableRequestCompression), + config.features.enabled(Feature::RuntimeMetrics), + Self::build_model_client_beta_features_header(config.as_ref()), + ), }; let sess = Arc::new(Session { @@ -959,12 +1079,16 @@ impl Session { sess.send_event_raw(event).await; } + // Start the watcher after SessionConfigured so it cannot emit earlier events. + sess.start_file_watcher_listener(); + // Construct sandbox_state before initialize() so it can be sent to each // MCP server immediately after it becomes ready (avoiding blocking). let sandbox_state = SandboxState { sandbox_policy: session_configuration.sandbox_policy.get().clone(), codex_linux_sandbox_exe: config.codex_linux_sandbox_exe.clone(), sandbox_cwd: session_configuration.cwd.clone(), + use_linux_sandbox_bwrap: config.features.enabled(Feature::UseLinuxSandboxBwrap), }; let cancel_token = sess.mcp_startup_cancellation_token().await; @@ -1062,7 +1186,7 @@ impl Session { None } }) { - let curr = turn_context.client.get_model(); + let curr = turn_context.model_info.slug.as_str(); if prev != curr { warn!( "resuming session with different model: previous={prev}, current={curr}" @@ -1214,6 +1338,9 @@ impl Session { sandbox_policy: per_turn_config.sandbox_policy.get().clone(), codex_linux_sandbox_exe: per_turn_config.codex_linux_sandbox_exe.clone(), sandbox_cwd: per_turn_config.cwd.clone(), + use_linux_sandbox_bwrap: per_turn_config + .features + .enabled(Feature::UseLinuxSandboxBwrap), }; if let Err(e) = self .services @@ -1242,14 +1369,15 @@ impl Session { &session_configuration, per_turn_config, model_info, - self.conversation_id, sub_id, - self.services.transport_manager.clone(), ); + if let Some(final_schema) = final_output_json_schema { turn_context.final_output_json_schema = final_schema; } - Arc::new(turn_context) + let turn_context = Arc::new(turn_context); + turn_context.spawn_turn_metadata_header_task(); + turn_context } pub(crate) async fn new_default_turn(&self) -> Arc { @@ -1337,8 +1465,8 @@ impl Session { if let Some(personality) = next.personality && next.personality != previous.personality { - let model_info = next.client.get_model_info(); - let personality_message = Self::personality_message_for(&model_info, personality); + let model_info = &next.model_info; + let personality_message = Self::personality_message_for(model_info, personality); personality_message.map(|personality_message| { DeveloperInstructions::personality_spec_message(personality_message).into() }) @@ -1902,7 +2030,7 @@ impl Session { if self.features.enabled(Feature::Personality) && let Some(personality) = turn_context.personality { - let model_info = turn_context.client.get_model_info(); + let model_info = turn_context.model_info.clone(); let has_baked_personality = model_info.supports_personality() && base_instructions == model_info.get_model_instructions(Some(personality)); if !has_baked_personality @@ -1955,10 +2083,8 @@ impl Session { { let mut state = self.state.lock().await; if let Some(token_usage) = token_usage { - state.update_token_info_from_usage( - token_usage, - turn_context.client.get_model_context_window(), - ); + state + .update_token_info_from_usage(token_usage, turn_context.model_context_window()); } } self.send_token_count_event(turn_context).await; @@ -1989,7 +2115,7 @@ impl Session { }; if info.model_context_window.is_none() { - info.model_context_window = turn_context.client.get_model_context_window(); + info.model_context_window = turn_context.model_context_window(); } state.set_token_info(Some(info)); @@ -2047,7 +2173,7 @@ impl Session { } pub(crate) async fn set_total_tokens_full(&self, turn_context: &TurnContext) { - if let Some(context_window) = turn_context.client.get_model_context_window() { + if let Some(context_window) = turn_context.model_context_window() { let mut state = self.state.lock().await; state.set_token_usage_full(context_window); } @@ -2295,6 +2421,7 @@ impl Session { sandbox_policy: turn_context.sandbox_policy.clone(), codex_linux_sandbox_exe: turn_context.codex_linux_sandbox_exe.clone(), sandbox_cwd: turn_context.cwd.clone(), + use_linux_sandbox_bwrap: turn_context.features.enabled(Feature::UseLinuxSandboxBwrap), }; let cancel_token = self.reset_mcp_startup_cancellation_token().await; @@ -2469,6 +2596,22 @@ async fn submission_loop(sess: Arc, config: Arc, rx_sub: Receiv Op::ListSkills { cwds, force_reload } => { handlers::list_skills(&sess, sub.id.clone(), cwds, force_reload).await; } + Op::ListRemoteSkills => { + handlers::list_remote_skills(&sess, &config, sub.id.clone()).await; + } + Op::DownloadRemoteSkill { + hazelnut_id, + is_preload, + } => { + handlers::download_remote_skill( + &sess, + &config, + sub.id.clone(), + hazelnut_id, + is_preload, + ) + .await; + } Op::Undo => { handlers::undo(&sess, sub.id.clone()).await; } @@ -2535,9 +2678,12 @@ mod handlers { use codex_protocol::protocol::Event; use codex_protocol::protocol::EventMsg; use codex_protocol::protocol::ListCustomPromptsResponseEvent; + use codex_protocol::protocol::ListRemoteSkillsResponseEvent; use codex_protocol::protocol::ListSkillsResponseEvent; use codex_protocol::protocol::McpServerRefreshConfig; use codex_protocol::protocol::Op; + use codex_protocol::protocol::RemoteSkillDownloadedEvent; + use codex_protocol::protocol::RemoteSkillSummary; use codex_protocol::protocol::ReviewDecision; use codex_protocol::protocol::ReviewRequest; use codex_protocol::protocol::SkillsListEntry; @@ -2642,10 +2788,7 @@ mod handlers { // new_turn_with_sub_id already emits the error event. return; }; - current_context - .client - .get_otel_manager() - .user_prompt(&items); + current_context.otel_manager.user_prompt(&items); // Attempt to inject input into current task if let Err(items) = sess.inject_input(items).await { @@ -2895,6 +3038,77 @@ mod handlers { sess.send_event_raw(event).await; } + pub async fn list_remote_skills(sess: &Session, config: &Arc, sub_id: String) { + let response = crate::skills::remote::list_remote_skills(config) + .await + .map(|skills| { + skills + .into_iter() + .map(|skill| RemoteSkillSummary { + id: skill.id, + name: skill.name, + description: skill.description, + }) + .collect::>() + }); + + match response { + Ok(skills) => { + let event = Event { + id: sub_id, + msg: EventMsg::ListRemoteSkillsResponse(ListRemoteSkillsResponseEvent { + skills, + }), + }; + sess.send_event_raw(event).await; + } + Err(err) => { + let event = Event { + id: sub_id, + msg: EventMsg::Error(ErrorEvent { + message: format!("failed to list remote skills: {err}"), + codex_error_info: Some(CodexErrorInfo::Other), + }), + }; + sess.send_event_raw(event).await; + } + } + } + + pub async fn download_remote_skill( + sess: &Session, + config: &Arc, + sub_id: String, + hazelnut_id: String, + is_preload: bool, + ) { + match crate::skills::remote::download_remote_skill(config, hazelnut_id.as_str(), is_preload) + .await + { + Ok(result) => { + let event = Event { + id: sub_id, + msg: EventMsg::RemoteSkillDownloaded(RemoteSkillDownloadedEvent { + id: result.id, + name: result.name, + path: result.path, + }), + }; + sess.send_event_raw(event).await; + } + Err(err) => { + let event = Event { + id: sub_id, + msg: EventMsg::Error(ErrorEvent { + message: format!("failed to download remote skill {hazelnut_id}: {err}"), + codex_error_info: Some(CodexErrorInfo::Other), + }), + }; + sess.send_event_raw(event).await; + } + } + } + pub async fn undo(sess: &Arc, sub_id: String) { let turn_context = sess.new_default_turn_with_sub_id(sub_id).await; sess.spawn_task(turn_context, Vec::new(), UndoTask::new()) @@ -3116,7 +3330,7 @@ async fn spawn_review_thread( let model = config .review_model .clone() - .unwrap_or_else(|| parent_turn_context.client.get_model()); + .unwrap_or_else(|| parent_turn_context.model_info.slug.clone()); let review_model_info = sess .services .models_manager @@ -3135,8 +3349,8 @@ async fn spawn_review_thread( }); let review_prompt = resolved.prompt.clone(); - let provider = parent_turn_context.client.get_provider(); - let auth_manager = parent_turn_context.client.get_auth_manager(); + let provider = parent_turn_context.provider.clone(); + let auth_manager = parent_turn_context.auth_manager.clone(); let model_info = review_model_info.clone(); // Build per‑turn client with the requested model/family. @@ -3146,28 +3360,30 @@ async fn spawn_review_thread( per_turn_config.web_search_mode = Some(review_web_search_mode); let otel_manager = parent_turn_context - .client - .get_otel_manager() + .otel_manager + .clone() .with_model(model.as_str(), review_model_info.slug.as_str()); + let auth_manager_for_context = auth_manager.clone(); + let provider_for_context = provider.clone(); + let otel_manager_for_context = otel_manager.clone(); + let reasoning_effort = per_turn_config.model_reasoning_effort; + let reasoning_summary = per_turn_config.model_reasoning_summary; + let session_source = parent_turn_context.session_source.clone(); let per_turn_config = Arc::new(per_turn_config); - let client = ModelClient::new( - per_turn_config.clone(), - auth_manager, - model_info.clone(), - otel_manager, - provider, - per_turn_config.model_reasoning_effort, - per_turn_config.model_reasoning_summary, - sess.conversation_id, - parent_turn_context.client.get_session_source(), - parent_turn_context.client.transport_manager(), - ); let review_turn_context = TurnContext { sub_id: sub_id.to_string(), - client, + config: per_turn_config, + auth_manager: auth_manager_for_context, + model_info: model_info.clone(), + otel_manager: otel_manager_for_context, + provider: provider_for_context, + reasoning_effort, + reasoning_summary, + session_source, tools_config, + features: parent_turn_context.features.clone(), ghost_snapshot: parent_turn_context.ghost_snapshot.clone(), developer_instructions: None, user_instructions: None, @@ -3184,6 +3400,7 @@ async fn spawn_review_thread( tool_call_gate: Arc::new(ReadinessFlag::new()), dynamic_tools: parent_turn_context.dynamic_tools.clone(), truncation_policy: model_info.truncation_policy.into(), + turn_metadata_header: parent_turn_context.turn_metadata_header.clone(), }; // Seed the child task with the review prompt as the initial user message. @@ -3282,12 +3499,12 @@ pub(crate) async fn run_turn( return None; } - let model_info = turn_context.client.get_model_info(); + let model_info = turn_context.model_info.clone(); let auto_compact_limit = model_info.auto_compact_token_limit().unwrap_or(i64::MAX); let total_usage_tokens = sess.get_total_token_usage().await; let event = EventMsg::TurnStarted(TurnStartedEvent { - model_context_window: turn_context.client.get_model_context_window(), + model_context_window: turn_context.model_context_window(), collaboration_mode_kind: turn_context.collaboration_mode.mode, }); sess.send_event(&turn_context, event).await; @@ -3306,7 +3523,7 @@ pub(crate) async fn run_turn( || (HashMap::new(), HashMap::new()), |outcome| build_skill_name_counts(&outcome.skills, &outcome.disabled_paths), ); - let connector_slug_counts = if turn_context.client.config().features.enabled(Feature::Apps) { + let connector_slug_counts = if turn_context.config.features.enabled(Feature::Apps) { let mcp_tools = match sess .services .mcp_connection_manager @@ -3335,7 +3552,7 @@ pub(crate) async fn run_turn( }); let explicit_app_paths = collect_explicit_app_paths(&input); - let config = turn_context.client.config(); + let config = turn_context.config.clone(); if config .features .enabled(Feature::SkillEnvVarDependencyPrompt) @@ -3352,9 +3569,9 @@ pub(crate) async fn run_turn( ) .await; - let otel_manager = turn_context.client.get_otel_manager(); + let otel_manager = turn_context.otel_manager.clone(); let thread_id = sess.conversation_id.to_string(); - let tracking = build_track_events_context(turn_context.client.get_model(), thread_id); + let tracking = build_track_events_context(turn_context.model_info.slug.clone(), thread_id); let SkillInjections { items: skill_items, warnings: skill_warnings, @@ -3388,27 +3605,44 @@ pub(crate) async fn run_turn( // many turns, from the perspective of the user, it is a single turn. let turn_diff_tracker = Arc::new(tokio::sync::Mutex::new(TurnDiffTracker::new())); - let mut client_session = turn_context - .client - .new_session(Some(turn_context.cwd.clone())); + let turn_metadata_header = turn_context.resolve_turn_metadata_header().await; + // `ModelClientSession` is turn-scoped and caches WebSocket + sticky routing state, so we reuse + // one instance across retries within this turn. + let mut client_session = sess.services.model_client.new_session(); loop { // Note that pending_input would be something like a message the user // submitted through the UI while the model was running. Though the UI // may support this, the model might not. - let pending_input = sess + let pending_response_items = sess .get_pending_input() .await .into_iter() .map(ResponseItem::from) .collect::>(); + if !pending_response_items.is_empty() { + for response_item in pending_response_items { + if let Some(TurnItem::UserMessage(user_message)) = parse_turn_item(&response_item) { + // todo(aibrahim): move pending input to be UserInput only to keep TextElements. context: https://github.com/openai/codex/pull/10656#discussion_r2765522480 + sess.record_user_prompt_and_emit_turn_item( + turn_context.as_ref(), + &user_message.content, + response_item, + ) + .await; + } else { + sess.record_conversation_items( + &turn_context, + std::slice::from_ref(&response_item), + ) + .await; + } + } + } + // Construct the input that we will send to the model. - let sampling_request_input: Vec = { - sess.record_conversation_items(&turn_context, &pending_input) - .await; - sess.clone_history().await.for_prompt() - }; + let sampling_request_input: Vec = { sess.clone_history().await.for_prompt() }; let sampling_request_input_messages = sampling_request_input .iter() @@ -3427,6 +3661,7 @@ pub(crate) async fn run_turn( Arc::clone(&turn_context), Arc::clone(&turn_diff_tracker), &mut client_session, + turn_metadata_header.as_deref(), sampling_request_input, tool_selection, cancellation_token.child_token(), @@ -3508,7 +3743,7 @@ pub(crate) async fn run_turn( } async fn run_auto_compact(sess: &Arc, turn_context: &Arc) { - if should_use_remote_compact_task(sess.as_ref(), &turn_context.client.get_provider()) { + if should_use_remote_compact_task(sess.as_ref(), &turn_context.provider) { run_inline_remote_auto_compact_task(Arc::clone(sess), Arc::clone(turn_context)).await; } else { run_inline_auto_compact_task(Arc::clone(sess), Arc::clone(turn_context)).await; @@ -3613,11 +3848,12 @@ struct SamplingRequestToolSelection<'a> { skill_name_counts_lower: &'a HashMap, } +#[allow(clippy::too_many_arguments)] #[instrument(level = "trace", skip_all, fields( turn_id = %turn_context.sub_id, - model = %turn_context.client.get_model(), + model = %turn_context.model_info.slug, cwd = %turn_context.cwd.display() ) )] @@ -3626,6 +3862,7 @@ async fn run_sampling_request( turn_context: Arc, turn_diff_tracker: SharedTurnDiffTracker, client_session: &mut ModelClientSession, + turn_metadata_header: Option<&str>, input: Vec, tool_selection: SamplingRequestToolSelection<'_>, cancellation_token: CancellationToken, @@ -3638,7 +3875,7 @@ async fn run_sampling_request( .list_all_tools() .or_cancel(&cancellation_token) .await?; - let connectors_for_tools = if turn_context.client.config().features.enabled(Feature::Apps) { + let connectors_for_tools = if turn_context.config.features.enabled(Feature::Apps) { let connectors = connectors::accessible_connectors_from_mcp_tools(&mcp_tools); Some(filter_connectors_for_input( connectors, @@ -3663,10 +3900,7 @@ async fn run_sampling_request( turn_context.dynamic_tools.as_slice(), )); - let model_supports_parallel = turn_context - .client - .get_model_info() - .supports_parallel_tool_calls; + let model_supports_parallel = turn_context.model_info.supports_parallel_tool_calls; let base_instructions = sess.get_base_instructions().await; @@ -3686,6 +3920,7 @@ async fn run_sampling_request( Arc::clone(&sess), Arc::clone(&turn_context), client_session, + turn_metadata_header, Arc::clone(&turn_diff_tracker), &prompt, cancellation_token.child_token(), @@ -3714,8 +3949,10 @@ async fn run_sampling_request( } // Use the configured provider-specific stream retry budget. - let max_retries = turn_context.client.get_provider().stream_max_retries(); - if retries >= max_retries && client_session.try_switch_fallback_transport() { + let max_retries = turn_context.provider.stream_max_retries(); + if retries >= max_retries + && client_session.try_switch_fallback_transport(&turn_context.otel_manager) + { sess.send_event( &turn_context, EventMsg::Warning(WarningEvent { @@ -4160,7 +4397,7 @@ async fn drain_in_flight( skip_all, fields( turn_id = %turn_context.sub_id, - model = %turn_context.client.get_model() + model = %turn_context.model_info.slug ) )] async fn try_run_sampling_request( @@ -4168,6 +4405,7 @@ async fn try_run_sampling_request( sess: Arc, turn_context: Arc, client_session: &mut ModelClientSession, + turn_metadata_header: Option<&str>, turn_diff_tracker: SharedTurnDiffTracker, prompt: &Prompt, cancellation_token: CancellationToken, @@ -4177,11 +4415,11 @@ async fn try_run_sampling_request( cwd: turn_context.cwd.clone(), approval_policy: turn_context.approval_policy, sandbox_policy: turn_context.sandbox_policy.clone(), - model: turn_context.client.get_model(), + model: turn_context.model_info.slug.clone(), personality: turn_context.personality, collaboration_mode: Some(collaboration_mode), - effort: turn_context.client.get_reasoning_effort(), - summary: turn_context.client.get_reasoning_summary(), + effort: turn_context.reasoning_effort, + summary: turn_context.reasoning_summary, user_instructions: turn_context.user_instructions.clone(), developer_instructions: turn_context.developer_instructions.clone(), final_output_json_schema: turn_context.final_output_json_schema.clone(), @@ -4189,17 +4427,29 @@ async fn try_run_sampling_request( }); feedback_tags!( - model = turn_context.client.get_model(), + model = turn_context.model_info.slug.clone(), approval_policy = turn_context.approval_policy, sandbox_policy = turn_context.sandbox_policy, - effort = turn_context.client.get_reasoning_effort(), + effort = turn_context.reasoning_effort, auth_mode = sess.services.auth_manager.get_auth_mode(), features = sess.features.enabled_features(), ); sess.persist_rollout_items(&[rollout_item]).await; + let web_search_eligible = !matches!( + turn_context.config.web_search_mode, + Some(WebSearchMode::Disabled) + ); let mut stream = client_session - .stream(prompt) + .stream( + prompt, + &turn_context.model_info, + &turn_context.otel_manager, + turn_context.reasoning_effort, + turn_context.reasoning_summary, + web_search_eligible, + turn_metadata_header, + ) .instrument(trace_span!("stream_request")) .or_cancel(&cancellation_token) .await??; @@ -4481,6 +4731,7 @@ mod tests { use crate::tools::format_exec_output_str; use codex_protocol::ThreadId; + use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use crate::protocol::CompactedItem; @@ -5114,13 +5365,14 @@ mod tests { let got = FunctionCallOutputPayload::from(&ctr); let expected = FunctionCallOutputPayload { - content: serde_json::to_string(&json!({ - "ok": true, - "value": 42 - })) - .unwrap(), + body: FunctionCallOutputBody::Text( + serde_json::to_string(&json!({ + "ok": true, + "value": 42 + })) + .unwrap(), + ), success: Some(true), - ..Default::default() }; assert_eq!(expected, got); @@ -5157,10 +5409,10 @@ mod tests { let got = FunctionCallOutputPayload::from(&ctr); let expected = FunctionCallOutputPayload { - content: serde_json::to_string(&vec![text_block("hello"), text_block("world")]) - .unwrap(), + body: FunctionCallOutputBody::Text( + serde_json::to_string(&vec![text_block("hello"), text_block("world")]).unwrap(), + ), success: Some(true), - ..Default::default() }; assert_eq!(expected, got); @@ -5177,9 +5429,10 @@ mod tests { let got = FunctionCallOutputPayload::from(&ctr); let expected = FunctionCallOutputPayload { - content: serde_json::to_string(&json!({ "message": "bad" })).unwrap(), + body: FunctionCallOutputBody::Text( + serde_json::to_string(&json!({ "message": "bad" })).unwrap(), + ), success: Some(false), - ..Default::default() }; assert_eq!(expected, got); @@ -5196,9 +5449,10 @@ mod tests { let got = FunctionCallOutputPayload::from(&ctr); let expected = FunctionCallOutputPayload { - content: serde_json::to_string(&vec![text_block("alpha")]).unwrap(), + body: FunctionCallOutputBody::Text( + serde_json::to_string(&vec![text_block("alpha")]).unwrap(), + ), success: Some(true), - ..Default::default() }; assert_eq!(expected, got); @@ -5340,6 +5594,7 @@ mod tests { mark_state_initial_context_seeded(&mut state); let skills_manager = Arc::new(SkillsManager::new(config.codex_home.clone())); + let file_watcher = Arc::new(FileWatcher::noop()); let services = SessionServices { mcp_connection_manager: Arc::new(RwLock::new(McpConnectionManager::default())), mcp_startup_cancellation_token: Mutex::new(CancellationToken::new()), @@ -5358,9 +5613,20 @@ mod tests { models_manager: Arc::clone(&models_manager), tool_approvals: Mutex::new(ApprovalStore::default()), skills_manager, + file_watcher, agent_control, state_db: None, - transport_manager: TransportManager::new(), + model_client: ModelClient::new( + Some(auth_manager.clone()), + conversation_id, + session_configuration.provider.clone(), + session_configuration.session_source.clone(), + config.model_verbosity, + config.features.enabled(Feature::ResponsesWebsockets), + config.features.enabled(Feature::EnableRequestCompression), + config.features.enabled(Feature::RuntimeMetrics), + Session::build_model_client_beta_features_header(config.as_ref()), + ), }; let turn_context = Session::make_turn_context( @@ -5370,9 +5636,7 @@ mod tests { &session_configuration, per_turn_config, model_info, - conversation_id, "turn_id".to_string(), - services.transport_manager.clone(), ); let session = Session { @@ -5460,6 +5724,7 @@ mod tests { mark_state_initial_context_seeded(&mut state); let skills_manager = Arc::new(SkillsManager::new(config.codex_home.clone())); + let file_watcher = Arc::new(FileWatcher::noop()); let services = SessionServices { mcp_connection_manager: Arc::new(RwLock::new(McpConnectionManager::default())), mcp_startup_cancellation_token: Mutex::new(CancellationToken::new()), @@ -5478,9 +5743,20 @@ mod tests { models_manager: Arc::clone(&models_manager), tool_approvals: Mutex::new(ApprovalStore::default()), skills_manager, + file_watcher, agent_control, state_db: None, - transport_manager: TransportManager::new(), + model_client: ModelClient::new( + Some(Arc::clone(&auth_manager)), + conversation_id, + session_configuration.provider.clone(), + session_configuration.session_source.clone(), + config.model_verbosity, + config.features.enabled(Feature::ResponsesWebsockets), + config.features.enabled(Feature::EnableRequestCompression), + config.features.enabled(Feature::RuntimeMetrics), + Session::build_model_client_beta_features_header(config.as_ref()), + ), }; let turn_context = Arc::new(Session::make_turn_context( @@ -5490,9 +5766,7 @@ mod tests { &session_configuration, per_turn_config, model_info, - conversation_id, "turn_id".to_string(), - services.transport_manager.clone(), )); let session = Arc::new(Session { @@ -5648,7 +5922,7 @@ mod tests { } #[tokio::test] - async fn abort_gracefuly_emits_turn_aborted_only() { + async fn abort_gracefully_emits_turn_aborted_only() { let (sess, tc, rx) = make_session_and_context_with_rx().await; let input = vec![UserInput::Text { text: "hello".to_string(), @@ -6039,7 +6313,10 @@ mod tests { .await; let output = match resp2.expect("expected Ok result") { - ToolOutput::Function { content, .. } => content, + ToolOutput::Function { + body: FunctionCallOutputBody::Text(content), + .. + } => content, _ => panic!("unexpected tool output"), }; diff --git a/codex-rs/core/src/codex_delegate.rs b/codex-rs/core/src/codex_delegate.rs index 6499370fc3..cb5a31a0cc 100644 --- a/codex-rs/core/src/codex_delegate.rs +++ b/codex-rs/core/src/codex_delegate.rs @@ -54,6 +54,7 @@ pub(crate) async fn run_codex_thread_interactive( auth_manager, models_manager, Arc::clone(&parent_session.services.skills_manager), + Arc::clone(&parent_session.services.file_watcher), initial_history.unwrap_or(InitialHistory::New), SessionSource::SubAgent(SubAgentSource::Review), parent_session.services.agent_control.clone(), diff --git a/codex-rs/core/src/compact.rs b/codex-rs/core/src/compact.rs index ee94e4994b..91b95759c0 100644 --- a/codex-rs/core/src/compact.rs +++ b/codex-rs/core/src/compact.rs @@ -2,6 +2,7 @@ use std::sync::Arc; use crate::ModelProviderInfo; use crate::Prompt; +use crate::client::ModelClientSession; use crate::client_common::ResponseEvent; use crate::codex::Session; use crate::codex::TurnContext; @@ -19,6 +20,7 @@ use crate::truncate::TruncationPolicy; use crate::truncate::approx_token_count; use crate::truncate::truncate_text; use crate::util::backoff; +use codex_protocol::config_types::WebSearchMode; use codex_protocol::items::ContextCompactionItem; use codex_protocol::items::TurnItem; use codex_protocol::models::ContentItem; @@ -60,7 +62,7 @@ pub(crate) async fn run_compact_task( input: Vec, ) { let start_event = EventMsg::TurnStarted(TurnStartedEvent { - model_context_window: turn_context.client.get_model_context_window(), + model_context_window: turn_context.model_context_window(), collaboration_mode_kind: turn_context.collaboration_mode.mode, }); sess.send_event(&turn_context, start_event).await; @@ -85,8 +87,12 @@ async fn run_compact_task_inner( let mut truncated_count = 0usize; - let max_retries = turn_context.client.get_provider().stream_max_retries(); + let max_retries = turn_context.provider.stream_max_retries(); let mut retries = 0; + let turn_metadata_header = turn_context.resolve_turn_metadata_header().await; + let mut client_session = sess.services.model_client.new_session(); + // Reuse one client session so turn-scoped state (sticky routing, websocket append tracking) + // survives retries within this compact turn. // TODO: If we need to guarantee the persisted mode always matches the prompt used for this // turn, capture it in TurnContext at creation time. Using SessionConfiguration here avoids @@ -97,11 +103,11 @@ async fn run_compact_task_inner( cwd: turn_context.cwd.clone(), approval_policy: turn_context.approval_policy, sandbox_policy: turn_context.sandbox_policy.clone(), - model: turn_context.client.get_model(), + model: turn_context.model_info.slug.clone(), personality: turn_context.personality, collaboration_mode: Some(collaboration_mode), - effort: turn_context.client.get_reasoning_effort(), - summary: turn_context.client.get_reasoning_summary(), + effort: turn_context.reasoning_effort, + summary: turn_context.reasoning_summary, user_instructions: turn_context.user_instructions.clone(), developer_instructions: turn_context.developer_instructions.clone(), final_output_json_schema: turn_context.final_output_json_schema.clone(), @@ -119,7 +125,14 @@ async fn run_compact_task_inner( personality: turn_context.personality, ..Default::default() }; - let attempt_result = drain_to_completed(&sess, turn_context.as_ref(), &prompt).await; + let attempt_result = drain_to_completed( + &sess, + turn_context.as_ref(), + &mut client_session, + turn_metadata_header.as_deref(), + &prompt, + ) + .await; match attempt_result { Ok(()) => { @@ -335,12 +348,25 @@ fn build_compacted_history_with_limit( async fn drain_to_completed( sess: &Session, turn_context: &TurnContext, + client_session: &mut ModelClientSession, + turn_metadata_header: Option<&str>, prompt: &Prompt, ) -> CodexResult<()> { - let mut client_session = turn_context - .client - .new_session(Some(turn_context.cwd.clone())); - let mut stream = client_session.stream(prompt).await?; + let web_search_eligible = !matches!( + turn_context.config.web_search_mode, + Some(WebSearchMode::Disabled) + ); + let mut stream = client_session + .stream( + prompt, + &turn_context.model_info, + &turn_context.otel_manager, + turn_context.reasoning_effort, + turn_context.reasoning_summary, + web_search_eligible, + turn_metadata_header, + ) + .await?; loop { let maybe_event = stream.next().await; let Some(event) = maybe_event else { diff --git a/codex-rs/core/src/compact_remote.rs b/codex-rs/core/src/compact_remote.rs index 12bc769ce2..f46084bd9c 100644 --- a/codex-rs/core/src/compact_remote.rs +++ b/codex-rs/core/src/compact_remote.rs @@ -24,7 +24,7 @@ pub(crate) async fn run_inline_remote_auto_compact_task( pub(crate) async fn run_remote_compact_task(sess: Arc, turn_context: Arc) { let start_event = EventMsg::TurnStarted(TurnStartedEvent { - model_context_window: turn_context.client.get_model_context_window(), + model_context_window: turn_context.model_context_window(), collaboration_mode_kind: turn_context.collaboration_mode.mode, }); sess.send_event(&turn_context, start_event).await; @@ -76,9 +76,14 @@ async fn run_remote_compact_task_inner_impl( output_schema: None, }; - let mut new_history = turn_context - .client - .compact_conversation_history(&prompt) + let mut new_history = sess + .services + .model_client + .compact_conversation_history( + &prompt, + &turn_context.model_info, + &turn_context.otel_manager, + ) .await?; if !ghost_snapshots.is_empty() { @@ -104,7 +109,7 @@ fn trim_function_call_history_to_fit_context_window( turn_context: &TurnContext, ) -> usize { let mut deleted_items = 0usize; - let Some(context_window) = turn_context.client.get_model_context_window() else { + let Some(context_window) = turn_context.model_context_window() else { return deleted_items; }; diff --git a/codex-rs/core/src/config/mod.rs b/codex-rs/core/src/config/mod.rs index 3e2ed804af..e22cfe65a1 100644 --- a/codex-rs/core/src/config/mod.rs +++ b/codex-rs/core/src/config/mod.rs @@ -1094,7 +1094,11 @@ impl ConfigToml { profile_sandbox_mode: Option, windows_sandbox_level: WindowsSandboxLevel, resolved_cwd: &Path, + sandbox_policy_constraint: Option<&Constrained>, ) -> SandboxPolicyResolution { + let sandbox_mode_was_explicit = sandbox_mode_override.is_some() + || profile_sandbox_mode.is_some() + || self.sandbox_mode.is_some(); let resolved_sandbox_mode = sandbox_mode_override .or(profile_sandbox_mode) .or(self.sandbox_mode) @@ -1128,13 +1132,30 @@ impl ConfigToml { SandboxMode::DangerFullAccess => SandboxPolicy::DangerFullAccess, }; let mut forced_auto_mode_downgraded_on_windows = false; - if cfg!(target_os = "windows") - && matches!(resolved_sandbox_mode, SandboxMode::WorkspaceWrite) - // If the experimental Windows sandbox is enabled, do not force a downgrade. - && windows_sandbox_level == codex_protocol::config_types::WindowsSandboxLevel::Disabled + let mut downgrade_workspace_write_if_unsupported = |policy: &mut SandboxPolicy| { + if cfg!(target_os = "windows") + // If the experimental Windows sandbox is enabled, do not force a downgrade. + && windows_sandbox_level + == codex_protocol::config_types::WindowsSandboxLevel::Disabled + && matches!(&*policy, SandboxPolicy::WorkspaceWrite { .. }) + { + *policy = SandboxPolicy::new_read_only_policy(); + forced_auto_mode_downgraded_on_windows = true; + } + }; + if matches!(resolved_sandbox_mode, SandboxMode::WorkspaceWrite) { + downgrade_workspace_write_if_unsupported(&mut sandbox_policy); + } + if !sandbox_mode_was_explicit + && let Some(constraint) = sandbox_policy_constraint + && let Err(err) = constraint.can_set(&sandbox_policy) { - sandbox_policy = SandboxPolicy::new_read_only_policy(); - forced_auto_mode_downgraded_on_windows = true; + tracing::warn!( + error = %err, + "default sandbox policy is disallowed by requirements; falling back to required default" + ); + sandbox_policy = constraint.get().clone(); + downgrade_workspace_write_if_unsupported(&mut sandbox_policy); } SandboxPolicyResolution { policy: sandbox_policy, @@ -1364,6 +1385,9 @@ impl Config { let active_project = cfg .get_active_project(&resolved_cwd) .unwrap_or(ProjectConfig { trust_level: None }); + let sandbox_mode_was_explicit = sandbox_mode.is_some() + || config_profile.sandbox_mode.is_some() + || cfg.sandbox_mode.is_some(); let windows_sandbox_level = WindowsSandboxLevel::from_features(&features); let SandboxPolicyResolution { @@ -1374,6 +1398,7 @@ impl Config { config_profile.sandbox_mode, windows_sandbox_level, &resolved_cwd, + Some(&requirements.sandbox_policy), ); if let SandboxPolicy::WorkspaceWrite { writable_roots, .. } = &mut sandbox_policy { for path in additional_writable_roots { @@ -1382,7 +1407,10 @@ impl Config { } } } - let approval_policy = approval_policy_override + let approval_policy_was_explicit = approval_policy_override.is_some() + || config_profile.approval_policy.is_some() + || cfg.approval_policy.is_some(); + let mut approval_policy = approval_policy_override .or(config_profile.approval_policy) .or(cfg.approval_policy) .unwrap_or_else(|| { @@ -1394,16 +1422,20 @@ impl Config { AskForApproval::default() } }); + if !approval_policy_was_explicit + && let Err(err) = requirements.approval_policy.can_set(&approval_policy) + { + tracing::warn!( + error = %err, + "default approval policy is disallowed by requirements; falling back to required default" + ); + approval_policy = requirements.approval_policy.value(); + } let web_search_mode = resolve_web_search_mode(&cfg, &config_profile, &features); // TODO(dylan): We should be able to leverage ConfigLayerStack so that // we can reliably check this at every config level. - let did_user_set_custom_approval_policy_or_sandbox_mode = approval_policy_override - .is_some() - || config_profile.approval_policy.is_some() - || cfg.approval_policy.is_some() - || sandbox_mode.is_some() - || config_profile.sandbox_mode.is_some() - || cfg.sandbox_mode.is_some(); + let did_user_set_custom_approval_policy_or_sandbox_mode = + approval_policy_was_explicit || sandbox_mode_was_explicit; let mut model_providers = built_in_model_providers(); // Merge user-defined providers into the built-in list. @@ -1556,9 +1588,9 @@ impl Config { model_provider_id, model_provider, cwd: resolved_cwd, - approval_policy: constrained_approval_policy, - sandbox_policy: constrained_sandbox_policy, - enforce_residency, + approval_policy: constrained_approval_policy.value, + sandbox_policy: constrained_sandbox_policy.value, + enforce_residency: enforce_residency.value, did_user_set_custom_approval_policy_or_sandbox_mode, forced_auto_mode_downgraded_on_windows, shell_environment_policy, @@ -1919,6 +1951,7 @@ network_access = false # This should be ignored. None, WindowsSandboxLevel::Disabled, &PathBuf::from("/tmp/test"), + None, ); assert_eq!( resolution, @@ -1943,6 +1976,7 @@ network_access = true # This should be ignored. None, WindowsSandboxLevel::Disabled, &PathBuf::from("/tmp/test"), + None, ); assert_eq!( resolution, @@ -1975,6 +2009,7 @@ exclude_slash_tmp = true None, WindowsSandboxLevel::Disabled, &PathBuf::from("/tmp/test"), + None, ); if cfg!(target_os = "windows") { assert_eq!( @@ -2024,6 +2059,7 @@ trust_level = "trusted" None, WindowsSandboxLevel::Disabled, &PathBuf::from("/tmp/test"), + None, ); if cfg!(target_os = "windows") { assert_eq!( @@ -3826,7 +3862,7 @@ model_verbosity = "high" forced_login_method: None, include_apply_patch_tool: false, web_search_mode: None, - use_experimental_unified_exec_tool: false, + use_experimental_unified_exec_tool: !cfg!(windows), ghost_snapshot: GhostSnapshotConfig::default(), features: Features::with_defaults(), suppress_unstable_features_warning: false, @@ -3911,7 +3947,7 @@ model_verbosity = "high" forced_login_method: None, include_apply_patch_tool: false, web_search_mode: None, - use_experimental_unified_exec_tool: false, + use_experimental_unified_exec_tool: !cfg!(windows), ghost_snapshot: GhostSnapshotConfig::default(), features: Features::with_defaults(), suppress_unstable_features_warning: false, @@ -4011,7 +4047,7 @@ model_verbosity = "high" forced_login_method: None, include_apply_patch_tool: false, web_search_mode: None, - use_experimental_unified_exec_tool: false, + use_experimental_unified_exec_tool: !cfg!(windows), ghost_snapshot: GhostSnapshotConfig::default(), features: Features::with_defaults(), suppress_unstable_features_warning: false, @@ -4097,7 +4133,7 @@ model_verbosity = "high" forced_login_method: None, include_apply_patch_tool: false, web_search_mode: None, - use_experimental_unified_exec_tool: false, + use_experimental_unified_exec_tool: !cfg!(windows), ghost_snapshot: GhostSnapshotConfig::default(), features: Features::with_defaults(), suppress_unstable_features_warning: false, @@ -4331,6 +4367,7 @@ trust_level = "untrusted" None, WindowsSandboxLevel::Disabled, &PathBuf::from("/tmp/test"), + None, ); // Verify that untrusted projects get WorkspaceWrite (or ReadOnly on Windows due to downgrade) @@ -4351,6 +4388,103 @@ trust_level = "untrusted" Ok(()) } + #[test] + fn derive_sandbox_policy_falls_back_to_constraint_value_for_implicit_defaults() + -> anyhow::Result<()> { + let project_dir = TempDir::new()?; + let project_path = project_dir.path().to_path_buf(); + let project_key = project_path.to_string_lossy().to_string(); + let cfg = ConfigToml { + projects: Some(HashMap::from([( + project_key, + ProjectConfig { + trust_level: Some(TrustLevel::Trusted), + }, + )])), + ..Default::default() + }; + let constrained = Constrained::new(SandboxPolicy::DangerFullAccess, |candidate| { + if matches!(candidate, SandboxPolicy::DangerFullAccess) { + Ok(()) + } else { + Err(ConstraintError::InvalidValue { + field_name: "sandbox_mode", + candidate: format!("{candidate:?}"), + allowed: "[DangerFullAccess]".to_string(), + requirement_source: RequirementSource::Unknown, + }) + } + })?; + + let resolution = cfg.derive_sandbox_policy( + None, + None, + WindowsSandboxLevel::Disabled, + &project_path, + Some(&constrained), + ); + + assert_eq!(resolution.policy, SandboxPolicy::DangerFullAccess); + Ok(()) + } + + #[test] + fn derive_sandbox_policy_preserves_windows_downgrade_for_unsupported_fallback() + -> anyhow::Result<()> { + let project_dir = TempDir::new()?; + let project_path = project_dir.path().to_path_buf(); + let project_key = project_path.to_string_lossy().to_string(); + let cfg = ConfigToml { + projects: Some(HashMap::from([( + project_key, + ProjectConfig { + trust_level: Some(TrustLevel::Trusted), + }, + )])), + ..Default::default() + }; + let constrained = + Constrained::new(SandboxPolicy::new_workspace_write_policy(), |candidate| { + if matches!(candidate, SandboxPolicy::WorkspaceWrite { .. }) { + Ok(()) + } else { + Err(ConstraintError::InvalidValue { + field_name: "sandbox_mode", + candidate: format!("{candidate:?}"), + allowed: "[WorkspaceWrite]".to_string(), + requirement_source: RequirementSource::Unknown, + }) + } + })?; + + let resolution = cfg.derive_sandbox_policy( + None, + None, + WindowsSandboxLevel::Disabled, + &project_path, + Some(&constrained), + ); + + if cfg!(target_os = "windows") { + assert_eq!( + resolution, + SandboxPolicyResolution { + policy: SandboxPolicy::ReadOnly, + forced_auto_mode_downgraded_on_windows: true, + } + ); + } else { + assert_eq!( + resolution, + SandboxPolicyResolution { + policy: SandboxPolicy::new_workspace_write_policy(), + forced_auto_mode_downgraded_on_windows: false, + } + ); + } + Ok(()) + } + #[test] fn test_resolve_oss_provider_explicit_override() { let config_toml = ConfigToml::default(); @@ -4506,6 +4640,125 @@ mcp_oauth_callback_port = 5678 Ok(()) } + + #[tokio::test] + async fn requirements_disallowing_default_sandbox_falls_back_to_required_default() + -> std::io::Result<()> { + let codex_home = TempDir::new()?; + + let config = ConfigBuilder::default() + .codex_home(codex_home.path().to_path_buf()) + .cloud_requirements(CloudRequirementsLoader::new(async { + Some(crate::config_loader::ConfigRequirementsToml { + allowed_sandbox_modes: Some(vec![ + crate::config_loader::SandboxModeRequirement::ReadOnly, + ]), + ..Default::default() + }) + })) + .build() + .await?; + + assert_eq!(*config.sandbox_policy.get(), SandboxPolicy::ReadOnly); + Ok(()) + } + + #[tokio::test] + async fn explicit_sandbox_mode_still_errors_when_disallowed_by_requirements() + -> std::io::Result<()> { + let codex_home = TempDir::new()?; + std::fs::write( + codex_home.path().join(CONFIG_TOML_FILE), + r#"sandbox_mode = "danger-full-access" +"#, + )?; + + let requirements = crate::config_loader::ConfigRequirementsToml { + allowed_approval_policies: None, + allowed_sandbox_modes: Some(vec![ + crate::config_loader::SandboxModeRequirement::ReadOnly, + ]), + mcp_servers: None, + rules: None, + enforce_residency: None, + }; + + let err = ConfigBuilder::default() + .codex_home(codex_home.path().to_path_buf()) + .fallback_cwd(Some(codex_home.path().to_path_buf())) + .cloud_requirements(CloudRequirementsLoader::new( + async move { Some(requirements) }, + )) + .build() + .await + .expect_err("explicit disallowed mode should still fail"); + assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput); + let message = err.to_string(); + assert!(message.contains("invalid value for `sandbox_mode`")); + assert!(message.contains("set by cloud requirements")); + Ok(()) + } + + #[tokio::test] + async fn requirements_disallowing_default_approval_falls_back_to_required_default() + -> std::io::Result<()> { + let codex_home = TempDir::new()?; + let workspace = TempDir::new()?; + let workspace_key = workspace.path().to_string_lossy().replace('\\', "\\\\"); + std::fs::write( + codex_home.path().join(CONFIG_TOML_FILE), + format!( + r#" +[projects."{workspace_key}"] +trust_level = "untrusted" +"# + ), + )?; + + let config = ConfigBuilder::default() + .codex_home(codex_home.path().to_path_buf()) + .fallback_cwd(Some(workspace.path().to_path_buf())) + .cloud_requirements(CloudRequirementsLoader::new(async { + Some(crate::config_loader::ConfigRequirementsToml { + allowed_approval_policies: Some(vec![AskForApproval::OnRequest]), + ..Default::default() + }) + })) + .build() + .await?; + + assert_eq!(config.approval_policy.value(), AskForApproval::OnRequest); + Ok(()) + } + + #[tokio::test] + async fn explicit_approval_policy_still_errors_when_disallowed_by_requirements() + -> std::io::Result<()> { + let codex_home = TempDir::new()?; + std::fs::write( + codex_home.path().join(CONFIG_TOML_FILE), + r#"approval_policy = "untrusted" +"#, + )?; + + let err = ConfigBuilder::default() + .codex_home(codex_home.path().to_path_buf()) + .fallback_cwd(Some(codex_home.path().to_path_buf())) + .cloud_requirements(CloudRequirementsLoader::new(async { + Some(crate::config_loader::ConfigRequirementsToml { + allowed_approval_policies: Some(vec![AskForApproval::OnRequest]), + ..Default::default() + }) + })) + .build() + .await + .expect_err("explicit disallowed approval policy should fail"); + assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput); + let message = err.to_string(); + assert!(message.contains("invalid value for `approval_policy`")); + assert!(message.contains("set by cloud requirements")); + Ok(()) + } } #[cfg(test)] diff --git a/codex-rs/core/src/config_loader/config_requirements.rs b/codex-rs/core/src/config_loader/config_requirements.rs index 478c74193e..b3e6043288 100644 --- a/codex-rs/core/src/config_loader/config_requirements.rs +++ b/codex-rs/core/src/config_loader/config_requirements.rs @@ -44,29 +44,67 @@ impl fmt::Display for RequirementSource { } } +#[derive(Debug, Clone, PartialEq)] +pub struct ConstrainedWithSource { + pub value: Constrained, + pub source: Option, +} + +impl ConstrainedWithSource { + pub fn new(value: Constrained, source: Option) -> Self { + Self { value, source } + } +} + +impl std::ops::Deref for ConstrainedWithSource { + type Target = Constrained; + + fn deref(&self) -> &Self::Target { + &self.value + } +} + +impl std::ops::DerefMut for ConstrainedWithSource { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.value + } +} + /// Normalized version of [`ConfigRequirementsToml`] after deserialization and /// normalization. #[derive(Debug, Clone, PartialEq)] pub struct ConfigRequirements { - pub approval_policy: Constrained, - pub sandbox_policy: Constrained, + pub approval_policy: ConstrainedWithSource, + pub sandbox_policy: ConstrainedWithSource, pub mcp_servers: Option>>, pub(crate) exec_policy: Option>, - pub enforce_residency: Constrained>, + pub enforce_residency: ConstrainedWithSource>, } impl Default for ConfigRequirements { fn default() -> Self { Self { - approval_policy: Constrained::allow_any_from_default(), - sandbox_policy: Constrained::allow_any(SandboxPolicy::ReadOnly), + approval_policy: ConstrainedWithSource::new( + Constrained::allow_any_from_default(), + None, + ), + sandbox_policy: ConstrainedWithSource::new( + Constrained::allow_any(SandboxPolicy::ReadOnly), + None, + ), mcp_servers: None, exec_policy: None, - enforce_residency: Constrained::allow_any(None), + enforce_residency: ConstrainedWithSource::new(Constrained::allow_any(None), None), } } } +impl ConfigRequirements { + pub fn exec_policy_source(&self) -> Option<&RequirementSource> { + self.exec_policy.as_ref().map(|policy| &policy.source) + } +} + #[derive(Deserialize, Debug, Clone, PartialEq, Eq)] #[serde(untagged)] pub enum McpServerIdentity { @@ -228,7 +266,7 @@ impl TryFrom for ConfigRequirements { enforce_residency, } = toml; - let approval_policy: Constrained = match allowed_approval_policies { + let approval_policy = match allowed_approval_policies { Some(Sourced { value: policies, source: requirement_source, @@ -237,7 +275,8 @@ impl TryFrom for ConfigRequirements { return Err(ConstraintError::empty_field("allowed_approval_policies")); }; - Constrained::new(initial_value, move |candidate| { + let requirement_source_for_error = requirement_source.clone(); + let constrained = Constrained::new(initial_value, move |candidate| { if policies.contains(candidate) { Ok(()) } else { @@ -245,12 +284,13 @@ impl TryFrom for ConfigRequirements { field_name: "approval_policy", candidate: format!("{candidate:?}"), allowed: format!("{policies:?}"), - requirement_source: requirement_source.clone(), + requirement_source: requirement_source_for_error.clone(), }) } - })? + })?; + ConstrainedWithSource::new(constrained, Some(requirement_source)) } - None => Constrained::allow_any_from_default(), + None => ConstrainedWithSource::new(Constrained::allow_any_from_default(), None), }; // TODO(gt): `ConfigRequirementsToml` should let the author specify the @@ -261,7 +301,7 @@ impl TryFrom for ConfigRequirements { // additional parameters. Ultimately, we should expand the config // format to allow specifying those parameters. let default_sandbox_policy = SandboxPolicy::ReadOnly; - let sandbox_policy: Constrained = match allowed_sandbox_modes { + let sandbox_policy = match allowed_sandbox_modes { Some(Sourced { value: modes, source: requirement_source, @@ -275,7 +315,8 @@ impl TryFrom for ConfigRequirements { }); }; - Constrained::new(default_sandbox_policy, move |candidate| { + let requirement_source_for_error = requirement_source.clone(); + let constrained = Constrained::new(default_sandbox_policy, move |candidate| { let mode = match candidate { SandboxPolicy::ReadOnly => SandboxModeRequirement::ReadOnly, SandboxPolicy::WorkspaceWrite { .. } => { @@ -293,12 +334,15 @@ impl TryFrom for ConfigRequirements { field_name: "sandbox_mode", candidate: format!("{mode:?}"), allowed: format!("{modes:?}"), - requirement_source: requirement_source.clone(), + requirement_source: requirement_source_for_error.clone(), }) } - })? + })?; + ConstrainedWithSource::new(constrained, Some(requirement_source)) + } + None => { + ConstrainedWithSource::new(Constrained::allow_any(default_sandbox_policy), None) } - None => Constrained::allow_any(default_sandbox_policy), }; let exec_policy = match rules { Some(Sourced { value, source }) => { @@ -313,13 +357,14 @@ impl TryFrom for ConfigRequirements { None => None, }; - let enforce_residency: Constrained> = match enforce_residency { + let enforce_residency = match enforce_residency { Some(Sourced { value: residency, source: requirement_source, }) => { let required = Some(residency); - Constrained::new(required, move |candidate| { + let requirement_source_for_error = requirement_source.clone(); + let constrained = Constrained::new(required, move |candidate| { if candidate == &required { Ok(()) } else { @@ -327,12 +372,13 @@ impl TryFrom for ConfigRequirements { field_name: "enforce_residency", candidate: format!("{candidate:?}"), allowed: format!("{required:?}"), - requirement_source: requirement_source.clone(), + requirement_source: requirement_source_for_error.clone(), }) } - })? + })?; + ConstrainedWithSource::new(constrained, Some(requirement_source)) } - None => Constrained::allow_any(None), + None => ConstrainedWithSource::new(Constrained::allow_any(None), None), }; Ok(ConfigRequirements { approval_policy, @@ -563,6 +609,34 @@ mod tests { Ok(()) } + #[test] + fn constrained_fields_store_requirement_source() -> Result<()> { + let source: ConfigRequirementsToml = from_str( + r#" + allowed_approval_policies = ["on-request"] + allowed_sandbox_modes = ["read-only"] + enforce_residency = "us" + "#, + )?; + + let source_location = RequirementSource::CloudRequirements; + let mut target = ConfigRequirementsWithSources::default(); + target.merge_unset_fields(source_location.clone(), source); + let requirements = ConfigRequirements::try_from(target)?; + + assert_eq!( + requirements.approval_policy.source, + Some(source_location.clone()) + ); + assert_eq!( + requirements.sandbox_policy.source, + Some(source_location.clone()) + ); + assert_eq!(requirements.enforce_residency.source, Some(source_location)); + + Ok(()) + } + #[test] fn deserialize_allowed_approval_policies() -> Result<()> { let toml_str = r#" diff --git a/codex-rs/core/src/config_loader/mod.rs b/codex-rs/core/src/config_loader/mod.rs index d8813e23d7..28fda87357 100644 --- a/codex-rs/core/src/config_loader/mod.rs +++ b/codex-rs/core/src/config_loader/mod.rs @@ -34,6 +34,7 @@ use toml::Value as TomlValue; pub use cloud_requirements::CloudRequirementsLoader; pub use config_requirements::ConfigRequirements; pub use config_requirements::ConfigRequirementsToml; +pub use config_requirements::ConstrainedWithSource; pub use config_requirements::McpServerIdentity; pub use config_requirements::McpServerRequirement; pub use config_requirements::RequirementSource; @@ -71,8 +72,8 @@ const DEFAULT_PROJECT_ROOT_MARKERS: &[&str] = &[".git"]; /// configuration layers in the following order, but a constraint defined in an /// earlier layer cannot be overridden by a later layer: /// -/// - admin: managed preferences (*) /// - cloud: managed cloud requirements +/// - admin: managed preferences (*) /// - system `/etc/codex/requirements.toml` /// /// For backwards compatibility, we also load from @@ -106,6 +107,11 @@ pub async fn load_config_layers_state( ) -> io::Result { let mut config_requirements_toml = ConfigRequirementsWithSources::default(); + if let Some(requirements) = cloud_requirements.get().await { + config_requirements_toml + .merge_unset_fields(RequirementSource::CloudRequirements, requirements); + } + #[cfg(target_os = "macos")] macos::load_managed_admin_requirements_toml( &mut config_requirements_toml, @@ -115,11 +121,6 @@ pub async fn load_config_layers_state( ) .await?; - if let Some(requirements) = cloud_requirements.get().await { - config_requirements_toml - .merge_unset_fields(RequirementSource::CloudRequirements, requirements); - } - // Honor /etc/codex/requirements.toml. if cfg!(unix) { load_requirements_toml( diff --git a/codex-rs/core/src/config_loader/tests.rs b/codex-rs/core/src/config_loader/tests.rs index cbd49f131c..2d7e36d6ef 100644 --- a/codex-rs/core/src/config_loader/tests.rs +++ b/codex-rs/core/src/config_loader/tests.rs @@ -487,6 +487,59 @@ enforce_residency = "us" Ok(()) } +#[cfg(target_os = "macos")] +#[tokio::test] +async fn cloud_requirements_take_precedence_over_mdm_requirements() -> anyhow::Result<()> { + use base64::Engine; + + let tmp = tempdir()?; + let state = load_config_layers_state( + tmp.path(), + Some(AbsolutePathBuf::try_from(tmp.path())?), + &[] as &[(String, TomlValue)], + LoaderOverrides { + macos_managed_config_requirements_base64: Some( + base64::prelude::BASE64_STANDARD.encode( + r#" +allowed_approval_policies = ["on-request"] +"# + .as_bytes(), + ), + ), + ..LoaderOverrides::default() + }, + CloudRequirementsLoader::new(async { + Some(ConfigRequirementsToml { + allowed_approval_policies: Some(vec![AskForApproval::Never]), + allowed_sandbox_modes: None, + mcp_servers: None, + rules: None, + enforce_residency: None, + }) + }), + ) + .await?; + + assert_eq!( + state.requirements().approval_policy.value(), + AskForApproval::Never + ); + assert_eq!( + state + .requirements() + .approval_policy + .can_set(&AskForApproval::OnRequest), + Err(ConstraintError::InvalidValue { + field_name: "approval_policy", + candidate: "OnRequest".into(), + allowed: "[Never]".into(), + requirement_source: RequirementSource::CloudRequirements, + }) + ); + + Ok(()) +} + #[tokio::test(flavor = "current_thread")] async fn cloud_requirements_are_not_overwritten_by_system_requirements() -> anyhow::Result<()> { let tmp = tempdir()?; diff --git a/codex-rs/core/src/connectors.rs b/codex-rs/core/src/connectors.rs index edf3c63aa5..afe03fbbe7 100644 --- a/codex-rs/core/src/connectors.rs +++ b/codex-rs/core/src/connectors.rs @@ -43,6 +43,7 @@ pub async fn list_accessible_connectors_from_mcp_tools( sandbox_policy: SandboxPolicy::ReadOnly, codex_linux_sandbox_exe: config.codex_linux_sandbox_exe.clone(), sandbox_cwd: env::current_dir().unwrap_or_else(|_| PathBuf::from("/")), + use_linux_sandbox_bwrap: config.features.enabled(Feature::UseLinuxSandboxBwrap), }; mcp_connection_manager diff --git a/codex-rs/core/src/context_manager/history.rs b/codex-rs/core/src/context_manager/history.rs index a29f7df7e0..b1bf4086da 100644 --- a/codex-rs/core/src/context_manager/history.rs +++ b/codex-rs/core/src/context_manager/history.rs @@ -10,6 +10,7 @@ use crate::truncate::truncate_function_output_items_with_policy; use crate::truncate::truncate_text; use crate::user_shell_command::is_user_shell_command_text; use codex_protocol::models::ContentItem; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputContentItem; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseItem; @@ -85,10 +86,8 @@ impl ContextManager { // Estimate token usage using byte-based heuristics from the truncation helpers. // This is a coarse lower bound, not a tokenizer-accurate count. pub(crate) fn estimate_token_count(&self, turn_context: &TurnContext) -> Option { - let model_info = turn_context.client.get_model_info(); - let personality = turn_context - .personality - .or(turn_context.client.config().personality); + let model_info = &turn_context.model_info; + let personality = turn_context.personality.or(turn_context.config.personality); let base_instructions = model_info.get_model_instructions(personality); let base_tokens = i64::try_from(approx_token_count(&base_instructions)).unwrap_or(i64::MAX); @@ -136,7 +135,7 @@ impl ContextManager { match &mut self.items[index] { ResponseItem::FunctionCallOutput { output, .. } => { - let Some(content_items) = output.content_items.as_mut() else { + let Some(content_items) = output.content_items_mut() else { return false; }; let mut replaced = false; @@ -270,19 +269,23 @@ impl ContextManager { let policy_with_serialization_budget = policy * 1.2; match item { ResponseItem::FunctionCallOutput { call_id, output } => { - let truncated = - truncate_text(output.content.as_str(), policy_with_serialization_budget); - let truncated_items = output.content_items.as_ref().map(|items| { - truncate_function_output_items_with_policy( - items, - policy_with_serialization_budget, - ) - }); + let body = match &output.body { + FunctionCallOutputBody::Text(content) => FunctionCallOutputBody::Text( + truncate_text(content, policy_with_serialization_budget), + ), + FunctionCallOutputBody::ContentItems(items) => { + FunctionCallOutputBody::ContentItems( + truncate_function_output_items_with_policy( + items, + policy_with_serialization_budget, + ), + ) + } + }; ResponseItem::FunctionCallOutput { call_id: call_id.clone(), output: FunctionCallOutputPayload { - content: truncated, - content_items: truncated_items, + body, success: output.success, }, } diff --git a/codex-rs/core/src/context_manager/history_tests.rs b/codex-rs/core/src/context_manager/history_tests.rs index a6eba62f1a..c1801e202d 100644 --- a/codex-rs/core/src/context_manager/history_tests.rs +++ b/codex-rs/core/src/context_manager/history_tests.rs @@ -3,6 +3,7 @@ use crate::truncate; use crate::truncate::TruncationPolicy; use codex_git::GhostCommit; use codex_protocol::models::ContentItem; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputContentItem; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::LocalShellAction; @@ -63,10 +64,7 @@ fn user_input_text_msg(text: &str) -> ResponseItem { fn function_call_output(call_id: &str, content: &str) -> ResponseItem { ResponseItem::FunctionCallOutput { call_id: call_id.to_string(), - output: FunctionCallOutputPayload { - content: content.to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text(content.to_string()), } } @@ -263,10 +261,7 @@ fn remove_first_item_removes_matching_output_for_function_call() { }, ResponseItem::FunctionCallOutput { call_id: "call-1".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, ]; let mut h = create_history_with_items(items); @@ -279,10 +274,7 @@ fn remove_first_item_removes_matching_call_for_output() { let items = vec![ ResponseItem::FunctionCallOutput { call_id: "call-2".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, ResponseItem::FunctionCall { id: None, @@ -308,10 +300,7 @@ fn remove_last_item_removes_matching_call_for_output() { }, ResponseItem::FunctionCallOutput { call_id: "call-delete-last".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, ]; let mut h = create_history_with_items(items); @@ -327,10 +316,11 @@ fn replace_last_turn_images_replaces_tool_output_images() { ResponseItem::FunctionCallOutput { call_id: "call-1".to_string(), output: FunctionCallOutputPayload { - content: "ok".to_string(), - content_items: Some(vec![FunctionCallOutputContentItem::InputImage { - image_url: "data:image/png;base64,AAA".to_string(), - }]), + body: FunctionCallOutputBody::ContentItems(vec![ + FunctionCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + ]), success: Some(true), }, }, @@ -346,10 +336,11 @@ fn replace_last_turn_images_replaces_tool_output_images() { ResponseItem::FunctionCallOutput { call_id: "call-1".to_string(), output: FunctionCallOutputPayload { - content: "ok".to_string(), - content_items: Some(vec![FunctionCallOutputContentItem::InputText { - text: "Invalid image".to_string(), - }]), + body: FunctionCallOutputBody::ContentItems(vec![ + FunctionCallOutputContentItem::InputText { + text: "Invalid image".to_string(), + }, + ]), success: Some(true), }, }, @@ -391,10 +382,7 @@ fn remove_first_item_handles_local_shell_pair() { }, ResponseItem::FunctionCallOutput { call_id: "call-3".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, ]; let mut h = create_history_with_items(items); @@ -560,10 +548,7 @@ fn normalization_retains_local_shell_outputs() { }, ResponseItem::FunctionCallOutput { call_id: "shell-1".to_string(), - output: FunctionCallOutputPayload { - content: "Total output lines: 1\n\nok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("Total output lines: 1\n\nok".to_string()), }, ]; @@ -583,9 +568,8 @@ fn record_items_truncates_function_call_output_content() { let item = ResponseItem::FunctionCallOutput { call_id: "call-100".to_string(), output: FunctionCallOutputPayload { - content: long_output.clone(), + body: FunctionCallOutputBody::Text(long_output.clone()), success: Some(true), - ..Default::default() }, }; @@ -594,16 +578,15 @@ fn record_items_truncates_function_call_output_content() { assert_eq!(history.items.len(), 1); match &history.items[0] { ResponseItem::FunctionCallOutput { output, .. } => { - assert_ne!(output.content, long_output); + let content = output.text_content().unwrap_or_default(); + assert_ne!(content, long_output); assert!( - output.content.contains("tokens truncated"), - "expected token-based truncation marker, got {}", - output.content + content.contains("tokens truncated"), + "expected token-based truncation marker, got {content}" ); assert!( - output.content.contains("tokens truncated"), - "expected truncation marker, got {}", - output.content + content.contains("tokens truncated"), + "expected truncation marker, got {content}" ); } other => panic!("unexpected history item: {other:?}"), @@ -648,9 +631,8 @@ fn record_items_respects_custom_token_limit() { let item = ResponseItem::FunctionCallOutput { call_id: "call-custom-limit".to_string(), output: FunctionCallOutputPayload { - content: long_output, + body: FunctionCallOutputBody::Text(long_output), success: Some(true), - ..Default::default() }, }; @@ -660,7 +642,11 @@ fn record_items_respects_custom_token_limit() { ResponseItem::FunctionCallOutput { output, .. } => output, other => panic!("unexpected history item: {other:?}"), }; - assert!(stored.content.contains("tokens truncated")); + assert!( + stored + .text_content() + .is_some_and(|content| content.contains("tokens truncated")) + ); } fn assert_truncated_message_matches(message: &str, line: &str, expected_removed: usize) { @@ -782,10 +768,7 @@ fn normalize_adds_missing_output_for_function_call() { }, ResponseItem::FunctionCallOutput { call_id: "call-x".to_string(), - output: FunctionCallOutputPayload { - content: "aborted".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("aborted".to_string()), }, ] ); @@ -859,10 +842,7 @@ fn normalize_adds_missing_output_for_local_shell_call_with_id() { }, ResponseItem::FunctionCallOutput { call_id: "shell-1".to_string(), - output: FunctionCallOutputPayload { - content: "aborted".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("aborted".to_string()), }, ] ); @@ -873,10 +853,7 @@ fn normalize_adds_missing_output_for_local_shell_call_with_id() { fn normalize_removes_orphan_function_call_output() { let items = vec![ResponseItem::FunctionCallOutput { call_id: "orphan-1".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }]; let mut h = create_history_with_items(items); @@ -913,10 +890,7 @@ fn normalize_mixed_inserts_and_removals() { // Orphan output that should be removed ResponseItem::FunctionCallOutput { call_id: "c2".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, // Will get an inserted custom tool output ResponseItem::CustomToolCall { @@ -955,10 +929,7 @@ fn normalize_mixed_inserts_and_removals() { }, ResponseItem::FunctionCallOutput { call_id: "c1".to_string(), - output: FunctionCallOutputPayload { - content: "aborted".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("aborted".to_string()), }, ResponseItem::CustomToolCall { id: None, @@ -985,10 +956,7 @@ fn normalize_mixed_inserts_and_removals() { }, ResponseItem::FunctionCallOutput { call_id: "s1".to_string(), - output: FunctionCallOutputPayload { - content: "aborted".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("aborted".to_string()), }, ] ); @@ -1015,10 +983,7 @@ fn normalize_adds_missing_output_for_function_call_inserts_output() { }, ResponseItem::FunctionCallOutput { call_id: "call-x".to_string(), - output: FunctionCallOutputPayload { - content: "aborted".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("aborted".to_string()), }, ] ); @@ -1065,10 +1030,7 @@ fn normalize_adds_missing_output_for_local_shell_call_with_id_panics_in_debug() fn normalize_removes_orphan_function_call_output_panics_in_debug() { let items = vec![ResponseItem::FunctionCallOutput { call_id: "orphan-1".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }]; let mut h = create_history_with_items(items); h.normalize_history(); @@ -1099,10 +1061,7 @@ fn normalize_mixed_inserts_and_removals_panics_in_debug() { }, ResponseItem::FunctionCallOutput { call_id: "c2".to_string(), - output: FunctionCallOutputPayload { - content: "ok".to_string(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".to_string()), }, ResponseItem::CustomToolCall { id: None, diff --git a/codex-rs/core/src/context_manager/normalize.rs b/codex-rs/core/src/context_manager/normalize.rs index 85e25e32aa..37e177900f 100644 --- a/codex-rs/core/src/context_manager/normalize.rs +++ b/codex-rs/core/src/context_manager/normalize.rs @@ -1,5 +1,6 @@ use std::collections::HashSet; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseItem; @@ -29,7 +30,7 @@ pub(crate) fn ensure_call_outputs_present(items: &mut Vec) { ResponseItem::FunctionCallOutput { call_id: call_id.clone(), output: FunctionCallOutputPayload { - content: "aborted".to_string(), + body: FunctionCallOutputBody::Text("aborted".to_string()), ..Default::default() }, }, @@ -76,7 +77,7 @@ pub(crate) fn ensure_call_outputs_present(items: &mut Vec) { ResponseItem::FunctionCallOutput { call_id: call_id.clone(), output: FunctionCallOutputPayload { - content: "aborted".to_string(), + body: FunctionCallOutputBody::Text("aborted".to_string()), ..Default::default() }, }, diff --git a/codex-rs/core/src/exec.rs b/codex-rs/core/src/exec.rs index e0d65c8305..3ac8d1956f 100644 --- a/codex-rs/core/src/exec.rs +++ b/codex-rs/core/src/exec.rs @@ -140,6 +140,7 @@ pub async fn process_exec_tool_call( sandbox_policy: &SandboxPolicy, sandbox_cwd: &Path, codex_linux_sandbox_exe: &Option, + use_linux_sandbox_bwrap: bool, stdout_stream: Option, ) -> Result { let windows_sandbox_level = params.windows_sandbox_level; @@ -184,14 +185,15 @@ pub async fn process_exec_tool_call( let manager = SandboxManager::new(); let exec_env = manager - .transform( + .transform(crate::sandboxing::SandboxTransformRequest { spec, - sandbox_policy, - sandbox_type, - sandbox_cwd, - codex_linux_sandbox_exe.as_ref(), + policy: sandbox_policy, + sandbox: sandbox_type, + sandbox_policy_cwd: sandbox_cwd, + codex_linux_sandbox_exe: codex_linux_sandbox_exe.as_ref(), + use_linux_sandbox_bwrap, windows_sandbox_level, - ) + }) .map_err(CodexErr::from)?; // Route through the sandboxing module for a single, unified execution path. @@ -1108,6 +1110,7 @@ mod tests { &SandboxPolicy::DangerFullAccess, cwd.as_path(), &None, + false, None, ) .await; diff --git a/codex-rs/core/src/features.rs b/codex-rs/core/src/features.rs index 1eb2a2dea5..f6c286c571 100644 --- a/codex-rs/core/src/features.rs +++ b/codex-rs/core/src/features.rs @@ -89,6 +89,8 @@ pub enum Feature { WebSearchCached, /// Gate the execpolicy enforcement for shell/unified exec. ExecPolicy, + /// Use the bubblewrap-based Linux sandbox pipeline. + UseLinuxSandboxBwrap, /// Allow the model to request approval and propose exec rules. RequestRule, /// Enable Windows sandbox (restricted token) on Windows. @@ -406,6 +408,12 @@ pub const FEATURES: &[FeatureSpec] = &[ stage: Stage::Stable, default_enabled: true, }, + FeatureSpec { + id: Feature::UnifiedExec, + key: "unified_exec", + stage: Stage::Stable, + default_enabled: !cfg!(windows), + }, FeatureSpec { id: Feature::WebSearchRequest, key: "web_search_request", @@ -419,16 +427,6 @@ pub const FEATURES: &[FeatureSpec] = &[ default_enabled: false, }, // Experimental program. Rendered in the `/experimental` menu for users. - FeatureSpec { - id: Feature::UnifiedExec, - key: "unified_exec", - stage: Stage::Experimental { - name: "Background terminal", - menu_description: "Run long-running terminal commands in the background.", - announcement: "NEW! Try Background terminals for long-running commands. Enable in /experimental!", - }, - default_enabled: false, - }, FeatureSpec { id: Feature::ShellSnapshot, key: "shell_snapshot", @@ -469,6 +467,12 @@ pub const FEATURES: &[FeatureSpec] = &[ stage: Stage::UnderDevelopment, default_enabled: true, }, + FeatureSpec { + id: Feature::UseLinuxSandboxBwrap, + key: "use_linux_sandbox_bwrap", + stage: Stage::UnderDevelopment, + default_enabled: false, + }, FeatureSpec { id: Feature::RequestRule, key: "request_rule", diff --git a/codex-rs/core/src/file_watcher.rs b/codex-rs/core/src/file_watcher.rs new file mode 100644 index 0000000000..afda5b6086 --- /dev/null +++ b/codex-rs/core/src/file_watcher.rs @@ -0,0 +1,407 @@ +//! Watches skill roots for changes and broadcasts coarse-grained +//! `FileWatcherEvent`s that higher-level components react to on the next turn. + +use std::collections::HashMap; +use std::collections::HashSet; +use std::path::Path; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::Mutex; +use std::sync::RwLock; +use std::time::Duration; + +use notify::Event; +use notify::RecommendedWatcher; +use notify::RecursiveMode; +use notify::Watcher; +use tokio::runtime::Handle; +use tokio::sync::broadcast; +use tokio::sync::mpsc; +use tokio::time::Instant; +use tokio::time::sleep_until; +use tracing::warn; + +use crate::config::Config; +use crate::skills::loader::skill_roots_from_layer_stack_with_agents; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum FileWatcherEvent { + SkillsChanged { paths: Vec }, +} + +struct WatchState { + skills_roots: HashSet, +} + +struct FileWatcherInner { + watcher: RecommendedWatcher, + watched_paths: HashMap, +} + +const WATCHER_THROTTLE_INTERVAL: Duration = Duration::from_secs(1); + +/// Coalesces bursts of paths and emits at most once per interval. +struct ThrottledPaths { + pending: HashSet, + next_allowed_at: Instant, +} + +impl ThrottledPaths { + fn new(now: Instant) -> Self { + Self { + pending: HashSet::new(), + next_allowed_at: now, + } + } + + fn add(&mut self, paths: Vec) { + self.pending.extend(paths); + } + + fn next_deadline(&self, now: Instant) -> Option { + (!self.pending.is_empty() && now < self.next_allowed_at).then_some(self.next_allowed_at) + } + + fn take_ready(&mut self, now: Instant) -> Option> { + if self.pending.is_empty() || now < self.next_allowed_at { + return None; + } + Some(self.take_with_next_allowed(now)) + } + + fn take_pending(&mut self, now: Instant) -> Option> { + if self.pending.is_empty() { + return None; + } + Some(self.take_with_next_allowed(now)) + } + + fn take_with_next_allowed(&mut self, now: Instant) -> Vec { + let mut paths: Vec = self.pending.drain().collect(); + paths.sort_unstable_by(|a, b| a.as_os_str().cmp(b.as_os_str())); + self.next_allowed_at = now + WATCHER_THROTTLE_INTERVAL; + paths + } +} + +pub(crate) struct FileWatcher { + inner: Option>, + state: Arc>, + tx: broadcast::Sender, +} + +impl FileWatcher { + pub(crate) fn new(_codex_home: PathBuf) -> notify::Result { + let (raw_tx, raw_rx) = mpsc::unbounded_channel(); + let raw_tx_clone = raw_tx; + let watcher = notify::recommended_watcher(move |res| { + let _ = raw_tx_clone.send(res); + })?; + let inner = FileWatcherInner { + watcher, + watched_paths: HashMap::new(), + }; + let (tx, _) = broadcast::channel(128); + let state = Arc::new(RwLock::new(WatchState { + skills_roots: HashSet::new(), + })); + let file_watcher = Self { + inner: Some(Mutex::new(inner)), + state: Arc::clone(&state), + tx: tx.clone(), + }; + file_watcher.spawn_event_loop(raw_rx, state, tx); + Ok(file_watcher) + } + + pub(crate) fn noop() -> Self { + let (tx, _) = broadcast::channel(1); + Self { + inner: None, + state: Arc::new(RwLock::new(WatchState { + skills_roots: HashSet::new(), + })), + tx, + } + } + + pub(crate) fn subscribe(&self) -> broadcast::Receiver { + self.tx.subscribe() + } + + pub(crate) fn register_config(&self, config: &Config) { + let roots = + skill_roots_from_layer_stack_with_agents(&config.config_layer_stack, &config.cwd); + for root in roots { + self.register_skills_root(root.path); + } + } + + // Bridge `notify`'s callback-based events into the Tokio runtime and + // broadcast coarse-grained change signals to subscribers. + fn spawn_event_loop( + &self, + mut raw_rx: mpsc::UnboundedReceiver>, + state: Arc>, + tx: broadcast::Sender, + ) { + if let Ok(handle) = Handle::try_current() { + handle.spawn(async move { + let now = Instant::now(); + let mut skills = ThrottledPaths::new(now); + + loop { + let now = Instant::now(); + let next_deadline = skills.next_deadline(now); + let timer_deadline = next_deadline + .unwrap_or_else(|| now + Duration::from_secs(60 * 60 * 24 * 365)); + let timer = sleep_until(timer_deadline); + tokio::pin!(timer); + + tokio::select! { + res = raw_rx.recv() => { + match res { + Some(Ok(event)) => { + let skills_paths = classify_event(&event, &state); + let now = Instant::now(); + skills.add(skills_paths); + + if let Some(paths) = skills.take_ready(now) { + let _ = tx.send(FileWatcherEvent::SkillsChanged { paths }); + } + } + Some(Err(err)) => { + warn!("file watcher error: {err}"); + } + None => { + // Flush any pending changes before shutdown so subscribers + // see the latest state. + let now = Instant::now(); + if let Some(paths) = skills.take_pending(now) { + let _ = tx.send(FileWatcherEvent::SkillsChanged { paths }); + } + break; + } + } + } + _ = &mut timer => { + let now = Instant::now(); + if let Some(paths) = skills.take_ready(now) { + let _ = tx.send(FileWatcherEvent::SkillsChanged { paths }); + } + } + } + } + }); + } else { + warn!("file watcher loop skipped: no Tokio runtime available"); + } + } + + fn register_skills_root(&self, root: PathBuf) { + { + let mut state = match self.state.write() { + Ok(state) => state, + Err(err) => err.into_inner(), + }; + state.skills_roots.insert(root.clone()); + } + self.watch_path(root, RecursiveMode::Recursive); + } + + fn watch_path(&self, path: PathBuf, mode: RecursiveMode) { + let Some(inner) = &self.inner else { + return; + }; + if !path.exists() { + return; + } + let watch_path = path; + let mut guard = match inner.lock() { + Ok(guard) => guard, + Err(err) => err.into_inner(), + }; + if let Some(existing) = guard.watched_paths.get(&watch_path) { + if *existing == RecursiveMode::Recursive || *existing == mode { + return; + } + if let Err(err) = guard.watcher.unwatch(&watch_path) { + warn!("failed to unwatch {}: {err}", watch_path.display()); + } + } + if let Err(err) = guard.watcher.watch(&watch_path, mode) { + warn!("failed to watch {}: {err}", watch_path.display()); + return; + } + guard.watched_paths.insert(watch_path, mode); + } +} + +fn classify_event(event: &Event, state: &RwLock) -> Vec { + let mut skills_paths = Vec::new(); + let skills_roots = match state.read() { + Ok(state) => state.skills_roots.clone(), + Err(err) => { + let state = err.into_inner(); + state.skills_roots.clone() + } + }; + + for path in &event.paths { + if is_skills_path(path, &skills_roots) { + skills_paths.push(path.clone()); + } + } + + skills_paths +} + +fn is_skills_path(path: &Path, roots: &HashSet) -> bool { + roots.iter().any(|root| path.starts_with(root)) +} + +#[cfg(test)] +mod tests { + use super::*; + use notify::EventKind; + use pretty_assertions::assert_eq; + use tokio::time::timeout; + + fn path(name: &str) -> PathBuf { + PathBuf::from(name) + } + + fn notify_event(paths: Vec) -> Event { + let mut event = Event::new(EventKind::Any); + for path in paths { + event = event.add_path(path); + } + event + } + + #[test] + fn throttles_and_coalesces_within_interval() { + let start = Instant::now(); + let mut throttled = ThrottledPaths::new(start); + + throttled.add(vec![path("a")]); + let first = throttled.take_ready(start).expect("first emit"); + assert_eq!(first, vec![path("a")]); + + throttled.add(vec![path("b"), path("c")]); + assert_eq!(throttled.take_ready(start), None); + + let second = throttled + .take_ready(start + WATCHER_THROTTLE_INTERVAL) + .expect("coalesced emit"); + assert_eq!(second, vec![path("b"), path("c")]); + } + + #[test] + fn flushes_pending_on_shutdown() { + let start = Instant::now(); + let mut throttled = ThrottledPaths::new(start); + + throttled.add(vec![path("a")]); + let _ = throttled.take_ready(start).expect("first emit"); + + throttled.add(vec![path("b")]); + assert_eq!(throttled.take_ready(start), None); + + let flushed = throttled + .take_pending(start) + .expect("shutdown flush emits pending paths"); + assert_eq!(flushed, vec![path("b")]); + } + + #[test] + fn classify_event_filters_to_skills_roots() { + let root = path("/tmp/skills"); + let state = RwLock::new(WatchState { + skills_roots: HashSet::from([root.clone()]), + }); + let event = notify_event(vec![ + root.join("demo/SKILL.md"), + path("/tmp/other/not-a-skill.txt"), + ]); + + let classified = classify_event(&event, &state); + assert_eq!(classified, vec![root.join("demo/SKILL.md")]); + } + + #[test] + fn classify_event_supports_multiple_roots_without_prefix_false_positives() { + let root_a = path("/tmp/skills"); + let root_b = path("/tmp/workspace/.codex/skills"); + let state = RwLock::new(WatchState { + skills_roots: HashSet::from([root_a.clone(), root_b.clone()]), + }); + let event = notify_event(vec![ + root_a.join("alpha/SKILL.md"), + path("/tmp/skills-extra/not-under-skills.txt"), + root_b.join("beta/SKILL.md"), + ]); + + let classified = classify_event(&event, &state); + assert_eq!( + classified, + vec![root_a.join("alpha/SKILL.md"), root_b.join("beta/SKILL.md")] + ); + } + + #[test] + fn register_skills_root_dedupes_state_entries() { + let watcher = FileWatcher::noop(); + let root = path("/tmp/skills"); + watcher.register_skills_root(root.clone()); + watcher.register_skills_root(root); + watcher.register_skills_root(path("/tmp/other-skills")); + + let state = watcher.state.read().expect("state lock"); + assert_eq!(state.skills_roots.len(), 2); + } + + #[tokio::test] + async fn spawn_event_loop_flushes_pending_changes_on_shutdown() { + let watcher = FileWatcher::noop(); + let root = path("/tmp/skills"); + { + let mut state = watcher.state.write().expect("state lock"); + state.skills_roots.insert(root.clone()); + } + + let (raw_tx, raw_rx) = mpsc::unbounded_channel(); + let (tx, mut rx) = broadcast::channel(8); + watcher.spawn_event_loop(raw_rx, Arc::clone(&watcher.state), tx); + + raw_tx + .send(Ok(notify_event(vec![root.join("a/SKILL.md")]))) + .expect("send first event"); + let first = timeout(Duration::from_secs(2), rx.recv()) + .await + .expect("first watcher event") + .expect("broadcast recv first"); + assert_eq!( + first, + FileWatcherEvent::SkillsChanged { + paths: vec![root.join("a/SKILL.md")] + } + ); + + raw_tx + .send(Ok(notify_event(vec![root.join("b/SKILL.md")]))) + .expect("send second event"); + drop(raw_tx); + + let second = timeout(Duration::from_secs(2), rx.recv()) + .await + .expect("second watcher event") + .expect("broadcast recv second"); + assert_eq!( + second, + FileWatcherEvent::SkillsChanged { + paths: vec![root.join("b/SKILL.md")] + } + ); + } +} diff --git a/codex-rs/core/src/landlock.rs b/codex-rs/core/src/landlock.rs index 340aebff2c..ea27f77f75 100644 --- a/codex-rs/core/src/landlock.rs +++ b/codex-rs/core/src/landlock.rs @@ -6,26 +6,34 @@ use std::path::Path; use std::path::PathBuf; use tokio::process::Child; -/// Spawn a shell tool command under the Linux Landlock+seccomp sandbox helper -/// (codex-linux-sandbox). +/// Spawn a shell tool command under the Linux sandbox helper +/// (codex-linux-sandbox), which currently uses bubblewrap for filesystem +/// isolation plus seccomp for network restrictions. /// /// Unlike macOS Seatbelt where we directly embed the policy text, the Linux /// helper accepts a list of `--sandbox-permission`/`-s` flags mirroring the /// public CLI. We convert the internal [`SandboxPolicy`] representation into /// the equivalent CLI options. +#[allow(clippy::too_many_arguments)] pub async fn spawn_command_under_linux_sandbox

( codex_linux_sandbox_exe: P, command: Vec, command_cwd: PathBuf, sandbox_policy: &SandboxPolicy, sandbox_policy_cwd: &Path, + use_bwrap_sandbox: bool, stdio_policy: StdioPolicy, env: HashMap, ) -> std::io::Result where P: AsRef, { - let args = create_linux_sandbox_command_args(command, sandbox_policy, sandbox_policy_cwd); + let args = create_linux_sandbox_command_args( + command, + sandbox_policy, + sandbox_policy_cwd, + use_bwrap_sandbox, + ); let arg0 = Some("codex-linux-sandbox"); spawn_child_async( codex_linux_sandbox_exe.as_ref().to_path_buf(), @@ -40,10 +48,14 @@ where } /// Converts the sandbox policy into the CLI invocation for `codex-linux-sandbox`. +/// +/// The helper performs the actual sandboxing (bubblewrap + seccomp) after +/// parsing these arguments. See `docs/linux_sandbox.md` for the Linux semantics. pub(crate) fn create_linux_sandbox_command_args( command: Vec, sandbox_policy: &SandboxPolicy, sandbox_policy_cwd: &Path, + use_bwrap_sandbox: bool, ) -> Vec { #[expect(clippy::expect_used)] let sandbox_policy_cwd = sandbox_policy_cwd @@ -60,13 +72,42 @@ pub(crate) fn create_linux_sandbox_command_args( sandbox_policy_cwd, "--sandbox-policy".to_string(), sandbox_policy_json, - // Separator so that command arguments starting with `-` are not parsed as - // options of the helper itself. - "--".to_string(), ]; + if use_bwrap_sandbox { + linux_cmd.push("--use-bwrap-sandbox".to_string()); + } + + // Separator so that command arguments starting with `-` are not parsed as + // options of the helper itself. + linux_cmd.push("--".to_string()); // Append the original tool command. linux_cmd.extend(command); linux_cmd } + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + + #[test] + fn bwrap_flags_are_feature_gated() { + let command = vec!["/bin/true".to_string()]; + let cwd = Path::new("/tmp"); + let policy = SandboxPolicy::ReadOnly; + + let with_bwrap = create_linux_sandbox_command_args(command.clone(), &policy, cwd, true); + assert_eq!( + with_bwrap.contains(&"--use-bwrap-sandbox".to_string()), + true + ); + + let without_bwrap = create_linux_sandbox_command_args(command, &policy, cwd, false); + assert_eq!( + without_bwrap.contains(&"--use-bwrap-sandbox".to_string()), + false + ); + } +} diff --git a/codex-rs/core/src/lib.rs b/codex-rs/core/src/lib.rs index c9d67e19f1..50a1f85387 100644 --- a/codex-rs/core/src/lib.rs +++ b/codex-rs/core/src/lib.rs @@ -32,6 +32,7 @@ pub mod exec; pub mod exec_env; mod exec_policy; pub mod features; +mod file_watcher; mod flags; pub mod git_info; pub mod instructions; @@ -39,7 +40,6 @@ pub mod landlock; pub mod mcp; mod mcp_connection_manager; pub mod models_manager; -mod transport_manager; pub use mcp_connection_manager::MCP_SANDBOX_STATE_CAPABILITY; pub use mcp_connection_manager::MCP_SANDBOX_STATE_METHOD; pub use mcp_connection_manager::SandboxState; @@ -61,6 +61,7 @@ pub mod token_data; mod truncate; mod unified_exec; pub mod windows_sandbox; +pub use client::X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER; pub use model_provider_info::DEFAULT_LMSTUDIO_PORT; pub use model_provider_info::DEFAULT_OLLAMA_PORT; pub use model_provider_info::LMSTUDIO_OSS_PROVIDER_ID; @@ -120,7 +121,6 @@ pub use rollout::list::read_head_for_summary; pub use rollout::list::read_session_meta_line; pub use rollout::rollout_date_parts; pub use rollout::session_index::find_thread_names_by_ids; -pub use transport_manager::TransportManager; mod function_tool; mod state; mod tasks; @@ -136,8 +136,10 @@ pub use command_safety::is_safe_command; pub use exec_policy::ExecPolicyError; pub use exec_policy::check_execpolicy_for_warnings; pub use exec_policy::load_exec_policy; +pub use file_watcher::FileWatcherEvent; pub use safety::get_platform_sandbox; pub use tools::spec::parse_tool_input_schema; +pub use turn_metadata::build_turn_metadata_header; // Re-export the protocol types from the standalone `codex-protocol` crate so existing // `codex_core::protocol::...` references continue to work across the workspace. pub use codex_protocol::protocol; @@ -159,4 +161,5 @@ pub use codex_protocol::models::ResponseItem; pub use compact::content_items_to_text; pub use event_mapping::parse_turn_item; pub mod compact; +pub mod memory_trace; pub mod otel_init; diff --git a/codex-rs/core/src/mcp/mod.rs b/codex-rs/core/src/mcp/mod.rs index cb0b5e2f26..970fa5db40 100644 --- a/codex-rs/core/src/mcp/mod.rs +++ b/codex-rs/core/src/mcp/mod.rs @@ -167,6 +167,7 @@ pub async fn collect_mcp_snapshot(config: &Config) -> McpListToolsResponseEvent sandbox_policy: SandboxPolicy::ReadOnly, codex_linux_sandbox_exe: config.codex_linux_sandbox_exe.clone(), sandbox_cwd: env::current_dir().unwrap_or_else(|_| PathBuf::from("/")), + use_linux_sandbox_bwrap: config.features.enabled(Feature::UseLinuxSandboxBwrap), }; mcp_connection_manager diff --git a/codex-rs/core/src/mcp/skill_dependencies.rs b/codex-rs/core/src/mcp/skill_dependencies.rs index 37620d73db..14b71fcf19 100644 --- a/codex-rs/core/src/mcp/skill_dependencies.rs +++ b/codex-rs/core/src/mcp/skill_dependencies.rs @@ -144,7 +144,7 @@ pub(crate) async fn maybe_prompt_and_install_mcp_dependencies( return; } - let config = turn_context.client.config(); + let config = turn_context.config.clone(); if mentioned_skills.is_empty() || !config.features.enabled(Feature::SkillMcpDependencyInstall) { return; } diff --git a/codex-rs/core/src/mcp_connection_manager.rs b/codex-rs/core/src/mcp_connection_manager.rs index 1a5ec559b8..d6447738c2 100644 --- a/codex-rs/core/src/mcp_connection_manager.rs +++ b/codex-rs/core/src/mcp_connection_manager.rs @@ -12,7 +12,10 @@ use std::env; use std::ffi::OsString; use std::path::PathBuf; use std::sync::Arc; +use std::sync::LazyLock; +use std::sync::Mutex as StdMutex; use std::time::Duration; +use std::time::Instant; use crate::mcp::CODEX_APPS_MCP_SERVER_NAME; use crate::mcp::auth::McpAuthStatusEntry; @@ -83,6 +86,8 @@ pub const DEFAULT_STARTUP_TIMEOUT: Duration = Duration::from_secs(10); /// Default timeout for individual tool calls. const DEFAULT_TOOL_TIMEOUT: Duration = Duration::from_secs(60); +const CODEX_APPS_TOOLS_CACHE_TTL: Duration = Duration::from_secs(3600); + /// The Responses API requires tool names to match `^[a-zA-Z0-9_-]+$`. /// MCP server/tool names are user-controlled, so sanitize the fully-qualified /// name we expose to the model by replacing any disallowed character with `_`. @@ -161,6 +166,15 @@ pub(crate) struct ToolInfo { pub(crate) connector_name: Option, } +#[derive(Clone)] +struct CachedCodexAppsTools { + expires_at: Instant, + tools: Vec, +} + +static CODEX_APPS_TOOLS_CACHE: LazyLock>> = + LazyLock::new(|| StdMutex::new(None)); + type ResponderMap = HashMap<(String, RequestId), oneshot::Sender>; #[derive(Clone, Default)] @@ -313,6 +327,8 @@ pub struct SandboxState { pub sandbox_policy: SandboxPolicy, pub codex_linux_sandbox_exe: Option, pub sandbox_cwd: PathBuf, + #[serde(default)] + pub use_linux_sandbox_bwrap: bool, } /// A thin wrapper around a set of running [`RmcpClient`] instances. @@ -464,20 +480,27 @@ impl McpConnectionManager { pub async fn list_all_tools(&self) -> HashMap { let mut tools = HashMap::new(); for (server_name, managed_client) in &self.clients { - let client = if server_name == CODEX_APPS_MCP_SERVER_NAME { - // Avoid blocking on codex_apps_mcp startup; use tools only when ready. - match managed_client.client.clone().now_or_never() { - Some(Ok(client)) => Some(client), - _ => None, - } - } else { - managed_client.client().await.ok() - }; + let client = managed_client.client().await.ok(); if let Some(client) = client { - tools.extend(qualify_tools(filter_tools( - client.tools, - client.tool_filter, - ))); + let rmcp_client = client.client; + let tool_timeout = client.tool_timeout; + let tool_filter = client.tool_filter; + let mut server_tools = client.tools; + + if server_name == CODEX_APPS_MCP_SERVER_NAME { + match list_tools_for_client(server_name, &rmcp_client, tool_timeout).await { + Ok(fresh_or_cached_tools) => { + server_tools = fresh_or_cached_tools; + } + Err(err) => { + warn!( + "Failed to refresh tools for MCP server '{server_name}', using startup snapshot: {err:#}" + ); + } + } + } + + tools.extend(qualify_tools(filter_tools(server_tools, tool_filter))); } } tools @@ -971,6 +994,50 @@ async fn list_tools_for_client( server_name: &str, client: &Arc, timeout: Option, +) -> Result> { + if server_name == CODEX_APPS_MCP_SERVER_NAME + && let Some(cached_tools) = read_cached_codex_apps_tools() + { + return Ok(cached_tools); + } + + let tools = list_tools_for_client_uncached(server_name, client, timeout).await?; + if server_name == CODEX_APPS_MCP_SERVER_NAME { + write_cached_codex_apps_tools(&tools); + } + Ok(tools) +} + +fn read_cached_codex_apps_tools() -> Option> { + let mut cache_guard = CODEX_APPS_TOOLS_CACHE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let now = Instant::now(); + + if let Some(cached) = cache_guard.as_ref() + && now < cached.expires_at + { + return Some(cached.tools.clone()); + } + + *cache_guard = None; + None +} + +fn write_cached_codex_apps_tools(tools: &[ToolInfo]) { + let mut cache_guard = CODEX_APPS_TOOLS_CACHE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + *cache_guard = Some(CachedCodexAppsTools { + expires_at: Instant::now() + CODEX_APPS_TOOLS_CACHE_TTL, + tools: tools.to_vec(), + }); +} + +async fn list_tools_for_client_uncached( + server_name: &str, + client: &Arc, + timeout: Option, ) -> Result> { let resp = client.list_tools_with_connector_ids(None, timeout).await?; Ok(resp diff --git a/codex-rs/core/src/mcp_tool_call.rs b/codex-rs/core/src/mcp_tool_call.rs index 75248f34cc..737b130247 100644 --- a/codex-rs/core/src/mcp_tool_call.rs +++ b/codex-rs/core/src/mcp_tool_call.rs @@ -11,15 +11,18 @@ use crate::protocol::McpInvocation; use crate::protocol::McpToolCallBeginEvent; use crate::protocol::McpToolCallEndEvent; use codex_protocol::mcp::CallToolResult; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseInputItem; use codex_protocol::protocol::AskForApproval; +use codex_protocol::protocol::ReviewDecision; use codex_protocol::protocol::SandboxPolicy; use codex_protocol::request_user_input::RequestUserInputArgs; use codex_protocol::request_user_input::RequestUserInputQuestion; use codex_protocol::request_user_input::RequestUserInputQuestionOption; use codex_protocol::request_user_input::RequestUserInputResponse; use rmcp::model::ToolAnnotations; +use serde::Serialize; use std::sync::Arc; /// Handles the specified tool call dispatches the appropriate @@ -44,9 +47,8 @@ pub(crate) async fn handle_mcp_tool_call( return ResponseInputItem::FunctionCallOutput { call_id: call_id.clone(), output: FunctionCallOutputPayload { - content: format!("err: {e}"), + body: FunctionCallOutputBody::Text(format!("err: {e}")), success: Some(false), - ..Default::default() }, }; } @@ -64,7 +66,7 @@ pub(crate) async fn handle_mcp_tool_call( .await { let result = match decision { - McpToolApprovalDecision::Accept => { + McpToolApprovalDecision::Accept | McpToolApprovalDecision::AcceptAndRemember => { let tool_call_begin_event = EventMsg::McpToolCallBegin(McpToolCallBeginEvent { call_id: call_id.clone(), invocation: invocation.clone(), @@ -120,8 +122,7 @@ pub(crate) async fn handle_mcp_tool_call( let status = if result.is_ok() { "ok" } else { "error" }; turn_context - .client - .get_otel_manager() + .otel_manager .counter("codex.mcp.call", 1, &[("status", status)]); return ResponseInputItem::McpToolCallOutput { call_id, result }; @@ -153,8 +154,7 @@ pub(crate) async fn handle_mcp_tool_call( let status = if result.is_ok() { "ok" } else { "error" }; turn_context - .client - .get_otel_manager() + .otel_manager .counter("codex.mcp.call", 1, &[("status", status)]); ResponseInputItem::McpToolCallOutput { call_id, result } @@ -167,21 +167,31 @@ async fn notify_mcp_tool_call_event(sess: &Session, turn_context: &TurnContext, #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum McpToolApprovalDecision { Accept, + AcceptAndRemember, Decline, Cancel, } struct McpToolApprovalMetadata { annotations: ToolAnnotations, + connector_id: Option, connector_name: Option, tool_title: Option, } const MCP_TOOL_APPROVAL_QUESTION_ID_PREFIX: &str = "mcp_tool_call_approval"; -const MCP_TOOL_APPROVAL_ACCEPT: &str = "Accept"; -const MCP_TOOL_APPROVAL_DECLINE: &str = "Decline"; +const MCP_TOOL_APPROVAL_ACCEPT: &str = "Approve Once"; +const MCP_TOOL_APPROVAL_ACCEPT_AND_REMEMBER: &str = "Approve this Session"; +const MCP_TOOL_APPROVAL_DECLINE: &str = "Deny"; const MCP_TOOL_APPROVAL_CANCEL: &str = "Cancel"; +#[derive(Debug, Serialize)] +struct McpToolApprovalKey { + server: String, + connector_id: String, + tool_name: String, +} + async fn maybe_request_mcp_tool_approval( sess: &Session, turn_context: &TurnContext, @@ -200,6 +210,19 @@ async fn maybe_request_mcp_tool_approval( if !requires_mcp_tool_approval(&metadata.annotations) { return None; } + let approval_key = metadata + .connector_id + .as_deref() + .map(|connector_id| McpToolApprovalKey { + server: server.to_string(), + connector_id: connector_id.to_string(), + tool_name: tool_name.to_string(), + }); + if let Some(key) = approval_key.as_ref() + && mcp_tool_approval_is_remembered(sess, key).await + { + return Some(McpToolApprovalDecision::Accept); + } let question_id = format!("{MCP_TOOL_APPROVAL_QUESTION_ID_PREFIX}_{call_id}"); let question = build_mcp_tool_approval_question( @@ -208,6 +231,7 @@ async fn maybe_request_mcp_tool_approval( metadata.tool_title.as_deref(), metadata.connector_name.as_deref(), &metadata.annotations, + approval_key.is_some(), ); let args = RequestUserInputArgs { questions: vec![question], @@ -215,7 +239,13 @@ async fn maybe_request_mcp_tool_approval( let response = sess .request_user_input(turn_context, call_id.to_string(), args) .await; - Some(parse_mcp_tool_approval_response(response, &question_id)) + let decision = parse_mcp_tool_approval_response(response, &question_id); + if matches!(decision, McpToolApprovalDecision::AcceptAndRemember) + && let Some(key) = approval_key + { + remember_mcp_tool_approval(sess, key).await; + } + Some(decision) } fn is_full_access_mode(turn_context: &TurnContext) -> bool { @@ -246,6 +276,7 @@ async fn lookup_mcp_tool_metadata( .annotations .map(|annotations| McpToolApprovalMetadata { annotations, + connector_id: tool_info.connector_id, connector_name: tool_info.connector_name, tool_title: tool_info.tool.title, }) @@ -261,6 +292,7 @@ fn build_mcp_tool_approval_question( tool_title: Option<&str>, connector_name: Option<&str>, annotations: &ToolAnnotations, + allow_remember_option: bool, ) -> RequestUserInputQuestion { let destructive = annotations.destructive_hint == Some(true); let open_world = annotations.open_world_hint == Some(true); @@ -279,26 +311,34 @@ fn build_mcp_tool_approval_question( "{app_label} wants to run the tool \"{tool_label}\", which {reason}. Allow this action?" ); + let mut options = vec![RequestUserInputQuestionOption { + label: MCP_TOOL_APPROVAL_ACCEPT.to_string(), + description: "Run the tool and continue.".to_string(), + }]; + if allow_remember_option { + options.push(RequestUserInputQuestionOption { + label: MCP_TOOL_APPROVAL_ACCEPT_AND_REMEMBER.to_string(), + description: "Run the tool and remember this choice for this session.".to_string(), + }); + } + options.extend([ + RequestUserInputQuestionOption { + label: MCP_TOOL_APPROVAL_DECLINE.to_string(), + description: "Decline this tool call and continue.".to_string(), + }, + RequestUserInputQuestionOption { + label: MCP_TOOL_APPROVAL_CANCEL.to_string(), + description: "Cancel this tool call".to_string(), + }, + ]); + RequestUserInputQuestion { id: question_id, header: "Approve app tool call?".to_string(), question, is_other: false, is_secret: false, - options: Some(vec![ - RequestUserInputQuestionOption { - label: MCP_TOOL_APPROVAL_ACCEPT.to_string(), - description: "Run the tool and continue.".to_string(), - }, - RequestUserInputQuestionOption { - label: MCP_TOOL_APPROVAL_DECLINE.to_string(), - description: "Decline this tool call and continue.".to_string(), - }, - RequestUserInputQuestionOption { - label: MCP_TOOL_APPROVAL_CANCEL.to_string(), - description: "Cancel this tool call".to_string(), - }, - ]), + options: Some(options), } } @@ -317,6 +357,11 @@ fn parse_mcp_tool_approval_response( return McpToolApprovalDecision::Cancel; }; if answers + .iter() + .any(|answer| answer == MCP_TOOL_APPROVAL_ACCEPT_AND_REMEMBER) + { + McpToolApprovalDecision::AcceptAndRemember + } else if answers .iter() .any(|answer| answer == MCP_TOOL_APPROVAL_ACCEPT) { @@ -331,6 +376,16 @@ fn parse_mcp_tool_approval_response( } } +async fn mcp_tool_approval_is_remembered(sess: &Session, key: &McpToolApprovalKey) -> bool { + let store = sess.services.tool_approvals.lock().await; + matches!(store.get(key), Some(ReviewDecision::ApprovedForSession)) +} + +async fn remember_mcp_tool_approval(sess: &Session, key: McpToolApprovalKey) { + let mut store = sess.services.tool_approvals.lock().await; + store.put(key, ReviewDecision::ApprovedForSession); +} + fn requires_mcp_tool_approval(annotations: &ToolAnnotations) -> bool { annotations.read_only_hint == Some(false) && (annotations.destructive_hint == Some(true) || annotations.open_world_hint == Some(true)) diff --git a/codex-rs/core/src/memory_trace.rs b/codex-rs/core/src/memory_trace.rs new file mode 100644 index 0000000000..6519980143 --- /dev/null +++ b/codex-rs/core/src/memory_trace.rs @@ -0,0 +1,303 @@ +use std::path::Path; +use std::path::PathBuf; + +use crate::ModelClient; +use crate::error::CodexErr; +use crate::error::Result; +use codex_api::MemoryTrace as ApiMemoryTrace; +use codex_api::MemoryTraceMetadata as ApiMemoryTraceMetadata; +use codex_otel::OtelManager; +use codex_protocol::openai_models::ModelInfo; +use codex_protocol::openai_models::ReasoningEffort as ReasoningEffortConfig; +use serde_json::Map; +use serde_json::Value; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct BuiltTraceMemory { + pub trace_id: String, + pub source_path: PathBuf, + pub trace_summary: String, + pub memory_summary: String, +} + +struct PreparedTrace { + trace_id: String, + source_path: PathBuf, + payload: ApiMemoryTrace, +} + +/// Loads raw trace files, normalizes trace items, and builds memory summaries. +/// +/// The request/response wiring mirrors the memory trace summarize E2E flow: +/// `/v1/memories/trace_summarize` with one output object per input trace. +/// +/// The caller provides the model selection, reasoning effort, and telemetry context explicitly so +/// the session-scoped [`ModelClient`] can be reused across turns. +pub async fn build_memories_from_trace_files( + client: &ModelClient, + trace_paths: &[PathBuf], + model_info: &ModelInfo, + effort: Option, + otel_manager: &OtelManager, +) -> Result> { + if trace_paths.is_empty() { + return Ok(Vec::new()); + } + + let mut prepared = Vec::with_capacity(trace_paths.len()); + for (index, path) in trace_paths.iter().enumerate() { + prepared.push(prepare_trace(index + 1, path).await?); + } + + let traces = prepared.iter().map(|trace| trace.payload.clone()).collect(); + let output = client + .summarize_memory_traces(traces, model_info, effort, otel_manager) + .await?; + if output.len() != prepared.len() { + return Err(CodexErr::InvalidRequest(format!( + "unexpected memory summarize output length: expected {}, got {}", + prepared.len(), + output.len() + ))); + } + + Ok(prepared + .into_iter() + .zip(output) + .map(|(trace, summary)| BuiltTraceMemory { + trace_id: trace.trace_id, + source_path: trace.source_path, + trace_summary: summary.trace_summary, + memory_summary: summary.memory_summary, + }) + .collect()) +} + +async fn prepare_trace(index: usize, path: &Path) -> Result { + let text = load_trace_text(path).await?; + let items = load_trace_items(path, &text)?; + let trace_id = build_trace_id(index, path); + let source_path = path.to_path_buf(); + + Ok(PreparedTrace { + trace_id: trace_id.clone(), + source_path: source_path.clone(), + payload: ApiMemoryTrace { + id: trace_id, + metadata: ApiMemoryTraceMetadata { + source_path: source_path.display().to_string(), + }, + items, + }, + }) +} + +async fn load_trace_text(path: &Path) -> Result { + let raw = tokio::fs::read(path).await?; + Ok(decode_trace_bytes(&raw)) +} + +fn decode_trace_bytes(raw: &[u8]) -> String { + if let Some(without_bom) = raw.strip_prefix(&[0xEF, 0xBB, 0xBF]) + && let Ok(text) = String::from_utf8(without_bom.to_vec()) + { + return text; + } + if let Ok(text) = String::from_utf8(raw.to_vec()) { + return text; + } + raw.iter().map(|b| char::from(*b)).collect() +} + +fn load_trace_items(path: &Path, text: &str) -> Result> { + if let Ok(Value::Array(items)) = serde_json::from_str::(text) { + let dict_items = items + .into_iter() + .filter(serde_json::Value::is_object) + .collect::>(); + if dict_items.is_empty() { + return Err(CodexErr::InvalidRequest(format!( + "no object items found in trace file: {}", + path.display() + ))); + } + return normalize_trace_items(dict_items, path); + } + + let mut parsed_items = Vec::new(); + for line in text.lines() { + let line = line.trim(); + if line.is_empty() || (!line.starts_with('{') && !line.starts_with('[')) { + continue; + } + + let Ok(obj) = serde_json::from_str::(line) else { + continue; + }; + + match obj { + Value::Object(_) => parsed_items.push(obj), + Value::Array(inner) => { + parsed_items.extend(inner.into_iter().filter(serde_json::Value::is_object)) + } + _ => {} + } + } + + if parsed_items.is_empty() { + return Err(CodexErr::InvalidRequest(format!( + "no JSON items parsed from trace file: {}", + path.display() + ))); + } + + normalize_trace_items(parsed_items, path) +} + +fn normalize_trace_items(items: Vec, path: &Path) -> Result> { + let mut normalized = Vec::new(); + + for item in items { + let Value::Object(obj) = item else { + continue; + }; + + if let Some(payload) = obj.get("payload") { + if obj.get("type").and_then(Value::as_str) != Some("response_item") { + continue; + } + + match payload { + Value::Object(payload_item) => { + if is_allowed_trace_item(payload_item) { + normalized.push(Value::Object(payload_item.clone())); + } + } + Value::Array(payload_items) => { + for payload_item in payload_items { + if let Value::Object(payload_item) = payload_item + && is_allowed_trace_item(payload_item) + { + normalized.push(Value::Object(payload_item.clone())); + } + } + } + _ => {} + } + continue; + } + + if is_allowed_trace_item(&obj) { + normalized.push(Value::Object(obj)); + } + } + + if normalized.is_empty() { + return Err(CodexErr::InvalidRequest(format!( + "no valid trace items after normalization: {}", + path.display() + ))); + } + Ok(normalized) +} + +fn is_allowed_trace_item(item: &Map) -> bool { + let Some(item_type) = item.get("type").and_then(Value::as_str) else { + return false; + }; + + if item_type == "message" { + return matches!( + item.get("role").and_then(Value::as_str), + Some("assistant" | "system" | "developer" | "user") + ); + } + + true +} + +fn build_trace_id(index: usize, path: &Path) -> String { + let stem = path + .file_stem() + .map(|stem| stem.to_string_lossy().into_owned()) + .filter(|stem| !stem.is_empty()) + .unwrap_or_else(|| "trace".to_string()); + format!("trace_{index}_{stem}") +} + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + use tempfile::tempdir; + + #[test] + fn normalize_trace_items_handles_payload_wrapper_and_message_role_filtering() { + let items = vec![ + serde_json::json!({ + "type": "response_item", + "payload": {"type": "message", "role": "assistant", "content": []} + }), + serde_json::json!({ + "type": "response_item", + "payload": [ + {"type": "message", "role": "user", "content": []}, + {"type": "message", "role": "tool", "content": []}, + {"type": "function_call", "name": "shell", "arguments": "{}", "call_id": "c1"} + ] + }), + serde_json::json!({ + "type": "not_response_item", + "payload": {"type": "message", "role": "assistant", "content": []} + }), + serde_json::json!({ + "type": "message", + "role": "developer", + "content": [] + }), + ]; + + let normalized = normalize_trace_items(items, Path::new("trace.json")).expect("normalize"); + let expected = vec![ + serde_json::json!({"type": "message", "role": "assistant", "content": []}), + serde_json::json!({"type": "message", "role": "user", "content": []}), + serde_json::json!({"type": "function_call", "name": "shell", "arguments": "{}", "call_id": "c1"}), + serde_json::json!({"type": "message", "role": "developer", "content": []}), + ]; + assert_eq!(normalized, expected); + } + + #[test] + fn load_trace_items_supports_jsonl_arrays_and_objects() { + let text = r#" +{"type":"response_item","payload":{"type":"message","role":"assistant","content":[]}} +[{"type":"message","role":"user","content":[]},{"type":"message","role":"tool","content":[]}] +"#; + let loaded = load_trace_items(Path::new("trace.jsonl"), text).expect("load"); + let expected = vec![ + serde_json::json!({"type":"message","role":"assistant","content":[]}), + serde_json::json!({"type":"message","role":"user","content":[]}), + ]; + assert_eq!(loaded, expected); + } + + #[tokio::test] + async fn load_trace_text_decodes_utf8_sig() { + let dir = tempdir().expect("tempdir"); + let path = dir.path().join("trace.json"); + tokio::fs::write( + &path, + [ + 0xEF, 0xBB, 0xBF, b'[', b'{', b'"', b't', b'y', b'p', b'e', b'"', b':', b'"', b'm', + b'e', b's', b's', b'a', b'g', b'e', b'"', b',', b'"', b'r', b'o', b'l', b'e', b'"', + b':', b'"', b'u', b's', b'e', b'r', b'"', b',', b'"', b'c', b'o', b'n', b't', b'e', + b'n', b't', b'"', b':', b'[', b']', b'}', b']', + ], + ) + .await + .expect("write"); + + let text = load_trace_text(&path).await.expect("decode"); + assert!(text.starts_with('[')); + } +} diff --git a/codex-rs/core/src/rollout/list.rs b/codex-rs/core/src/rollout/list.rs index 3f057649a6..2cac898d42 100644 --- a/codex-rs/core/src/rollout/list.rs +++ b/codex-rs/core/src/rollout/list.rs @@ -261,6 +261,14 @@ impl<'de> serde::Deserialize<'de> for Cursor { } } +impl From for Cursor { + fn from(anchor: codex_state::Anchor) -> Self { + let ts = OffsetDateTime::from_unix_timestamp(anchor.ts.timestamp()) + .unwrap_or(OffsetDateTime::UNIX_EPOCH); + Self::new(ts, anchor.id) + } +} + /// Retrieve recorded thread file paths with token pagination. The returned `next_cursor` /// can be supplied on the next call to resume after the last returned item, resilient to /// concurrent new sessions being appended. Ordering is stable by the requested sort key @@ -989,7 +997,6 @@ async fn read_head_summary(path: &Path, head_limit: usize) -> io::Result bool { | EventMsg::McpStartupComplete(_) | EventMsg::ListCustomPromptsResponse(_) | EventMsg::ListSkillsResponse(_) + | EventMsg::ListRemoteSkillsResponse(_) + | EventMsg::RemoteSkillDownloaded(_) | EventMsg::PlanUpdate(_) | EventMsg::ShutdownComplete | EventMsg::ViewImageToolCall(_) diff --git a/codex-rs/core/src/rollout/recorder.rs b/codex-rs/core/src/rollout/recorder.rs index 265ff2bde2..465bcbbcdf 100644 --- a/codex-rs/core/src/rollout/recorder.rs +++ b/codex-rs/core/src/rollout/recorder.rs @@ -6,6 +6,7 @@ use std::io::Error as IoError; use std::path::Path; use std::path::PathBuf; +use chrono::SecondsFormat; use codex_protocol::ThreadId; use codex_protocol::dynamic_tools::DynamicToolSpec; use codex_protocol::models::BaseInstructions; @@ -23,12 +24,14 @@ use tracing::warn; use super::ARCHIVED_SESSIONS_SUBDIR; use super::SESSIONS_SUBDIR; use super::list::Cursor; +use super::list::ThreadItem; use super::list::ThreadListConfig; use super::list::ThreadListLayout; use super::list::ThreadSortKey; use super::list::ThreadsPage; use super::list::get_threads; use super::list::get_threads_in_root; +use super::list::read_head_for_summary; use super::metadata; use super::policy::is_persisted_response_item; use crate::config::Config; @@ -120,8 +123,7 @@ impl RolloutRecorder { model_providers: Option<&[String]>, default_provider: &str, ) -> std::io::Result { - let stage = "list_threads"; - let page = get_threads( + Self::list_threads_with_db_fallback( codex_home, page_size, cursor, @@ -129,35 +131,9 @@ impl RolloutRecorder { allowed_sources, model_providers, default_provider, - ) - .await?; - - // TODO(jif): drop after sqlite migration phase 1 - let state_db_ctx = state_db::open_if_present(codex_home, default_provider).await; - if let Some(db_ids) = state_db::list_thread_ids_db( - state_db_ctx.as_deref(), - codex_home, - page_size, - cursor, - sort_key, - allowed_sources, - model_providers, false, - stage, ) .await - { - if page.items.len() != db_ids.len() { - state_db::record_discrepancy(stage, "bad_len"); - return Ok(page); - } - for (id, item) in db_ids.iter().zip(page.items.iter()) { - if !item.path.display().to_string().contains(&id.to_string()) { - state_db::record_discrepancy(stage, "bad_id"); - } - } - } - Ok(page) } /// List archived threads (rollout files) under the archived sessions directory. @@ -170,25 +146,32 @@ impl RolloutRecorder { model_providers: Option<&[String]>, default_provider: &str, ) -> std::io::Result { - let stage = "list_archived_threads"; - let root = codex_home.join(ARCHIVED_SESSIONS_SUBDIR); - let page = get_threads_in_root( - root, + Self::list_threads_with_db_fallback( + codex_home, page_size, cursor, sort_key, - ThreadListConfig { - allowed_sources, - model_providers, - default_provider, - layout: ThreadListLayout::Flat, - }, + allowed_sources, + model_providers, + default_provider, + true, ) - .await?; + .await + } - // TODO(jif): drop after sqlite migration phase 1 + #[allow(clippy::too_many_arguments)] + async fn list_threads_with_db_fallback( + codex_home: &Path, + page_size: usize, + cursor: Option<&Cursor>, + sort_key: ThreadSortKey, + allowed_sources: &[SessionSource], + model_providers: Option<&[String]>, + default_provider: &str, + archived: bool, + ) -> std::io::Result { let state_db_ctx = state_db::open_if_present(codex_home, default_provider).await; - if let Some(db_ids) = state_db::list_thread_ids_db( + if let Some(db_page) = state_db::list_threads_db( state_db_ctx.as_deref(), codex_home, page_size, @@ -196,22 +179,44 @@ impl RolloutRecorder { sort_key, allowed_sources, model_providers, - true, - stage, + archived, ) .await { - if page.items.len() != db_ids.len() { - state_db::record_discrepancy(stage, "bad_len"); - return Ok(page); - } - for (id, item) in db_ids.iter().zip(page.items.iter()) { - if !item.path.display().to_string().contains(&id.to_string()) { - state_db::record_discrepancy(stage, "bad_id"); - } - } + let mut page: ThreadsPage = db_page.into(); + populate_thread_heads(page.items.as_mut_slice()).await; + return Ok(page); } - Ok(page) + tracing::error!("Falling back on rollout system"); + state_db::record_discrepancy("list_threads_with_db_fallback", "falling_back"); + + if archived { + let root = codex_home.join(ARCHIVED_SESSIONS_SUBDIR); + return get_threads_in_root( + root, + page_size, + cursor, + sort_key, + ThreadListConfig { + allowed_sources, + model_providers, + default_provider, + layout: ThreadListLayout::Flat, + }, + ) + .await; + } + + get_threads( + codex_home, + page_size, + cursor, + sort_key, + allowed_sources, + model_providers, + default_provider, + ) + .await } /// Find the newest recorded thread path, optionally filtering to a matching cwd. @@ -645,6 +650,41 @@ impl JsonlWriter { } } +impl From for ThreadsPage { + fn from(db_page: codex_state::ThreadsPage) -> Self { + let items = db_page + .items + .into_iter() + .map(|item| ThreadItem { + path: item.rollout_path, + head: Vec::new(), + created_at: Some(item.created_at.to_rfc3339_opts(SecondsFormat::Secs, true)), + updated_at: Some(item.updated_at.to_rfc3339_opts(SecondsFormat::Secs, true)), + }) + .collect(); + Self { + items, + next_cursor: db_page.next_anchor.map(Into::into), + num_scanned_files: db_page.num_scanned_rows, + reached_scan_cap: false, + } + } +} + +async fn populate_thread_heads(items: &mut [ThreadItem]) { + for item in items { + item.head = read_head_for_summary(item.path.as_path()) + .await + .unwrap_or_else(|err| { + warn!( + "failed to read rollout head from state db path: {} ({err})", + item.path.display() + ); + Vec::new() + }); + } +} + fn select_resume_path(page: &ThreadsPage, filter_cwd: Option<&Path>) -> Option { match filter_cwd { Some(cwd) => page.items.iter().find_map(|item| { diff --git a/codex-rs/core/src/rollout/tests.rs b/codex-rs/core/src/rollout/tests.rs index ee750b126d..89df2897db 100644 --- a/codex-rs/core/src/rollout/tests.rs +++ b/codex-rs/core/src/rollout/tests.rs @@ -7,6 +7,7 @@ use std::fs::{self}; use std::io::Write; use std::path::Path; +use chrono::TimeZone; use pretty_assertions::assert_eq; use tempfile::TempDir; use time::Duration; @@ -22,6 +23,7 @@ use crate::rollout::list::ThreadItem; use crate::rollout::list::ThreadSortKey; use crate::rollout::list::ThreadsPage; use crate::rollout::list::get_threads; +use crate::rollout::recorder::RolloutRecorder; use crate::rollout::rollout_date_parts; use anyhow::Result; use codex_protocol::ThreadId; @@ -45,6 +47,212 @@ fn provider_vec(providers: &[&str]) -> Vec { .collect() } +async fn insert_state_db_thread( + home: &Path, + thread_id: ThreadId, + rollout_path: &Path, + archived: bool, +) { + let runtime = + codex_state::StateRuntime::init(home.to_path_buf(), TEST_PROVIDER.to_string(), None) + .await + .expect("state db should initialize"); + let created_at = chrono::Utc + .with_ymd_and_hms(2025, 1, 3, 12, 0, 0) + .single() + .expect("valid datetime"); + let mut builder = codex_state::ThreadMetadataBuilder::new( + thread_id, + rollout_path.to_path_buf(), + created_at, + SessionSource::Cli, + ); + builder.model_provider = Some(TEST_PROVIDER.to_string()); + builder.cwd = home.to_path_buf(); + if archived { + builder.archived_at = Some(created_at); + } + let mut metadata = builder.build(TEST_PROVIDER); + metadata.has_user_event = true; + runtime + .upsert_thread(&metadata) + .await + .expect("state db upsert should succeed"); +} + +#[tokio::test] +async fn list_threads_prefers_state_db_when_available() { + let temp = TempDir::new().unwrap(); + let home = temp.path(); + let fs_uuid = Uuid::from_u128(101); + write_session_file( + home, + "2025-01-03T13-00-00", + fs_uuid, + 1, + Some(SessionSource::Cli), + ) + .unwrap(); + + let db_uuid = Uuid::from_u128(102); + let db_thread_id = ThreadId::from_string(&db_uuid.to_string()).expect("valid thread id"); + let db_rollout_path = home.join(format!( + "sessions/2025/01/03/rollout-2025-01-03T12-00-00-{db_uuid}.jsonl" + )); + insert_state_db_thread(home, db_thread_id, db_rollout_path.as_path(), false).await; + + let page = RolloutRecorder::list_threads( + home, + 10, + None, + ThreadSortKey::CreatedAt, + NO_SOURCE_FILTER, + None, + TEST_PROVIDER, + ) + .await + .expect("thread listing should succeed"); + + assert_eq!(page.items.len(), 1); + assert_eq!(page.items[0].path, db_rollout_path); +} + +#[tokio::test] +async fn list_archived_threads_prefers_state_db_when_available() { + let temp = TempDir::new().unwrap(); + let home = temp.path(); + let archived_root = home.join("archived_sessions"); + fs::create_dir_all(&archived_root).unwrap(); + let fs_uuid = Uuid::from_u128(201); + let fs_path = archived_root.join(format!("rollout-2025-01-03T13-00-00-{fs_uuid}.jsonl")); + fs::write(&fs_path, "{\"type\":\"session_meta\",\"payload\":{}}\n").unwrap(); + + let db_uuid = Uuid::from_u128(202); + let db_thread_id = ThreadId::from_string(&db_uuid.to_string()).expect("valid thread id"); + let db_rollout_path = + archived_root.join(format!("rollout-2025-01-03T12-00-00-{db_uuid}.jsonl")); + insert_state_db_thread(home, db_thread_id, db_rollout_path.as_path(), true).await; + + let page = RolloutRecorder::list_archived_threads( + home, + 10, + None, + ThreadSortKey::CreatedAt, + NO_SOURCE_FILTER, + None, + TEST_PROVIDER, + ) + .await + .expect("archived thread listing should succeed"); + + assert_eq!(page.items.len(), 1); + assert_eq!(page.items[0].path, db_rollout_path); +} + +#[tokio::test] +async fn list_threads_db_excludes_archived_entries() { + let temp = TempDir::new().unwrap(); + let home = temp.path(); + let sessions_root = home.join("sessions/2025/01/03"); + let archived_root = home.join("archived_sessions"); + fs::create_dir_all(&sessions_root).unwrap(); + fs::create_dir_all(&archived_root).unwrap(); + + let active_uuid = Uuid::from_u128(211); + let active_thread_id = + ThreadId::from_string(&active_uuid.to_string()).expect("valid active thread id"); + let active_rollout_path = + sessions_root.join(format!("rollout-2025-01-03T12-00-00-{active_uuid}.jsonl")); + insert_state_db_thread(home, active_thread_id, active_rollout_path.as_path(), false).await; + + let archived_uuid = Uuid::from_u128(212); + let archived_thread_id = + ThreadId::from_string(&archived_uuid.to_string()).expect("valid archived thread id"); + let archived_rollout_path = + archived_root.join(format!("rollout-2025-01-03T11-00-00-{archived_uuid}.jsonl")); + insert_state_db_thread( + home, + archived_thread_id, + archived_rollout_path.as_path(), + true, + ) + .await; + + let page = RolloutRecorder::list_threads( + home, + 10, + None, + ThreadSortKey::CreatedAt, + NO_SOURCE_FILTER, + None, + TEST_PROVIDER, + ) + .await + .expect("thread listing should succeed"); + + assert_eq!(page.items.len(), 1); + assert_eq!(page.items[0].path, active_rollout_path); +} + +#[tokio::test] +async fn list_threads_falls_back_to_files_when_state_db_is_unavailable() { + let temp = TempDir::new().unwrap(); + let home = temp.path(); + let fs_uuid = Uuid::from_u128(301); + write_session_file( + home, + "2025-01-03T13-00-00", + fs_uuid, + 1, + Some(SessionSource::Cli), + ) + .unwrap(); + + let page = RolloutRecorder::list_threads( + home, + 10, + None, + ThreadSortKey::CreatedAt, + NO_SOURCE_FILTER, + None, + TEST_PROVIDER, + ) + .await + .expect("thread listing should succeed"); + + assert_eq!(page.items.len(), 1); + let file_name = page.items[0] + .path + .file_name() + .and_then(|value| value.to_str()) + .expect("rollout file name should be utf8"); + assert!( + file_name.contains(&fs_uuid.to_string()), + "expected file path from filesystem listing, got: {file_name}" + ); +} + +#[tokio::test] +async fn find_thread_path_falls_back_when_db_path_is_stale() { + let temp = TempDir::new().unwrap(); + let home = temp.path(); + let uuid = Uuid::from_u128(302); + let thread_id = ThreadId::from_string(&uuid.to_string()).expect("valid thread id"); + let ts = "2025-01-03T13-00-00"; + write_session_file(home, ts, uuid, 1, Some(SessionSource::Cli)).unwrap(); + let fs_rollout_path = home.join(format!("sessions/2025/01/03/rollout-{ts}-{uuid}.jsonl")); + + let stale_db_path = home.join(format!( + "sessions/2099/01/01/rollout-2099-01-01T00-00-00-{uuid}.jsonl" + )); + insert_state_db_thread(home, thread_id, stale_db_path.as_path(), false).await; + + let found = crate::rollout::find_thread_path_by_id_str(home, &uuid.to_string()) + .await + .expect("lookup should succeed"); + assert_eq!(found, Some(fs_rollout_path)); +} + #[test] fn rollout_date_parts_extracts_directory_components() { let file_name = OsStr::new("rollout-2025-03-01T09-00-00-123.jsonl"); diff --git a/codex-rs/core/src/sandboxing/mod.rs b/codex-rs/core/src/sandboxing/mod.rs index fca7adda29..bc914abe5b 100644 --- a/codex-rs/core/src/sandboxing/mod.rs +++ b/codex-rs/core/src/sandboxing/mod.rs @@ -51,6 +51,19 @@ pub struct ExecEnv { pub arg0: Option, } +/// Bundled arguments for sandbox transformation. +/// +/// This keeps call sites self-documenting when several fields are optional. +pub(crate) struct SandboxTransformRequest<'a> { + pub spec: CommandSpec, + pub policy: &'a SandboxPolicy, + pub sandbox: SandboxType, + pub sandbox_policy_cwd: &'a Path, + pub codex_linux_sandbox_exe: Option<&'a PathBuf>, + pub use_linux_sandbox_bwrap: bool, + pub windows_sandbox_level: WindowsSandboxLevel, +} + pub enum SandboxPreference { Auto, Require, @@ -104,13 +117,17 @@ impl SandboxManager { pub(crate) fn transform( &self, - mut spec: CommandSpec, - policy: &SandboxPolicy, - sandbox: SandboxType, - sandbox_policy_cwd: &Path, - codex_linux_sandbox_exe: Option<&PathBuf>, - windows_sandbox_level: WindowsSandboxLevel, + request: SandboxTransformRequest<'_>, ) -> Result { + let SandboxTransformRequest { + mut spec, + policy, + sandbox, + sandbox_policy_cwd, + codex_linux_sandbox_exe, + use_linux_sandbox_bwrap, + windows_sandbox_level, + } = request; let mut env = spec.env; if !policy.has_full_network_access() { env.insert( @@ -141,8 +158,12 @@ impl SandboxManager { SandboxType::LinuxSeccomp => { let exe = codex_linux_sandbox_exe .ok_or(SandboxTransformError::MissingLinuxSandboxExecutable)?; - let mut args = - create_linux_sandbox_command_args(command.clone(), policy, sandbox_policy_cwd); + let mut args = create_linux_sandbox_command_args( + command.clone(), + policy, + sandbox_policy_cwd, + use_linux_sandbox_bwrap, + ); let mut full_command = Vec::with_capacity(1 + args.len()); full_command.push(exe.to_string_lossy().to_string()); full_command.append(&mut args); diff --git a/codex-rs/core/src/skills/manager.rs b/codex-rs/core/src/skills/manager.rs index 85e0bf20eb..61dcd98f52 100644 --- a/codex-rs/core/src/skills/manager.rs +++ b/codex-rs/core/src/skills/manager.rs @@ -6,6 +6,7 @@ use std::sync::RwLock; use codex_utils_absolute_path::AbsolutePathBuf; use toml::Value as TomlValue; +use tracing::info; use tracing::warn; use crate::config::Config; @@ -51,14 +52,11 @@ impl SkillsManager { skill_roots_from_layer_stack_with_agents(&config.config_layer_stack, &config.cwd); let mut outcome = load_skills_from_roots(roots); outcome.disabled_paths = disabled_paths_from_stack(&config.config_layer_stack); - match self.cache_by_cwd.write() { - Ok(mut cache) => { - cache.insert(cwd.to_path_buf(), outcome.clone()); - } - Err(err) => { - err.into_inner().insert(cwd.to_path_buf(), outcome.clone()); - } - } + let mut cache = match self.cache_by_cwd.write() { + Ok(cache) => cache, + Err(err) => err.into_inner(), + }; + cache.insert(cwd.to_path_buf(), outcome.clone()); outcome } @@ -109,22 +107,22 @@ impl SkillsManager { let roots = skill_roots_from_layer_stack_with_agents(&config_layer_stack, cwd); let mut outcome = load_skills_from_roots(roots); outcome.disabled_paths = disabled_paths_from_stack(&config_layer_stack); - match self.cache_by_cwd.write() { - Ok(mut cache) => { - cache.insert(cwd.to_path_buf(), outcome.clone()); - } - Err(err) => { - err.into_inner().insert(cwd.to_path_buf(), outcome.clone()); - } - } + let mut cache = match self.cache_by_cwd.write() { + Ok(cache) => cache, + Err(err) => err.into_inner(), + }; + cache.insert(cwd.to_path_buf(), outcome.clone()); outcome } pub fn clear_cache(&self) { - match self.cache_by_cwd.write() { - Ok(mut cache) => cache.clear(), - Err(err) => err.into_inner().clear(), - } + let mut cache = match self.cache_by_cwd.write() { + Ok(cache) => cache, + Err(err) => err.into_inner(), + }; + let cleared = cache.len(); + cache.clear(); + info!("skills cache cleared ({cleared} entries)"); } } diff --git a/codex-rs/core/src/skills/mod.rs b/codex-rs/core/src/skills/mod.rs index 4d79bca258..6148092a12 100644 --- a/codex-rs/core/src/skills/mod.rs +++ b/codex-rs/core/src/skills/mod.rs @@ -3,6 +3,7 @@ pub mod injection; pub mod loader; pub mod manager; pub mod model; +pub mod remote; pub mod render; pub mod system; diff --git a/codex-rs/core/src/skills/remote.rs b/codex-rs/core/src/skills/remote.rs new file mode 100644 index 0000000000..af6dd5593a --- /dev/null +++ b/codex-rs/core/src/skills/remote.rs @@ -0,0 +1,314 @@ +use anyhow::Context; +use anyhow::Result; +use serde::Deserialize; +use std::collections::HashMap; +use std::collections::HashSet; +use std::path::Component; +use std::path::Path; +use std::path::PathBuf; +use std::time::Duration; + +use crate::config::Config; +use crate::default_client::build_reqwest_client; + +const REMOTE_SKILLS_API_TIMEOUT: Duration = Duration::from_secs(30); + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RemoteSkillSummary { + pub id: String, + pub name: String, + pub description: String, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RemoteSkillDownload { + pub id: String, + pub name: String, + pub base_sediment_id: String, + pub files: HashMap, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RemoteSkillDownloadResult { + pub id: String, + pub name: String, + pub path: PathBuf, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RemoteSkillFileRange { + pub start: u64, + pub length: u64, +} + +#[derive(Debug, Deserialize)] +struct RemoteSkillsResponse { + hazelnuts: Vec, +} + +#[derive(Debug, Deserialize)] +struct RemoteSkill { + id: String, + name: String, + description: String, +} + +#[derive(Debug, Deserialize)] +struct RemoteSkillsDownloadResponse { + hazelnuts: Vec, +} + +#[derive(Debug, Deserialize)] +struct RemoteSkillDownloadPayload { + id: String, + name: String, + #[serde(rename = "base_sediment_id")] + base_sediment_id: String, + files: HashMap, +} + +#[derive(Debug, Deserialize)] +struct RemoteSkillFileRangePayload { + start: u64, + length: u64, +} + +pub async fn list_remote_skills(config: &Config) -> Result> { + let base_url = config.chatgpt_base_url.trim_end_matches('/'); + let base_url = base_url.strip_suffix("/backend-api").unwrap_or(base_url); + let url = format!("{base_url}/public-api/hazelnuts/"); + + let client = build_reqwest_client(); + let response = client + .get(&url) + .timeout(REMOTE_SKILLS_API_TIMEOUT) + .query(&[("product_surface", "codex")]) + .send() + .await + .with_context(|| format!("Failed to send request to {url}"))?; + + let status = response.status(); + let body = response.text().await.unwrap_or_default(); + if !status.is_success() { + anyhow::bail!("Request failed with status {status} from {url}: {body}"); + } + + let parsed: RemoteSkillsResponse = + serde_json::from_str(&body).context("Failed to parse skills response")?; + + Ok(parsed + .hazelnuts + .into_iter() + .map(|skill| RemoteSkillSummary { + id: skill.id, + name: skill.name, + description: skill.description, + }) + .collect()) +} + +pub async fn download_remote_skill( + config: &Config, + hazelnut_id: &str, + is_preload: bool, +) -> Result { + let hazelnut = fetch_remote_skill(config, hazelnut_id).await?; + + let client = build_reqwest_client(); + let base_url = config.chatgpt_base_url.trim_end_matches('/'); + let base_url = base_url.strip_suffix("/backend-api").unwrap_or(base_url); + let url = format!("{base_url}/public-api/hazelnuts/{hazelnut_id}/export"); + let response = client + .get(&url) + .timeout(REMOTE_SKILLS_API_TIMEOUT) + .send() + .await + .with_context(|| format!("Failed to send download request to {url}"))?; + + let status = response.status(); + let body = response.bytes().await.context("Failed to read download")?; + if !status.is_success() { + let body_text = String::from_utf8_lossy(&body); + anyhow::bail!("Download failed with status {status} from {url}: {body_text}"); + } + + if !is_zip_payload(&body) { + anyhow::bail!("Downloaded remote skill payload is not a zip archive"); + } + + let preferred_dir_name = if hazelnut.name.trim().is_empty() { + None + } else { + Some(hazelnut.name.as_str()) + }; + let dir_name = preferred_dir_name + .and_then(validate_dir_name_format) + .or_else(|| validate_dir_name_format(&hazelnut.id)) + .ok_or_else(|| anyhow::anyhow!("Remote skill has no valid directory name"))?; + let output_root = if is_preload { + config + .codex_home + .join("vendor_imports") + .join("skills") + .join("skills") + .join(".curated") + } else { + config.codex_home.join("skills").join("downloaded") + }; + let output_dir = output_root.join(dir_name); + tokio::fs::create_dir_all(&output_dir) + .await + .context("Failed to create downloaded skills directory")?; + + let allowed_files = hazelnut.files.keys().cloned().collect::>(); + let zip_bytes = body.to_vec(); + let output_dir_clone = output_dir.clone(); + let prefix_candidates = vec![hazelnut.name.clone(), hazelnut.id.clone()]; + tokio::task::spawn_blocking(move || { + extract_zip_to_dir( + zip_bytes, + &output_dir_clone, + &allowed_files, + &prefix_candidates, + ) + }) + .await + .context("Zip extraction task failed")??; + + Ok(RemoteSkillDownloadResult { + id: hazelnut.id, + name: hazelnut.name, + path: output_dir, + }) +} + +fn safe_join(base: &Path, name: &str) -> Result { + let path = Path::new(name); + for component in path.components() { + match component { + Component::Normal(_) => {} + _ => { + anyhow::bail!("Invalid file path in remote skill payload: {name}"); + } + } + } + Ok(base.join(path)) +} + +fn validate_dir_name_format(name: &str) -> Option { + let mut components = Path::new(name).components(); + match (components.next(), components.next()) { + (Some(Component::Normal(component)), None) => { + let value = component.to_string_lossy().to_string(); + if value.is_empty() { None } else { Some(value) } + } + _ => None, + } +} + +fn is_zip_payload(bytes: &[u8]) -> bool { + bytes.starts_with(b"PK\x03\x04") + || bytes.starts_with(b"PK\x05\x06") + || bytes.starts_with(b"PK\x07\x08") +} + +fn extract_zip_to_dir( + bytes: Vec, + output_dir: &Path, + allowed_files: &HashSet, + prefix_candidates: &[String], +) -> Result<()> { + let cursor = std::io::Cursor::new(bytes); + let mut archive = zip::ZipArchive::new(cursor).context("Failed to open zip archive")?; + for i in 0..archive.len() { + let mut file = archive.by_index(i).context("Failed to read zip entry")?; + if file.is_dir() { + continue; + } + let raw_name = file.name().to_string(); + let normalized = normalize_zip_name(&raw_name, prefix_candidates); + let Some(normalized) = normalized else { + continue; + }; + if !allowed_files.contains(&normalized) { + continue; + } + let file_path = safe_join(output_dir, &normalized)?; + if let Some(parent) = file_path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("Failed to create parent dir for {normalized}"))?; + } + let mut out = std::fs::File::create(&file_path) + .with_context(|| format!("Failed to create file {normalized}"))?; + std::io::copy(&mut file, &mut out) + .with_context(|| format!("Failed to write skill file {normalized}"))?; + } + Ok(()) +} + +fn normalize_zip_name(name: &str, prefix_candidates: &[String]) -> Option { + let mut trimmed = name.trim_start_matches("./"); + for prefix in prefix_candidates { + if prefix.is_empty() { + continue; + } + let prefix = format!("{prefix}/"); + if let Some(rest) = trimmed.strip_prefix(&prefix) { + trimmed = rest; + break; + } + } + if trimmed.is_empty() { + None + } else { + Some(trimmed.to_string()) + } +} + +async fn fetch_remote_skill(config: &Config, hazelnut_id: &str) -> Result { + let base_url = config.chatgpt_base_url.trim_end_matches('/'); + let base_url = base_url.strip_suffix("/backend-api").unwrap_or(base_url); + let url = format!("{base_url}/public-api/hazelnuts/"); + + let client = build_reqwest_client(); + let response = client + .get(&url) + .timeout(REMOTE_SKILLS_API_TIMEOUT) + .query(&[("product_surface", "codex")]) + .send() + .await + .with_context(|| format!("Failed to send request to {url}"))?; + + let status = response.status(); + let body = response.text().await.unwrap_or_default(); + if !status.is_success() { + anyhow::bail!("Request failed with status {status} from {url}: {body}"); + } + + let parsed: RemoteSkillsDownloadResponse = + serde_json::from_str(&body).context("Failed to parse skills response")?; + let hazelnut = parsed + .hazelnuts + .into_iter() + .find(|hazelnut| hazelnut.id == hazelnut_id) + .ok_or_else(|| anyhow::anyhow!("Remote skill {hazelnut_id} not found"))?; + + Ok(RemoteSkillDownload { + id: hazelnut.id, + name: hazelnut.name, + base_sediment_id: hazelnut.base_sediment_id, + files: hazelnut + .files + .into_iter() + .map(|(name, range)| { + ( + name, + RemoteSkillFileRange { + start: range.start, + length: range.length, + }, + ) + }) + .collect(), + }) +} diff --git a/codex-rs/core/src/state/service.rs b/codex-rs/core/src/state/service.rs index d7788f71cb..0e9c038038 100644 --- a/codex-rs/core/src/state/service.rs +++ b/codex-rs/core/src/state/service.rs @@ -4,13 +4,14 @@ use crate::AuthManager; use crate::RolloutRecorder; use crate::agent::AgentControl; use crate::analytics_client::AnalyticsEventsClient; +use crate::client::ModelClient; use crate::exec_policy::ExecPolicyManager; +use crate::file_watcher::FileWatcher; use crate::mcp_connection_manager::McpConnectionManager; use crate::models_manager::manager::ModelsManager; use crate::skills::SkillsManager; use crate::state_db::StateDbHandle; use crate::tools::sandboxing::ApprovalStore; -use crate::transport_manager::TransportManager; use crate::unified_exec::UnifiedExecProcessManager; use crate::user_notification::UserNotifier; use codex_otel::OtelManager; @@ -33,7 +34,9 @@ pub(crate) struct SessionServices { pub(crate) otel_manager: OtelManager, pub(crate) tool_approvals: Mutex, pub(crate) skills_manager: Arc, + pub(crate) file_watcher: Arc, pub(crate) agent_control: AgentControl, pub(crate) state_db: Option, - pub(crate) transport_manager: TransportManager, + /// Session-scoped model client shared across turns. + pub(crate) model_client: ModelClient, } diff --git a/codex-rs/core/src/state_db.rs b/codex-rs/core/src/state_db.rs index ff95ed946f..c36d8afe4b 100644 --- a/codex-rs/core/src/state_db.rs +++ b/codex-rs/core/src/state_db.rs @@ -14,7 +14,6 @@ use codex_protocol::protocol::RolloutItem; use codex_protocol::protocol::SessionSource; use codex_state::DB_METRIC_COMPARE_ERROR; pub use codex_state::LogEntry; -use codex_state::STATE_DB_FILENAME; use codex_state::ThreadMetadataBuilder; use serde_json::Value; use std::path::Path; @@ -32,7 +31,7 @@ pub(crate) async fn init_if_enabled( config: &Config, otel: Option<&OtelManager>, ) -> Option { - let state_path = config.codex_home.join(STATE_DB_FILENAME); + let state_path = codex_state::state_db_path(config.codex_home.as_path()); if !config.features.enabled(Feature::Sqlite) { return None; } @@ -74,7 +73,7 @@ pub(crate) async fn init_if_enabled( /// Get the DB if the feature is enabled and the DB exists. pub async fn get_state_db(config: &Config, otel: Option<&OtelManager>) -> Option { - let state_path = config.codex_home.join(STATE_DB_FILENAME); + let state_path = codex_state::state_db_path(config.codex_home.as_path()); if !config.features.enabled(Feature::Sqlite) || !tokio::fs::try_exists(&state_path).await.unwrap_or(false) { @@ -93,7 +92,7 @@ pub async fn get_state_db(config: &Config, otel: Option<&OtelManager>) -> Option /// /// This is used for parity checks during the SQLite migration phase. pub async fn open_if_present(codex_home: &Path, default_provider: &str) -> Option { - let db_path = codex_home.join(STATE_DB_FILENAME); + let db_path = codex_state::state_db_path(codex_home); if !tokio::fs::try_exists(&db_path).await.unwrap_or(false) { return None; } @@ -181,6 +180,59 @@ pub async fn list_thread_ids_db( } } +/// List thread metadata from SQLite without rollout directory traversal. +#[allow(clippy::too_many_arguments)] +pub async fn list_threads_db( + context: Option<&codex_state::StateRuntime>, + codex_home: &Path, + page_size: usize, + cursor: Option<&Cursor>, + sort_key: ThreadSortKey, + allowed_sources: &[SessionSource], + model_providers: Option<&[String]>, + archived: bool, +) -> Option { + let ctx = context?; + if ctx.codex_home() != codex_home { + warn!( + "state db codex_home mismatch: expected {}, got {}", + ctx.codex_home().display(), + codex_home.display() + ); + } + + let anchor = cursor_to_anchor(cursor); + let allowed_sources: Vec = allowed_sources + .iter() + .map(|value| match serde_json::to_value(value) { + Ok(Value::String(s)) => s, + Ok(other) => other.to_string(), + Err(_) => String::new(), + }) + .collect(); + let model_providers = model_providers.map(<[String]>::to_vec); + match ctx + .list_threads( + page_size, + anchor.as_ref(), + match sort_key { + ThreadSortKey::CreatedAt => codex_state::SortKey::CreatedAt, + ThreadSortKey::UpdatedAt => codex_state::SortKey::UpdatedAt, + }, + allowed_sources.as_slice(), + model_providers.as_deref(), + archived, + ) + .await + { + Ok(page) => Some(page), + Err(err) => { + warn!("state db list_threads failed: {err}"); + None + } + } +} + /// Look up the rollout path for a thread id using SQLite. pub async fn find_rollout_path_by_id( context: Option<&codex_state::StateRuntime>, @@ -228,6 +280,60 @@ pub async fn persist_dynamic_tools( } } +/// Get memory summaries for a thread id using SQLite. +pub async fn get_thread_memory( + context: Option<&codex_state::StateRuntime>, + thread_id: ThreadId, + stage: &str, +) -> Option { + let ctx = context?; + match ctx.get_thread_memory(thread_id).await { + Ok(memory) => memory, + Err(err) => { + warn!("state db get_thread_memory failed during {stage}: {err}"); + None + } + } +} + +/// Upsert memory summaries for a thread id using SQLite. +pub async fn upsert_thread_memory( + context: Option<&codex_state::StateRuntime>, + thread_id: ThreadId, + trace_summary: &str, + memory_summary: &str, + stage: &str, +) -> Option { + let ctx = context?; + match ctx + .upsert_thread_memory(thread_id, trace_summary, memory_summary) + .await + { + Ok(memory) => Some(memory), + Err(err) => { + warn!("state db upsert_thread_memory failed during {stage}: {err}"); + None + } + } +} + +/// Get the last N memories corresponding to a cwd using an exact path match. +pub async fn get_last_n_thread_memories_for_cwd( + context: Option<&codex_state::StateRuntime>, + cwd: &Path, + n: usize, + stage: &str, +) -> Option> { + let ctx = context?; + match ctx.get_last_n_thread_memories_for_cwd(cwd, n).await { + Ok(memories) => Some(memories), + Err(err) => { + warn!("state db get_last_n_thread_memories_for_cwd failed during {stage}: {err}"); + None + } + } +} + /// Reconcile rollout items into SQLite, falling back to scanning the rollout file. pub async fn reconcile_rollout( context: Option<&codex_state::StateRuntime>, diff --git a/codex-rs/core/src/stream_events_utils.rs b/codex-rs/core/src/stream_events_utils.rs index 02d9822510..394f4b9329 100644 --- a/codex-rs/core/src/stream_events_utils.rs +++ b/codex-rs/core/src/stream_events_utils.rs @@ -14,6 +14,7 @@ use crate::parse_turn_item; use crate::proposed_plan_parser::strip_proposed_plan_blocks; use crate::tools::parallel::ToolCallRuntime; use crate::tools::router::ToolRouter; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseInputItem; use codex_protocol::models::ResponseItem; @@ -100,15 +101,14 @@ pub(crate) async fn handle_output_item_done( Err(FunctionCallError::MissingLocalShellCallId) => { let msg = "LocalShellCall without call_id or id"; ctx.turn_context - .client - .get_otel_manager() + .otel_manager .log_tool_failed("local_shell", msg); tracing::error!(msg); let response = ResponseInputItem::FunctionCallOutput { call_id: String::new(), output: FunctionCallOutputPayload { - content: msg.to_string(), + body: FunctionCallOutputBody::Text(msg.to_string()), ..Default::default() }, }; @@ -131,7 +131,7 @@ pub(crate) async fn handle_output_item_done( let response = ResponseInputItem::FunctionCallOutput { call_id: String::new(), output: FunctionCallOutputPayload { - content: message, + body: FunctionCallOutputBody::Text(message), ..Default::default() }, }; @@ -236,9 +236,8 @@ pub(crate) fn response_input_to_response_item(input: &ResponseInputItem) -> Opti let output = match result { Ok(call_tool_result) => FunctionCallOutputPayload::from(call_tool_result), Err(err) => FunctionCallOutputPayload { - content: err.clone(), + body: FunctionCallOutputBody::Text(err.clone()), success: Some(false), - ..Default::default() }, }; Some(ResponseItem::FunctionCallOutput { diff --git a/codex-rs/core/src/tasks/compact.rs b/codex-rs/core/src/tasks/compact.rs index 00d882f607..e1fdbfd91f 100644 --- a/codex-rs/core/src/tasks/compact.rs +++ b/codex-rs/core/src/tasks/compact.rs @@ -25,10 +25,7 @@ impl SessionTask for CompactTask { _cancellation_token: CancellationToken, ) -> Option { let session = session.clone_session(); - if crate::compact::should_use_remote_compact_task( - session.as_ref(), - &ctx.client.get_provider(), - ) { + if crate::compact::should_use_remote_compact_task(session.as_ref(), &ctx.provider) { let _ = session.services.otel_manager.counter( "codex.task.compact", 1, diff --git a/codex-rs/core/src/tasks/mod.rs b/codex-rs/core/src/tasks/mod.rs index d5d6f60582..0ab6e2f49b 100644 --- a/codex-rs/core/src/tasks/mod.rs +++ b/codex-rs/core/src/tasks/mod.rs @@ -158,8 +158,7 @@ impl Session { }; let timer = turn_context - .client - .get_otel_manager() + .otel_manager .start_timer("codex.turn.e2e_duration_ms", &[]) .ok(); diff --git a/codex-rs/core/src/tasks/review.rs b/codex-rs/core/src/tasks/review.rs index cfc554f856..87d6c41933 100644 --- a/codex-rs/core/src/tasks/review.rs +++ b/codex-rs/core/src/tasks/review.rs @@ -82,7 +82,7 @@ async fn start_review_conversation( input: Vec, cancellation_token: CancellationToken, ) -> Option> { - let config = ctx.client.config(); + let config = ctx.config.clone(); let mut sub_agent_config = config.as_ref().clone(); // Carry over review-only feature restrictions so the delegate cannot // re-enable blocked tools (web search, view image). @@ -94,7 +94,7 @@ async fn start_review_conversation( let model = config .review_model .clone() - .unwrap_or_else(|| ctx.client.get_model()); + .unwrap_or_else(|| ctx.model_info.slug.clone()); sub_agent_config.model = Some(model); (run_codex_thread_one_shot( sub_agent_config, diff --git a/codex-rs/core/src/tasks/user_shell.rs b/codex-rs/core/src/tasks/user_shell.rs index a3d38afd2c..6c37cb150e 100644 --- a/codex-rs/core/src/tasks/user_shell.rs +++ b/codex-rs/core/src/tasks/user_shell.rs @@ -66,7 +66,7 @@ impl SessionTask for UserShellCommandTask { .counter("codex.task.user_shell", 1, &[]); let event = EventMsg::TurnStarted(TurnStartedEvent { - model_context_window: turn_context.client.get_model_context_window(), + model_context_window: turn_context.model_context_window(), collaboration_mode_kind: turn_context.collaboration_mode.mode, }); let session = session.clone_session(); diff --git a/codex-rs/core/src/thread_manager.rs b/codex-rs/core/src/thread_manager.rs index 37bd0efabc..e702d9a004 100644 --- a/codex-rs/core/src/thread_manager.rs +++ b/codex-rs/core/src/thread_manager.rs @@ -11,6 +11,8 @@ use crate::codex_thread::CodexThread; use crate::config::Config; use crate::error::CodexErr; use crate::error::Result as CodexResult; +use crate::file_watcher::FileWatcher; +use crate::file_watcher::FileWatcherEvent; use crate::models_manager::manager::ModelsManager; use crate::protocol::Event; use crate::protocol::EventMsg; @@ -31,12 +33,56 @@ use std::path::PathBuf; use std::sync::Arc; #[cfg(any(test, feature = "test-support"))] use tempfile::TempDir; +use tokio::runtime::Handle; +#[cfg(any(test, feature = "test-support"))] +use tokio::runtime::RuntimeFlavor; use tokio::sync::RwLock; use tokio::sync::broadcast; use tracing::warn; const THREAD_CREATED_CHANNEL_CAPACITY: usize = 1024; +fn build_file_watcher(codex_home: PathBuf, skills_manager: Arc) -> Arc { + #[cfg(any(test, feature = "test-support"))] + if let Ok(handle) = Handle::try_current() + && handle.runtime_flavor() == RuntimeFlavor::CurrentThread + { + // The real watcher spins background tasks that can starve the + // current-thread test runtime and cause event waits to time out. + // Integration tests compile with the `test-support` feature. + warn!("using noop file watcher under current-thread test runtime"); + return Arc::new(FileWatcher::noop()); + } + + let file_watcher = match FileWatcher::new(codex_home) { + Ok(file_watcher) => Arc::new(file_watcher), + Err(err) => { + warn!("failed to initialize file watcher: {err}"); + Arc::new(FileWatcher::noop()) + } + }; + + let mut rx = file_watcher.subscribe(); + let skills_manager = Arc::clone(&skills_manager); + if let Ok(handle) = Handle::try_current() { + handle.spawn(async move { + loop { + match rx.recv().await { + Ok(FileWatcherEvent::SkillsChanged { .. }) => { + skills_manager.clear_cache(); + } + Err(broadcast::error::RecvError::Closed) => break, + Err(broadcast::error::RecvError::Lagged(_)) => continue, + } + } + }); + } else { + warn!("file watcher listener skipped: no Tokio runtime available"); + } + + file_watcher +} + /// Represents a newly created Codex thread (formerly called a conversation), including the first event /// (which is [`EventMsg::SessionConfigured`]). pub struct NewThread { @@ -62,6 +108,7 @@ pub(crate) struct ThreadManagerState { auth_manager: Arc, models_manager: Arc, skills_manager: Arc, + file_watcher: Arc, session_source: SessionSource, #[cfg(any(test, feature = "test-support"))] #[allow(dead_code)] @@ -76,15 +123,15 @@ impl ThreadManager { session_source: SessionSource, ) -> Self { let (thread_created_tx, _) = broadcast::channel(THREAD_CREATED_CHANNEL_CAPACITY); + let skills_manager = Arc::new(SkillsManager::new(codex_home.clone())); + let file_watcher = build_file_watcher(codex_home.clone(), Arc::clone(&skills_manager)); Self { state: Arc::new(ThreadManagerState { threads: Arc::new(RwLock::new(HashMap::new())), thread_created_tx, - models_manager: Arc::new(ModelsManager::new( - codex_home.clone(), - auth_manager.clone(), - )), - skills_manager: Arc::new(SkillsManager::new(codex_home)), + models_manager: Arc::new(ModelsManager::new(codex_home, auth_manager.clone())), + skills_manager, + file_watcher, auth_manager, session_source, #[cfg(any(test, feature = "test-support"))] @@ -116,16 +163,19 @@ impl ThreadManager { ) -> Self { let auth_manager = AuthManager::from_auth_for_testing(auth); let (thread_created_tx, _) = broadcast::channel(THREAD_CREATED_CHANNEL_CAPACITY); + let skills_manager = Arc::new(SkillsManager::new(codex_home.clone())); + let file_watcher = build_file_watcher(codex_home.clone(), Arc::clone(&skills_manager)); Self { state: Arc::new(ThreadManagerState { threads: Arc::new(RwLock::new(HashMap::new())), thread_created_tx, models_manager: Arc::new(ModelsManager::with_provider( - codex_home.clone(), + codex_home, auth_manager.clone(), provider, )), - skills_manager: Arc::new(SkillsManager::new(codex_home)), + skills_manager, + file_watcher, auth_manager, session_source: SessionSource::Exec, #[cfg(any(test, feature = "test-support"))] @@ -143,6 +193,10 @@ impl ThreadManager { self.state.skills_manager.clone() } + pub fn subscribe_file_watcher(&self) -> broadcast::Receiver { + self.state.file_watcher.subscribe() + } + pub fn get_models_manager(&self) -> Arc { self.state.models_manager.clone() } @@ -380,6 +434,7 @@ impl ThreadManagerState { session_source: SessionSource, dynamic_tools: Vec, ) -> CodexResult { + self.file_watcher.register_config(&config); let CodexSpawnOk { codex, thread_id, .. } = Codex::spawn( @@ -387,6 +442,7 @@ impl ThreadManagerState { auth_manager, Arc::clone(&self.models_manager), Arc::clone(&self.skills_manager), + Arc::clone(&self.file_watcher), initial_history, session_source, agent_control, diff --git a/codex-rs/core/src/tools/context.rs b/codex-rs/core/src/tools/context.rs index f0bbb158f5..e9edd7db46 100644 --- a/codex-rs/core/src/tools/context.rs +++ b/codex-rs/core/src/tools/context.rs @@ -5,7 +5,7 @@ use crate::tools::TELEMETRY_PREVIEW_MAX_LINES; use crate::tools::TELEMETRY_PREVIEW_TRUNCATION_NOTICE; use crate::turn_diff_tracker::TurnDiffTracker; use codex_protocol::mcp::CallToolResult; -use codex_protocol::models::FunctionCallOutputContentItem; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseInputItem; use codex_protocol::models::ShellToolCallParams; @@ -58,10 +58,9 @@ impl ToolPayload { #[derive(Clone)] pub enum ToolOutput { Function { - // Plain text representation of the tool output. - content: String, - // Some tool calls such as MCP calls may return structured content that can get parsed into an array of polymorphic content items. - content_items: Option>, + // Canonical output body for function-style tools. This may be plain text + // or structured content items. + body: FunctionCallOutputBody, success: Option, }, Mcp { @@ -72,7 +71,9 @@ pub enum ToolOutput { impl ToolOutput { pub fn log_preview(&self) -> String { match self { - ToolOutput::Function { content, .. } => telemetry_preview(content), + ToolOutput::Function { body, .. } => { + telemetry_preview(&body.to_text().unwrap_or_default()) + } ToolOutput::Mcp { result } => format!("{result:?}"), } } @@ -86,27 +87,28 @@ impl ToolOutput { pub fn into_response(self, call_id: &str, payload: &ToolPayload) -> ResponseInputItem { match self { - ToolOutput::Function { - content, - content_items, - success, - } => { + ToolOutput::Function { body, success } => { + // `custom_tool_call` is the Responses API item type for freeform + // tools (`ToolSpec::Freeform`, e.g. freeform `apply_patch`). + // Those payloads must round-trip as `custom_tool_call_output` + // with plain string output. if matches!(payload, ToolPayload::Custom { .. }) { - ResponseInputItem::CustomToolCallOutput { + // Freeform/custom tools (`custom_tool_call`) use the custom + // output wire shape and remain string-only. + return ResponseInputItem::CustomToolCallOutput { call_id: call_id.to_string(), - output: content, - } - } else { - ResponseInputItem::FunctionCallOutput { - call_id: call_id.to_string(), - output: FunctionCallOutputPayload { - content, - content_items, - success, - }, - } + output: body.to_text().unwrap_or_default(), + }; + } + + // Function-style outputs (JSON function tools, including dynamic + // tools and MCP adaptation) preserve the exact body shape. + ResponseInputItem::FunctionCallOutput { + call_id: call_id.to_string(), + output: FunctionCallOutputPayload { body, success }, } } + // Direct MCP response path for MCP tool result envelopes. ToolOutput::Mcp { result } => ResponseInputItem::McpToolCallOutput { call_id: call_id.to_string(), result, @@ -158,6 +160,7 @@ fn telemetry_preview(content: &str) -> String { #[cfg(test)] mod tests { use super::*; + use codex_protocol::models::FunctionCallOutputContentItem; use pretty_assertions::assert_eq; #[test] @@ -166,8 +169,7 @@ mod tests { input: "patch".to_string(), }; let response = ToolOutput::Function { - content: "patched".to_string(), - content_items: None, + body: FunctionCallOutputBody::Text("patched".to_string()), success: Some(true), } .into_response("call-42", &payload); @@ -187,8 +189,7 @@ mod tests { arguments: "{}".to_string(), }; let response = ToolOutput::Function { - content: "ok".to_string(), - content_items: None, + body: FunctionCallOutputBody::Text("ok".to_string()), success: Some(true), } .into_response("fn-1", &payload); @@ -196,14 +197,58 @@ mod tests { match response { ResponseInputItem::FunctionCallOutput { call_id, output } => { assert_eq!(call_id, "fn-1"); - assert_eq!(output.content, "ok"); - assert!(output.content_items.is_none()); + assert_eq!(output.text_content(), Some("ok")); + assert!(output.content_items().is_none()); assert_eq!(output.success, Some(true)); } other => panic!("expected FunctionCallOutput, got {other:?}"), } } + #[test] + fn custom_tool_calls_can_derive_text_from_content_items() { + let payload = ToolPayload::Custom { + input: "patch".to_string(), + }; + let response = ToolOutput::Function { + body: FunctionCallOutputBody::ContentItems(vec![ + FunctionCallOutputContentItem::InputText { + text: "line 1".to_string(), + }, + FunctionCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + FunctionCallOutputContentItem::InputText { + text: "line 2".to_string(), + }, + ]), + success: Some(true), + } + .into_response("call-99", &payload); + + match response { + ResponseInputItem::CustomToolCallOutput { call_id, output } => { + assert_eq!(call_id, "call-99"); + assert_eq!(output, "line 1\nline 2"); + } + other => panic!("expected CustomToolCallOutput, got {other:?}"), + } + } + + #[test] + fn log_preview_uses_content_items_when_plain_text_is_missing() { + let output = ToolOutput::Function { + body: FunctionCallOutputBody::ContentItems(vec![ + FunctionCallOutputContentItem::InputText { + text: "preview".to_string(), + }, + ]), + success: Some(true), + }; + + assert_eq!(output.log_preview(), "preview"); + } + #[test] fn telemetry_preview_returns_original_within_limits() { let content = "short output"; diff --git a/codex-rs/core/src/tools/handlers/apply_patch.rs b/codex-rs/core/src/tools/handlers/apply_patch.rs index 46723decf6..aced569ce4 100644 --- a/codex-rs/core/src/tools/handlers/apply_patch.rs +++ b/codex-rs/core/src/tools/handlers/apply_patch.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::collections::BTreeMap; use std::path::Path; @@ -109,8 +110,7 @@ impl ToolHandler for ApplyPatchHandler { InternalApplyPatchInvocation::Output(item) => { let content = item?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } @@ -155,8 +155,7 @@ impl ToolHandler for ApplyPatchHandler { ); let content = emitter.finish(event_ctx, out).await?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } @@ -205,8 +204,7 @@ pub(crate) async fn intercept_apply_patch( InternalApplyPatchInvocation::Output(item) => { let content = item?; Ok(Some(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), })) } @@ -242,8 +240,7 @@ pub(crate) async fn intercept_apply_patch( ToolEventCtx::new(session, turn, call_id, tracker.as_ref().copied()); let content = emitter.finish(event_ctx, out).await?; Ok(Some(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), })) } diff --git a/codex-rs/core/src/tools/handlers/collab.rs b/codex-rs/core/src/tools/handlers/collab.rs index 61ccc1932e..3110204d29 100644 --- a/codex-rs/core/src/tools/handlers/collab.rs +++ b/codex-rs/core/src/tools/handlers/collab.rs @@ -15,6 +15,7 @@ use crate::tools::registry::ToolKind; use async_trait::async_trait; use codex_protocol::ThreadId; use codex_protocol::models::BaseInstructions; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::protocol::CollabAgentInteractionBeginEvent; use codex_protocol::protocol::CollabAgentInteractionEndEvent; use codex_protocol::protocol::CollabAgentSpawnBeginEvent; @@ -114,7 +115,7 @@ mod spawn { "Empty message can't be sent to an agent".to_string(), )); } - let session_source = turn.client.get_session_source(); + let session_source = turn.session_source.clone(); let child_depth = next_thread_spawn_depth(&session_source); if exceeds_thread_spawn_depth_limit(child_depth) { return Err(FunctionCallError::RespondToModel( @@ -184,9 +185,8 @@ mod spawn { })?; Ok(ToolOutput::Function { - content, + body: FunctionCallOutputBody::Text(content), success: Some(true), - content_items: None, }) } } @@ -273,9 +273,8 @@ mod send_input { })?; Ok(ToolOutput::Function { - content, + body: FunctionCallOutputBody::Text(content), success: Some(true), - content_items: None, }) } } @@ -441,9 +440,8 @@ mod wait { })?; Ok(ToolOutput::Function { - content, + body: FunctionCallOutputBody::Text(content), success: None, - content_items: None, }) } @@ -552,9 +550,8 @@ pub mod close_agent { })?; Ok(ToolOutput::Function { - content, + body: FunctionCallOutputBody::Text(content), success: Some(true), - content_items: None, }) } } @@ -593,13 +590,13 @@ fn build_agent_spawn_config( turn: &TurnContext, child_depth: i32, ) -> Result { - let base_config = turn.client.config(); + let base_config = turn.config.clone(); let mut config = (*base_config).clone(); config.base_instructions = Some(base_instructions.text.clone()); - config.model = Some(turn.client.get_model()); - config.model_provider = turn.client.get_provider(); - config.model_reasoning_effort = turn.client.get_reasoning_effort(); - config.model_reasoning_summary = turn.client.get_reasoning_summary(); + config.model = Some(turn.model_info.slug.clone()); + config.model_provider = turn.provider.clone(); + config.model_reasoning_effort = turn.reasoning_effort; + config.model_reasoning_summary = turn.reasoning_summary; config.developer_instructions = turn.developer_instructions.clone(); config.compact_prompt = turn.compact_prompt.clone(); config.shell_environment_policy = turn.shell_environment_policy.clone(); @@ -633,7 +630,6 @@ mod tests { use crate::ThreadManager; use crate::agent::MAX_THREAD_SPAWN_DEPTH; use crate::built_in_model_providers; - use crate::client::ModelClient; use crate::codex::make_session_and_context; use crate::config::types::ShellEnvironmentPolicy; use crate::function_tool::FunctionCallError; @@ -767,22 +763,10 @@ mod tests { let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let session_source = SessionSource::SubAgent(SubAgentSource::ThreadSpawn { + turn.session_source = SessionSource::SubAgent(SubAgentSource::ThreadSpawn { parent_thread_id: session.conversation_id, depth: MAX_THREAD_SPAWN_DEPTH, }); - turn.client = ModelClient::new( - turn.client.config(), - Some(session.services.auth_manager.clone()), - turn.client.get_model_info(), - turn.client.get_otel_manager(), - turn.client.get_provider(), - turn.client.get_reasoning_effort(), - turn.client.get_reasoning_summary(), - session.conversation_id, - session_source, - session.services.transport_manager.clone(), - ); let invocation = invocation( Arc::new(session), @@ -865,7 +849,7 @@ mod tests { let (mut session, turn) = make_session_and_context().await; let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let config = turn.client.config().as_ref().clone(); + let config = turn.config.as_ref().clone(); let thread = manager.start_thread(config).await.expect("start thread"); let agent_id = thread.thread_id; let invocation = invocation( @@ -983,7 +967,9 @@ mod tests { .await .expect("wait should succeed"); let ToolOutput::Function { - content, success, .. + body: FunctionCallOutputBody::Text(content), + success, + .. } = output else { panic!("expected function output"); @@ -1008,7 +994,7 @@ mod tests { let (mut session, turn) = make_session_and_context().await; let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let config = turn.client.config().as_ref().clone(); + let config = turn.config.as_ref().clone(); let thread = manager.start_thread(config).await.expect("start thread"); let agent_id = thread.thread_id; let invocation = invocation( @@ -1025,7 +1011,9 @@ mod tests { .await .expect("wait should succeed"); let ToolOutput::Function { - content, success, .. + body: FunctionCallOutputBody::Text(content), + success, + .. } = output else { panic!("expected function output"); @@ -1053,7 +1041,7 @@ mod tests { let (mut session, turn) = make_session_and_context().await; let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let config = turn.client.config().as_ref().clone(); + let config = turn.config.as_ref().clone(); let thread = manager.start_thread(config).await.expect("start thread"); let agent_id = thread.thread_id; let invocation = invocation( @@ -1084,7 +1072,7 @@ mod tests { let (mut session, turn) = make_session_and_context().await; let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let config = turn.client.config().as_ref().clone(); + let config = turn.config.as_ref().clone(); let thread = manager.start_thread(config).await.expect("start thread"); let agent_id = thread.thread_id; let mut status_rx = manager @@ -1116,7 +1104,9 @@ mod tests { .await .expect("wait should succeed"); let ToolOutput::Function { - content, success, .. + body: FunctionCallOutputBody::Text(content), + success, + .. } = output else { panic!("expected function output"); @@ -1138,7 +1128,7 @@ mod tests { let (mut session, turn) = make_session_and_context().await; let manager = thread_manager(); session.services.agent_control = manager.agent_control(); - let config = turn.client.config().as_ref().clone(); + let config = turn.config.as_ref().clone(); let thread = manager.start_thread(config).await.expect("start thread"); let agent_id = thread.thread_id; let status_before = manager.agent_control().get_status(agent_id).await; @@ -1154,7 +1144,9 @@ mod tests { .await .expect("close_agent should succeed"); let ToolOutput::Function { - content, success, .. + body: FunctionCallOutputBody::Text(content), + success, + .. } = output else { panic!("expected function output"); @@ -1193,12 +1185,12 @@ mod tests { turn.sandbox_policy = SandboxPolicy::DangerFullAccess; let config = build_agent_spawn_config(&base_instructions, &turn, 0).expect("spawn config"); - let mut expected = (*turn.client.config()).clone(); + let mut expected = (*turn.config).clone(); expected.base_instructions = Some(base_instructions.text); - expected.model = Some(turn.client.get_model()); - expected.model_provider = turn.client.get_provider(); - expected.model_reasoning_effort = turn.client.get_reasoning_effort(); - expected.model_reasoning_summary = turn.client.get_reasoning_summary(); + expected.model = Some(turn.model_info.slug.clone()); + expected.model_provider = turn.provider.clone(); + expected.model_reasoning_effort = turn.reasoning_effort; + expected.model_reasoning_summary = turn.reasoning_summary; expected.developer_instructions = turn.developer_instructions.clone(); expected.compact_prompt = turn.compact_prompt.clone(); expected.shell_environment_policy = turn.shell_environment_policy.clone(); @@ -1217,24 +1209,11 @@ mod tests { #[tokio::test] async fn build_agent_spawn_config_preserves_base_user_instructions() { - let (session, mut turn) = make_session_and_context().await; - let session_source = turn.client.get_session_source(); - let mut base_config = (*turn.client.config()).clone(); + let (_session, mut turn) = make_session_and_context().await; + let mut base_config = (*turn.config).clone(); base_config.user_instructions = Some("base-user".to_string()); turn.user_instructions = Some("resolved-user".to_string()); - let transport_manager = turn.client.transport_manager(); - turn.client = ModelClient::new( - Arc::new(base_config.clone()), - Some(session.services.auth_manager.clone()), - turn.client.get_model_info(), - turn.client.get_otel_manager(), - turn.client.get_provider(), - turn.client.get_reasoning_effort(), - turn.client.get_reasoning_summary(), - session.conversation_id, - session_source, - transport_manager, - ); + turn.config = Arc::new(base_config.clone()); let base_instructions = BaseInstructions { text: "base".to_string(), }; diff --git a/codex-rs/core/src/tools/handlers/dynamic.rs b/codex-rs/core/src/tools/handlers/dynamic.rs index a68c70b98d..51dc8db56f 100644 --- a/codex-rs/core/src/tools/handlers/dynamic.rs +++ b/codex-rs/core/src/tools/handlers/dynamic.rs @@ -10,6 +10,8 @@ use crate::tools::registry::ToolKind; use async_trait::async_trait; use codex_protocol::dynamic_tools::DynamicToolCallRequest; use codex_protocol::dynamic_tools::DynamicToolResponse; +use codex_protocol::models::FunctionCallOutputBody; +use codex_protocol::models::FunctionCallOutputContentItem; use codex_protocol::protocol::EventMsg; use serde_json::Value; use tokio::sync::oneshot; @@ -55,10 +57,19 @@ impl ToolHandler for DynamicToolHandler { ) })?; + let DynamicToolResponse { + content_items, + success, + } = response; + let body = content_items + .into_iter() + .map(FunctionCallOutputContentItem::from) + .collect::>(); + let body = FunctionCallOutputBody::ContentItems(body); + Ok(ToolOutput::Function { - content: response.output, - content_items: None, - success: Some(response.success), + body, + success: Some(success), }) } } diff --git a/codex-rs/core/src/tools/handlers/grep_files.rs b/codex-rs/core/src/tools/handlers/grep_files.rs index a3e89af6a8..9fbc6c17a3 100644 --- a/codex-rs/core/src/tools/handlers/grep_files.rs +++ b/codex-rs/core/src/tools/handlers/grep_files.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::path::Path; use std::time::Duration; @@ -86,14 +87,12 @@ impl ToolHandler for GrepFilesHandler { if search_results.is_empty() { Ok(ToolOutput::Function { - content: "No matches found.".to_string(), - content_items: None, + body: FunctionCallOutputBody::Text("No matches found.".to_string()), success: Some(false), }) } else { Ok(ToolOutput::Function { - content: search_results.join("\n"), - content_items: None, + body: FunctionCallOutputBody::Text(search_results.join("\n")), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/list_dir.rs b/codex-rs/core/src/tools/handlers/list_dir.rs index a06fca3d14..5535ce0ba6 100644 --- a/codex-rs/core/src/tools/handlers/list_dir.rs +++ b/codex-rs/core/src/tools/handlers/list_dir.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::collections::VecDeque; use std::ffi::OsStr; use std::fs::FileType; @@ -102,8 +103,7 @@ impl ToolHandler for ListDirHandler { output.push(format!("Absolute path: {}", path.display())); output.extend(entries); Ok(ToolOutput::Function { - content: output.join("\n"), - content_items: None, + body: FunctionCallOutputBody::Text(output.join("\n")), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/mcp.rs b/codex-rs/core/src/tools/handlers/mcp.rs index 5138b3cd0d..8ea46b4ed8 100644 --- a/codex-rs/core/src/tools/handlers/mcp.rs +++ b/codex-rs/core/src/tools/handlers/mcp.rs @@ -8,6 +8,7 @@ use crate::tools::context::ToolOutput; use crate::tools::context::ToolPayload; use crate::tools::registry::ToolHandler; use crate::tools::registry::ToolKind; +use codex_protocol::models::ResponseInputItem; pub struct McpHandler; @@ -53,20 +54,11 @@ impl ToolHandler for McpHandler { .await; match response { - codex_protocol::models::ResponseInputItem::McpToolCallOutput { result, .. } => { - Ok(ToolOutput::Mcp { result }) - } - codex_protocol::models::ResponseInputItem::FunctionCallOutput { output, .. } => { - let codex_protocol::models::FunctionCallOutputPayload { - content, - content_items, - success, - } = output; - Ok(ToolOutput::Function { - content, - content_items, - success, - }) + ResponseInputItem::McpToolCallOutput { result, .. } => Ok(ToolOutput::Mcp { result }), + ResponseInputItem::FunctionCallOutput { output, .. } => { + let success = output.success; + let body = output.body; + Ok(ToolOutput::Function { body, success }) } _ => Err(FunctionCallError::RespondToModel( "mcp handler received unexpected response variant".to_string(), diff --git a/codex-rs/core/src/tools/handlers/mcp_resource.rs b/codex-rs/core/src/tools/handlers/mcp_resource.rs index df421bd6d3..fe541197cd 100644 --- a/codex-rs/core/src/tools/handlers/mcp_resource.rs +++ b/codex-rs/core/src/tools/handlers/mcp_resource.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; @@ -296,12 +297,10 @@ async fn handle_list_resources( match payload_result { Ok(payload) => match serialize_function_output(payload) { Ok(output) => { - let ToolOutput::Function { - content, success, .. - } = &output - else { + let ToolOutput::Function { body, success } = &output else { unreachable!("MCP resource handler should return function output"); }; + let content = body.to_text().unwrap_or_default(); let duration = start.elapsed(); emit_tool_call_end( &session, @@ -309,7 +308,7 @@ async fn handle_list_resources( &call_id, invocation, duration, - Ok(call_tool_result_from_content(content, *success)), + Ok(call_tool_result_from_content(&content, *success)), ) .await; Ok(output) @@ -405,12 +404,10 @@ async fn handle_list_resource_templates( match payload_result { Ok(payload) => match serialize_function_output(payload) { Ok(output) => { - let ToolOutput::Function { - content, success, .. - } = &output - else { + let ToolOutput::Function { body, success } = &output else { unreachable!("MCP resource handler should return function output"); }; + let content = body.to_text().unwrap_or_default(); let duration = start.elapsed(); emit_tool_call_end( &session, @@ -418,7 +415,7 @@ async fn handle_list_resource_templates( &call_id, invocation, duration, - Ok(call_tool_result_from_content(content, *success)), + Ok(call_tool_result_from_content(&content, *success)), ) .await; Ok(output) @@ -494,12 +491,10 @@ async fn handle_read_resource( match payload_result { Ok(payload) => match serialize_function_output(payload) { Ok(output) => { - let ToolOutput::Function { - content, success, .. - } = &output - else { + let ToolOutput::Function { body, success } = &output else { unreachable!("MCP resource handler should return function output"); }; + let content = body.to_text().unwrap_or_default(); let duration = start.elapsed(); emit_tool_call_end( &session, @@ -507,7 +502,7 @@ async fn handle_read_resource( &call_id, invocation, duration, - Ok(call_tool_result_from_content(content, *success)), + Ok(call_tool_result_from_content(&content, *success)), ) .await; Ok(output) @@ -622,8 +617,7 @@ where })?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/plan.rs b/codex-rs/core/src/tools/handlers/plan.rs index 88e0b0dc27..2b43429cc8 100644 --- a/codex-rs/core/src/tools/handlers/plan.rs +++ b/codex-rs/core/src/tools/handlers/plan.rs @@ -11,6 +11,7 @@ use crate::tools::registry::ToolKind; use crate::tools::spec::JsonSchema; use async_trait::async_trait; use codex_protocol::config_types::ModeKind; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::plan_tool::UpdatePlanArgs; use codex_protocol::protocol::EventMsg; use std::collections::BTreeMap; @@ -88,8 +89,7 @@ impl ToolHandler for PlanHandler { handle_update_plan(session.as_ref(), turn.as_ref(), arguments, call_id).await?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/read_file.rs b/codex-rs/core/src/tools/handlers/read_file.rs index eb5ce647ce..59c6ced7f5 100644 --- a/codex-rs/core/src/tools/handlers/read_file.rs +++ b/codex-rs/core/src/tools/handlers/read_file.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::collections::VecDeque; use std::path::PathBuf; @@ -146,8 +147,7 @@ impl ToolHandler for ReadFileHandler { } }; Ok(ToolOutput::Function { - content: collected.join("\n"), - content_items: None, + body: FunctionCallOutputBody::Text(collected.join("\n")), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/request_user_input.rs b/codex-rs/core/src/tools/handlers/request_user_input.rs index 6d014755b5..6e1da5641f 100644 --- a/codex-rs/core/src/tools/handlers/request_user_input.rs +++ b/codex-rs/core/src/tools/handlers/request_user_input.rs @@ -1,4 +1,5 @@ use async_trait::async_trait; +use codex_protocol::models::FunctionCallOutputBody; use crate::function_tool::FunctionCallError; use crate::tools::context::ToolInvocation; @@ -120,8 +121,7 @@ impl ToolHandler for RequestUserInputHandler { })?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/shell.rs b/codex-rs/core/src/tools/handlers/shell.rs index f62caea556..cd1aa9056a 100644 --- a/codex-rs/core/src/tools/handlers/shell.rs +++ b/codex-rs/core/src/tools/handlers/shell.rs @@ -1,5 +1,6 @@ use async_trait::async_trait; use codex_protocol::ThreadId; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::ShellCommandToolCallParams; use codex_protocol::models::ShellToolCallParams; use std::sync::Arc; @@ -329,8 +330,7 @@ impl ShellHandler { let event_ctx = ToolEventCtx::new(session.as_ref(), turn.as_ref(), &call_id, None); let content = emitter.finish(event_ctx, out).await?; Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/test_sync.rs b/codex-rs/core/src/tools/handlers/test_sync.rs index 643cb464fa..4d8fe1025b 100644 --- a/codex-rs/core/src/tools/handlers/test_sync.rs +++ b/codex-rs/core/src/tools/handlers/test_sync.rs @@ -1,3 +1,4 @@ +use codex_protocol::models::FunctionCallOutputBody; use std::collections::HashMap; use std::collections::hash_map::Entry; use std::sync::Arc; @@ -91,8 +92,7 @@ impl ToolHandler for TestSyncHandler { } Ok(ToolOutput::Function { - content: "ok".to_string(), - content_items: None, + body: FunctionCallOutputBody::Text("ok".to_string()), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/unified_exec.rs b/codex-rs/core/src/tools/handlers/unified_exec.rs index 2e331bae59..1eb51dd733 100644 --- a/codex-rs/core/src/tools/handlers/unified_exec.rs +++ b/codex-rs/core/src/tools/handlers/unified_exec.rs @@ -18,6 +18,7 @@ use crate::unified_exec::UnifiedExecProcessManager; use crate::unified_exec::UnifiedExecResponse; use crate::unified_exec::WriteStdinRequest; use async_trait::async_trait; +use codex_protocol::models::FunctionCallOutputBody; use serde::Deserialize; use std::path::PathBuf; use std::sync::Arc; @@ -238,8 +239,7 @@ impl ToolHandler for UnifiedExecHandler { let content = format_response(&response); Ok(ToolOutput::Function { - content, - content_items: None, + body: FunctionCallOutputBody::Text(content), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/handlers/view_image.rs b/codex-rs/core/src/tools/handlers/view_image.rs index 87dd7207b1..fe62522180 100644 --- a/codex-rs/core/src/tools/handlers/view_image.rs +++ b/codex-rs/core/src/tools/handlers/view_image.rs @@ -1,4 +1,5 @@ use async_trait::async_trait; +use codex_protocol::models::FunctionCallOutputBody; use serde::Deserialize; use tokio::fs; @@ -92,8 +93,7 @@ impl ToolHandler for ViewImageHandler { .await; Ok(ToolOutput::Function { - content: "attached local image path".to_string(), - content_items: None, + body: FunctionCallOutputBody::Text("attached local image path".to_string()), success: Some(true), }) } diff --git a/codex-rs/core/src/tools/orchestrator.rs b/codex-rs/core/src/tools/orchestrator.rs index e9fdd6208b..381f8ce136 100644 --- a/codex-rs/core/src/tools/orchestrator.rs +++ b/codex-rs/core/src/tools/orchestrator.rs @@ -8,6 +8,7 @@ retry without sandbox on denial (no re‑approval thanks to caching). use crate::error::CodexErr; use crate::error::SandboxErr; use crate::exec::ExecToolCallOutput; +use crate::features::Feature; use crate::sandboxing::SandboxManager; use crate::tools::sandboxing::ApprovalCtx; use crate::tools::sandboxing::ExecApprovalRequirement; @@ -43,7 +44,7 @@ impl ToolOrchestrator { where T: ToolRuntime, { - let otel = turn_ctx.client.get_otel_manager(); + let otel = turn_ctx.otel_manager.clone(); let otel_tn = &tool_ctx.tool_name; let otel_ci = &tool_ctx.call_id; let otel_user = ToolDecisionSource::User; @@ -97,12 +98,14 @@ impl ToolOrchestrator { // Platform-specific flag gating is handled by SandboxManager::select_initial // via crate::safety::get_platform_sandbox(..). + let use_linux_sandbox_bwrap = turn_ctx.features.enabled(Feature::UseLinuxSandboxBwrap); let initial_attempt = SandboxAttempt { sandbox: initial_sandbox, policy: &turn_ctx.sandbox_policy, manager: &self.sandbox, sandbox_cwd: &turn_ctx.cwd, codex_linux_sandbox_exe: turn_ctx.codex_linux_sandbox_exe.as_ref(), + use_linux_sandbox_bwrap, windows_sandbox_level: turn_ctx.windows_sandbox_level, }; @@ -154,6 +157,7 @@ impl ToolOrchestrator { manager: &self.sandbox, sandbox_cwd: &turn_ctx.cwd, codex_linux_sandbox_exe: None, + use_linux_sandbox_bwrap, windows_sandbox_level: turn_ctx.windows_sandbox_level, }; diff --git a/codex-rs/core/src/tools/parallel.rs b/codex-rs/core/src/tools/parallel.rs index dcd3ae40ad..ca08048bd8 100644 --- a/codex-rs/core/src/tools/parallel.rs +++ b/codex-rs/core/src/tools/parallel.rs @@ -17,6 +17,7 @@ use crate::tools::context::SharedTurnDiffTracker; use crate::tools::context::ToolPayload; use crate::tools::router::ToolCall; use crate::tools::router::ToolRouter; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::FunctionCallOutputPayload; use codex_protocol::models::ResponseInputItem; @@ -119,7 +120,7 @@ impl ToolCallRuntime { _ => ResponseInputItem::FunctionCallOutput { call_id: call.call_id.clone(), output: FunctionCallOutputPayload { - content: Self::abort_message(call, secs), + body: FunctionCallOutputBody::Text(Self::abort_message(call, secs)), ..Default::default() }, }, diff --git a/codex-rs/core/src/tools/registry.rs b/codex-rs/core/src/tools/registry.rs index 72b848e315..f67db63943 100644 --- a/codex-rs/core/src/tools/registry.rs +++ b/codex-rs/core/src/tools/registry.rs @@ -70,7 +70,7 @@ impl ToolRegistry { ) -> Result { let tool_name = invocation.tool_name.clone(); let call_id_owned = invocation.call_id.clone(); - let otel = invocation.turn.client.get_otel_manager(); + let otel = invocation.turn.otel_manager.clone(); let payload_for_response = invocation.payload.clone(); let log_payload = payload_for_response.log_payload(); diff --git a/codex-rs/core/src/tools/router.rs b/codex-rs/core/src/tools/router.rs index 51328ccc9f..1eb6190bcb 100644 --- a/codex-rs/core/src/tools/router.rs +++ b/codex-rs/core/src/tools/router.rs @@ -11,6 +11,7 @@ use crate::tools::registry::ToolRegistry; use crate::tools::spec::ToolsConfig; use crate::tools::spec::build_specs; use codex_protocol::dynamic_tools::DynamicToolSpec; +use codex_protocol::models::FunctionCallOutputBody; use codex_protocol::models::LocalShellAction; use codex_protocol::models::ResponseInputItem; use codex_protocol::models::ResponseItem; @@ -181,9 +182,8 @@ impl ToolRouter { ResponseInputItem::FunctionCallOutput { call_id, output: codex_protocol::models::FunctionCallOutputPayload { - content: message, + body: FunctionCallOutputBody::Text(message), success: Some(false), - ..Default::default() }, } } diff --git a/codex-rs/core/src/tools/sandboxing.rs b/codex-rs/core/src/tools/sandboxing.rs index a7d2bca62a..d50e592530 100644 --- a/codex-rs/core/src/tools/sandboxing.rs +++ b/codex-rs/core/src/tools/sandboxing.rs @@ -274,6 +274,7 @@ pub(crate) struct SandboxAttempt<'a> { pub(crate) manager: &'a SandboxManager, pub(crate) sandbox_cwd: &'a Path, pub codex_linux_sandbox_exe: Option<&'a std::path::PathBuf>, + pub use_linux_sandbox_bwrap: bool, pub windows_sandbox_level: codex_protocol::config_types::WindowsSandboxLevel, } @@ -282,14 +283,16 @@ impl<'a> SandboxAttempt<'a> { &self, spec: CommandSpec, ) -> Result { - self.manager.transform( - spec, - self.policy, - self.sandbox, - self.sandbox_cwd, - self.codex_linux_sandbox_exe, - self.windows_sandbox_level, - ) + self.manager + .transform(crate::sandboxing::SandboxTransformRequest { + spec, + policy: self.policy, + sandbox: self.sandbox, + sandbox_policy_cwd: self.sandbox_cwd, + codex_linux_sandbox_exe: self.codex_linux_sandbox_exe, + use_linux_sandbox_bwrap: self.use_linux_sandbox_bwrap, + windows_sandbox_level: self.windows_sandbox_level, + }) } } diff --git a/codex-rs/core/src/tools/spec.rs b/codex-rs/core/src/tools/spec.rs index 8851a157a2..ddac191a9b 100644 --- a/codex-rs/core/src/tools/spec.rs +++ b/codex-rs/core/src/tools/spec.rs @@ -1687,6 +1687,22 @@ mod tests { assert_eq!(&tool_names, &expected_tools,); } + fn assert_default_model_tools( + model_slug: &str, + features: &Features, + web_search_mode: Option, + shell_tool: &'static str, + expected_tail: &[&str], + ) { + let mut expected = if features.enabled(Feature::UnifiedExec) { + vec!["exec_command", "write_stdin"] + } else { + vec![shell_tool] + }; + expected.extend(expected_tail); + assert_model_tools(model_slug, features, web_search_mode, &expected); + } + #[test] fn web_search_mode_cached_sets_external_web_access_false() { let config = test_config(); @@ -1735,12 +1751,12 @@ mod tests { fn test_build_specs_gpt5_codex_default() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "gpt-5-codex", &features, Some(WebSearchMode::Cached), + "shell_command", &[ - "shell_command", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -1757,12 +1773,12 @@ mod tests { fn test_build_specs_gpt51_codex_default() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "gpt-5.1-codex", &features, Some(WebSearchMode::Cached), + "shell_command", &[ - "shell_command", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -1827,12 +1843,12 @@ mod tests { fn test_codex_mini_defaults() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "codex-mini-latest", &features, Some(WebSearchMode::Cached), + "local_shell", &[ - "local_shell", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -1848,12 +1864,12 @@ mod tests { fn test_codex_5_1_mini_defaults() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "gpt-5.1-codex-mini", &features, Some(WebSearchMode::Cached), + "shell_command", &[ - "shell_command", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -1870,12 +1886,12 @@ mod tests { fn test_gpt_5_defaults() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "gpt-5", &features, Some(WebSearchMode::Cached), + "shell", &[ - "shell", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -1891,12 +1907,12 @@ mod tests { fn test_gpt_5_1_defaults() { let mut features = Features::with_defaults(); features.enable(Feature::CollaborationModes); - assert_model_tools( + assert_default_model_tools( "gpt-5.1", &features, Some(WebSearchMode::Cached), + "shell_command", &[ - "shell_command", "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", diff --git a/codex-rs/core/src/transport_manager.rs b/codex-rs/core/src/transport_manager.rs deleted file mode 100644 index 2b5d095000..0000000000 --- a/codex-rs/core/src/transport_manager.rs +++ /dev/null @@ -1,22 +0,0 @@ -use std::sync::Arc; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::Ordering; - -#[derive(Clone, Debug, Default)] -pub struct TransportManager { - disable_websockets: Arc, -} - -impl TransportManager { - pub fn new() -> Self { - Self::default() - } - - pub fn disable_websockets(&self) -> bool { - self.disable_websockets.load(Ordering::Relaxed) - } - - pub fn activate_http_fallback(&self, websocket_enabled: bool) -> bool { - websocket_enabled && !self.disable_websockets.swap(true, Ordering::Relaxed) - } -} diff --git a/codex-rs/core/src/turn_metadata.rs b/codex-rs/core/src/turn_metadata.rs index 58b394a2f2..6d4878c7a3 100644 --- a/codex-rs/core/src/turn_metadata.rs +++ b/codex-rs/core/src/turn_metadata.rs @@ -20,7 +20,7 @@ struct TurnMetadata { workspaces: BTreeMap, } -pub(crate) async fn build_turn_metadata_header(cwd: &Path) -> Option { +pub async fn build_turn_metadata_header(cwd: &Path) -> Option { let repo_root = get_git_repo_root(cwd)?; let (latest_git_commit_hash, associated_remote_urls) = tokio::join!( diff --git a/codex-rs/core/tests/common/lib.rs b/codex-rs/core/tests/common/lib.rs index 5df2a74edd..3feadb0722 100644 --- a/codex-rs/core/tests/common/lib.rs +++ b/codex-rs/core/tests/common/lib.rs @@ -1,7 +1,6 @@ #![expect(clippy::expect_used)] use codex_utils_cargo_bin::CargoBinError; -use codex_utils_cargo_bin::find_resource; use tempfile::TempDir; use codex_core::CodexThread; @@ -147,40 +146,6 @@ pub fn load_sse_fixture_with_id_from_str(raw: &str, id: &str) -> String { .collect() } -/// Same as [`load_sse_fixture`], but replaces the placeholder `__ID__` in the -/// fixture template with the supplied identifier before parsing. This lets a -/// single JSON template be reused by multiple tests that each need a unique -/// `response_id`. -pub fn load_sse_fixture_with_id(path: impl AsRef, id: &str) -> String { - let p = path.as_ref(); - let full_path = match find_resource!(p) { - Ok(p) => p, - Err(err) => panic!( - "failed to find fixture template at {:?}: {err}", - path.as_ref() - ), - }; - - let raw = std::fs::read_to_string(full_path).expect("read fixture template"); - let replaced = raw.replace("__ID__", id); - let events: Vec = - serde_json::from_str(&replaced).expect("parse JSON fixture"); - events - .into_iter() - .map(|e| { - let kind = e - .get("type") - .and_then(|v| v.as_str()) - .expect("fixture event missing type"); - if e.as_object().map(|o| o.len() == 1).unwrap_or(false) { - format!("event: {kind}\n\n") - } else { - format!("event: {kind}\ndata: {e}\n\n") - } - }) - .collect() -} - pub async fn wait_for_event(codex: &CodexThread, predicate: F) -> codex_core::protocol::EventMsg where F: FnMut(&codex_core::protocol::EventMsg) -> bool, diff --git a/codex-rs/core/tests/common/responses.rs b/codex-rs/core/tests/common/responses.rs index 86a332874e..7d3b077085 100644 --- a/codex-rs/core/tests/common/responses.rs +++ b/codex-rs/core/tests/common/responses.rs @@ -375,6 +375,10 @@ pub fn sse(events: Vec) -> String { out } +pub fn sse_completed(id: &str) -> String { + sse(vec![ev_response_created(id), ev_completed(id)]) +} + /// Convenience: SSE event for a completed response with a specific id. pub fn ev_completed(id: &str) -> Value { serde_json::json!({ diff --git a/codex-rs/core/tests/fixtures/completed_template.json b/codex-rs/core/tests/fixtures/completed_template.json deleted file mode 100644 index 1774dc5e84..0000000000 --- a/codex-rs/core/tests/fixtures/completed_template.json +++ /dev/null @@ -1,16 +0,0 @@ -[ - { - "type": "response.completed", - "response": { - "id": "__ID__", - "usage": { - "input_tokens": 0, - "input_tokens_details": null, - "output_tokens": 0, - "output_tokens_details": null, - "total_tokens": 0 - }, - "output": [] - } - } -] diff --git a/codex-rs/core/tests/responses_headers.rs b/codex-rs/core/tests/responses_headers.rs index 86e511c472..9e70264bf9 100644 --- a/codex-rs/core/tests/responses_headers.rs +++ b/codex-rs/core/tests/responses_headers.rs @@ -10,7 +10,6 @@ use codex_core::ModelProviderInfo; use codex_core::Prompt; use codex_core::ResponseEvent; use codex_core::ResponseItem; -use codex_core::TransportManager; use codex_core::WEB_SEARCH_ELIGIBLE_HEADER; use codex_core::WireApi; use codex_core::models_manager::manager::ModelsManager; @@ -88,19 +87,19 @@ async fn responses_stream_includes_subagent_header_on_review() { session_source.clone(), ); - let mut client_session = ModelClient::new( - Arc::clone(&config), + let web_search_eligible = !matches!(config.web_search_mode, Some(WebSearchMode::Disabled)); + let client = ModelClient::new( None, - model_info, - otel_manager, - provider, - effort, - summary, conversation_id, + provider.clone(), session_source, - TransportManager::new(), - ) - .new_session(None); + config.model_verbosity, + false, + false, + false, + None, + ); + let mut client_session = client.new_session(); let mut prompt = Prompt::default(); prompt.input = vec![ResponseItem::Message { @@ -113,7 +112,18 @@ async fn responses_stream_includes_subagent_header_on_review() { phase: None, }]; - let mut stream = client_session.stream(&prompt).await.expect("stream failed"); + let mut stream = client_session + .stream( + &prompt, + &model_info, + &otel_manager, + effort, + summary, + web_search_eligible, + None, + ) + .await + .expect("stream failed"); while let Some(event) = stream.next().await { if matches!(event, Ok(ResponseEvent::Completed { .. })) { break; @@ -188,19 +198,19 @@ async fn responses_stream_includes_subagent_header_on_other() { session_source.clone(), ); - let mut client_session = ModelClient::new( - Arc::clone(&config), + let web_search_eligible = !matches!(config.web_search_mode, Some(WebSearchMode::Disabled)); + let client = ModelClient::new( None, - model_info, - otel_manager, - provider, - effort, - summary, conversation_id, + provider.clone(), session_source, - TransportManager::new(), - ) - .new_session(None); + config.model_verbosity, + false, + false, + false, + None, + ); + let mut client_session = client.new_session(); let mut prompt = Prompt::default(); prompt.input = vec![ResponseItem::Message { @@ -213,7 +223,18 @@ async fn responses_stream_includes_subagent_header_on_other() { phase: None, }]; - let mut stream = client_session.stream(&prompt).await.expect("stream failed"); + let mut stream = client_session + .stream( + &prompt, + &model_info, + &otel_manager, + effort, + summary, + web_search_eligible, + None, + ) + .await + .expect("stream failed"); while let Some(event) = stream.next().await { if matches!(event, Ok(ResponseEvent::Completed { .. })) { break; @@ -346,19 +367,19 @@ async fn responses_respects_model_info_overrides_from_config() { session_source.clone(), ); - let mut client = ModelClient::new( - Arc::clone(&config), + let web_search_eligible = !matches!(config.web_search_mode, Some(WebSearchMode::Disabled)); + let client = ModelClient::new( None, - model_info, - otel_manager, - provider, - effort, - summary, conversation_id, + provider.clone(), session_source, - TransportManager::new(), - ) - .new_session(None); + config.model_verbosity, + false, + false, + false, + None, + ); + let mut client_session = client.new_session(); let mut prompt = Prompt::default(); prompt.input = vec![ResponseItem::Message { @@ -371,7 +392,18 @@ async fn responses_respects_model_info_overrides_from_config() { phase: None, }]; - let mut stream = client.stream(&prompt).await.expect("stream failed"); + let mut stream = client_session + .stream( + &prompt, + &model_info, + &otel_manager, + effort, + summary, + web_search_eligible, + None, + ) + .await + .expect("stream failed"); while let Some(event) = stream.next().await { if matches!(event, Ok(ResponseEvent::Completed { .. })) { break; @@ -408,88 +440,14 @@ async fn responses_stream_includes_turn_metadata_header_for_git_workspace_e2e() responses::ev_response_created("resp-1"), responses::ev_completed("resp-1"), ]); - let provider = ModelProviderInfo { - name: "mock".into(), - base_url: Some(format!("{}/v1", server.uri())), - env_key: None, - env_key_instructions: None, - experimental_bearer_token: None, - wire_api: WireApi::Responses, - query_params: None, - http_headers: None, - env_http_headers: None, - request_max_retries: Some(0), - stream_max_retries: Some(0), - stream_idle_timeout_ms: Some(5_000), - requires_openai_auth: false, - supports_websockets: false, - }; - let codex_home = TempDir::new().expect("failed to create TempDir"); - let mut config = load_default_config_for_test(&codex_home).await; - config.model_provider_id = provider.name.clone(); - config.model_provider = provider.clone(); - let effort = config.model_reasoning_effort; - let summary = config.model_reasoning_summary; - let model = ModelsManager::get_model_offline(config.model.as_deref()); - config.model = Some(model.clone()); - let config = Arc::new(config); - - let conversation_id = ThreadId::new(); - let auth_mode = AuthMode::Chatgpt; - let session_source = - SessionSource::SubAgent(SubAgentSource::Other("turn-metadata-e2e".to_string())); - let model_info = ModelsManager::construct_model_info_offline(model.as_str(), &config); - let otel_manager = OtelManager::new( - conversation_id, - model.as_str(), - model_info.slug.as_str(), - None, - Some("test@test.com".to_string()), - Some(auth_mode), - false, - "test".to_string(), - session_source.clone(), - ); - - let client = ModelClient::new( - Arc::clone(&config), - None, - model_info, - otel_manager, - provider, - effort, - summary, - conversation_id, - session_source, - TransportManager::new(), - ); - - let workspace = TempDir::new().expect("workspace tempdir"); - let cwd = workspace.path(); - - let mut prompt = Prompt::default(); - prompt.input = vec![ResponseItem::Message { - id: None, - role: "user".into(), - content: vec![ContentItem::InputText { - text: "hello".into(), - }], - end_turn: None, - phase: None, - }]; + let test = test_codex().build(&server).await.expect("build test codex"); + let cwd = test.cwd_path(); let first_request = responses::mount_sse_once(&server, response_body.clone()).await; - let mut first_session = client.new_session(Some(cwd.to_path_buf())); - let mut first_stream = first_session - .stream(&prompt) + test.submit_turn("hello") .await - .expect("stream first turn"); - while let Some(event) = first_stream.next().await { - if matches!(event, Ok(ResponseEvent::Completed { .. })) { - break; - } - } + .expect("submit first turn prompt"); assert_eq!( first_request .single_request() @@ -539,21 +497,13 @@ async fn responses_stream_includes_turn_metadata_header_for_git_workspace_e2e() .trim() .to_string(); - let repo_root = std::fs::canonicalize(cwd) - .unwrap_or_else(|_| cwd.to_path_buf()) - .to_string_lossy() - .into_owned(); let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); loop { let request_recorder = responses::mount_sse_once(&server, response_body.clone()).await; - let mut session = client.new_session(Some(cwd.to_path_buf())); tokio::time::sleep(std::time::Duration::from_millis(50)).await; - let mut stream = session.stream(&prompt).await.expect("stream post-git turn"); - while let Some(event) = stream.next().await { - if matches!(event, Ok(ResponseEvent::Completed { .. })) { - break; - } - } + test.submit_turn("hello") + .await + .expect("submit post-git turn prompt"); let maybe_header = request_recorder .single_request() @@ -561,11 +511,14 @@ async fn responses_stream_includes_turn_metadata_header_for_git_workspace_e2e() if let Some(header_value) = maybe_header { let parsed: serde_json::Value = serde_json::from_str(&header_value) .expect("x-codex-turn-metadata should be valid JSON"); - let workspace = parsed + let workspaces = parsed .get("workspaces") .and_then(serde_json::Value::as_object) - .and_then(|workspaces| workspaces.get(&repo_root)) - .expect("metadata should include cwd repo root workspace entry"); + .expect("metadata should include workspaces"); + let workspace = workspaces + .values() + .next() + .expect("metadata should include at least one workspace entry"); assert_eq!( workspace diff --git a/codex-rs/core/tests/suite/client.rs b/codex-rs/core/tests/suite/client.rs index bd50708a2c..731ce97c0d 100644 --- a/codex-rs/core/tests/suite/client.rs +++ b/codex-rs/core/tests/suite/client.rs @@ -11,7 +11,6 @@ use codex_core::Prompt; use codex_core::ResponseEvent; use codex_core::ResponseItem; use codex_core::ThreadManager; -use codex_core::TransportManager; use codex_core::WireApi; use codex_core::auth::AuthCredentialsStoreMode; use codex_core::built_in_model_providers; @@ -36,8 +35,9 @@ use codex_protocol::models::WebSearchAction; use codex_protocol::openai_models::ReasoningEffort; use codex_protocol::user_input::UserInput; use core_test_support::load_default_config_for_test; -use core_test_support::load_sse_fixture_with_id; +use core_test_support::responses::ev_completed; use core_test_support::responses::ev_completed_with_tokens; +use core_test_support::responses::ev_response_created; use core_test_support::responses::mount_sse_once; use core_test_support::responses::mount_sse_once_match; use core_test_support::responses::mount_sse_sequence; @@ -64,11 +64,6 @@ use wiremock::matchers::method; use wiremock::matchers::path; use wiremock::matchers::query_param; -/// Build minimal SSE stream with completed marker using the JSON fixture. -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[expect(clippy::unwrap_used)] fn assert_message_role(request_body: &serde_json::Value, role: &str) { assert_eq!(request_body["role"].as_str().unwrap(), role); @@ -259,7 +254,11 @@ async fn resume_includes_initial_messages_and_sends_prior_items() { // Mock server that will receive the resumed request let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; // Configure Codex to resume from our file let codex_home = Arc::new(TempDir::new().unwrap()); @@ -377,7 +376,11 @@ async fn includes_conversation_id_and_model_headers_in_request() { // Mock server let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex().with_auth(CodexAuth::from_api_key("Test API Key")); let test = builder @@ -418,7 +421,11 @@ async fn includes_base_instructions_override_in_request() { skip_if_no_network!(); // Mock server let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex() .with_auth(CodexAuth::from_api_key("Test API Key")) @@ -462,7 +469,11 @@ async fn chatgpt_auth_sends_correct_request() { // Mock server let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut model_provider = built_in_model_providers()["openai"].clone(); model_provider.base_url = Some(format!("{}/api/codex", server.uri())); @@ -524,7 +535,10 @@ async fn prefers_apikey_when_config_prefers_apikey_even_with_chatgpt_tokens() { let first = ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") - .set_body_raw(sse_completed("resp1"), "text/event-stream"); + .set_body_raw( + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + "text/event-stream", + ); // Expect API key header, no ChatGPT account header required. Mock::given(method("POST")) @@ -590,7 +604,11 @@ async fn includes_user_instructions_message_in_request() { skip_if_no_network!(); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex() .with_auth(CodexAuth::from_api_key("Test API Key")) @@ -652,7 +670,11 @@ async fn skills_append_to_instructions() { skip_if_no_network!(); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let codex_home = Arc::new(TempDir::new().unwrap()); let skill_dir = codex_home.path().join("skills/demo"); @@ -720,7 +742,11 @@ async fn includes_configured_effort_in_request() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_model("gpt-5.1-codex") .with_config(|config| { @@ -761,7 +787,11 @@ async fn includes_no_effort_in_request() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_model("gpt-5.1-codex") .build(&server) @@ -800,7 +830,11 @@ async fn includes_default_reasoning_effort_in_request_when_defined_by_model_info skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex().with_model("gpt-5.1").build(&server).await?; codex @@ -835,7 +869,11 @@ async fn user_turn_collaboration_mode_overrides_model_and_effort() -> anyhow::Re skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, config, @@ -893,7 +931,11 @@ async fn configured_reasoning_summary_is_sent() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_config(|config| { config.model_reasoning_summary = ReasoningSummary::Concise; @@ -933,7 +975,11 @@ async fn reasoning_summary_is_omitted_when_disabled() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_config(|config| { config.model_reasoning_summary = ReasoningSummary::None; @@ -972,7 +1018,11 @@ async fn includes_default_verbosity_in_request() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex().with_model("gpt-5.1").build(&server).await?; codex @@ -1007,7 +1057,11 @@ async fn configured_verbosity_not_sent_for_models_without_support() -> anyhow::R skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_model("gpt-5.1-codex") .with_config(|config| { @@ -1047,7 +1101,11 @@ async fn configured_verbosity_is_sent() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_model("gpt-5.1") .with_config(|config| { @@ -1088,7 +1146,11 @@ async fn includes_developer_instructions_message_in_request() { skip_if_no_network!(); let server = MockServer::start().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex() .with_auth(CodexAuth::from_api_key("Test API Key")) .with_config(|config| { @@ -1201,19 +1263,18 @@ async fn azure_responses_request_includes_store_and_reasoning_ids() { SessionSource::Exec, ); - let mut client = ModelClient::new( - Arc::clone(&config), + let client = ModelClient::new( None, - model_info, - otel_manager, - provider, - effort, - summary, conversation_id, + provider.clone(), SessionSource::Exec, - TransportManager::new(), - ) - .new_session(None); + config.model_verbosity, + false, + false, + false, + None, + ); + let mut client_session = client.new_session(); let mut prompt = Prompt::default(); prompt.input.push(ResponseItem::Reasoning { @@ -1251,10 +1312,7 @@ async fn azure_responses_request_includes_store_and_reasoning_ids() { }); prompt.input.push(ResponseItem::FunctionCallOutput { call_id: "function-call-id".into(), - output: FunctionCallOutputPayload { - content: "ok".into(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".into()), }); prompt.input.push(ResponseItem::LocalShellCall { id: Some("local-shell-id".into()), @@ -1280,8 +1338,16 @@ async fn azure_responses_request_includes_store_and_reasoning_ids() { output: "ok".into(), }); - let mut stream = client - .stream(&prompt) + let mut stream = client_session + .stream( + &prompt, + &model_info, + &otel_manager, + effort, + summary, + true, + None, + ) .await .expect("responses stream to start"); @@ -1570,7 +1636,10 @@ async fn context_window_error_sets_total_tokens_to_model_window() -> anyhow::Res mount_sse_once_match( &server, body_string_contains("seed turn"), - sse_completed("resp_seed"), + sse(vec![ + ev_response_created("resp_seed"), + ev_completed("resp_seed"), + ]), ) .await; @@ -1656,7 +1725,10 @@ async fn azure_overrides_assign_properties_used_for_responses_url() { // First request – must NOT include `previous_response_id`. let first = ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") - .set_body_raw(sse_completed("resp1"), "text/event-stream"); + .set_body_raw( + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + "text/event-stream", + ); // Expect POST to /openai/responses with api-version query param Mock::given(method("POST")) @@ -1737,7 +1809,10 @@ async fn env_var_overrides_loaded_auth() { // First request – must NOT include `previous_response_id`. let first = ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") - .set_body_raw(sse_completed("resp1"), "text/event-stream"); + .set_body_raw( + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + "text/event-stream", + ); // Expect POST to /openai/responses with api-version query param Mock::given(method("POST")) diff --git a/codex-rs/core/tests/suite/client_websockets.rs b/codex-rs/core/tests/suite/client_websockets.rs index 51c94edd7b..e3a1ef0a8e 100644 --- a/codex-rs/core/tests/suite/client_websockets.rs +++ b/codex-rs/core/tests/suite/client_websockets.rs @@ -8,8 +8,8 @@ use codex_core::ModelProviderInfo; use codex_core::Prompt; use codex_core::ResponseEvent; use codex_core::ResponseItem; -use codex_core::TransportManager; use codex_core::WireApi; +use codex_core::X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER; use codex_core::features::Feature; use codex_core::models_manager::manager::ModelsManager; use codex_core::protocol::SessionSource; @@ -17,7 +17,10 @@ use codex_otel::OtelManager; use codex_otel::metrics::MetricsClient; use codex_otel::metrics::MetricsConfig; use codex_protocol::ThreadId; +use codex_protocol::account::PlanType; use codex_protocol::config_types::ReasoningSummary; +use codex_protocol::openai_models::ModelInfo; +use codex_protocol::openai_models::ReasoningEffort as ReasoningEffortConfig; use core_test_support::load_default_config_for_test; use core_test_support::responses::WebSocketConnectionConfig; use core_test_support::responses::WebSocketTestServer; @@ -29,6 +32,7 @@ use core_test_support::skip_if_no_network; use futures::StreamExt; use opentelemetry_sdk::metrics::InMemoryMetricExporter; use pretty_assertions::assert_eq; +use serde_json::json; use std::sync::Arc; use std::time::Duration; use tempfile::TempDir; @@ -39,6 +43,10 @@ const MODEL: &str = "gpt-5.2-codex"; struct WebsocketTestHarness { _codex_home: TempDir, client: ModelClient, + model_info: ModelInfo, + effort: Option, + summary: ReasoningSummary, + web_search_eligible: bool, otel_manager: OtelManager, } @@ -53,10 +61,10 @@ async fn responses_websocket_streams_request() { .await; let harness = websocket_harness(&server).await; - let mut session = harness.client.new_session(None); + let mut client_session = harness.client.new_session(); let prompt = prompt_with_input(vec![message_item("hello")]); - stream_until_complete(&mut session, &prompt).await; + stream_until_complete(&mut client_session, &harness, &prompt).await; let connection = server.single_connection(); assert_eq!(connection.len(), 1); @@ -83,10 +91,10 @@ async fn responses_websocket_emits_websocket_telemetry_events() { let harness = websocket_harness(&server).await; harness.otel_manager.reset_runtime_metrics(); - let mut session = harness.client.new_session(None); + let mut client_session = harness.client.new_session(); let prompt = prompt_with_input(vec![message_item("hello")]); - stream_until_complete(&mut session, &prompt).await; + stream_until_complete(&mut client_session, &harness, &prompt).await; tokio::time::sleep(Duration::from_millis(10)).await; @@ -102,6 +110,72 @@ async fn responses_websocket_emits_websocket_telemetry_events() { server.shutdown().await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn responses_websocket_includes_timing_metrics_header_when_runtime_metrics_enabled() { + skip_if_no_network!(); + + let server = start_websocket_server(vec![vec![vec![ + ev_response_created("resp-1"), + serde_json::json!({ + "type": "responsesapi.websocket_timing", + "timing_metrics": { + "responses_duration_excl_engine_and_client_tool_time_ms": 120, + "engine_service_total_ms": 450 + } + }), + ev_completed("resp-1"), + ]]]) + .await; + + let harness = websocket_harness_with_runtime_metrics(&server, true).await; + harness.otel_manager.reset_runtime_metrics(); + let mut client_session = harness.client.new_session(); + let prompt = prompt_with_input(vec![message_item("hello")]); + + stream_until_complete(&mut client_session, &harness, &prompt).await; + tokio::time::sleep(Duration::from_millis(10)).await; + + let handshake = server.single_handshake(); + assert_eq!( + handshake.header(X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER), + Some("true".to_string()) + ); + + let summary = harness + .otel_manager + .runtime_metrics_summary() + .expect("runtime metrics summary"); + assert_eq!(summary.responses_api_overhead_ms, 120); + assert_eq!(summary.responses_api_inference_time_ms, 450); + + server.shutdown().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn responses_websocket_omits_timing_metrics_header_when_runtime_metrics_disabled() { + skip_if_no_network!(); + + let server = start_websocket_server(vec![vec![vec![ + ev_response_created("resp-1"), + ev_completed("resp-1"), + ]]]) + .await; + + let harness = websocket_harness_with_runtime_metrics(&server, false).await; + let mut client_session = harness.client.new_session(); + let prompt = prompt_with_input(vec![message_item("hello")]); + + stream_until_complete(&mut client_session, &harness, &prompt).await; + + let handshake = server.single_handshake(); + assert_eq!( + handshake.header(X_RESPONSESAPI_INCLUDE_TIMING_METRICS_HEADER), + None + ); + + server.shutdown().await; +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn responses_websocket_emits_reasoning_included_event() { skip_if_no_network!(); @@ -113,11 +187,19 @@ async fn responses_websocket_emits_reasoning_included_event() { .await; let harness = websocket_harness(&server).await; - let mut session = harness.client.new_session(None); + let mut client_session = harness.client.new_session(); let prompt = prompt_with_input(vec![message_item("hello")]); - let mut stream = session - .stream(&prompt) + let mut stream = client_session + .stream( + &prompt, + &harness.model_info, + &harness.otel_manager, + harness.effort, + harness.summary, + harness.web_search_eligible, + None, + ) .await .expect("websocket stream failed"); @@ -136,6 +218,98 @@ async fn responses_websocket_emits_reasoning_included_event() { server.shutdown().await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn responses_websocket_emits_rate_limit_events() { + skip_if_no_network!(); + + let rate_limit_event = json!({ + "type": "codex.rate_limits", + "plan_type": "plus", + "rate_limits": { + "allowed": true, + "limit_reached": false, + "primary": { + "used_percent": 42, + "window_minutes": 60, + "reset_at": 1700000000 + }, + "secondary": null + }, + "code_review_rate_limits": null, + "credits": { + "has_credits": true, + "unlimited": false, + "balance": "123" + }, + "promo": null + }); + + let server = start_websocket_server_with_headers(vec![WebSocketConnectionConfig { + requests: vec![vec![ + rate_limit_event, + ev_response_created("resp-1"), + ev_completed("resp-1"), + ]], + response_headers: vec![ + ("X-Models-Etag".to_string(), "etag-123".to_string()), + ("X-Reasoning-Included".to_string(), "true".to_string()), + ], + }]) + .await; + + let harness = websocket_harness(&server).await; + let mut client_session = harness.client.new_session(); + let prompt = prompt_with_input(vec![message_item("hello")]); + + let mut stream = client_session + .stream( + &prompt, + &harness.model_info, + &harness.otel_manager, + harness.effort, + harness.summary, + harness.web_search_eligible, + None, + ) + .await + .expect("websocket stream failed"); + + let mut saw_rate_limits = None; + let mut saw_models_etag = None; + let mut saw_reasoning_included = false; + + while let Some(event) = stream.next().await { + match event.expect("event") { + ResponseEvent::RateLimits(snapshot) => { + saw_rate_limits = Some(snapshot); + } + ResponseEvent::ModelsEtag(etag) => { + saw_models_etag = Some(etag); + } + ResponseEvent::ServerReasoningIncluded(true) => { + saw_reasoning_included = true; + } + ResponseEvent::Completed { .. } => break, + _ => {} + } + } + + let rate_limits = saw_rate_limits.expect("missing rate limits"); + let primary = rate_limits.primary.expect("missing primary window"); + assert_eq!(primary.used_percent, 42.0); + assert_eq!(primary.window_minutes, Some(60)); + assert_eq!(primary.resets_at, Some(1_700_000_000)); + assert_eq!(rate_limits.plan_type, Some(PlanType::Plus)); + let credits = rate_limits.credits.expect("missing credits"); + assert!(credits.has_credits); + assert!(!credits.unlimited); + assert_eq!(credits.balance.as_deref(), Some("123")); + assert_eq!(saw_models_etag.as_deref(), Some("etag-123")); + assert!(saw_reasoning_included); + + server.shutdown().await; +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn responses_websocket_appends_on_prefix() { skip_if_no_network!(); @@ -147,12 +321,12 @@ async fn responses_websocket_appends_on_prefix() { .await; let harness = websocket_harness(&server).await; - let mut session = harness.client.new_session(None); + let mut client_session = harness.client.new_session(); let prompt_one = prompt_with_input(vec![message_item("hello")]); let prompt_two = prompt_with_input(vec![message_item("hello"), message_item("second")]); - stream_until_complete(&mut session, &prompt_one).await; - stream_until_complete(&mut session, &prompt_two).await; + stream_until_complete(&mut client_session, &harness, &prompt_one).await; + stream_until_complete(&mut client_session, &harness, &prompt_two).await; let connection = server.single_connection(); assert_eq!(connection.len(), 2); @@ -183,12 +357,12 @@ async fn responses_websocket_creates_on_non_prefix() { .await; let harness = websocket_harness(&server).await; - let mut session = harness.client.new_session(None); + let mut client_session = harness.client.new_session(); let prompt_one = prompt_with_input(vec![message_item("hello")]); let prompt_two = prompt_with_input(vec![message_item("different")]); - stream_until_complete(&mut session, &prompt_one).await; - stream_until_complete(&mut session, &prompt_two).await; + stream_until_complete(&mut client_session, &harness, &prompt_one).await; + stream_until_complete(&mut client_session, &harness, &prompt_two).await; let connection = server.single_connection(); assert_eq!(connection.len(), 2); @@ -241,11 +415,21 @@ fn websocket_provider(server: &WebSocketTestServer) -> ModelProviderInfo { } async fn websocket_harness(server: &WebSocketTestServer) -> WebsocketTestHarness { + websocket_harness_with_runtime_metrics(server, false).await +} + +async fn websocket_harness_with_runtime_metrics( + server: &WebSocketTestServer, + runtime_metrics_enabled: bool, +) -> WebsocketTestHarness { let provider = websocket_provider(server); let codex_home = TempDir::new().unwrap(); let mut config = load_default_config_for_test(&codex_home).await; config.model = Some(MODEL.to_string()); config.features.enable(Feature::ResponsesWebsockets); + if runtime_metrics_enabled { + config.features.enable(Feature::RuntimeMetrics); + } let config = Arc::new(config); let model_info = ModelsManager::construct_model_info_offline(MODEL, &config); let conversation_id = ThreadId::new(); @@ -268,29 +452,47 @@ async fn websocket_harness(server: &WebSocketTestServer) -> WebsocketTestHarness SessionSource::Exec, ) .with_metrics(metrics); + let effort = None; + let summary = ReasoningSummary::Auto; + let web_search_eligible = true; let client = ModelClient::new( - Arc::clone(&config), None, - model_info, - otel_manager.clone(), - provider.clone(), - None, - ReasoningSummary::Auto, conversation_id, + provider.clone(), SessionSource::Exec, - TransportManager::new(), + config.model_verbosity, + true, + false, + runtime_metrics_enabled, + None, ); WebsocketTestHarness { _codex_home: codex_home, client, + model_info, + effort, + summary, + web_search_eligible, otel_manager, } } -async fn stream_until_complete(session: &mut ModelClientSession, prompt: &Prompt) { - let mut stream = session - .stream(prompt) +async fn stream_until_complete( + client_session: &mut ModelClientSession, + harness: &WebsocketTestHarness, + prompt: &Prompt, +) { + let mut stream = client_session + .stream( + prompt, + &harness.model_info, + &harness.otel_manager, + harness.effort, + harness.summary, + harness.web_search_eligible, + None, + ) .await .expect("websocket stream failed"); diff --git a/codex-rs/core/tests/suite/collaboration_instructions.rs b/codex-rs/core/tests/suite/collaboration_instructions.rs index 21a0b27613..dc9905379e 100644 --- a/codex-rs/core/tests/suite/collaboration_instructions.rs +++ b/codex-rs/core/tests/suite/collaboration_instructions.rs @@ -18,10 +18,6 @@ use core_test_support::wait_for_event; use pretty_assertions::assert_eq; use serde_json::Value; -fn sse_completed(id: &str) -> String { - sse(vec![ev_response_created(id), ev_completed(id)]) -} - fn collab_mode_with_mode_and_instructions( mode: ModeKind, instructions: Option<&str>, @@ -72,7 +68,11 @@ async fn no_collaboration_instructions_by_default() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; @@ -100,7 +100,11 @@ async fn user_input_includes_collaboration_instructions_after_override() -> Resu skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; @@ -144,7 +148,11 @@ async fn collaboration_instructions_added_on_user_turn() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; let collab_text = "turn instructions"; @@ -182,7 +190,11 @@ async fn override_then_next_turn_uses_updated_collaboration_instructions() -> Re skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; let collab_text = "override instructions"; @@ -226,7 +238,11 @@ async fn user_turn_overrides_collaboration_instructions_after_override() -> Resu skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; let base_text = "base instructions"; @@ -282,8 +298,16 @@ async fn collaboration_mode_update_emits_new_instruction_message() -> Result<()> skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let test = test_codex().build(&server).await?; let first_text = "first instructions"; @@ -354,8 +378,16 @@ async fn collaboration_mode_update_noop_does_not_append() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let test = test_codex().build(&server).await?; let collab_text = "same instructions"; @@ -423,8 +455,16 @@ async fn collaboration_mode_update_emits_new_instruction_message_when_mode_chang skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let test = test_codex().build(&server).await?; let default_text = "default mode instructions"; @@ -501,8 +541,16 @@ async fn collaboration_mode_update_noop_does_not_append_when_mode_is_unchanged() skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let test = test_codex().build(&server).await?; let collab_text = "mode-stable instructions"; @@ -576,8 +624,16 @@ async fn resume_replays_collaboration_instructions() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let mut builder = test_codex(); let initial = builder.build(&server).await?; @@ -642,7 +698,11 @@ async fn empty_collaboration_instructions_are_ignored() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let test = test_codex().build(&server).await?; diff --git a/codex-rs/core/tests/suite/exec.rs b/codex-rs/core/tests/suite/exec.rs index cdf597a4e9..b70062d7ed 100644 --- a/codex-rs/core/tests/suite/exec.rs +++ b/codex-rs/core/tests/suite/exec.rs @@ -44,7 +44,7 @@ async fn run_test_cmd(tmp: TempDir, cmd: Vec<&str>) -> Result String { - core_test_support::load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn fork_thread_twice_drops_to_first_message() { skip_if_no_network!(); // Start a mock server that completes three turns. let server = MockServer::start().await; - let sse = sse_completed("resp"); + let sse = sse(vec![ev_response_created("resp"), ev_completed("resp")]); let first = ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") .set_body_raw(sse.clone(), "text/event-stream"); diff --git a/codex-rs/core/tests/suite/hierarchical_agents.rs b/codex-rs/core/tests/suite/hierarchical_agents.rs index 5dfa636489..5f1f84340d 100644 --- a/codex-rs/core/tests/suite/hierarchical_agents.rs +++ b/codex-rs/core/tests/suite/hierarchical_agents.rs @@ -1,20 +1,22 @@ use codex_core::features::Feature; -use core_test_support::load_sse_fixture_with_id; +use core_test_support::responses::ev_completed; +use core_test_support::responses::ev_response_created; use core_test_support::responses::mount_sse_once; +use core_test_support::responses::sse; use core_test_support::responses::start_mock_server; use core_test_support::test_codex::test_codex; const HIERARCHICAL_AGENTS_SNIPPET: &str = "Files called AGENTS.md commonly appear in many places inside a container"; -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn hierarchical_agents_appends_to_project_doc_in_user_instructions() { let server = start_mock_server().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex().with_config(|config| { config.features.enable(Feature::ChildAgentsMd); @@ -49,7 +51,11 @@ async fn hierarchical_agents_appends_to_project_doc_in_user_instructions() { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn hierarchical_agents_emits_when_no_project_doc() { let server = start_mock_server().await; - let resp_mock = mount_sse_once(&server, sse_completed("resp1")).await; + let resp_mock = mount_sse_once( + &server, + sse(vec![ev_response_created("resp1"), ev_completed("resp1")]), + ) + .await; let mut builder = test_codex().with_config(|config| { config.features.enable(Feature::ChildAgentsMd); diff --git a/codex-rs/core/tests/suite/live_reload.rs b/codex-rs/core/tests/suite/live_reload.rs new file mode 100644 index 0000000000..e46c77128f --- /dev/null +++ b/codex-rs/core/tests/suite/live_reload.rs @@ -0,0 +1,151 @@ +#![allow(clippy::expect_used, clippy::unwrap_used)] + +use std::fs; +use std::path::Path; +use std::path::PathBuf; +use std::time::Duration; + +use anyhow::Result; +use codex_core::config::ProjectConfig; +use codex_core::protocol::AskForApproval; +use codex_core::protocol::EventMsg; +use codex_core::protocol::Op; +use codex_core::protocol::SandboxPolicy; +use codex_protocol::config_types::ReasoningSummary; +use codex_protocol::config_types::TrustLevel; +use codex_protocol::user_input::UserInput; +use core_test_support::responses; +use core_test_support::responses::ResponsesRequest; +use core_test_support::responses::mount_sse_sequence; +use core_test_support::responses::start_mock_server; +use core_test_support::test_codex::TestCodex; +use core_test_support::test_codex::test_codex; +use core_test_support::wait_for_event; +use tokio::time::timeout; + +fn enable_trusted_project(config: &mut codex_core::config::Config) { + config.active_project = ProjectConfig { + trust_level: Some(TrustLevel::Trusted), + }; +} + +fn write_skill(home: &Path, name: &str, description: &str, body: &str) -> PathBuf { + let skill_dir = home.join("skills").join(name); + fs::create_dir_all(&skill_dir).expect("create skill dir"); + let contents = format!("---\nname: {name}\ndescription: {description}\n---\n\n{body}\n"); + let path = skill_dir.join("SKILL.md"); + fs::write(&path, contents).expect("write skill"); + path +} + +fn contains_skill_body(request: &ResponsesRequest, skill_body: &str) -> bool { + request + .message_input_texts("user") + .iter() + .any(|text| text.contains(skill_body) && text.contains("")) +} + +async fn submit_skill_turn(test: &TestCodex, skill_path: PathBuf, prompt: &str) -> Result<()> { + let session_model = test.session_configured.model.clone(); + test.codex + .submit(Op::UserTurn { + items: vec![ + UserInput::Text { + text: prompt.to_string(), + text_elements: Vec::new(), + }, + UserInput::Skill { + name: "demo".to_string(), + path: skill_path, + }, + ], + final_output_json_schema: None, + cwd: test.cwd_path().to_path_buf(), + approval_policy: AskForApproval::Never, + sandbox_policy: SandboxPolicy::DangerFullAccess, + model: session_model, + effort: None, + summary: ReasoningSummary::Auto, + collaboration_mode: None, + personality: None, + }) + .await?; + + wait_for_event(test.codex.as_ref(), |event| { + matches!(event, EventMsg::TurnComplete(_)) + }) + .await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn live_skills_reload_refreshes_skill_cache_after_skill_change() -> Result<()> { + let server = start_mock_server().await; + let responses = mount_sse_sequence( + &server, + vec![ + responses::sse(vec![responses::ev_completed("resp-1")]), + responses::sse(vec![responses::ev_completed("resp-2")]), + ], + ) + .await; + + let skill_v1 = "skill body v1"; + let skill_v2 = "skill body v2"; + let mut builder = test_codex() + .with_pre_build_hook(move |home| { + write_skill(home, "demo", "demo skill", skill_v1); + }) + .with_config(|config| { + enable_trusted_project(config); + }); + let test = builder.build(&server).await?; + + let skill_path = std::fs::canonicalize(test.codex_home_path().join("skills/demo/SKILL.md"))?; + + submit_skill_turn(&test, skill_path.clone(), "please use $demo").await?; + let first_request = responses + .requests() + .first() + .cloned() + .expect("first request captured"); + assert!( + contains_skill_body(&first_request, skill_v1), + "expected initial skill body in request" + ); + + write_skill(test.codex_home_path(), "demo", "demo skill", skill_v2); + + let saw_skills_update = timeout(Duration::from_secs(5), async { + loop { + match test.codex.next_event().await { + Ok(event) => { + if matches!(event.msg, EventMsg::SkillsUpdateAvailable) { + break; + } + } + Err(err) => panic!("event stream ended unexpectedly: {err}"), + } + } + }) + .await; + + if saw_skills_update.is_err() { + // Some environments do not reliably surface file watcher events for + // skill changes. Clear the cache explicitly so we can still validate + // that the updated skill body is injected on the next turn. + test.thread_manager.skills_manager().clear_cache(); + } + + submit_skill_turn(&test, skill_path.clone(), "please use $demo again").await?; + let last_request = responses + .last_request() + .expect("request captured after skill update"); + + assert!( + contains_skill_body(&last_request, skill_v2), + "expected updated skill body after reload" + ); + + Ok(()) +} diff --git a/codex-rs/core/tests/suite/mod.rs b/codex-rs/core/tests/suite/mod.rs index f34ca09b40..8b057c5e16 100644 --- a/codex-rs/core/tests/suite/mod.rs +++ b/codex-rs/core/tests/suite/mod.rs @@ -1,16 +1,57 @@ // Aggregates all former standalone integration tests as modules. +use std::ffi::OsString; + +use codex_arg0::Arg0PathEntryGuard; use codex_arg0::arg0_dispatch; use ctor::ctor; use tempfile::TempDir; +struct TestCodexAliasesGuard { + _codex_home: TempDir, + _arg0: Arg0PathEntryGuard, + _previous_codex_home: Option, +} + +const CODEX_HOME_ENV_VAR: &str = "CODEX_HOME"; + // This code runs before any other tests are run. // It allows the test binary to behave like codex and dispatch to apply_patch and codex-linux-sandbox // based on the arg0. // NOTE: this doesn't work on ARM #[ctor] -pub static CODEX_ALIASES_TEMP_DIR: TempDir = unsafe { +pub static CODEX_ALIASES_TEMP_DIR: TestCodexAliasesGuard = unsafe { #[allow(clippy::unwrap_used)] - arg0_dispatch().unwrap() + let codex_home = tempfile::Builder::new() + .prefix("codex-core-tests") + .tempdir() + .unwrap(); + let previous_codex_home = std::env::var_os(CODEX_HOME_ENV_VAR); + // arg0_dispatch() creates helper links under CODEX_HOME/tmp. Point it at a + // test-owned temp dir so startup never mutates the developer's real ~/.codex. + // + // Safety: #[ctor] runs before tests start, so no test threads exist yet. + unsafe { + std::env::set_var(CODEX_HOME_ENV_VAR, codex_home.path()); + } + + #[allow(clippy::unwrap_used)] + let arg0 = arg0_dispatch().unwrap(); + // Restore the process environment immediately so later tests observe the + // same CODEX_HOME state they started with. + match previous_codex_home.as_ref() { + Some(value) => unsafe { + std::env::set_var(CODEX_HOME_ENV_VAR, value); + }, + None => unsafe { + std::env::remove_var(CODEX_HOME_ENV_VAR); + }, + } + + TestCodexAliasesGuard { + _codex_home: codex_home, + _arg0: arg0, + _previous_codex_home: previous_codex_home, + } }; #[cfg(not(target_os = "windows"))] @@ -40,6 +81,7 @@ mod json_result; mod list_dir; mod list_models; mod live_cli; +mod live_reload; mod model_info_overrides; mod model_overrides; mod model_tools; diff --git a/codex-rs/core/tests/suite/model_tools.rs b/codex-rs/core/tests/suite/model_tools.rs index b08b5281c8..d74ab0d7be 100644 --- a/codex-rs/core/tests/suite/model_tools.rs +++ b/codex-rs/core/tests/suite/model_tools.rs @@ -2,16 +2,11 @@ use codex_core::features::Feature; use codex_protocol::config_types::WebSearchMode; -use core_test_support::load_sse_fixture_with_id; use core_test_support::responses; use core_test_support::responses::start_mock_server; use core_test_support::skip_if_no_network; use core_test_support::test_codex::test_codex; -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[allow(clippy::expect_used)] fn tool_identifiers(body: &serde_json::Value) -> Vec { body["tools"] @@ -31,7 +26,10 @@ fn tool_identifiers(body: &serde_json::Value) -> Vec { #[allow(clippy::expect_used)] async fn collect_tool_identifiers_for_model(model: &str) -> Vec { let server = start_mock_server().await; - let sse = sse_completed(model); + let sse = responses::sse(vec![ + responses::ev_response_created(model), + responses::ev_completed(model), + ]); let resp_mock = responses::mount_sse_once(&server, sse).await; let mut builder = test_codex() @@ -52,6 +50,16 @@ async fn collect_tool_identifiers_for_model(model: &str) -> Vec { tool_identifiers(&body) } +fn expected_default_tools(shell_tool: &str, tail: &[&str]) -> Vec { + let mut tools = if cfg!(windows) { + vec![shell_tool.to_string()] + } else { + vec!["exec_command".to_string(), "write_stdin".to_string()] + }; + tools.extend(tail.iter().map(|tool| (*tool).to_string())); + tools +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn model_selects_expected_tools() { skip_if_no_network!(); @@ -60,83 +68,93 @@ async fn model_selects_expected_tools() { let codex_tools = collect_tool_identifiers_for_model("codex-mini-latest").await; assert_eq!( codex_tools, - vec![ - "local_shell".to_string(), - "list_mcp_resources".to_string(), - "list_mcp_resource_templates".to_string(), - "read_mcp_resource".to_string(), - "update_plan".to_string(), - "request_user_input".to_string(), - "web_search".to_string(), - "view_image".to_string() - ], + expected_default_tools( + "local_shell", + &[ + "list_mcp_resources", + "list_mcp_resource_templates", + "read_mcp_resource", + "update_plan", + "request_user_input", + "web_search", + "view_image", + ], + ), "codex-mini-latest should expose the local shell tool", ); let gpt5_codex_tools = collect_tool_identifiers_for_model("gpt-5-codex").await; assert_eq!( gpt5_codex_tools, - vec![ - "shell_command".to_string(), - "list_mcp_resources".to_string(), - "list_mcp_resource_templates".to_string(), - "read_mcp_resource".to_string(), - "update_plan".to_string(), - "request_user_input".to_string(), - "apply_patch".to_string(), - "web_search".to_string(), - "view_image".to_string() - ], + expected_default_tools( + "shell_command", + &[ + "list_mcp_resources", + "list_mcp_resource_templates", + "read_mcp_resource", + "update_plan", + "request_user_input", + "apply_patch", + "web_search", + "view_image", + ], + ), "gpt-5-codex should expose the apply_patch tool", ); let gpt51_codex_tools = collect_tool_identifiers_for_model("gpt-5.1-codex").await; assert_eq!( gpt51_codex_tools, - vec![ - "shell_command".to_string(), - "list_mcp_resources".to_string(), - "list_mcp_resource_templates".to_string(), - "read_mcp_resource".to_string(), - "update_plan".to_string(), - "request_user_input".to_string(), - "apply_patch".to_string(), - "web_search".to_string(), - "view_image".to_string() - ], + expected_default_tools( + "shell_command", + &[ + "list_mcp_resources", + "list_mcp_resource_templates", + "read_mcp_resource", + "update_plan", + "request_user_input", + "apply_patch", + "web_search", + "view_image", + ], + ), "gpt-5.1-codex should expose the apply_patch tool", ); let gpt5_tools = collect_tool_identifiers_for_model("gpt-5").await; assert_eq!( gpt5_tools, - vec![ - "shell".to_string(), - "list_mcp_resources".to_string(), - "list_mcp_resource_templates".to_string(), - "read_mcp_resource".to_string(), - "update_plan".to_string(), - "request_user_input".to_string(), - "web_search".to_string(), - "view_image".to_string() - ], + expected_default_tools( + "shell", + &[ + "list_mcp_resources", + "list_mcp_resource_templates", + "read_mcp_resource", + "update_plan", + "request_user_input", + "web_search", + "view_image", + ], + ), "gpt-5 should expose the apply_patch tool", ); let gpt51_tools = collect_tool_identifiers_for_model("gpt-5.1").await; assert_eq!( gpt51_tools, - vec![ - "shell_command".to_string(), - "list_mcp_resources".to_string(), - "list_mcp_resource_templates".to_string(), - "read_mcp_resource".to_string(), - "update_plan".to_string(), - "request_user_input".to_string(), - "apply_patch".to_string(), - "web_search".to_string(), - "view_image".to_string() - ], + expected_default_tools( + "shell_command", + &[ + "list_mcp_resources", + "list_mcp_resource_templates", + "read_mcp_resource", + "update_plan", + "request_user_input", + "apply_patch", + "web_search", + "view_image", + ], + ), "gpt-5.1 should expose the apply_patch tool", ); let exp_tools = collect_tool_identifiers_for_model("exp-5.1").await; diff --git a/codex-rs/core/tests/suite/pending_input.rs b/codex-rs/core/tests/suite/pending_input.rs index ad662d901b..1fcff4da3b 100644 --- a/codex-rs/core/tests/suite/pending_input.rs +++ b/codex-rs/core/tests/suite/pending_input.rs @@ -125,6 +125,11 @@ async fn injected_user_input_triggers_follow_up_request_with_deltas() { let _ = gate_completed_tx.send(()); + let _ = wait_for_event(&codex, |event| { + matches!(event, EventMsg::UserMessage(message) if message.message == "second prompt") + }) + .await; + wait_for_event(&codex, |event| matches!(event, EventMsg::TurnComplete(_))).await; let requests = server.requests().await; diff --git a/codex-rs/core/tests/suite/permissions_messages.rs b/codex-rs/core/tests/suite/permissions_messages.rs index c54203b4f1..fc1f0fa0b6 100644 --- a/codex-rs/core/tests/suite/permissions_messages.rs +++ b/codex-rs/core/tests/suite/permissions_messages.rs @@ -43,16 +43,16 @@ fn permissions_texts(input: &[serde_json::Value]) -> Vec { .collect() } -fn sse_completed(id: &str) -> String { - sse(vec![ev_response_created(id), ev_completed(id)]) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn permissions_message_sent_once_on_start() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let mut builder = test_codex().with_config(move |config| { config.approval_policy = Constrained::allow_any(AskForApproval::OnRequest); @@ -84,8 +84,16 @@ async fn permissions_message_added_on_override_change() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let mut builder = test_codex().with_config(move |config| { config.approval_policy = Constrained::allow_any(AskForApproval::OnRequest); @@ -148,8 +156,16 @@ async fn permissions_message_not_added_when_no_change() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let mut builder = test_codex().with_config(move |config| { config.approval_policy = Constrained::allow_any(AskForApproval::OnRequest); @@ -197,9 +213,21 @@ async fn resume_replays_permissions_messages() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let _req2 = mount_sse_once(&server, sse_completed("resp-2")).await; - let req3 = mount_sse_once(&server, sse_completed("resp-3")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let _req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; + let req3 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-3"), ev_completed("resp-3")]), + ) + .await; let mut builder = test_codex().with_config(|config| { config.approval_policy = Constrained::allow_any(AskForApproval::OnRequest); @@ -279,10 +307,26 @@ async fn resume_and_fork_append_permissions_messages() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let _req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; - let req3 = mount_sse_once(&server, sse_completed("resp-3")).await; - let req4 = mount_sse_once(&server, sse_completed("resp-4")).await; + let _req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; + let req3 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-3"), ev_completed("resp-3")]), + ) + .await; + let req4 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-4"), ev_completed("resp-4")]), + ) + .await; let mut builder = test_codex().with_config(|config| { config.approval_policy = Constrained::allow_any(AskForApproval::OnRequest); @@ -404,7 +448,11 @@ async fn permissions_message_includes_writable_roots() -> Result<()> { skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let writable = TempDir::new()?; let writable_root = AbsolutePathBuf::try_from(writable.path())?; let sandbox_policy = SandboxPolicy::WorkspaceWrite { diff --git a/codex-rs/core/tests/suite/personality.rs b/codex-rs/core/tests/suite/personality.rs index 87978ceb5a..ff2c0cfec0 100644 --- a/codex-rs/core/tests/suite/personality.rs +++ b/codex-rs/core/tests/suite/personality.rs @@ -19,12 +19,10 @@ use codex_protocol::openai_models::TruncationPolicyConfig; use codex_protocol::openai_models::default_input_modalities; use codex_protocol::user_input::UserInput; use core_test_support::load_default_config_for_test; -use core_test_support::responses::ev_completed; -use core_test_support::responses::ev_response_created; use core_test_support::responses::mount_models_once; use core_test_support::responses::mount_sse_once; use core_test_support::responses::mount_sse_sequence; -use core_test_support::responses::sse; +use core_test_support::responses::sse_completed; use core_test_support::responses::start_mock_server; use core_test_support::skip_if_no_network; use core_test_support::test_codex::test_codex; @@ -42,10 +40,6 @@ const LOCAL_FRIENDLY_TEMPLATE: &str = "You optimize for team morale and being a supportive teammate as much as code quality."; const LOCAL_PRAGMATIC_TEMPLATE: &str = "You are a deeply pragmatic, effective software engineer."; -fn sse_completed(id: &str) -> String { - sse(vec![ev_response_created(id), ev_completed(id)]) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn personality_does_not_mutate_base_instructions_without_template() { let codex_home = TempDir::new().expect("create temp dir"); @@ -177,6 +171,111 @@ async fn config_personality_some_sets_instructions_template() -> anyhow::Result< Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn config_personality_none_sends_no_personality() -> anyhow::Result<()> { + skip_if_no_network!(Ok(())); + + let server = start_mock_server().await; + let resp_mock = mount_sse_once(&server, sse_completed("resp-1")).await; + let mut builder = test_codex() + .with_model("gpt-5.2-codex") + .with_config(|config| { + config.features.disable(Feature::RemoteModels); + config.features.enable(Feature::Personality); + config.personality = Some(Personality::None); + }); + let test = builder.build(&server).await?; + + test.codex + .submit(Op::UserTurn { + items: vec![UserInput::Text { + text: "hello".into(), + text_elements: Vec::new(), + }], + final_output_json_schema: None, + cwd: test.cwd_path().to_path_buf(), + approval_policy: test.config.approval_policy.value(), + sandbox_policy: SandboxPolicy::ReadOnly, + model: test.session_configured.model.clone(), + effort: test.config.model_reasoning_effort, + summary: ReasoningSummary::Auto, + collaboration_mode: None, + personality: None, + }) + .await?; + + wait_for_event(&test.codex, |ev| matches!(ev, EventMsg::TurnComplete(_))).await; + + let request = resp_mock.single_request(); + let instructions_text = request.instructions_text(); + assert!( + !instructions_text.contains(LOCAL_FRIENDLY_TEMPLATE), + "expected no friendly personality template, got: {instructions_text:?}" + ); + assert!( + !instructions_text.contains(LOCAL_PRAGMATIC_TEMPLATE), + "expected no pragmatic personality template, got: {instructions_text:?}" + ); + assert!( + !instructions_text.contains("{{ personality }}"), + "expected personality placeholder to be removed, got: {instructions_text:?}" + ); + + let developer_texts = request.message_input_texts("developer"); + assert!( + !developer_texts + .iter() + .any(|text| text.contains("")), + "did not expect a personality update message when personality is None" + ); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn default_personality_is_friendly_without_config_toml() -> anyhow::Result<()> { + skip_if_no_network!(Ok(())); + + let server = start_mock_server().await; + let resp_mock = mount_sse_once(&server, sse_completed("resp-1")).await; + let mut builder = test_codex() + .with_model("gpt-5.2-codex") + .with_config(|config| { + config.features.disable(Feature::RemoteModels); + config.features.enable(Feature::Personality); + }); + let test = builder.build(&server).await?; + + test.codex + .submit(Op::UserTurn { + items: vec![UserInput::Text { + text: "hello".into(), + text_elements: Vec::new(), + }], + final_output_json_schema: None, + cwd: test.cwd_path().to_path_buf(), + approval_policy: test.config.approval_policy.value(), + sandbox_policy: SandboxPolicy::ReadOnly, + model: test.session_configured.model.clone(), + effort: test.config.model_reasoning_effort, + summary: ReasoningSummary::Auto, + collaboration_mode: None, + personality: None, + }) + .await?; + + wait_for_event(&test.codex, |ev| matches!(ev, EventMsg::TurnComplete(_))).await; + + let request = resp_mock.single_request(); + let instructions_text = request.instructions_text(); + assert!( + instructions_text.contains(LOCAL_FRIENDLY_TEMPLATE), + "expected default friendly template, got: {instructions_text:?}" + ); + + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn user_turn_personality_some_adds_update_message() -> anyhow::Result<()> { skip_if_no_network!(Ok(())); diff --git a/codex-rs/core/tests/suite/prompt_caching.rs b/codex-rs/core/tests/suite/prompt_caching.rs index bf1819cbf1..acff97381a 100644 --- a/codex-rs/core/tests/suite/prompt_caching.rs +++ b/codex-rs/core/tests/suite/prompt_caching.rs @@ -18,8 +18,10 @@ use codex_protocol::config_types::WebSearchMode; use codex_protocol::openai_models::ReasoningEffort; use codex_protocol::user_input::UserInput; use codex_utils_absolute_path::AbsolutePathBuf; -use core_test_support::load_sse_fixture_with_id; +use core_test_support::responses::ev_completed; +use core_test_support::responses::ev_response_created; use core_test_support::responses::mount_sse_once; +use core_test_support::responses::sse; use core_test_support::responses::start_mock_server; use core_test_support::skip_if_no_network; use core_test_support::test_codex::TestCodex; @@ -46,11 +48,6 @@ fn default_env_context_str(cwd: &str, shell: &Shell) -> String { ) } -/// Build minimal SSE stream with completed marker using the JSON fixture. -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - fn assert_tool_names(body: &serde_json::Value, expected_names: &[&str]) { assert_eq!( body["tools"] @@ -79,8 +76,16 @@ async fn prompt_tools_are_consistent_across_requests() -> anyhow::Result<()> { use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, @@ -131,8 +136,12 @@ async fn prompt_tools_are_consistent_across_requests() -> anyhow::Result<()> { .await?; wait_for_event(&codex, |ev| matches!(ev, EventMsg::TurnComplete(_))).await; - let expected_tools_names = vec![ - "shell_command", + let mut expected_tools_names = if cfg!(windows) { + vec!["shell_command"] + } else { + vec!["exec_command", "write_stdin"] + }; + expected_tools_names.extend([ "list_mcp_resources", "list_mcp_resource_templates", "read_mcp_resource", @@ -141,7 +150,7 @@ async fn prompt_tools_are_consistent_across_requests() -> anyhow::Result<()> { "apply_patch", "web_search", "view_image", - ]; + ]); let body0 = req1.single_request().body_json(); let expected_instructions = if expected_tools_names.contains(&"apply_patch") { @@ -172,8 +181,16 @@ async fn codex_mini_latest_tools() -> anyhow::Result<()> { use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_config(|config| { @@ -238,8 +255,16 @@ async fn prefixes_context_and_instructions_once_and_consistently_across_requests use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, config, .. } = test_codex() .with_config(|config| { @@ -315,8 +340,16 @@ async fn overrides_turn_context_but_keeps_cached_prefix_and_key_constant() -> an use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_config(|config| { @@ -407,7 +440,11 @@ async fn override_before_first_turn_emits_environment_context() -> anyhow::Resul skip_if_no_network!(Ok(())); let server = start_mock_server().await; - let req = mount_sse_once(&server, sse_completed("resp-1")).await; + let req = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; let TestCodex { codex, .. } = test_codex().build(&server).await?; @@ -542,8 +579,16 @@ async fn per_turn_overrides_keep_cached_prefix_and_key_constant() -> anyhow::Res use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, .. } = test_codex() .with_config(|config| { @@ -646,8 +691,16 @@ async fn send_user_turn_with_no_changes_does_not_send_environment_context() -> a use pretty_assertions::assert_eq; let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, @@ -747,8 +800,16 @@ async fn send_user_turn_with_changes_sends_environment_context() -> anyhow::Resu let server = start_mock_server().await; - let req1 = mount_sse_once(&server, sse_completed("resp-1")).await; - let req2 = mount_sse_once(&server, sse_completed("resp-2")).await; + let req1 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + ) + .await; + let req2 = mount_sse_once( + &server, + sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ) + .await; let TestCodex { codex, config, diff --git a/codex-rs/core/tests/suite/rollout_list_find.rs b/codex-rs/core/tests/suite/rollout_list_find.rs index 8786c9be21..5b545384b9 100644 --- a/codex-rs/core/tests/suite/rollout_list_find.rs +++ b/codex-rs/core/tests/suite/rollout_list_find.rs @@ -104,9 +104,11 @@ async fn find_prefers_sqlite_path_by_id() { let home = TempDir::new().unwrap(); let id = Uuid::new_v4(); let thread_id = ThreadId::from_string(&id.to_string()).unwrap(); - let db_path = home - .path() - .join("sessions/2030/12/30/rollout-2030-12-30T00-00-00-db.jsonl"); + let db_path = home.path().join(format!( + "sessions/2030/12/30/rollout-2030-12-30T00-00-00-{id}.jsonl" + )); + std::fs::create_dir_all(db_path.parent().unwrap()).unwrap(); + std::fs::write(&db_path, "").unwrap(); write_minimal_rollout_with_id(home.path(), id); upsert_thread_metadata(home.path(), thread_id, db_path.clone()).await; diff --git a/codex-rs/core/tests/suite/sqlite_state.rs b/codex-rs/core/tests/suite/sqlite_state.rs index 218da34825..07268f4fdb 100644 --- a/codex-rs/core/tests/suite/sqlite_state.rs +++ b/codex-rs/core/tests/suite/sqlite_state.rs @@ -9,8 +9,6 @@ use codex_protocol::protocol::SessionMeta; use codex_protocol::protocol::SessionMetaLine; use codex_protocol::protocol::SessionSource; use codex_protocol::protocol::UserMessageEvent; -use codex_state::STATE_DB_FILENAME; -use core_test_support::load_sse_fixture_with_id; use core_test_support::responses; use core_test_support::responses::ev_completed; use core_test_support::responses::ev_function_call; @@ -25,10 +23,6 @@ use tokio::time::Duration; use tracing_subscriber::prelude::*; use uuid::Uuid; -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn new_thread_is_recorded_in_state_db() -> Result<()> { let server = start_mock_server().await; @@ -39,7 +33,7 @@ async fn new_thread_is_recorded_in_state_db() -> Result<()> { let thread_id = test.session_configured.session_id; let rollout_path = test.codex.rollout_path().expect("rollout path"); - let db_path = test.config.codex_home.join(STATE_DB_FILENAME); + let db_path = codex_state::state_db_path(test.config.codex_home.as_path()); for _ in 0..100 { if tokio::fs::try_exists(&db_path).await.unwrap_or(false) { @@ -149,7 +143,7 @@ async fn backfill_scans_existing_rollouts() -> Result<()> { let test = builder.build(&server).await?; - let db_path = test.config.codex_home.join(STATE_DB_FILENAME); + let db_path = codex_state::state_db_path(test.config.codex_home.as_path()); let rollout_path = test.config.codex_home.join(&rollout_rel_path); let default_provider = test.config.model_provider_id.clone(); @@ -196,7 +190,10 @@ async fn user_messages_persist_in_state_db() -> Result<()> { let server = start_mock_server().await; mount_sse_sequence( &server, - vec![sse_completed("resp-1"), sse_completed("resp-2")], + vec![ + responses::sse(vec![ev_response_created("resp-1"), ev_completed("resp-1")]), + responses::sse(vec![ev_response_created("resp-2"), ev_completed("resp-2")]), + ], ) .await; @@ -205,7 +202,7 @@ async fn user_messages_persist_in_state_db() -> Result<()> { }); let test = builder.build(&server).await?; - let db_path = test.config.codex_home.join(STATE_DB_FILENAME); + let db_path = codex_state::state_db_path(test.config.codex_home.as_path()); for _ in 0..100 { if tokio::fs::try_exists(&db_path).await.unwrap_or(false) { break; diff --git a/codex-rs/core/tests/suite/stream_error_allows_next_turn.rs b/codex-rs/core/tests/suite/stream_error_allows_next_turn.rs index ec18816235..3562ec2500 100644 --- a/codex-rs/core/tests/suite/stream_error_allows_next_turn.rs +++ b/codex-rs/core/tests/suite/stream_error_allows_next_turn.rs @@ -3,7 +3,9 @@ use codex_core::WireApi; use codex_core::protocol::EventMsg; use codex_core::protocol::Op; use codex_protocol::user_input::UserInput; -use core_test_support::load_sse_fixture_with_id; +use core_test_support::responses::ev_completed; +use core_test_support::responses::ev_response_created; +use core_test_support::responses::sse; use core_test_support::skip_if_no_network; use core_test_support::test_codex::TestCodex; use core_test_support::test_codex::test_codex; @@ -15,10 +17,6 @@ use wiremock::matchers::body_string_contains; use wiremock::matchers::method; use wiremock::matchers::path; -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn continue_after_stream_error() { skip_if_no_network!(); @@ -46,7 +44,13 @@ async fn continue_after_stream_error() { let ok = ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") - .set_body_raw(sse_completed("resp_ok2"), "text/event-stream"); + .set_body_raw( + sse(vec![ + ev_response_created("resp_ok2"), + ev_completed("resp_ok2"), + ]), + "text/event-stream", + ); Mock::given(method("POST")) .and(path("/v1/responses")) diff --git a/codex-rs/core/tests/suite/stream_no_completed.rs b/codex-rs/core/tests/suite/stream_no_completed.rs index 68b7763eca..6528e6277d 100644 --- a/codex-rs/core/tests/suite/stream_no_completed.rs +++ b/codex-rs/core/tests/suite/stream_no_completed.rs @@ -7,7 +7,9 @@ use codex_core::protocol::EventMsg; use codex_core::protocol::Op; use codex_protocol::user_input::UserInput; use core_test_support::load_sse_fixture; -use core_test_support::load_sse_fixture_with_id; +use core_test_support::responses::ev_completed; +use core_test_support::responses::ev_response_created; +use core_test_support::responses::sse; use core_test_support::skip_if_no_network; use core_test_support::test_codex::TestCodex; use core_test_support::test_codex::test_codex; @@ -24,10 +26,6 @@ fn sse_incomplete() -> String { load_sse_fixture("tests/fixtures/incomplete_sse.json") } -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn retries_on_early_close() { skip_if_no_network!(); @@ -48,7 +46,13 @@ async fn retries_on_early_close() { } else { ResponseTemplate::new(200) .insert_header("content-type", "text/event-stream") - .set_body_raw(sse_completed("resp_ok"), "text/event-stream") + .set_body_raw( + sse(vec![ + ev_response_created("resp_ok"), + ev_completed("resp_ok"), + ]), + "text/event-stream", + ) } } } @@ -67,7 +71,7 @@ async fn retries_on_early_close() { name: "openai".into(), base_url: Some(format!("{}/v1", server.uri())), // Environment variable that should exist in the test environment. - // ModelClientSession will return an error if the environment variable for the + // ModelClient will return an error if the environment variable for the // provider is not set. env_key: Some("PATH".into()), env_key_instructions: None, diff --git a/codex-rs/core/tests/suite/tool_parallelism.rs b/codex-rs/core/tests/suite/tool_parallelism.rs index 955b2f7ec3..0df8f8cbc9 100644 --- a/codex-rs/core/tests/suite/tool_parallelism.rs +++ b/codex-rs/core/tests/suite/tool_parallelism.rs @@ -149,6 +149,8 @@ async fn shell_tools_run_in_parallel() -> anyhow::Result<()> { let shell_args = json!({ "command": "sleep 0.3", + // Avoid user-specific shell startup cost (e.g. zsh profile scripts) in timing assertions. + "login": false, "timeout_ms": 1_000, }); let args_one = serde_json::to_string(&shell_args)?; diff --git a/codex-rs/core/tests/suite/web_search.rs b/codex-rs/core/tests/suite/web_search.rs index 9397e88edb..8b1cc2a4e7 100644 --- a/codex-rs/core/tests/suite/web_search.rs +++ b/codex-rs/core/tests/suite/web_search.rs @@ -5,7 +5,6 @@ use codex_core::built_in_model_providers; use codex_core::features::Feature; use codex_core::protocol::SandboxPolicy; use codex_protocol::config_types::WebSearchMode; -use core_test_support::load_sse_fixture_with_id; use core_test_support::responses; use core_test_support::responses::start_mock_server; use core_test_support::skip_if_no_network; @@ -13,10 +12,6 @@ use core_test_support::test_codex::test_codex; use pretty_assertions::assert_eq; use serde_json::Value; -fn sse_completed(id: &str) -> String { - load_sse_fixture_with_id("../fixtures/completed_template.json", id) -} - #[allow(clippy::expect_used)] fn find_web_search_tool(body: &Value) -> &Value { body["tools"] @@ -41,7 +36,10 @@ async fn web_search_mode_cached_sets_external_web_access_false() { skip_if_no_network!(); let server = start_mock_server().await; - let sse = sse_completed("resp-1"); + let sse = responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_completed("resp-1"), + ]); let resp_mock = responses::mount_sse_once(&server, sse).await; let mut builder = test_codex() @@ -72,7 +70,10 @@ async fn web_search_mode_takes_precedence_over_legacy_flags() { skip_if_no_network!(); let server = start_mock_server().await; - let sse = sse_completed("resp-1"); + let sse = responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_completed("resp-1"), + ]); let resp_mock = responses::mount_sse_once(&server, sse).await; let mut builder = test_codex() @@ -104,7 +105,10 @@ async fn web_search_mode_defaults_to_cached_when_unset() { skip_if_no_network!(); let server = start_mock_server().await; - let sse = sse_completed("resp-1"); + let sse = responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_completed("resp-1"), + ]); let resp_mock = responses::mount_sse_once(&server, sse).await; let mut builder = test_codex() @@ -139,7 +143,16 @@ async fn web_search_mode_updates_between_turns_with_sandbox_policy() { let server = start_mock_server().await; let resp_mock = responses::mount_sse_sequence( &server, - vec![sse_completed("resp-1"), sse_completed("resp-2")], + vec![ + responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_completed("resp-1"), + ]), + responses::sse(vec![ + responses::ev_response_created("resp-2"), + responses::ev_completed("resp-2"), + ]), + ], ) .await; @@ -191,7 +204,10 @@ async fn web_search_mode_defaults_to_disabled_for_azure_responses() { skip_if_no_network!(); let server = start_mock_server().await; - let sse = sse_completed("resp-1"); + let sse = responses::sse(vec![ + responses::ev_response_created("resp-1"), + responses::ev_completed("resp-1"), + ]); let resp_mock = responses::mount_sse_once(&server, sse).await; let mut builder = test_codex() diff --git a/codex-rs/docs/codex_mcp_interface.md b/codex-rs/docs/codex_mcp_interface.md index edd693d5ed..8b1b5f7045 100644 --- a/codex-rs/docs/codex_mcp_interface.md +++ b/codex-rs/docs/codex_mcp_interface.md @@ -73,6 +73,8 @@ Send input to the active turn: - `sendUserMessage` β†’ enqueue items to the conversation - `sendUserTurn` β†’ structured turn with explicit `cwd`, `approvalPolicy`, `sandboxPolicy`, `model`, optional `effort`, `summary`, optional `personality`, and optional `outputSchema` (JSON Schema for the final assistant message) +Valid `personality` values are `friendly`, `pragmatic`, and `none`. When `none` is selected, the personality placeholder is replaced with an empty string. + For v2 threads, `turn/start` also accepts `outputSchema` to constrain the final assistant message for that turn. Interrupt a running turn: `interruptConversation`. @@ -100,6 +102,7 @@ Each response yields: - `defaultReasoningEffort` – suggested effort for the UI - `supportsPersonality` – whether the model supports personality-specific instructions - `isDefault` – whether the model is recommended for most users + - `upgrade` – optional recommended upgrade model id - `nextCursor` – pass into the next request to continue paging (optional) ## Collaboration modes (experimental) diff --git a/codex-rs/docs/protocol_v1.md b/codex-rs/docs/protocol_v1.md index 8f68ecc9a6..4d4e5c147c 100644 --- a/codex-rs/docs/protocol_v1.md +++ b/codex-rs/docs/protocol_v1.md @@ -72,6 +72,9 @@ For complete documentation of the `Op` and `EventMsg` variants, refer to [protoc - `Op::UserInputAnswer` – Provide answers for a `request_user_input` tool call - `Op::ListSkills` – Request skills for one or more cwd values (optionally `force_reload`) - `Op::UserTurn` and `Op::OverrideTurnContext` accept an optional `personality` override that updates the model’s communication style + +Valid `personality` values are `friendly`, `pragmatic`, and `none`. When `none` is selected, the personality placeholder is replaced with an empty string. + - `EventMsg` - `EventMsg::AgentMessage` – Messages from the `Model` - `EventMsg::AgentMessageContentDelta` – Streaming assistant text diff --git a/codex-rs/exec-server/src/posix/escalate_server.rs b/codex-rs/exec-server/src/posix/escalate_server.rs index ef991ce8cd..a2f247daae 100644 --- a/codex-rs/exec-server/src/posix/escalate_server.rs +++ b/codex-rs/exec-server/src/posix/escalate_server.rs @@ -95,6 +95,7 @@ impl EscalateServer { &sandbox_state.sandbox_policy, &sandbox_state.sandbox_cwd, &sandbox_state.codex_linux_sandbox_exe, + sandbox_state.use_linux_sandbox_bwrap, None, ) .await?; diff --git a/codex-rs/exec-server/src/posix/mcp.rs b/codex-rs/exec-server/src/posix/mcp.rs index 620d332e71..e48b70a2a2 100644 --- a/codex-rs/exec-server/src/posix/mcp.rs +++ b/codex-rs/exec-server/src/posix/mcp.rs @@ -126,6 +126,7 @@ impl ExecTool { sandbox_policy: SandboxPolicy::ReadOnly, codex_linux_sandbox_exe: None, sandbox_cwd: PathBuf::from(¶ms.workdir), + use_linux_sandbox_bwrap: false, }); let escalate_server = EscalateServer::new( self.bash_path.clone(), diff --git a/codex-rs/exec-server/tests/common/lib.rs b/codex-rs/exec-server/tests/common/lib.rs index 562d3504f6..66bb4b7f9c 100644 --- a/codex-rs/exec-server/tests/common/lib.rs +++ b/codex-rs/exec-server/tests/common/lib.rs @@ -91,6 +91,7 @@ where sandbox_policy: SandboxPolicy::ReadOnly, codex_linux_sandbox_exe, sandbox_cwd: sandbox_cwd.as_ref().to_path_buf(), + use_linux_sandbox_bwrap: false, }; send_sandbox_state_update(sandbox_state, service).await } @@ -118,6 +119,7 @@ where }, codex_linux_sandbox_exe, sandbox_cwd: writable_folder.as_ref().to_path_buf(), + use_linux_sandbox_bwrap: false, }; send_sandbox_state_update(sandbox_state, service).await } diff --git a/codex-rs/exec/src/event_processor_with_human_output.rs b/codex-rs/exec/src/event_processor_with_human_output.rs index 3b4a3bf86b..cbe45b92f1 100644 --- a/codex-rs/exec/src/event_processor_with_human_output.rs +++ b/codex-rs/exec/src/event_processor_with_human_output.rs @@ -593,17 +593,20 @@ impl EventProcessor for EventProcessorWithHumanOutput { view.path.display() ); } - EventMsg::TurnAborted(abort_reason) => match abort_reason.reason { - TurnAbortReason::Interrupted => { - ts_msg!(self, "task interrupted"); + EventMsg::TurnAborted(abort_reason) => { + match abort_reason.reason { + TurnAbortReason::Interrupted => { + ts_msg!(self, "task interrupted"); + } + TurnAbortReason::Replaced => { + ts_msg!(self, "task aborted: replaced by a new task"); + } + TurnAbortReason::ReviewEnded => { + ts_msg!(self, "task aborted: review ended"); + } } - TurnAbortReason::Replaced => { - ts_msg!(self, "task aborted: replaced by a new task"); - } - TurnAbortReason::ReviewEnded => { - ts_msg!(self, "task aborted: review ended"); - } - }, + return CodexStatus::InitiateShutdown; + } EventMsg::ContextCompacted(_) => { ts_msg!(self, "context compacted"); } @@ -773,6 +776,8 @@ impl EventProcessor for EventProcessorWithHumanOutput { | EventMsg::McpListToolsResponse(_) | EventMsg::ListCustomPromptsResponse(_) | EventMsg::ListSkillsResponse(_) + | EventMsg::ListRemoteSkillsResponse(_) + | EventMsg::RemoteSkillDownloaded(_) | EventMsg::RawResponseItem(_) | EventMsg::UserMessage(_) | EventMsg::EnteredReviewMode(_) diff --git a/codex-rs/exec/src/event_processor_with_jsonl_output.rs b/codex-rs/exec/src/event_processor_with_jsonl_output.rs index 823fa7b810..9675651ef7 100644 --- a/codex-rs/exec/src/event_processor_with_jsonl_output.rs +++ b/codex-rs/exec/src/event_processor_with_jsonl_output.rs @@ -871,6 +871,7 @@ impl EventProcessor for EventProcessorWithJsonOutput { } CodexStatus::InitiateShutdown } + protocol::EventMsg::TurnAborted(_) => CodexStatus::InitiateShutdown, protocol::EventMsg::ShutdownComplete => CodexStatus::Shutdown, _ => CodexStatus::Running, } diff --git a/codex-rs/exec/tests/suite/sandbox.rs b/codex-rs/exec/tests/suite/sandbox.rs index 303023b158..2d0ad42caa 100644 --- a/codex-rs/exec/tests/suite/sandbox.rs +++ b/codex-rs/exec/tests/suite/sandbox.rs @@ -50,6 +50,7 @@ async fn spawn_command_under_sandbox( command_cwd, sandbox_policy, sandbox_cwd, + false, stdio_policy, env, ) diff --git a/codex-rs/linux-sandbox/Cargo.toml b/codex-rs/linux-sandbox/Cargo.toml index 3feb1eb7b8..ff16a569fe 100644 --- a/codex-rs/linux-sandbox/Cargo.toml +++ b/codex-rs/linux-sandbox/Cargo.toml @@ -23,7 +23,6 @@ landlock = { workspace = true } libc = { workspace = true } seccompiler = { workspace = true } serde_json = { workspace = true } -which = "8.0.0" [target.'cfg(target_os = "linux")'.dev-dependencies] pretty_assertions = { workspace = true } diff --git a/codex-rs/linux-sandbox/README.md b/codex-rs/linux-sandbox/README.md index 676f234954..f14fc5f13d 100644 --- a/codex-rs/linux-sandbox/README.md +++ b/codex-rs/linux-sandbox/README.md @@ -6,3 +6,28 @@ This crate is responsible for producing: - a lib crate that exposes the business logic of the executable as `run_main()` so that - the `codex-exec` CLI can check if its arg0 is `codex-linux-sandbox` and, if so, execute as if it were `codex-linux-sandbox` - this should also be true of the `codex` multitool CLI + +On Linux, the bubblewrap pipeline uses the vendored bubblewrap path compiled +into this binary. + +**Current Behavior** +- Legacy Landlock + mount protections remain available as the legacy pipeline. +- The bubblewrap pipeline is standardized on the vendored path. +- During rollout, the bubblewrap pipeline is gated by the temporary feature + flag `use_linux_sandbox_bwrap` (CLI `-c` alias for + `features.use_linux_sandbox_bwrap`; legacy remains default when off). +- When enabled, the bubblewrap pipeline applies `PR_SET_NO_NEW_PRIVS` and a + seccomp network filter in-process. +- When enabled, the filesystem is read-only by default via `--ro-bind / /`. +- When enabled, writable roots are layered with `--bind `. +- When enabled, protected subpaths under writable roots (for example `.git`, + resolved `gitdir:`, and `.codex`) are re-applied as read-only via `--ro-bind`. +- When enabled, symlink-in-path and non-existent protected paths inside + writable roots are blocked by mounting `/dev/null` on the symlink or first + missing component. +- When enabled, the helper isolates the PID namespace via `--unshare-pid`. +- When enabled, it mounts a fresh `/proc` via `--proc /proc` by default, but + you can skip this in restrictive container environments with `--no-proc`. + +**Notes** +- The CLI surface still uses legacy names like `codex debug landlock`. diff --git a/codex-rs/linux-sandbox/src/bwrap.rs b/codex-rs/linux-sandbox/src/bwrap.rs index c1e0732e08..4a60c65dd3 100644 --- a/codex-rs/linux-sandbox/src/bwrap.rs +++ b/codex-rs/linux-sandbox/src/bwrap.rs @@ -44,45 +44,6 @@ pub(crate) fn create_bwrap_command_args( sandbox_policy: &SandboxPolicy, cwd: &Path, options: BwrapOptions, - bwrap_path: Option<&Path>, -) -> Result> { - if sandbox_policy.has_full_disk_write_access() { - return Ok(command); - } - - let bwrap_path = match bwrap_path { - Some(path) => { - if path.exists() { - path.to_path_buf() - } else { - return Err(CodexErr::UnsupportedOperation(format!( - "bubblewrap (bwrap) not found at configured path: {}", - path.display() - ))); - } - } - None => which::which("bwrap").map_err(|err| { - CodexErr::UnsupportedOperation(format!("bubblewrap (bwrap) not found on PATH: {err}")) - })?, - }; - - let mut args = Vec::new(); - args.push(path_to_string(&bwrap_path)); - args.extend(create_bwrap_flags(command, sandbox_policy, cwd, options)?); - Ok(args) -} - -/// Doc-hidden helper that builds bubblewrap arguments without a program path. -/// -/// This is intended for experiments where we call a build-time bubblewrap -/// `main` symbol via FFI rather than exec'ing the `bwrap` binary. The caller -/// is responsible for providing a suitable `argv[0]`. -#[doc(hidden)] -pub(crate) fn create_bwrap_command_args_vendored( - command: Vec, - sandbox_policy: &SandboxPolicy, - cwd: &Path, - options: BwrapOptions, ) -> Result> { if sandbox_policy.has_full_disk_write_access() { return Ok(command); diff --git a/codex-rs/linux-sandbox/src/landlock.rs b/codex-rs/linux-sandbox/src/landlock.rs index 772176fc90..4853011792 100644 --- a/codex-rs/linux-sandbox/src/landlock.rs +++ b/codex-rs/linux-sandbox/src/landlock.rs @@ -1,3 +1,7 @@ +//! In-process Linux sandbox primitives: `no_new_privs` and seccomp. +//! +//! Filesystem restrictions are enforced by bubblewrap in `linux_run_main`. +//! Landlock helpers remain available here as legacy/backup utilities. use std::collections::BTreeMap; use std::path::Path; @@ -8,6 +12,7 @@ use codex_core::protocol::SandboxPolicy; use codex_utils_absolute_path::AbsolutePathBuf; use landlock::ABI; +#[allow(unused_imports)] use landlock::Access; use landlock::AccessFs; use landlock::CompatLevel; @@ -27,11 +32,24 @@ use seccompiler::apply_filter; /// Apply sandbox policies inside this thread so only the child inherits /// them, not the entire CLI process. +/// +/// This function is responsible for: +/// - enabling `PR_SET_NO_NEW_PRIVS` when restrictions apply, and +/// - installing the network seccomp filter when network access is disabled. +/// +/// Filesystem restrictions are intentionally handled by bubblewrap. pub(crate) fn apply_sandbox_policy_to_current_thread( sandbox_policy: &SandboxPolicy, cwd: &Path, + apply_landlock_fs: bool, ) -> Result<()> { - if !sandbox_policy.has_full_disk_write_access() || !sandbox_policy.has_full_network_access() { + // `PR_SET_NO_NEW_PRIVS` is required for seccomp, but it also prevents + // setuid privilege elevation. Many `bwrap` deployments rely on setuid, so + // we avoid this unless we need seccomp or we are explicitly using the + // legacy Landlock filesystem pipeline. + if !sandbox_policy.has_full_network_access() + || (apply_landlock_fs && !sandbox_policy.has_full_disk_write_access()) + { set_no_new_privs()?; } @@ -39,7 +57,7 @@ pub(crate) fn apply_sandbox_policy_to_current_thread( install_network_seccomp_filter_on_current_thread()?; } - if !sandbox_policy.has_full_disk_write_access() { + if apply_landlock_fs && !sandbox_policy.has_full_disk_write_access() { let writable_roots = sandbox_policy .get_writable_roots_with_cwd(cwd) .into_iter() @@ -54,6 +72,7 @@ pub(crate) fn apply_sandbox_policy_to_current_thread( Ok(()) } +/// Enable `PR_SET_NO_NEW_PRIVS` so seccomp can be applied safely. fn set_no_new_privs() -> Result<()> { let result = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; if result != 0 { @@ -68,6 +87,9 @@ fn set_no_new_privs() -> Result<()> { /// /// # Errors /// Returns [`CodexErr::Sandbox`] variants when the ruleset fails to apply. +/// +/// Note: this is currently unused because filesystem sandboxing is performed +/// via bubblewrap. It is kept for reference and potential fallback use. fn install_filesystem_landlock_rules_on_current_thread( writable_roots: Vec, ) -> Result<()> { @@ -98,6 +120,9 @@ fn install_filesystem_landlock_rules_on_current_thread( /// Installs a seccomp filter that blocks outbound network access except for /// AF_UNIX domain sockets. +/// +/// The filter is applied to the current thread so only the sandboxed child +/// inherits it. fn install_network_seccomp_filter_on_current_thread() -> std::result::Result<(), SandboxErr> { // Build rule map. let mut rules: BTreeMap> = BTreeMap::new(); diff --git a/codex-rs/linux-sandbox/src/linux_run_main.rs b/codex-rs/linux-sandbox/src/linux_run_main.rs index 3f6cd3ac86..f5f0d9887a 100644 --- a/codex-rs/linux-sandbox/src/linux_run_main.rs +++ b/codex-rs/linux-sandbox/src/linux_run_main.rs @@ -1,13 +1,16 @@ use clap::Parser; use std::ffi::CString; +use std::fs::File; +use std::io::Read; +use std::os::fd::FromRawFd; use std::path::Path; use std::path::PathBuf; use crate::bwrap::BwrapOptions; use crate::bwrap::create_bwrap_command_args; -use crate::bwrap::create_bwrap_command_args_vendored; use crate::landlock::apply_sandbox_policy_to_current_thread; use crate::vendored_bwrap::exec_vendored_bwrap; +use crate::vendored_bwrap::run_vendored_bwrap_main; #[derive(Debug, Parser)] /// CLI surface for the Linux sandbox helper. @@ -29,18 +32,6 @@ pub struct LandlockCommand { #[arg(long = "use-bwrap-sandbox", hide = true, default_value_t = false)] pub use_bwrap_sandbox: bool, - /// Optional explicit path to the `bwrap` binary to use. - /// - /// When provided, this implies bubblewrap opt-in and avoids PATH lookups. - #[arg(long = "bwrap-path", hide = true)] - pub bwrap_path: Option, - - /// Experimental: call a build-time bubblewrap `main()` via FFI. - /// - /// This is opt-in and only works when the build script compiles bwrap. - #[arg(long = "use-vendored-bwrap", hide = true, default_value_t = false)] - pub use_vendored_bwrap: bool, - /// Internal: apply seccomp and `no_new_privs` in the already-sandboxed /// process, then exec the user command. /// @@ -72,13 +63,10 @@ pub fn run_main() -> ! { sandbox_policy_cwd, sandbox_policy, use_bwrap_sandbox, - bwrap_path, - use_vendored_bwrap, apply_seccomp_then_exec, no_proc, command, } = LandlockCommand::parse(); - let use_bwrap_sandbox = use_bwrap_sandbox || bwrap_path.is_some() || use_vendored_bwrap; if command.is_empty() { panic!("No command specified to execute."); @@ -87,82 +75,199 @@ pub fn run_main() -> ! { // Inner stage: apply seccomp/no_new_privs after bubblewrap has already // established the filesystem view. if apply_seccomp_then_exec { - if let Err(e) = apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd) + if let Err(e) = + apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd, false) { panic!("error applying Linux sandbox restrictions: {e:?}"); } exec_or_panic(command); } - let command = if sandbox_policy.has_full_disk_write_access() { - if let Err(e) = apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd) + if sandbox_policy.has_full_disk_write_access() { + if let Err(e) = + apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd, false) { panic!("error applying Linux sandbox restrictions: {e:?}"); } - command - } else if use_bwrap_sandbox { + exec_or_panic(command); + } + + if use_bwrap_sandbox { // Outer stage: bubblewrap first, then re-enter this binary in the - // sandboxed environment to apply seccomp. + // sandboxed environment to apply seccomp. This path never falls back + // to legacy Landlock on failure. let inner = build_inner_seccomp_command( &sandbox_policy_cwd, &sandbox_policy, use_bwrap_sandbox, - bwrap_path.as_deref(), command, ); - let options = BwrapOptions { - mount_proc: !no_proc, - }; - if use_vendored_bwrap { - let mut argv0 = bwrap_path - .as_deref() - .map(|path| path.to_string_lossy().to_string()) - .unwrap_or_else(|| "bwrap".to_string()); - if argv0.is_empty() { - argv0 = "bwrap".to_string(); - } - - let mut argv = vec![argv0]; - argv.extend( - create_bwrap_command_args_vendored( - inner, - &sandbox_policy, - &sandbox_policy_cwd, - options, - ) - .unwrap_or_else(|err| { - panic!("error building build-time bubblewrap command: {err:?}") - }), - ); - exec_vendored_bwrap(argv); - } - ensure_bwrap_available(bwrap_path.as_deref()); - create_bwrap_command_args( - inner, - &sandbox_policy, - &sandbox_policy_cwd, - options, - bwrap_path.as_deref(), - ) - .unwrap_or_else(|err| panic!("error building bubblewrap command: {err:?}")) - } else { - // Legacy path: Landlock enforcement only. - if let Err(e) = apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd) - { - panic!("error applying legacy Linux sandbox restrictions: {e:?}"); - } - command - }; + run_bwrap_with_proc_fallback(&sandbox_policy_cwd, &sandbox_policy, inner, !no_proc); + } + // Legacy path: Landlock enforcement only, when bwrap sandboxing is not enabled. + if let Err(e) = + apply_sandbox_policy_to_current_thread(&sandbox_policy, &sandbox_policy_cwd, true) + { + panic!("error applying legacy Linux sandbox restrictions: {e:?}"); + } exec_or_panic(command); } +fn run_bwrap_with_proc_fallback( + sandbox_policy_cwd: &Path, + sandbox_policy: &codex_core::protocol::SandboxPolicy, + inner: Vec, + mount_proc: bool, +) -> ! { + let mut mount_proc = mount_proc; + + if mount_proc && !preflight_proc_mount_support(sandbox_policy_cwd, sandbox_policy) { + eprintln!("codex-linux-sandbox: bwrap could not mount /proc; retrying with --no-proc"); + mount_proc = false; + } + + let options = BwrapOptions { mount_proc }; + let argv = build_bwrap_argv(inner, sandbox_policy, sandbox_policy_cwd, options); + exec_vendored_bwrap(argv); +} + +fn build_bwrap_argv( + inner: Vec, + sandbox_policy: &codex_core::protocol::SandboxPolicy, + sandbox_policy_cwd: &Path, + options: BwrapOptions, +) -> Vec { + let mut args = create_bwrap_command_args(inner, sandbox_policy, sandbox_policy_cwd, options) + .unwrap_or_else(|err| panic!("error building bubblewrap command: {err:?}")); + + let command_separator_index = args + .iter() + .position(|arg| arg == "--") + .unwrap_or_else(|| panic!("bubblewrap argv is missing command separator '--'")); + args.splice( + command_separator_index..command_separator_index, + ["--argv0".to_string(), "codex-linux-sandbox".to_string()], + ); + + let mut argv = vec!["bwrap".to_string()]; + argv.extend(args); + argv +} + +fn preflight_proc_mount_support( + sandbox_policy_cwd: &Path, + sandbox_policy: &codex_core::protocol::SandboxPolicy, +) -> bool { + let preflight_command = vec![resolve_true_command()]; + let preflight_argv = build_bwrap_argv( + preflight_command, + sandbox_policy, + sandbox_policy_cwd, + BwrapOptions { mount_proc: true }, + ); + let stderr = run_bwrap_in_child_capture_stderr(preflight_argv); + !is_proc_mount_failure(stderr.as_str()) +} + +fn resolve_true_command() -> String { + for candidate in ["/usr/bin/true", "/bin/true"] { + if Path::new(candidate).exists() { + return candidate.to_string(); + } + } + "true".to_string() +} + +/// Run a short-lived bubblewrap preflight in a child process and capture stderr. +/// +/// Strategy: +/// - This is used only by `preflight_proc_mount_support`, which runs `/bin/true` +/// under bubblewrap with `--proc /proc`. +/// - The goal is to detect environments where mounting `/proc` fails (for +/// example, restricted containers), so we can retry the real run with +/// `--no-proc`. +/// - We capture stderr from that preflight to match known mount-failure text. +/// We do not stream it because this is a one-shot probe with a trivial +/// command, and reads are bounded to a fixed max size. +fn run_bwrap_in_child_capture_stderr(argv: Vec) -> String { + const MAX_PREFLIGHT_STDERR_BYTES: u64 = 64 * 1024; + + let mut pipe_fds = [0; 2]; + let pipe_res = unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) }; + if pipe_res < 0 { + let err = std::io::Error::last_os_error(); + panic!("failed to create stderr pipe for bubblewrap: {err}"); + } + let read_fd = pipe_fds[0]; + let write_fd = pipe_fds[1]; + + let pid = unsafe { libc::fork() }; + if pid < 0 { + let err = std::io::Error::last_os_error(); + panic!("failed to fork for bubblewrap: {err}"); + } + + if pid == 0 { + // Child: redirect stderr to the pipe, then run bubblewrap. + unsafe { + close_fd_or_panic(read_fd, "close read end in bubblewrap child"); + if libc::dup2(write_fd, libc::STDERR_FILENO) < 0 { + let err = std::io::Error::last_os_error(); + panic!("failed to redirect stderr for bubblewrap: {err}"); + } + close_fd_or_panic(write_fd, "close write end in bubblewrap child"); + } + + let exit_code = run_vendored_bwrap_main(&argv); + std::process::exit(exit_code); + } + + // Parent: close the write end and read stderr while the child runs. + close_fd_or_panic(write_fd, "close write end in bubblewrap parent"); + + // SAFETY: `read_fd` is a valid owned fd in the parent. + let mut read_file = unsafe { File::from_raw_fd(read_fd) }; + let mut stderr_bytes = Vec::new(); + let mut limited_reader = (&mut read_file).take(MAX_PREFLIGHT_STDERR_BYTES); + if let Err(err) = limited_reader.read_to_end(&mut stderr_bytes) { + panic!("failed to read bubblewrap stderr: {err}"); + } + + let mut status: libc::c_int = 0; + let wait_res = unsafe { libc::waitpid(pid, &mut status as *mut libc::c_int, 0) }; + if wait_res < 0 { + let err = std::io::Error::last_os_error(); + panic!("waitpid failed for bubblewrap child: {err}"); + } + + String::from_utf8_lossy(&stderr_bytes).into_owned() +} + +/// Close an owned file descriptor and panic with context on failure. +/// +/// We use explicit close() checks here (instead of ignoring return codes) +/// because this code runs in low-level sandbox setup paths where fd leaks or +/// close errors can mask the root cause of later failures. +fn close_fd_or_panic(fd: libc::c_int, context: &str) { + let close_res = unsafe { libc::close(fd) }; + if close_res < 0 { + let err = std::io::Error::last_os_error(); + panic!("{context}: {err}"); + } +} + +fn is_proc_mount_failure(stderr: &str) -> bool { + stderr.contains("Can't mount proc") + && stderr.contains("/newroot/proc") + && stderr.contains("Invalid argument") +} + /// Build the inner command that applies seccomp after bubblewrap. fn build_inner_seccomp_command( sandbox_policy_cwd: &Path, sandbox_policy: &codex_core::protocol::SandboxPolicy, use_bwrap_sandbox: bool, - bwrap_path: Option<&Path>, command: Vec, ) -> Vec { let current_exe = match std::env::current_exe() { @@ -185,10 +290,6 @@ fn build_inner_seccomp_command( inner.push("--use-bwrap-sandbox".to_string()); inner.push("--apply-seccomp-then-exec".to_string()); } - if let Some(bwrap_path) = bwrap_path { - inner.push("--bwrap-path".to_string()); - inner.push(bwrap_path.to_string_lossy().to_string()); - } inner.push("--".to_string()); inner.extend(command); inner @@ -217,32 +318,52 @@ fn exec_or_panic(command: Vec) -> ! { panic!("Failed to execvp {}: {err}", command[0].as_str()); } -/// Ensure the `bwrap` binary is available when the sandbox needs it. -fn ensure_bwrap_available(bwrap_path: Option<&Path>) { - if let Some(path) = bwrap_path { - if path.exists() { - return; - } - panic!( - "bubblewrap (bwrap) is required for Linux filesystem sandboxing but was not found at the configured path: {}\n\ -Install it and retry. Examples:\n\ -- Debian/Ubuntu: apt-get install bubblewrap\n\ -- Fedora/RHEL: dnf install bubblewrap\n\ -- Arch: pacman -S bubblewrap\n\ -If you are running the Codex Node package, ensure bwrap is installed on the host system.", - path.display() - ); - } - if which::which("bwrap").is_ok() { - return; +#[cfg(test)] +mod tests { + use super::*; + use codex_core::protocol::SandboxPolicy; + use pretty_assertions::assert_eq; + + #[test] + fn detects_proc_mount_invalid_argument_failure() { + let stderr = "bwrap: Can't mount proc on /newroot/proc: Invalid argument"; + assert_eq!(is_proc_mount_failure(stderr), true); } - panic!( - "bubblewrap (bwrap) is required for Linux filesystem sandboxing but was not found on PATH.\n\ -Install it and retry. Examples:\n\ -- Debian/Ubuntu: apt-get install bubblewrap\n\ -- Fedora/RHEL: dnf install bubblewrap\n\ -- Arch: pacman -S bubblewrap\n\ -If you are running the Codex Node package, ensure bwrap is installed on the host system." - ); + #[test] + fn ignores_non_proc_mount_errors() { + let stderr = "bwrap: Can't bind mount /dev/null: Operation not permitted"; + assert_eq!(is_proc_mount_failure(stderr), false); + } + + #[test] + fn inserts_bwrap_argv0_before_command_separator() { + let argv = build_bwrap_argv( + vec!["/bin/true".to_string()], + &SandboxPolicy::ReadOnly, + Path::new("/"), + BwrapOptions { mount_proc: true }, + ); + assert_eq!( + argv, + vec![ + "bwrap".to_string(), + "--new-session".to_string(), + "--die-with-parent".to_string(), + "--ro-bind".to_string(), + "/".to_string(), + "/".to_string(), + "--dev-bind".to_string(), + "/dev/null".to_string(), + "/dev/null".to_string(), + "--unshare-pid".to_string(), + "--proc".to_string(), + "/proc".to_string(), + "--argv0".to_string(), + "codex-linux-sandbox".to_string(), + "--".to_string(), + "/bin/true".to_string(), + ] + ); + } } diff --git a/codex-rs/linux-sandbox/src/mounts.rs b/codex-rs/linux-sandbox/src/mounts.rs deleted file mode 100644 index 04f3651dff..0000000000 --- a/codex-rs/linux-sandbox/src/mounts.rs +++ /dev/null @@ -1,339 +0,0 @@ -#![allow(dead_code)] - -use std::ffi::CString; -use std::os::unix::ffi::OsStrExt; -use std::path::Path; - -use codex_core::error::CodexErr; -use codex_core::error::Result; -use codex_core::protocol::SandboxPolicy; -use codex_core::protocol::WritableRoot; -use codex_utils_absolute_path::AbsolutePathBuf; - -/// Apply read-only bind mounts for protected subpaths before Landlock. -/// -/// This unshares mount namespaces (and user namespaces for non-root) so the -/// read-only remounts do not affect the host, then bind-mounts each protected -/// target onto itself and remounts it read-only. -pub(crate) fn apply_read_only_mounts(sandbox_policy: &SandboxPolicy, cwd: &Path) -> Result<()> { - let writable_roots = sandbox_policy.get_writable_roots_with_cwd(cwd); - let mount_targets = collect_read_only_mount_targets(&writable_roots)?; - if mount_targets.is_empty() { - return Ok(()); - } - - // Root can unshare the mount namespace directly; non-root needs a user - // namespace to gain capabilities for remounting. - if is_running_as_root() { - unshare_mount_namespace()?; - } else { - let original_euid = unsafe { libc::geteuid() }; - let original_egid = unsafe { libc::getegid() }; - unshare_user_and_mount_namespaces()?; - write_user_namespace_maps(original_euid, original_egid)?; - } - make_mounts_private()?; - - for target in mount_targets { - // Bind and remount read-only works for both files and directories. - bind_mount_read_only(target.as_path())?; - } - - // Drop ambient capabilities acquired from the user namespace so the - // sandboxed command cannot remount or create new bind mounts. - if !is_running_as_root() { - drop_caps()?; - } - - Ok(()) -} - -/// Collect read-only mount targets, resolving worktree `.git` pointer files. -fn collect_read_only_mount_targets( - writable_roots: &[WritableRoot], -) -> Result> { - let mut targets = Vec::new(); - for writable_root in writable_roots { - for ro_subpath in &writable_root.read_only_subpaths { - // The policy expects these paths to exist; surface actionable errors - // rather than silently skipping protections. - if !ro_subpath.as_path().exists() { - return Err(CodexErr::UnsupportedOperation(format!( - "Sandbox expected to protect {path}, but it does not exist. Ensure the repository contains this path or create it before running Codex.", - path = ro_subpath.as_path().display() - ))); - } - targets.push(ro_subpath.clone()); - // Worktrees and submodules store `.git` as a pointer file; add the - // referenced gitdir as an extra read-only target. - if is_git_pointer_file(ro_subpath) { - let gitdir = resolve_gitdir_from_file(ro_subpath)?; - if !targets - .iter() - .any(|target| target.as_path() == gitdir.as_path()) - { - targets.push(gitdir); - } - } - } - } - Ok(targets) -} - -/// Detect a `.git` pointer file used by worktrees and submodules. -fn is_git_pointer_file(path: &AbsolutePathBuf) -> bool { - path.as_path().is_file() && path.as_path().file_name() == Some(std::ffi::OsStr::new(".git")) -} - -/// Resolve a worktree `.git` pointer file to its gitdir path. -fn resolve_gitdir_from_file(dot_git: &AbsolutePathBuf) -> Result { - let contents = std::fs::read_to_string(dot_git.as_path()).map_err(CodexErr::from)?; - let trimmed = contents.trim(); - let (_, gitdir_raw) = trimmed.split_once(':').ok_or_else(|| { - CodexErr::UnsupportedOperation(format!( - "Expected {path} to contain a gitdir pointer, but it did not match `gitdir: `.", - path = dot_git.as_path().display() - )) - })?; - // `gitdir: ` may be relative to the directory containing `.git`. - let gitdir_raw = gitdir_raw.trim(); - if gitdir_raw.is_empty() { - return Err(CodexErr::UnsupportedOperation(format!( - "Expected {path} to contain a gitdir pointer, but it was empty.", - path = dot_git.as_path().display() - ))); - } - let base = dot_git.as_path().parent().ok_or_else(|| { - CodexErr::UnsupportedOperation(format!( - "Unable to resolve parent directory for {path}.", - path = dot_git.as_path().display() - )) - })?; - let gitdir_path = AbsolutePathBuf::resolve_path_against_base(gitdir_raw, base)?; - if !gitdir_path.as_path().exists() { - return Err(CodexErr::UnsupportedOperation(format!( - "Resolved gitdir path {path} does not exist.", - path = gitdir_path.as_path().display() - ))); - } - Ok(gitdir_path) -} - -/// Unshare the mount namespace so mount changes are isolated to the sandboxed process. -fn unshare_mount_namespace() -> Result<()> { - let result = unsafe { libc::unshare(libc::CLONE_NEWNS) }; - if result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - Ok(()) -} - -/// Unshare user + mount namespaces so the process can remount read-only without privileges. -fn unshare_user_and_mount_namespaces() -> Result<()> { - let result = unsafe { libc::unshare(libc::CLONE_NEWUSER | libc::CLONE_NEWNS) }; - if result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - Ok(()) -} - -fn is_running_as_root() -> bool { - unsafe { libc::geteuid() == 0 } -} - -#[repr(C)] -struct CapUserHeader { - version: u32, - pid: i32, -} - -#[repr(C)] -struct CapUserData { - effective: u32, - permitted: u32, - inheritable: u32, -} - -const LINUX_CAPABILITY_VERSION_3: u32 = 0x2008_0522; - -/// Map the provided uid/gid to root inside the user namespace. -fn write_user_namespace_maps(uid: libc::uid_t, gid: libc::gid_t) -> Result<()> { - write_proc_file("/proc/self/setgroups", "deny\n")?; - - write_proc_file("/proc/self/uid_map", format!("0 {uid} 1\n"))?; - write_proc_file("/proc/self/gid_map", format!("0 {gid} 1\n"))?; - Ok(()) -} - -/// Drop all capabilities in the current user namespace. -fn drop_caps() -> Result<()> { - let mut header = CapUserHeader { - version: LINUX_CAPABILITY_VERSION_3, - pid: 0, - }; - let data = [ - CapUserData { - effective: 0, - permitted: 0, - inheritable: 0, - }, - CapUserData { - effective: 0, - permitted: 0, - inheritable: 0, - }, - ]; - - // Use syscall directly to avoid libc capability symbols that are missing on musl. - let result = unsafe { libc::syscall(libc::SYS_capset, &mut header, data.as_ptr()) }; - if result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - Ok(()) -} - -/// Write a small procfs file, returning a sandbox error on failure. -fn write_proc_file(path: &str, contents: impl AsRef<[u8]>) -> Result<()> { - std::fs::write(path, contents)?; - Ok(()) -} - -/// Ensure mounts are private so remounting does not propagate outside the namespace. -fn make_mounts_private() -> Result<()> { - let root = CString::new("/").map_err(|_| { - CodexErr::UnsupportedOperation("Sandbox mount path contains NUL byte: /".to_string()) - })?; - let result = unsafe { - libc::mount( - std::ptr::null(), - root.as_ptr(), - std::ptr::null(), - libc::MS_REC | libc::MS_PRIVATE, - std::ptr::null(), - ) - }; - if result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - Ok(()) -} - -/// Bind-mount a path onto itself and remount read-only. -fn bind_mount_read_only(path: &Path) -> Result<()> { - let c_path = CString::new(path.as_os_str().as_bytes()).map_err(|_| { - CodexErr::UnsupportedOperation(format!( - "Sandbox mount path contains NUL byte: {path}", - path = path.display() - )) - })?; - - let bind_result = unsafe { - libc::mount( - c_path.as_ptr(), - c_path.as_ptr(), - std::ptr::null(), - libc::MS_BIND, - std::ptr::null(), - ) - }; - if bind_result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - - let remount_result = unsafe { - libc::mount( - c_path.as_ptr(), - c_path.as_ptr(), - std::ptr::null(), - libc::MS_BIND | libc::MS_REMOUNT | libc::MS_RDONLY, - std::ptr::null(), - ) - }; - if remount_result != 0 { - return Err(std::io::Error::last_os_error().into()); - } - - Ok(()) -} - -#[cfg(test)] -mod tests { - use super::*; - use pretty_assertions::assert_eq; - - #[test] - fn collect_read_only_mount_targets_errors_on_missing_path() { - let tempdir = tempfile::tempdir().expect("tempdir"); - let missing = AbsolutePathBuf::try_from(tempdir.path().join("missing").as_path()) - .expect("missing path"); - let root = AbsolutePathBuf::try_from(tempdir.path()).expect("root"); - let writable_root = WritableRoot { - root, - read_only_subpaths: vec![missing], - }; - - let err = collect_read_only_mount_targets(&[writable_root]) - .expect_err("expected missing path error"); - let message = match err { - CodexErr::UnsupportedOperation(message) => message, - other => panic!("unexpected error: {other:?}"), - }; - assert_eq!( - message, - format!( - "Sandbox expected to protect {path}, but it does not exist. Ensure the repository contains this path or create it before running Codex.", - path = tempdir.path().join("missing").display() - ) - ); - } - - #[test] - fn collect_read_only_mount_targets_adds_gitdir_for_pointer_file() { - let tempdir = tempfile::tempdir().expect("tempdir"); - let gitdir = tempdir.path().join("actual-gitdir"); - std::fs::create_dir_all(&gitdir).expect("create gitdir"); - let dot_git = tempdir.path().join(".git"); - std::fs::write(&dot_git, format!("gitdir: {}\n", gitdir.display())) - .expect("write gitdir pointer"); - let root = AbsolutePathBuf::try_from(tempdir.path()).expect("root"); - let writable_root = WritableRoot { - root, - read_only_subpaths: vec![ - AbsolutePathBuf::try_from(dot_git.as_path()).expect("dot git"), - ], - }; - - let targets = collect_read_only_mount_targets(&[writable_root]).expect("collect targets"); - assert_eq!(targets.len(), 2); - assert_eq!(targets[0].as_path(), dot_git.as_path()); - assert_eq!(targets[1].as_path(), gitdir.as_path()); - } - - #[test] - fn collect_read_only_mount_targets_errors_on_invalid_gitdir_pointer() { - let tempdir = tempfile::tempdir().expect("tempdir"); - let dot_git = tempdir.path().join(".git"); - std::fs::write(&dot_git, "not-a-pointer\n").expect("write invalid pointer"); - let root = AbsolutePathBuf::try_from(tempdir.path()).expect("root"); - let writable_root = WritableRoot { - root, - read_only_subpaths: vec![ - AbsolutePathBuf::try_from(dot_git.as_path()).expect("dot git"), - ], - }; - - let err = collect_read_only_mount_targets(&[writable_root]) - .expect_err("expected invalid pointer error"); - let message = match err { - CodexErr::UnsupportedOperation(message) => message, - other => panic!("unexpected error: {other:?}"), - }; - assert_eq!( - message, - format!( - "Expected {path} to contain a gitdir pointer, but it did not match `gitdir: `.", - path = dot_git.display() - ) - ); - } -} diff --git a/codex-rs/linux-sandbox/src/vendored_bwrap.rs b/codex-rs/linux-sandbox/src/vendored_bwrap.rs index ab4fb959ef..c2816061ea 100644 --- a/codex-rs/linux-sandbox/src/vendored_bwrap.rs +++ b/codex-rs/linux-sandbox/src/vendored_bwrap.rs @@ -13,22 +13,35 @@ mod imp { fn bwrap_main(argc: libc::c_int, argv: *const *const c_char) -> libc::c_int; } - /// Execute the build-time bubblewrap `main` function with the given argv. - pub(crate) fn exec_vendored_bwrap(argv: Vec) -> ! { + fn argv_to_cstrings(argv: &[String]) -> Vec { let mut cstrings: Vec = Vec::with_capacity(argv.len()); - for arg in &argv { + for arg in argv { match CString::new(arg.as_str()) { Ok(value) => cstrings.push(value), Err(err) => panic!("failed to convert argv to CString: {err}"), } } + cstrings + } + + /// Run the build-time bubblewrap `main` function and return its exit code. + /// + /// On success, bubblewrap will `execve` into the target program and this + /// function will never return. A return value therefore implies failure. + pub(crate) fn run_vendored_bwrap_main(argv: &[String]) -> libc::c_int { + let cstrings = argv_to_cstrings(argv); let mut argv_ptrs: Vec<*const c_char> = cstrings.iter().map(|arg| arg.as_ptr()).collect(); argv_ptrs.push(std::ptr::null()); // SAFETY: We provide a null-terminated argv vector whose pointers // remain valid for the duration of the call. - let exit_code = unsafe { bwrap_main(cstrings.len() as libc::c_int, argv_ptrs.as_ptr()) }; + unsafe { bwrap_main(cstrings.len() as libc::c_int, argv_ptrs.as_ptr()) } + } + + /// Execute the build-time bubblewrap `main` function with the given argv. + pub(crate) fn exec_vendored_bwrap(argv: Vec) -> ! { + let exit_code = run_vendored_bwrap_main(&argv); std::process::exit(exit_code); } } @@ -36,7 +49,7 @@ mod imp { #[cfg(not(vendored_bwrap_available))] mod imp { /// Panics with a clear error when the build-time bwrap path is not enabled. - pub(crate) fn exec_vendored_bwrap(_argv: Vec) -> ! { + pub(crate) fn run_vendored_bwrap_main(_argv: &[String]) -> libc::c_int { panic!( "build-time bubblewrap is not available in this build.\n\ Rebuild codex-linux-sandbox on Linux with CODEX_BWRAP_ENABLE_FFI=1.\n\ @@ -49,6 +62,13 @@ Notes:\n\ - bubblewrap sources expected at codex-rs/vendor/bubblewrap (default)" ); } + + /// Panics with a clear error when the build-time bwrap path is not enabled. + pub(crate) fn exec_vendored_bwrap(_argv: Vec) -> ! { + let _ = run_vendored_bwrap_main(&[]); + unreachable!("run_vendored_bwrap_main should always panic in this configuration") + } } pub(crate) use imp::exec_vendored_bwrap; +pub(crate) use imp::run_vendored_bwrap_main; diff --git a/codex-rs/linux-sandbox/tests/suite/landlock.rs b/codex-rs/linux-sandbox/tests/suite/landlock.rs index 75551d4d91..52cd402a13 100644 --- a/codex-rs/linux-sandbox/tests/suite/landlock.rs +++ b/codex-rs/linux-sandbox/tests/suite/landlock.rs @@ -2,6 +2,7 @@ #![allow(clippy::unwrap_used)] use codex_core::config::types::ShellEnvironmentPolicy; use codex_core::error::CodexErr; +use codex_core::error::Result; use codex_core::error::SandboxErr; use codex_core::exec::ExecParams; use codex_core::exec::process_exec_tool_call; @@ -32,6 +33,8 @@ const NETWORK_TIMEOUT_MS: u64 = 2_000; #[cfg(target_arch = "aarch64")] const NETWORK_TIMEOUT_MS: u64 = 10_000; +const BWRAP_UNAVAILABLE_ERR: &str = "build-time bubblewrap is not available in this build."; + fn create_env_from_core_vars() -> HashMap { let policy = ShellEnvironmentPolicy::default(); create_env(&policy, None) @@ -47,12 +50,24 @@ async fn run_cmd(cmd: &[&str], writable_roots: &[PathBuf], timeout_ms: u64) { } } -#[expect(clippy::expect_used, clippy::unwrap_used)] +#[expect(clippy::expect_used)] async fn run_cmd_output( cmd: &[&str], writable_roots: &[PathBuf], timeout_ms: u64, ) -> codex_core::exec::ExecToolCallOutput { + run_cmd_result_with_writable_roots(cmd, writable_roots, timeout_ms, false) + .await + .expect("sandboxed command should execute") +} + +#[expect(clippy::expect_used)] +async fn run_cmd_result_with_writable_roots( + cmd: &[&str], + writable_roots: &[PathBuf], + timeout_ms: u64, + use_bwrap_sandbox: bool, +) -> Result { let cwd = std::env::current_dir().expect("cwd should exist"); let sandbox_cwd = cwd.clone(); let params = ExecParams { @@ -86,10 +101,48 @@ async fn run_cmd_output( &sandbox_policy, sandbox_cwd.as_path(), &codex_linux_sandbox_exe, + use_bwrap_sandbox, None, ) .await - .unwrap() +} + +fn is_bwrap_unavailable_output(output: &codex_core::exec::ExecToolCallOutput) -> bool { + output.stderr.text.contains(BWRAP_UNAVAILABLE_ERR) +} + +async fn should_skip_bwrap_tests() -> bool { + match run_cmd_result_with_writable_roots( + &["bash", "-lc", "true"], + &[], + NETWORK_TIMEOUT_MS, + true, + ) + .await + { + Ok(output) => is_bwrap_unavailable_output(&output), + Err(CodexErr::Sandbox(SandboxErr::Denied { output })) => { + is_bwrap_unavailable_output(&output) + } + // Probe timeouts are not actionable for the bwrap-specific assertions below; + // skip rather than fail the whole suite. + Err(CodexErr::Sandbox(SandboxErr::Timeout { .. })) => true, + Err(err) => panic!("bwrap availability probe failed unexpectedly: {err:?}"), + } +} + +fn expect_denied( + result: Result, + context: &str, +) -> codex_core::exec::ExecToolCallOutput { + match result { + Ok(output) => { + assert_ne!(output.exit_code, 0, "{context}: expected nonzero exit code"); + output + } + Err(CodexErr::Sandbox(SandboxErr::Denied { output })) => *output, + Err(err) => panic!("{context}: {err:?}"), + } } #[tokio::test] @@ -192,6 +245,7 @@ async fn assert_network_blocked(cmd: &[&str]) { &sandbox_policy, sandbox_cwd.as_path(), &codex_linux_sandbox_exe, + false, None, ) .await; @@ -242,6 +296,90 @@ async fn sandbox_blocks_nc() { assert_network_blocked(&["nc", "-z", "127.0.0.1", "80"]).await; } +#[tokio::test] +async fn sandbox_blocks_git_and_codex_writes_inside_writable_root() { + if should_skip_bwrap_tests().await { + eprintln!("skipping bwrap test: vendored bwrap was not built in this environment"); + return; + } + + let tmpdir = tempfile::tempdir().expect("tempdir"); + let dot_git = tmpdir.path().join(".git"); + let dot_codex = tmpdir.path().join(".codex"); + std::fs::create_dir_all(&dot_git).expect("create .git"); + std::fs::create_dir_all(&dot_codex).expect("create .codex"); + + let git_target = dot_git.join("config"); + let codex_target = dot_codex.join("config.toml"); + + let git_output = expect_denied( + run_cmd_result_with_writable_roots( + &[ + "bash", + "-lc", + &format!("echo denied > {}", git_target.to_string_lossy()), + ], + &[tmpdir.path().to_path_buf()], + LONG_TIMEOUT_MS, + true, + ) + .await, + ".git write should be denied under bubblewrap", + ); + + let codex_output = expect_denied( + run_cmd_result_with_writable_roots( + &[ + "bash", + "-lc", + &format!("echo denied > {}", codex_target.to_string_lossy()), + ], + &[tmpdir.path().to_path_buf()], + LONG_TIMEOUT_MS, + true, + ) + .await, + ".codex write should be denied under bubblewrap", + ); + assert_ne!(git_output.exit_code, 0); + assert_ne!(codex_output.exit_code, 0); +} + +#[tokio::test] +async fn sandbox_blocks_codex_symlink_replacement_attack() { + if should_skip_bwrap_tests().await { + eprintln!("skipping bwrap test: vendored bwrap was not built in this environment"); + return; + } + + use std::os::unix::fs::symlink; + + let tmpdir = tempfile::tempdir().expect("tempdir"); + let decoy = tmpdir.path().join("decoy-codex"); + std::fs::create_dir_all(&decoy).expect("create decoy dir"); + + let dot_codex = tmpdir.path().join(".codex"); + symlink(&decoy, &dot_codex).expect("create .codex symlink"); + + let codex_target = dot_codex.join("config.toml"); + + let codex_output = expect_denied( + run_cmd_result_with_writable_roots( + &[ + "bash", + "-lc", + &format!("echo denied > {}", codex_target.to_string_lossy()), + ], + &[tmpdir.path().to_path_buf()], + LONG_TIMEOUT_MS, + true, + ) + .await, + ".codex symlink replacement should be denied", + ); + assert_ne!(codex_output.exit_code, 0); +} + #[tokio::test] async fn sandbox_blocks_ssh() { // Force ssh to attempt a real TCP connection but fail quickly. `BatchMode` diff --git a/codex-rs/mcp-server/src/codex_tool_runner.rs b/codex-rs/mcp-server/src/codex_tool_runner.rs index 6273607a22..6513c9e6f7 100644 --- a/codex-rs/mcp-server/src/codex_tool_runner.rs +++ b/codex-rs/mcp-server/src/codex_tool_runner.rs @@ -322,6 +322,8 @@ async fn run_codex_tool_session_inner( | EventMsg::McpListToolsResponse(_) | EventMsg::ListCustomPromptsResponse(_) | EventMsg::ListSkillsResponse(_) + | EventMsg::ListRemoteSkillsResponse(_) + | EventMsg::RemoteSkillDownloaded(_) | EventMsg::ExecCommandBegin(_) | EventMsg::TerminalInteraction(_) | EventMsg::ExecCommandOutputDelta(_) diff --git a/codex-rs/otel/src/metrics/names.rs b/codex-rs/otel/src/metrics/names.rs index b8ff9364ad..97c4f4e6ed 100644 --- a/codex-rs/otel/src/metrics/names.rs +++ b/codex-rs/otel/src/metrics/names.rs @@ -8,3 +8,7 @@ pub(crate) const WEBSOCKET_REQUEST_COUNT_METRIC: &str = "codex.websocket.request pub(crate) const WEBSOCKET_REQUEST_DURATION_METRIC: &str = "codex.websocket.request.duration_ms"; pub(crate) const WEBSOCKET_EVENT_COUNT_METRIC: &str = "codex.websocket.event"; pub(crate) const WEBSOCKET_EVENT_DURATION_METRIC: &str = "codex.websocket.event.duration_ms"; +pub(crate) const RESPONSES_API_OVERHEAD_DURATION_METRIC: &str = + "codex.responses_api_overhead.duration_ms"; +pub(crate) const RESPONSES_API_INFERENCE_TIME_DURATION_METRIC: &str = + "codex.responses_api_inference_time.duration_ms"; diff --git a/codex-rs/otel/src/metrics/runtime_metrics.rs b/codex-rs/otel/src/metrics/runtime_metrics.rs index dbd28010f6..d59ff4b1b7 100644 --- a/codex-rs/otel/src/metrics/runtime_metrics.rs +++ b/codex-rs/otel/src/metrics/runtime_metrics.rs @@ -1,5 +1,7 @@ use crate::metrics::names::API_CALL_COUNT_METRIC; use crate::metrics::names::API_CALL_DURATION_METRIC; +use crate::metrics::names::RESPONSES_API_INFERENCE_TIME_DURATION_METRIC; +use crate::metrics::names::RESPONSES_API_OVERHEAD_DURATION_METRIC; use crate::metrics::names::SSE_EVENT_COUNT_METRIC; use crate::metrics::names::SSE_EVENT_DURATION_METRIC; use crate::metrics::names::TOOL_CALL_COUNT_METRIC; @@ -32,6 +34,8 @@ pub struct RuntimeMetricsSummary { pub streaming_events: RuntimeMetricTotals, pub websocket_calls: RuntimeMetricTotals, pub websocket_events: RuntimeMetricTotals, + pub responses_api_overhead_ms: u64, + pub responses_api_inference_time_ms: u64, } impl RuntimeMetricsSummary { @@ -41,6 +45,8 @@ impl RuntimeMetricsSummary { && self.streaming_events.is_empty() && self.websocket_calls.is_empty() && self.websocket_events.is_empty() + && self.responses_api_overhead_ms == 0 + && self.responses_api_inference_time_ms == 0 } pub(crate) fn from_snapshot(snapshot: &ResourceMetrics) -> Self { @@ -64,12 +70,18 @@ impl RuntimeMetricsSummary { count: sum_counter(snapshot, WEBSOCKET_EVENT_COUNT_METRIC), duration_ms: sum_histogram_ms(snapshot, WEBSOCKET_EVENT_DURATION_METRIC), }; + let responses_api_overhead_ms = + sum_histogram_ms(snapshot, RESPONSES_API_OVERHEAD_DURATION_METRIC); + let responses_api_inference_time_ms = + sum_histogram_ms(snapshot, RESPONSES_API_INFERENCE_TIME_DURATION_METRIC); Self { tool_calls, api_calls, streaming_events, websocket_calls, websocket_events, + responses_api_overhead_ms, + responses_api_inference_time_ms, } } } diff --git a/codex-rs/otel/src/traces/otel_manager.rs b/codex-rs/otel/src/traces/otel_manager.rs index c585ec67d5..bde9515c20 100644 --- a/codex-rs/otel/src/traces/otel_manager.rs +++ b/codex-rs/otel/src/traces/otel_manager.rs @@ -1,5 +1,7 @@ use crate::metrics::names::API_CALL_COUNT_METRIC; use crate::metrics::names::API_CALL_DURATION_METRIC; +use crate::metrics::names::RESPONSES_API_INFERENCE_TIME_DURATION_METRIC; +use crate::metrics::names::RESPONSES_API_OVERHEAD_DURATION_METRIC; use crate::metrics::names::SSE_EVENT_COUNT_METRIC; use crate::metrics::names::SSE_EVENT_DURATION_METRIC; use crate::metrics::names::TOOL_CALL_COUNT_METRIC; @@ -42,6 +44,10 @@ pub use crate::ToolDecisionSource; const SSE_UNKNOWN_KIND: &str = "unknown"; const WEBSOCKET_UNKNOWN_KIND: &str = "unknown"; +const RESPONSES_WEBSOCKET_TIMING_KIND: &str = "responsesapi.websocket_timing"; +const RESPONSES_WEBSOCKET_TIMING_METRICS_FIELD: &str = "timing_metrics"; +const RESPONSES_API_OVERHEAD_FIELD: &str = "responses_duration_excl_engine_and_client_tool_time_ms"; +const RESPONSES_API_INFERENCE_FIELD: &str = "engine_service_total_ms"; impl OtelManager { #[allow(clippy::too_many_arguments)] @@ -252,6 +258,9 @@ impl OtelManager { .get("type") .and_then(|value| value.as_str()) .map(std::string::ToString::to_string); + if kind.as_deref() == Some(RESPONSES_WEBSOCKET_TIMING_KIND) { + self.record_responses_websocket_timing_metrics(&value); + } if kind.as_deref() == Some("response.failed") { success = false; error_message = value @@ -651,6 +660,22 @@ impl OtelManager { ); } + fn record_responses_websocket_timing_metrics(&self, value: &serde_json::Value) { + let timing_metrics = value.get(RESPONSES_WEBSOCKET_TIMING_METRICS_FIELD); + + let overhead_value = + timing_metrics.and_then(|value| value.get(RESPONSES_API_OVERHEAD_FIELD)); + if let Some(duration) = duration_from_ms_value(overhead_value) { + self.record_duration(RESPONSES_API_OVERHEAD_DURATION_METRIC, duration, &[]); + } + + let inference_value = + timing_metrics.and_then(|value| value.get(RESPONSES_API_INFERENCE_FIELD)); + if let Some(duration) = duration_from_ms_value(inference_value) { + self.record_duration(RESPONSES_API_INFERENCE_TIME_DURATION_METRIC, duration, &[]); + } + } + fn responses_type(event: &ResponseEvent) -> String { match event { ResponseEvent::Created => "created".into(), @@ -689,3 +714,16 @@ impl OtelManager { fn timestamp() -> String { Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true) } + +fn duration_from_ms_value(value: Option<&serde_json::Value>) -> Option { + let value = value?; + let ms = value + .as_f64() + .or_else(|| value.as_i64().map(|v| v as f64)) + .or_else(|| value.as_u64().map(|v| v as f64))?; + if !ms.is_finite() || ms < 0.0 { + return None; + } + let clamped = ms.min(u64::MAX as f64); + Some(Duration::from_millis(clamped.round() as u64)) +} diff --git a/codex-rs/otel/tests/suite/runtime_summary.rs b/codex-rs/otel/tests/suite/runtime_summary.rs index 78aed8a0a7..811c801fce 100644 --- a/codex-rs/otel/tests/suite/runtime_summary.rs +++ b/codex-rs/otel/tests/suite/runtime_summary.rs @@ -62,6 +62,14 @@ fn runtime_metrics_summary_collects_tool_api_and_streaming_metrics() -> Result<( r#"{"type":"response.created"}"#.into(), )))); manager.record_websocket_event(&ws_response, Duration::from_millis(80)); + let ws_timing_response: std::result::Result< + Option>, + codex_api::ApiError, + > = Ok(Some(Ok(Message::Text( + r#"{"type":"responsesapi.websocket_timing","timing_metrics":{"responses_duration_excl_engine_and_client_tool_time_ms":124,"engine_service_total_ms":457}}"# + .into(), + )))); + manager.record_websocket_event(&ws_timing_response, Duration::from_millis(20)); let summary = manager .runtime_metrics_summary() @@ -84,9 +92,11 @@ fn runtime_metrics_summary_collects_tool_api_and_streaming_metrics() -> Result<( duration_ms: 400, }, websocket_events: RuntimeMetricTotals { - count: 1, - duration_ms: 80, + count: 2, + duration_ms: 100, }, + responses_api_overhead_ms: 124, + responses_api_inference_time_ms: 457, }; assert_eq!(summary, expected); diff --git a/codex-rs/protocol/src/config_types.rs b/codex-rs/protocol/src/config_types.rs index aa28e1a6fa..f7d7b4f431 100644 --- a/codex-rs/protocol/src/config_types.rs +++ b/codex-rs/protocol/src/config_types.rs @@ -96,6 +96,7 @@ pub enum WindowsSandboxLevel { #[serde(rename_all = "lowercase")] #[strum(serialize_all = "lowercase")] pub enum Personality { + None, Friendly, Pragmatic, } diff --git a/codex-rs/protocol/src/dynamic_tools.rs b/codex-rs/protocol/src/dynamic_tools.rs index e55d372d8e..8b5405f307 100644 --- a/codex-rs/protocol/src/dynamic_tools.rs +++ b/codex-rs/protocol/src/dynamic_tools.rs @@ -24,7 +24,16 @@ pub struct DynamicToolCallRequest { #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, JsonSchema, TS)] #[serde(rename_all = "camelCase")] pub struct DynamicToolResponse { - pub call_id: String, - pub output: String, + pub content_items: Vec, pub success: bool, } + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, JsonSchema, TS)] +#[serde(tag = "type", rename_all = "camelCase")] +#[ts(tag = "type")] +pub enum DynamicToolCallOutputContentItem { + #[serde(rename_all = "camelCase")] + InputText { text: String }, + #[serde(rename_all = "camelCase")] + InputImage { image_url: String }, +} diff --git a/codex-rs/protocol/src/models.rs b/codex-rs/protocol/src/models.rs index fe63666e38..15114d970b 100644 --- a/codex-rs/protocol/src/models.rs +++ b/codex-rs/protocol/src/models.rs @@ -129,11 +129,11 @@ pub enum ResponseItem { arguments: String, call_id: String, }, - // NOTE: The input schema for `function_call_output` objects that clients send to the - // OpenAI /v1/responses endpoint is NOT the same shape as the objects the server returns on the - // SSE stream. When *sending* we must wrap the string output inside an object that includes a - // required `success` boolean. To ensure we serialize exactly the expected shape we introduce - // a dedicated payload struct and flatten it here. + // NOTE: The `output` field for `function_call_output` uses a dedicated payload type with + // custom serialization. On the wire it is either: + // - a plain string (`content`) + // - an array of structured content items (`content_items`) + // We keep this behavior centralized in `FunctionCallOutputPayload`. FunctionCallOutput { call_id: String, output: FunctionCallOutputPayload, @@ -249,7 +249,7 @@ impl DeveloperInstructions { match command_prefixes { Some(prefixes) => { format!( - "{APPROVAL_POLICY_ON_REQUEST_RULE}\nApproved command prefixes:\n{prefixes}" + "{APPROVAL_POLICY_ON_REQUEST_RULE}\n## Approved command prefixes\nThe following prefix rules have already been approved: {prefixes}" ) } None => APPROVAL_POLICY_ON_REQUEST_RULE.to_string(), @@ -617,9 +617,8 @@ impl From for ResponseItem { let output = match result { Ok(result) => FunctionCallOutputPayload::from(&result), Err(tool_call_err) => FunctionCallOutputPayload { - content: format!("err: {tool_call_err:?}"), + body: FunctionCallOutputBody::Text(format!("err: {tool_call_err:?}")), success: Some(false), - ..Default::default() }, }; Self::FunctionCallOutput { call_id, output } @@ -780,39 +779,146 @@ pub enum FunctionCallOutputContentItem { InputImage { image_url: String }, } +/// Converts structured function-call output content into plain text for +/// human-readable surfaces. +/// +/// This conversion is intentionally lossy: +/// - only `input_text` items are included +/// - image items are ignored +/// +/// We use this helper where callers still need a string representation (for +/// example telemetry previews or legacy string-only output paths) while keeping +/// the original multimodal `content_items` as the authoritative payload sent to +/// the model. +pub fn function_call_output_content_items_to_text( + content_items: &[FunctionCallOutputContentItem], +) -> Option { + let text_segments = content_items + .iter() + .filter_map(|item| match item { + FunctionCallOutputContentItem::InputText { text } if !text.trim().is_empty() => { + Some(text.as_str()) + } + FunctionCallOutputContentItem::InputText { .. } + | FunctionCallOutputContentItem::InputImage { .. } => None, + }) + .collect::>(); + + if text_segments.is_empty() { + None + } else { + Some(text_segments.join("\n")) + } +} + +impl From + for FunctionCallOutputContentItem +{ + fn from(item: crate::dynamic_tools::DynamicToolCallOutputContentItem) -> Self { + match item { + crate::dynamic_tools::DynamicToolCallOutputContentItem::InputText { text } => { + Self::InputText { text } + } + crate::dynamic_tools::DynamicToolCallOutputContentItem::InputImage { image_url } => { + Self::InputImage { image_url } + } + } + } +} + /// The payload we send back to OpenAI when reporting a tool call result. /// -/// `content` preserves the historical plain-string payload so downstream -/// integrations (tests, logging, etc.) can keep treating tool output as -/// `String`. When an MCP server returns richer data we additionally populate -/// `content_items` with the structured form that the Responses API understands. +/// `body` serializes directly as the wire value for `function_call_output.output`. +/// `success` remains internal metadata for downstream handling. #[derive(Debug, Default, Clone, PartialEq, JsonSchema, TS)] pub struct FunctionCallOutputPayload { - pub content: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub content_items: Option>, + pub body: FunctionCallOutputBody, pub success: Option, } -#[derive(Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, JsonSchema, TS)] #[serde(untagged)] -enum FunctionCallOutputPayloadSerde { +pub enum FunctionCallOutputBody { Text(String), - Items(Vec), + ContentItems(Vec), } -// The Responses API expects two *different* shapes depending on success vs failure: -// β€’ success β†’ output is a plain string (no nested object) -// β€’ failure β†’ output is an object { content, success:false } +impl FunctionCallOutputBody { + /// Best-effort conversion of a function-call output body to plain text for + /// human-readable surfaces. + /// + /// This conversion is intentionally lossy when the body contains content + /// items: image entries are dropped and text entries are joined with + /// newlines. + pub fn to_text(&self) -> Option { + match self { + Self::Text(content) => Some(content.clone()), + Self::ContentItems(items) => function_call_output_content_items_to_text(items), + } + } +} + +impl Default for FunctionCallOutputBody { + fn default() -> Self { + Self::Text(String::new()) + } +} + +impl FunctionCallOutputPayload { + pub fn from_text(content: String) -> Self { + Self { + body: FunctionCallOutputBody::Text(content), + success: None, + } + } + + pub fn from_content_items(content_items: Vec) -> Self { + Self { + body: FunctionCallOutputBody::ContentItems(content_items), + success: None, + } + } + + pub fn text_content(&self) -> Option<&str> { + match &self.body { + FunctionCallOutputBody::Text(content) => Some(content), + FunctionCallOutputBody::ContentItems(_) => None, + } + } + + pub fn text_content_mut(&mut self) -> Option<&mut String> { + match &mut self.body { + FunctionCallOutputBody::Text(content) => Some(content), + FunctionCallOutputBody::ContentItems(_) => None, + } + } + + pub fn content_items(&self) -> Option<&[FunctionCallOutputContentItem]> { + match &self.body { + FunctionCallOutputBody::Text(_) => None, + FunctionCallOutputBody::ContentItems(items) => Some(items), + } + } + + pub fn content_items_mut(&mut self) -> Option<&mut Vec> { + match &mut self.body { + FunctionCallOutputBody::Text(_) => None, + FunctionCallOutputBody::ContentItems(items) => Some(items), + } + } +} + +// `function_call_output.output` is encoded as either: +// - an array of structured content items +// - a plain string impl Serialize for FunctionCallOutputPayload { fn serialize(&self, serializer: S) -> Result where S: Serializer, { - if let Some(items) = &self.content_items { - items.serialize(serializer) - } else { - serializer.serialize_str(&self.content) + match &self.body { + FunctionCallOutputBody::Text(content) => serializer.serialize_str(content), + FunctionCallOutputBody::ContentItems(items) => items.serialize(serializer), } } } @@ -822,20 +928,11 @@ impl<'de> Deserialize<'de> for FunctionCallOutputPayload { where D: Deserializer<'de>, { - match FunctionCallOutputPayloadSerde::deserialize(deserializer)? { - FunctionCallOutputPayloadSerde::Text(content) => Ok(FunctionCallOutputPayload { - content, - ..Default::default() - }), - FunctionCallOutputPayloadSerde::Items(items) => { - let content = serde_json::to_string(&items).map_err(serde::de::Error::custom)?; - Ok(FunctionCallOutputPayload { - content, - content_items: Some(items), - success: None, - }) - } - } + let body = FunctionCallOutputBody::deserialize(deserializer)?; + Ok(FunctionCallOutputPayload { + body, + success: None, + }) } } @@ -856,16 +953,14 @@ impl From<&CallToolResult> for FunctionCallOutputPayload { match serde_json::to_string(structured_content) { Ok(serialized_structured_content) => { return FunctionCallOutputPayload { - content: serialized_structured_content, + body: FunctionCallOutputBody::Text(serialized_structured_content), success: Some(is_success), - ..Default::default() }; } Err(err) => { return FunctionCallOutputPayload { - content: err.to_string(), + body: FunctionCallOutputBody::Text(err.to_string()), success: Some(false), - ..Default::default() }; } } @@ -875,18 +970,21 @@ impl From<&CallToolResult> for FunctionCallOutputPayload { Ok(serialized_content) => serialized_content, Err(err) => { return FunctionCallOutputPayload { - content: err.to_string(), + body: FunctionCallOutputBody::Text(err.to_string()), success: Some(false), - ..Default::default() }; } }; let content_items = convert_mcp_content_to_items(content); + let body = match content_items { + Some(content_items) => FunctionCallOutputBody::ContentItems(content_items), + None => FunctionCallOutputBody::Text(serialized_content), + }; + FunctionCallOutputPayload { - content: serialized_content, - content_items, + body, success: Some(is_success), } } @@ -937,19 +1035,18 @@ fn convert_mcp_content_to_items( } // Implement Display so callers can treat the payload like a plain string when logging or doing -// trivial substring checks in tests (existing tests call `.contains()` on the output). Display -// returns the raw `content` field. +// trivial substring checks in tests (existing tests call `.contains()` on the output). For +// `ContentItems`, Display emits a JSON representation. impl std::fmt::Display for FunctionCallOutputPayload { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.write_str(&self.content) - } -} - -impl std::ops::Deref for FunctionCallOutputPayload { - type Target = str; - fn deref(&self) -> &Self::Target { - &self.content + match &self.body { + FunctionCallOutputBody::Text(content) => f.write_str(content), + FunctionCallOutputBody::ContentItems(items) => { + let content = serde_json::to_string(items).unwrap_or_default(); + f.write_str(content.as_str()) + } + } } } @@ -1010,6 +1107,61 @@ mod tests { assert_eq!(convert_mcp_content_to_items(&contents), None); } + #[test] + fn function_call_output_content_items_to_text_joins_text_segments() { + let content_items = vec![ + FunctionCallOutputContentItem::InputText { + text: "line 1".to_string(), + }, + FunctionCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + FunctionCallOutputContentItem::InputText { + text: "line 2".to_string(), + }, + ]; + + let text = function_call_output_content_items_to_text(&content_items); + assert_eq!(text, Some("line 1\nline 2".to_string())); + } + + #[test] + fn function_call_output_content_items_to_text_ignores_blank_text_and_images() { + let content_items = vec![ + FunctionCallOutputContentItem::InputText { + text: " ".to_string(), + }, + FunctionCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + ]; + + let text = function_call_output_content_items_to_text(&content_items); + assert_eq!(text, None); + } + + #[test] + fn function_call_output_body_to_text_returns_plain_text_content() { + let body = FunctionCallOutputBody::Text("ok".to_string()); + let text = body.to_text(); + assert_eq!(text, Some("ok".to_string())); + } + + #[test] + fn function_call_output_body_to_text_uses_content_item_fallback() { + let body = FunctionCallOutputBody::ContentItems(vec![ + FunctionCallOutputContentItem::InputText { + text: "line 1".to_string(), + }, + FunctionCallOutputContentItem::InputImage { + image_url: "data:image/png;base64,AAA".to_string(), + }, + ]); + + let text = body.to_text(); + assert_eq!(text, Some("line 1".to_string())); + } + #[test] fn converts_sandbox_mode_into_developer_instructions() { let workspace_write: DeveloperInstructions = SandboxMode::WorkspaceWrite.into(); @@ -1156,10 +1308,7 @@ mod tests { fn serializes_success_as_plain_string() -> Result<()> { let item = ResponseInputItem::FunctionCallOutput { call_id: "call1".into(), - output: FunctionCallOutputPayload { - content: "ok".into(), - ..Default::default() - }, + output: FunctionCallOutputPayload::from_text("ok".into()), }; let json = serde_json::to_string(&item)?; @@ -1175,9 +1324,8 @@ mod tests { let item = ResponseInputItem::FunctionCallOutput { call_id: "call1".into(), output: FunctionCallOutputPayload { - content: "bad".into(), + body: FunctionCallOutputBody::Text("bad".into()), success: Some(false), - ..Default::default() }, }; @@ -1202,7 +1350,10 @@ mod tests { let payload = FunctionCallOutputPayload::from(&call_tool_result); assert_eq!(payload.success, Some(true)); - let items = payload.content_items.clone().expect("content items"); + let Some(items) = payload.content_items() else { + panic!("expected content items"); + }; + let items = items.to_vec(); assert_eq!( items, vec![ @@ -1243,9 +1394,10 @@ mod tests { }; let payload = FunctionCallOutputPayload::from(&call_tool_result); - let Some(items) = payload.content_items else { + let Some(items) = payload.content_items() else { panic!("expected content items"); }; + let items = items.to_vec(); assert_eq!( items, vec![FunctionCallOutputContentItem::InputImage { @@ -1274,10 +1426,14 @@ mod tests { image_url: "data:image/png;base64,XYZ".into(), }, ]; - assert_eq!(payload.content_items, Some(expected_items.clone())); - - let expected_content = serde_json::to_string(&expected_items)?; - assert_eq!(payload.content, expected_content); + assert_eq!( + payload.body, + FunctionCallOutputBody::ContentItems(expected_items.clone()) + ); + assert_eq!( + serde_json::to_string(&payload)?, + serde_json::to_string(&expected_items)? + ); Ok(()) } diff --git a/codex-rs/protocol/src/openai_models.rs b/codex-rs/protocol/src/openai_models.rs index 90cf34f393..36298e9ca5 100644 --- a/codex-rs/protocol/src/openai_models.rs +++ b/codex-rs/protocol/src/openai_models.rs @@ -335,6 +335,7 @@ impl ModelInstructionsVariables { pub fn get_personality_message(&self, personality: Option) -> Option { if let Some(personality) = personality { match personality { + Personality::None => Some(String::new()), Personality::Friendly => self.personality_friendly.clone(), Personality::Pragmatic => self.personality_pragmatic.clone(), } @@ -546,6 +547,10 @@ mod tests { model.get_model_instructions(Some(Personality::Pragmatic)), "Hello\n" ); + assert_eq!( + model.get_model_instructions(Some(Personality::None)), + "Hello\n" + ); assert_eq!(model.get_model_instructions(None), "Hello\n"); let model_no_personality = test_model(Some(ModelMessages { @@ -564,6 +569,10 @@ mod tests { model_no_personality.get_model_instructions(Some(Personality::Pragmatic)), "Hello\n" ); + assert_eq!( + model_no_personality.get_model_instructions(Some(Personality::None)), + "Hello\n" + ); assert_eq!(model_no_personality.get_model_instructions(None), "Hello\n"); } @@ -603,6 +612,10 @@ mod tests { personality_variables.get_personality_message(Some(Personality::Pragmatic)), Some("pragmatic".to_string()) ); + assert_eq!( + personality_variables.get_personality_message(Some(Personality::None)), + Some(String::new()) + ); assert_eq!( personality_variables.get_personality_message(None), Some("default".to_string()) @@ -621,6 +634,10 @@ mod tests { personality_variables.get_personality_message(Some(Personality::Pragmatic)), None ); + assert_eq!( + personality_variables.get_personality_message(Some(Personality::None)), + Some(String::new()) + ); assert_eq!( personality_variables.get_personality_message(None), Some("default".to_string()) @@ -639,6 +656,10 @@ mod tests { personality_variables.get_personality_message(Some(Personality::Pragmatic)), Some("pragmatic".to_string()) ); + assert_eq!( + personality_variables.get_personality_message(Some(Personality::None)), + Some(String::new()) + ); assert_eq!(personality_variables.get_personality_message(None), None); } } diff --git a/codex-rs/protocol/src/prompts/permissions/approval_policy/on_request_rule.md b/codex-rs/protocol/src/prompts/permissions/approval_policy/on_request_rule.md index 336f1fb748..96d962d12d 100644 --- a/codex-rs/protocol/src/prompts/permissions/approval_policy/on_request_rule.md +++ b/codex-rs/protocol/src/prompts/permissions/approval_policy/on_request_rule.md @@ -25,9 +25,9 @@ IMPORTANT: To request approval to execute a command that will require escalated - Provide the `sandbox_permissions` parameter with the value `"require_escalated"` - Include a short question asking the user if they want to allow the action in `justification` parameter. e.g. "Do you want to download and install dependencies for this project?" -- Suggest a `prefix_rule` - this will be shown to the user with an option to persist the rule approval for future sessions. +- Optionally suggest a `prefix_rule` - this will be shown to the user with an option to persist the rule approval for future sessions. -If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with "require_escalated". ALWAYS proceed to use the `justification` and `prefix_rule` parameters - do not message the user before requesting approval for the command. +If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with "require_escalated". ALWAYS proceed to use the `justification` parameter - do not message the user before requesting approval for the command. ## When to request escalation @@ -37,25 +37,21 @@ While commands are running inside the sandbox, here are some scenarios that will - You need to run a GUI app (e.g., open/xdg-open/osascript) to open browsers or files. - If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with `require_escalated`. ALWAYS proceed to use the `sandbox_permissions` and `justification` parameters. do not message the user before requesting approval for the command. - You are about to take a potentially destructive action such as an `rm` or `git reset` that the user did not explicitly ask for. - -Only run commands that require approval if it is absolutely necessary to solve the user's query, don't try and circumvent approvals by using other tools. +- Be judicious with escalating, but if completing the user's request requires it, you should do so - don't try and circumvent approvals by using other tools. ## prefix_rule guidance -When choosing a `prefix_rule`, request one that will allow you to fulfill similar requests from the user in the future without re-requesting escalation. It should be categorical and reasonably scoped to similar capabilities. You MUST NOT pass the entire command into `prefix_rule`. +When choosing a `prefix_rule`, request one that will allow you to fulfill similar requests from the user in the future without re-requesting escalation. It should be categorical and reasonably scoped to similar capabilities. You should rarely pass the entire command into `prefix_rule`. + +### Banned prefix_rules +Avoid requesting overly broad prefixes that the user would be ill-advised to approve. For example, do not request ["python3"], ["python", "-"], or other similar prefixes. +NEVER provide a prefix_rule argument for destructive commands like rm. +NEVER provide a prefix_rule if your command uses a heredoc or herestring. + +### Examples +Good examples of prefixes: +- ["npm", "run", "dev"] +- ["gh", "pr", "check"] +- ["pytest"] +- ["cargo", "test"] - -["npm", "run", "dev"] - - -["gh", "pr", "checks"] - - -["pytest"] - - -["cargo", "test", "-p", "codex-app-server"] - -["cargo", "test"] - - diff --git a/codex-rs/protocol/src/protocol.rs b/codex-rs/protocol/src/protocol.rs index 140d3931e0..078ea31d18 100644 --- a/codex-rs/protocol/src/protocol.rs +++ b/codex-rs/protocol/src/protocol.rs @@ -269,6 +269,15 @@ pub enum Op { force_reload: bool, }, + /// Request the list of remote skills available via ChatGPT sharing. + ListRemoteSkills, + + /// Download a remote skill by id into the local skills cache. + DownloadRemoteSkill { + hazelnut_id: String, + is_preload: bool, + }, + /// Request the agent to summarize the current conversation context. /// The agent will use its existing context (either conversation history or previous response id) /// to generate a summary which will be returned as an AgentMessage event. @@ -822,6 +831,12 @@ pub enum EventMsg { /// List of skills available to the agent. ListSkillsResponse(ListSkillsResponseEvent), + /// List of remote skills available to the agent. + ListRemoteSkillsResponse(ListRemoteSkillsResponseEvent), + + /// Remote skill downloaded to local cache. + RemoteSkillDownloaded(RemoteSkillDownloadedEvent), + /// Notification that skill data may have been updated and clients may want to reload. SkillsUpdateAvailable, @@ -2127,6 +2142,27 @@ pub struct ListSkillsResponseEvent { pub skills: Vec, } +#[derive(Debug, Clone, Deserialize, Serialize, JsonSchema, TS)] +pub struct RemoteSkillSummary { + pub id: String, + pub name: String, + pub description: String, +} + +/// Response payload for `Op::ListRemoteSkills`. +#[derive(Debug, Clone, Deserialize, Serialize, JsonSchema, TS)] +pub struct ListRemoteSkillsResponseEvent { + pub skills: Vec, +} + +/// Response payload for `Op::DownloadRemoteSkill`. +#[derive(Debug, Clone, Deserialize, Serialize, JsonSchema, TS)] +pub struct RemoteSkillDownloadedEvent { + pub id: String, + pub name: String, + pub path: PathBuf, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema, TS)] #[serde(rename_all = "snake_case")] #[ts(rename_all = "snake_case")] diff --git a/codex-rs/state/migrations/0006_thread_memory.sql b/codex-rs/state/migrations/0006_thread_memory.sql new file mode 100644 index 0000000000..fe90ab6679 --- /dev/null +++ b/codex-rs/state/migrations/0006_thread_memory.sql @@ -0,0 +1,9 @@ +CREATE TABLE thread_memory ( + thread_id TEXT PRIMARY KEY, + trace_summary TEXT NOT NULL, + memory_summary TEXT NOT NULL, + updated_at INTEGER NOT NULL, + FOREIGN KEY(thread_id) REFERENCES threads(id) ON DELETE CASCADE +); + +CREATE INDEX idx_thread_memory_updated_at ON thread_memory(updated_at DESC, thread_id DESC); diff --git a/codex-rs/state/src/bin/logs_client.rs b/codex-rs/state/src/bin/logs_client.rs index d1329a1a89..796a968cb8 100644 --- a/codex-rs/state/src/bin/logs_client.rs +++ b/codex-rs/state/src/bin/logs_client.rs @@ -6,14 +6,13 @@ use chrono::DateTime; use clap::Parser; use codex_state::LogQuery; use codex_state::LogRow; -use codex_state::STATE_DB_FILENAME; use codex_state::StateRuntime; use dirs::home_dir; use owo_colors::OwoColorize; #[derive(Debug, Parser)] #[command(name = "codex-state-logs")] -#[command(about = "Tail Codex logs from state.sqlite with simple filters")] +#[command(about = "Tail Codex logs from the state SQLite DB with simple filters")] struct Args { /// Path to CODEX_HOME. Defaults to $CODEX_HOME or ~/.codex. #[arg(long, env = "CODEX_HOME")] @@ -104,7 +103,7 @@ fn resolve_db_path(args: &Args) -> anyhow::Result { } let codex_home = args.codex_home.clone().unwrap_or_else(default_codex_home); - Ok(codex_home.join(STATE_DB_FILENAME)) + Ok(codex_state::state_db_path(codex_home.as_path())) } fn default_codex_home() -> PathBuf { diff --git a/codex-rs/state/src/extract.rs b/codex-rs/state/src/extract.rs index ad40118086..753d8e426d 100644 --- a/codex-rs/state/src/extract.rs +++ b/codex-rs/state/src/extract.rs @@ -1,8 +1,5 @@ use crate::model::ThreadMetadata; -use codex_protocol::models::ContentItem; use codex_protocol::models::ResponseItem; -use codex_protocol::models::is_local_image_close_tag_text; -use codex_protocol::models::is_local_image_open_tag_text; use codex_protocol::protocol::EventMsg; use codex_protocol::protocol::RolloutItem; use codex_protocol::protocol::SessionMetaLine; @@ -73,39 +70,8 @@ fn apply_event_msg(metadata: &mut ThreadMetadata, event: &EventMsg) { } } -fn apply_response_item(metadata: &mut ThreadMetadata, item: &ResponseItem) { - if let Some(text) = extract_user_message_text(item) { - metadata.has_user_event = true; - if metadata.title.is_empty() { - metadata.title = text; - } - } -} - -fn extract_user_message_text(item: &ResponseItem) -> Option { - let ResponseItem::Message { role, content, .. } = item else { - return None; - }; - if role != "user" { - return None; - } - let texts: Vec<&str> = content - .iter() - .filter_map(|content_item| match content_item { - ContentItem::InputText { text } => Some(text.as_str()), - ContentItem::InputImage { .. } | ContentItem::OutputText { .. } => None, - }) - .filter(|text| !is_local_image_open_tag_text(text) && !is_local_image_close_tag_text(text)) - .collect(); - if texts.is_empty() { - return None; - } - let joined = texts.join("\n"); - Some( - strip_user_message_prefix(joined.as_str()) - .trim() - .to_string(), - ) +fn apply_response_item(_metadata: &mut ThreadMetadata, _item: &ResponseItem) { + // Title and has_user_event are derived from EventMsg::UserMessage only. } fn strip_user_message_prefix(text: &str) -> &str { @@ -125,43 +91,61 @@ pub(crate) fn enum_to_string(value: &T) -> String { #[cfg(test)] mod tests { - use super::extract_user_message_text; + use super::apply_rollout_item; use crate::model::ThreadMetadata; use chrono::DateTime; use chrono::Utc; use codex_protocol::ThreadId; use codex_protocol::models::ContentItem; use codex_protocol::models::ResponseItem; + use codex_protocol::protocol::EventMsg; + use codex_protocol::protocol::RolloutItem; use codex_protocol::protocol::USER_MESSAGE_BEGIN; + use codex_protocol::protocol::UserMessageEvent; + use pretty_assertions::assert_eq; use std::path::PathBuf; use uuid::Uuid; #[test] - fn extracts_user_message_text() { - let item = ResponseItem::Message { + fn response_item_user_messages_do_not_set_title_or_has_user_event() { + let mut metadata = metadata_for_test(); + let item = RolloutItem::ResponseItem(ResponseItem::Message { id: None, role: "user".to_string(), - content: vec![ - ContentItem::InputText { - text: format!(" {USER_MESSAGE_BEGIN}actual question"), - }, - ContentItem::InputImage { - image_url: "https://example.com/image.png".to_string(), - }, - ], + content: vec![ContentItem::InputText { + text: "hello from response item".to_string(), + }], end_turn: None, phase: None, - }; - let actual = extract_user_message_text(&item); - assert_eq!(actual.as_deref(), Some("actual question")); + }); + + apply_rollout_item(&mut metadata, &item, "test-provider"); + + assert_eq!(metadata.has_user_event, false); + assert_eq!(metadata.title, ""); } #[test] - fn diff_fields_detects_changes() { - let id = ThreadId::from_string(&Uuid::now_v7().to_string()).expect("thread id"); + fn event_msg_user_messages_set_title_and_has_user_event() { + let mut metadata = metadata_for_test(); + let item = RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent { + message: format!("{USER_MESSAGE_BEGIN} actual user request"), + images: Some(vec![]), + local_images: vec![], + text_elements: vec![], + })); + + apply_rollout_item(&mut metadata, &item, "test-provider"); + + assert_eq!(metadata.has_user_event, true); + assert_eq!(metadata.title, "actual user request"); + } + + fn metadata_for_test() -> ThreadMetadata { + let id = ThreadId::from_string(&Uuid::from_u128(42).to_string()).expect("thread id"); let created_at = DateTime::::from_timestamp(1_735_689_600, 0).expect("timestamp"); - let base = ThreadMetadata { + ThreadMetadata { id, rollout_path: PathBuf::from("/tmp/a.jsonl"), created_at, @@ -169,7 +153,7 @@ mod tests { source: "cli".to_string(), model_provider: "openai".to_string(), cwd: PathBuf::from("/tmp"), - title: "hello".to_string(), + title: String::new(), sandbox_policy: "read-only".to_string(), approval_mode: "on-request".to_string(), tokens_used: 1, @@ -178,7 +162,14 @@ mod tests { git_sha: None, git_branch: None, git_origin_url: None, - }; + } + } + + #[test] + fn diff_fields_detects_changes() { + let mut base = metadata_for_test(); + base.id = ThreadId::from_string(&Uuid::now_v7().to_string()).expect("thread id"); + base.title = "hello".to_string(); let mut other = base.clone(); other.tokens_used = 2; other.title = "world".to_string(); diff --git a/codex-rs/state/src/lib.rs b/codex-rs/state/src/lib.rs index c08c76a1cf..7df3374096 100644 --- a/codex-rs/state/src/lib.rs +++ b/codex-rs/state/src/lib.rs @@ -25,10 +25,14 @@ pub use model::Anchor; pub use model::BackfillStats; pub use model::ExtractionOutcome; pub use model::SortKey; +pub use model::ThreadMemory; pub use model::ThreadMetadata; pub use model::ThreadMetadataBuilder; pub use model::ThreadsPage; pub use runtime::STATE_DB_FILENAME; +pub use runtime::STATE_DB_VERSION; +pub use runtime::state_db_filename; +pub use runtime::state_db_path; /// Errors encountered during DB operations. Tags: [stage] pub const DB_ERROR_METRIC: &str = "codex.db.error"; diff --git a/codex-rs/state/src/model/mod.rs b/codex-rs/state/src/model/mod.rs index bd615d7561..d937019ec1 100644 --- a/codex-rs/state/src/model/mod.rs +++ b/codex-rs/state/src/model/mod.rs @@ -1,9 +1,11 @@ mod log; +mod thread_memory; mod thread_metadata; pub use log::LogEntry; pub use log::LogQuery; pub use log::LogRow; +pub use thread_memory::ThreadMemory; pub use thread_metadata::Anchor; pub use thread_metadata::BackfillStats; pub use thread_metadata::ExtractionOutcome; @@ -12,6 +14,7 @@ pub use thread_metadata::ThreadMetadata; pub use thread_metadata::ThreadMetadataBuilder; pub use thread_metadata::ThreadsPage; +pub(crate) use thread_memory::ThreadMemoryRow; pub(crate) use thread_metadata::ThreadRow; pub(crate) use thread_metadata::anchor_from_item; pub(crate) use thread_metadata::datetime_to_epoch_seconds; diff --git a/codex-rs/state/src/model/thread_memory.rs b/codex-rs/state/src/model/thread_memory.rs new file mode 100644 index 0000000000..6e3a34c21d --- /dev/null +++ b/codex-rs/state/src/model/thread_memory.rs @@ -0,0 +1,52 @@ +use anyhow::Result; +use chrono::DateTime; +use chrono::Utc; +use codex_protocol::ThreadId; +use sqlx::Row; +use sqlx::sqlite::SqliteRow; + +/// Stored memory summaries for a single thread. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ThreadMemory { + pub thread_id: ThreadId, + pub trace_summary: String, + pub memory_summary: String, + pub updated_at: DateTime, +} + +#[derive(Debug)] +pub(crate) struct ThreadMemoryRow { + thread_id: String, + trace_summary: String, + memory_summary: String, + updated_at: i64, +} + +impl ThreadMemoryRow { + pub(crate) fn try_from_row(row: &SqliteRow) -> Result { + Ok(Self { + thread_id: row.try_get("thread_id")?, + trace_summary: row.try_get("trace_summary")?, + memory_summary: row.try_get("memory_summary")?, + updated_at: row.try_get("updated_at")?, + }) + } +} + +impl TryFrom for ThreadMemory { + type Error = anyhow::Error; + + fn try_from(row: ThreadMemoryRow) -> std::result::Result { + Ok(Self { + thread_id: ThreadId::try_from(row.thread_id)?, + trace_summary: row.trace_summary, + memory_summary: row.memory_summary, + updated_at: epoch_seconds_to_datetime(row.updated_at)?, + }) + } +} + +fn epoch_seconds_to_datetime(secs: i64) -> Result> { + DateTime::::from_timestamp(secs, 0) + .ok_or_else(|| anyhow::anyhow!("invalid unix timestamp: {secs}")) +} diff --git a/codex-rs/state/src/runtime.rs b/codex-rs/state/src/runtime.rs index 9a750f1d5d..25c666fa6e 100644 --- a/codex-rs/state/src/runtime.rs +++ b/codex-rs/state/src/runtime.rs @@ -3,11 +3,13 @@ use crate::LogEntry; use crate::LogQuery; use crate::LogRow; use crate::SortKey; +use crate::ThreadMemory; use crate::ThreadMetadata; use crate::ThreadMetadataBuilder; use crate::ThreadsPage; use crate::apply_rollout_item; use crate::migrations::MIGRATOR; +use crate::model::ThreadMemoryRow; use crate::model::ThreadRow; use crate::model::anchor_from_item; use crate::model::datetime_to_epoch_seconds; @@ -35,7 +37,8 @@ use std::sync::Arc; use std::time::Duration; use tracing::warn; -pub const STATE_DB_FILENAME: &str = "state.sqlite"; +pub const STATE_DB_FILENAME: &str = "state"; +pub const STATE_DB_VERSION: u32 = 2; const METRIC_DB_INIT: &str = "codex.db.init"; @@ -56,7 +59,8 @@ impl StateRuntime { otel: Option, ) -> anyhow::Result> { tokio::fs::create_dir_all(&codex_home).await?; - let state_path = codex_home.join(STATE_DB_FILENAME); + remove_legacy_state_files(&codex_home).await; + let state_path = state_db_path(codex_home.as_path()); let existed = tokio::fs::try_exists(&state_path).await.unwrap_or(false); let pool = match open_sqlite(&state_path).await { Ok(db) => Arc::new(db), @@ -151,6 +155,26 @@ ORDER BY position ASC Ok(Some(tools)) } + /// Get memory summaries for a thread, if present. + pub async fn get_thread_memory( + &self, + thread_id: ThreadId, + ) -> anyhow::Result> { + let row = sqlx::query( + r#" +SELECT thread_id, trace_summary, memory_summary, updated_at +FROM thread_memory +WHERE thread_id = ? + "#, + ) + .bind(thread_id.to_string()) + .fetch_optional(self.pool.as_ref()) + .await?; + + row.map(|row| ThreadMemoryRow::try_from_row(&row).and_then(ThreadMemory::try_from)) + .transpose() + } + /// Find a rollout path by thread id using the underlying database. pub async fn find_rollout_path_by_id( &self, @@ -403,6 +427,83 @@ ON CONFLICT(id) DO UPDATE SET Ok(()) } + /// Insert or update memory summaries for a thread. + /// + /// This method always advances `updated_at`, even if summaries are unchanged. + pub async fn upsert_thread_memory( + &self, + thread_id: ThreadId, + trace_summary: &str, + memory_summary: &str, + ) -> anyhow::Result { + if self.get_thread(thread_id).await?.is_none() { + return Err(anyhow::anyhow!("thread not found: {thread_id}")); + } + + let updated_at = Utc::now().timestamp(); + sqlx::query( + r#" +INSERT INTO thread_memory ( + thread_id, + trace_summary, + memory_summary, + updated_at +) VALUES (?, ?, ?, ?) +ON CONFLICT(thread_id) DO UPDATE SET + trace_summary = excluded.trace_summary, + memory_summary = excluded.memory_summary, + updated_at = CASE + WHEN excluded.updated_at <= thread_memory.updated_at THEN thread_memory.updated_at + 1 + ELSE excluded.updated_at + END + "#, + ) + .bind(thread_id.to_string()) + .bind(trace_summary) + .bind(memory_summary) + .bind(updated_at) + .execute(self.pool.as_ref()) + .await?; + + self.get_thread_memory(thread_id) + .await? + .ok_or_else(|| anyhow::anyhow!("failed to load upserted thread memory: {thread_id}")) + } + + /// Get the last `n` memories for threads with an exact cwd match. + pub async fn get_last_n_thread_memories_for_cwd( + &self, + cwd: &Path, + n: usize, + ) -> anyhow::Result> { + if n == 0 { + return Ok(Vec::new()); + } + + let rows = sqlx::query( + r#" +SELECT + m.thread_id, + m.trace_summary, + m.memory_summary, + m.updated_at +FROM thread_memory AS m +INNER JOIN threads AS t ON t.id = m.thread_id +WHERE t.cwd = ? +ORDER BY m.updated_at DESC, m.thread_id DESC +LIMIT ? + "#, + ) + .bind(cwd.display().to_string()) + .bind(n as i64) + .fetch_all(self.pool.as_ref()) + .await?; + + rows.into_iter() + .map(|row| ThreadMemoryRow::try_from_row(&row).and_then(ThreadMemory::try_from)) + .collect() + } + /// Persist dynamic tools for a thread if none have been stored yet. /// /// Dynamic tools are defined at thread start and should not change afterward. @@ -419,6 +520,7 @@ ON CONFLICT(id) DO UPDATE SET return Ok(()); } let thread_id = thread_id.to_string(); + let mut tx = self.pool.begin().await?; for (idx, tool) in tools.iter().enumerate() { let position = i64::try_from(idx).unwrap_or(i64::MAX); let input_schema = serde_json::to_string(&tool.input_schema)?; @@ -439,9 +541,10 @@ ON CONFLICT(thread_id, position) DO NOTHING .bind(tool.name.as_str()) .bind(tool.description.as_str()) .bind(input_schema) - .execute(self.pool.as_ref()) + .execute(&mut *tx) .await?; } + tx.commit().await?; Ok(()) } @@ -622,6 +725,77 @@ async fn open_sqlite(path: &Path) -> anyhow::Result { Ok(pool) } +pub fn state_db_filename() -> String { + format!("{STATE_DB_FILENAME}_{STATE_DB_VERSION}.sqlite") +} + +pub fn state_db_path(codex_home: &Path) -> PathBuf { + codex_home.join(state_db_filename()) +} + +async fn remove_legacy_state_files(codex_home: &Path) { + let current_name = state_db_filename(); + let mut entries = match tokio::fs::read_dir(codex_home).await { + Ok(entries) => entries, + Err(err) => { + warn!( + "failed to read codex_home for state db cleanup {}: {err}", + codex_home.display() + ); + return; + } + }; + while let Ok(Some(entry)) = entries.next_entry().await { + if !entry + .file_type() + .await + .map(|file_type| file_type.is_file()) + .unwrap_or(false) + { + continue; + } + let file_name = entry.file_name(); + let file_name = file_name.to_string_lossy(); + if !should_remove_state_file(file_name.as_ref(), current_name.as_str()) { + continue; + } + + let legacy_path = entry.path(); + if let Err(err) = tokio::fs::remove_file(&legacy_path).await { + warn!( + "failed to remove legacy state db file {}: {err}", + legacy_path.display() + ); + } + } +} + +fn should_remove_state_file(file_name: &str, current_name: &str) -> bool { + let mut base_name = file_name; + for suffix in ["-wal", "-shm", "-journal"] { + if let Some(stripped) = file_name.strip_suffix(suffix) { + base_name = stripped; + break; + } + } + if base_name == current_name { + return false; + } + let unversioned_name = format!("{STATE_DB_FILENAME}.sqlite"); + if base_name == unversioned_name { + return true; + } + + let Some(version_with_extension) = base_name.strip_prefix(&format!("{STATE_DB_FILENAME}_")) + else { + return false; + }; + let Some(version_suffix) = version_with_extension.strip_suffix(".sqlite") else { + return false; + }; + !version_suffix.is_empty() && version_suffix.chars().all(|ch| ch.is_ascii_digit()) +} + fn push_thread_filters<'a>( builder: &mut QueryBuilder<'a, Sqlite>, archived_only: bool, @@ -690,3 +864,396 @@ fn push_thread_order_and_limit( builder.push(" LIMIT "); builder.push_bind(limit as i64); } + +#[cfg(test)] +mod tests { + use super::STATE_DB_FILENAME; + use super::STATE_DB_VERSION; + use super::StateRuntime; + use super::ThreadMetadata; + use super::state_db_filename; + use chrono::DateTime; + use chrono::Utc; + use codex_protocol::ThreadId; + use codex_protocol::protocol::AskForApproval; + use codex_protocol::protocol::SandboxPolicy; + use pretty_assertions::assert_eq; + use sqlx::Row; + use std::path::Path; + use std::path::PathBuf; + use std::time::SystemTime; + use std::time::UNIX_EPOCH; + use uuid::Uuid; + + fn unique_temp_dir() -> PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |duration| duration.as_nanos()); + std::env::temp_dir().join(format!("codex-state-runtime-test-{nanos}")) + } + + #[tokio::test] + async fn init_removes_legacy_state_db_files() { + let codex_home = unique_temp_dir(); + tokio::fs::create_dir_all(&codex_home) + .await + .expect("create codex_home"); + + let current_name = state_db_filename(); + let previous_version = STATE_DB_VERSION.saturating_sub(1); + let unversioned_name = format!("{STATE_DB_FILENAME}.sqlite"); + for suffix in ["", "-wal", "-shm", "-journal"] { + let path = codex_home.join(format!("{unversioned_name}{suffix}")); + tokio::fs::write(path, b"legacy") + .await + .expect("write legacy"); + let old_version_path = codex_home.join(format!( + "{STATE_DB_FILENAME}_{previous_version}.sqlite{suffix}" + )); + tokio::fs::write(old_version_path, b"old_version") + .await + .expect("write old version"); + } + let unrelated_path = codex_home.join("state.sqlite_backup"); + tokio::fs::write(&unrelated_path, b"keep") + .await + .expect("write unrelated"); + let numeric_path = codex_home.join("123"); + tokio::fs::write(&numeric_path, b"keep") + .await + .expect("write numeric"); + + let _runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + for suffix in ["", "-wal", "-shm", "-journal"] { + let legacy_path = codex_home.join(format!("{unversioned_name}{suffix}")); + assert_eq!( + tokio::fs::try_exists(&legacy_path) + .await + .expect("check legacy path"), + false + ); + let old_version_path = codex_home.join(format!( + "{STATE_DB_FILENAME}_{previous_version}.sqlite{suffix}" + )); + assert_eq!( + tokio::fs::try_exists(&old_version_path) + .await + .expect("check old version path"), + false + ); + } + assert_eq!( + tokio::fs::try_exists(codex_home.join(current_name)) + .await + .expect("check new db path"), + true + ); + assert_eq!( + tokio::fs::try_exists(&unrelated_path) + .await + .expect("check unrelated path"), + true + ); + assert_eq!( + tokio::fs::try_exists(&numeric_path) + .await + .expect("check numeric path"), + true + ); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn upsert_and_get_thread_memory() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let thread_id = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let metadata = test_thread_metadata(&codex_home, thread_id, codex_home.join("a")); + runtime + .upsert_thread(&metadata) + .await + .expect("upsert thread"); + + assert_eq!( + runtime + .get_thread_memory(thread_id) + .await + .expect("get memory before insert"), + None + ); + + let inserted = runtime + .upsert_thread_memory(thread_id, "trace one", "memory one") + .await + .expect("upsert memory"); + assert_eq!(inserted.thread_id, thread_id); + assert_eq!(inserted.trace_summary, "trace one"); + assert_eq!(inserted.memory_summary, "memory one"); + + let updated = runtime + .upsert_thread_memory(thread_id, "trace two", "memory two") + .await + .expect("update memory"); + assert_eq!(updated.thread_id, thread_id); + assert_eq!(updated.trace_summary, "trace two"); + assert_eq!(updated.memory_summary, "memory two"); + assert!( + updated.updated_at >= inserted.updated_at, + "updated_at should not move backward" + ); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn get_last_n_thread_memories_for_cwd_matches_exactly() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let cwd_a = codex_home.join("workspace-a"); + let cwd_b = codex_home.join("workspace-b"); + let t1 = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let t2 = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let t3 = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + runtime + .upsert_thread(&test_thread_metadata(&codex_home, t1, cwd_a.clone())) + .await + .expect("upsert thread t1"); + runtime + .upsert_thread(&test_thread_metadata(&codex_home, t2, cwd_a.clone())) + .await + .expect("upsert thread t2"); + runtime + .upsert_thread(&test_thread_metadata(&codex_home, t3, cwd_b.clone())) + .await + .expect("upsert thread t3"); + + let first = runtime + .upsert_thread_memory(t1, "trace-1", "memory-1") + .await + .expect("upsert t1 memory"); + runtime + .upsert_thread_memory(t2, "trace-2", "memory-2") + .await + .expect("upsert t2 memory"); + runtime + .upsert_thread_memory(t3, "trace-3", "memory-3") + .await + .expect("upsert t3 memory"); + // Ensure deterministic ordering even when updates happen in the same second. + runtime + .upsert_thread_memory(t1, "trace-1b", "memory-1b") + .await + .expect("upsert t1 memory again"); + + let cwd_a_memories = runtime + .get_last_n_thread_memories_for_cwd(cwd_a.as_path(), 2) + .await + .expect("list cwd a memories"); + assert_eq!(cwd_a_memories.len(), 2); + assert_eq!(cwd_a_memories[0].thread_id, t1); + assert_eq!(cwd_a_memories[0].trace_summary, "trace-1b"); + assert_eq!(cwd_a_memories[0].memory_summary, "memory-1b"); + assert_eq!(cwd_a_memories[1].thread_id, t2); + assert!(cwd_a_memories[0].updated_at >= first.updated_at); + + let cwd_b_memories = runtime + .get_last_n_thread_memories_for_cwd(cwd_b.as_path(), 10) + .await + .expect("list cwd b memories"); + assert_eq!(cwd_b_memories.len(), 1); + assert_eq!(cwd_b_memories[0].thread_id, t3); + + let none = runtime + .get_last_n_thread_memories_for_cwd(codex_home.join("missing").as_path(), 10) + .await + .expect("list missing cwd memories"); + assert_eq!(none, Vec::new()); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn upsert_thread_memory_errors_for_unknown_thread() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let unknown_thread_id = + ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let err = runtime + .upsert_thread_memory(unknown_thread_id, "trace", "memory") + .await + .expect_err("unknown thread should fail"); + assert!( + err.to_string().contains("thread not found"), + "error should mention missing thread: {err}" + ); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn get_last_n_thread_memories_for_cwd_zero_returns_empty() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let thread_id = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let cwd = codex_home.join("workspace"); + runtime + .upsert_thread(&test_thread_metadata(&codex_home, thread_id, cwd.clone())) + .await + .expect("upsert thread"); + runtime + .upsert_thread_memory(thread_id, "trace", "memory") + .await + .expect("upsert memory"); + + let memories = runtime + .get_last_n_thread_memories_for_cwd(cwd.as_path(), 0) + .await + .expect("query memories"); + assert_eq!(memories, Vec::new()); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn get_last_n_thread_memories_for_cwd_does_not_prefix_match() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let cwd_exact = codex_home.join("workspace"); + let cwd_prefix = codex_home.join("workspace-child"); + let t_exact = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let t_prefix = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + runtime + .upsert_thread(&test_thread_metadata( + &codex_home, + t_exact, + cwd_exact.clone(), + )) + .await + .expect("upsert exact thread"); + runtime + .upsert_thread(&test_thread_metadata( + &codex_home, + t_prefix, + cwd_prefix.clone(), + )) + .await + .expect("upsert prefix thread"); + runtime + .upsert_thread_memory(t_exact, "trace-exact", "memory-exact") + .await + .expect("upsert exact memory"); + runtime + .upsert_thread_memory(t_prefix, "trace-prefix", "memory-prefix") + .await + .expect("upsert prefix memory"); + + let exact_only = runtime + .get_last_n_thread_memories_for_cwd(cwd_exact.as_path(), 10) + .await + .expect("query exact cwd"); + assert_eq!(exact_only.len(), 1); + assert_eq!(exact_only[0].thread_id, t_exact); + assert_eq!(exact_only[0].memory_summary, "memory-exact"); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + #[tokio::test] + async fn deleting_thread_cascades_thread_memory() { + let codex_home = unique_temp_dir(); + let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string(), None) + .await + .expect("initialize runtime"); + + let thread_id = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("thread id"); + let cwd = codex_home.join("workspace"); + runtime + .upsert_thread(&test_thread_metadata(&codex_home, thread_id, cwd)) + .await + .expect("upsert thread"); + runtime + .upsert_thread_memory(thread_id, "trace", "memory") + .await + .expect("upsert memory"); + + let count_before = + sqlx::query("SELECT COUNT(*) AS count FROM thread_memory WHERE thread_id = ?") + .bind(thread_id.to_string()) + .fetch_one(runtime.pool.as_ref()) + .await + .expect("count before delete") + .try_get::("count") + .expect("count value"); + assert_eq!(count_before, 1); + + sqlx::query("DELETE FROM threads WHERE id = ?") + .bind(thread_id.to_string()) + .execute(runtime.pool.as_ref()) + .await + .expect("delete thread"); + + let count_after = + sqlx::query("SELECT COUNT(*) AS count FROM thread_memory WHERE thread_id = ?") + .bind(thread_id.to_string()) + .fetch_one(runtime.pool.as_ref()) + .await + .expect("count after delete") + .try_get::("count") + .expect("count value"); + assert_eq!(count_after, 0); + assert_eq!( + runtime + .get_thread_memory(thread_id) + .await + .expect("get memory after delete"), + None + ); + + let _ = tokio::fs::remove_dir_all(codex_home).await; + } + + fn test_thread_metadata( + codex_home: &Path, + thread_id: ThreadId, + cwd: PathBuf, + ) -> ThreadMetadata { + let now = DateTime::::from_timestamp(1_700_000_000, 0).expect("timestamp"); + ThreadMetadata { + id: thread_id, + rollout_path: codex_home.join(format!("rollout-{thread_id}.jsonl")), + created_at: now, + updated_at: now, + source: "cli".to_string(), + model_provider: "test-provider".to_string(), + cwd, + title: String::new(), + sandbox_policy: crate::extract::enum_to_string(&SandboxPolicy::ReadOnly), + approval_mode: crate::extract::enum_to_string(&AskForApproval::OnRequest), + tokens_used: 0, + has_user_event: true, + archived_at: None, + git_sha: None, + git_branch: None, + git_origin_url: None, + } + } +} diff --git a/codex-rs/tui/src/app.rs b/codex-rs/tui/src/app.rs index 0160e8f23d..99164efa61 100644 --- a/codex-rs/tui/src/app.rs +++ b/codex-rs/tui/src/app.rs @@ -102,6 +102,11 @@ use toml::Value as TomlValue; const EXTERNAL_EDITOR_HINT: &str = "Save and close external editor to continue."; const THREAD_EVENT_CHANNEL_CAPACITY: usize = 32768; +/// Baseline cadence for periodic stream commit animation ticks. +/// +/// Smooth-mode streaming drains one line per tick, so this interval controls +/// perceived typing speed for non-backlogged output. +const COMMIT_ANIMATION_TICK: Duration = tui::TARGET_FRAME_INTERVAL; #[derive(Debug, Clone)] pub struct AppExitInfo { @@ -1485,7 +1490,7 @@ impl App { let running = self.commit_anim_running.clone(); thread::spawn(move || { while running.load(Ordering::Relaxed) { - thread::sleep(Duration::from_millis(50)); + thread::sleep(COMMIT_ANIMATION_TICK); tx.send(AppEvent::CommitTick); } }); @@ -2321,6 +2326,7 @@ impl App { fn personality_label(personality: Personality) -> &'static str { match personality { + Personality::None => "None", Personality::Friendly => "Friendly", Personality::Pragmatic => "Pragmatic", } diff --git a/codex-rs/tui/src/bottom_pane/list_selection_view.rs b/codex-rs/tui/src/bottom_pane/list_selection_view.rs index 095a095500..9bc046f010 100644 --- a/codex-rs/tui/src/bottom_pane/list_selection_view.rs +++ b/codex-rs/tui/src/bottom_pane/list_selection_view.rs @@ -23,14 +23,25 @@ use super::CancellationEvent; use super::bottom_pane_view::BottomPaneView; use super::popup_consts::MAX_POPUP_ROWS; use super::scroll_state::ScrollState; +pub(crate) use super::selection_popup_common::ColumnWidthMode; use super::selection_popup_common::GenericDisplayRow; use super::selection_popup_common::measure_rows_height; +use super::selection_popup_common::measure_rows_height_stable_col_widths; +use super::selection_popup_common::measure_rows_height_with_col_width_mode; use super::selection_popup_common::render_rows; +use super::selection_popup_common::render_rows_stable_col_widths; +use super::selection_popup_common::render_rows_with_col_width_mode; use unicode_width::UnicodeWidthStr; /// One selectable item in the generic selection list. pub(crate) type SelectionAction = Box; +/// One row in a [`ListSelectionView`] selection list. +/// +/// This is the source-of-truth model for row state before filtering and +/// formatting into render rows. A row is treated as disabled when either +/// `is_disabled` is true or `disabled_reason` is present; disabled rows cannot +/// be accepted and are skipped by keyboard navigation. #[derive(Default)] pub(crate) struct SelectionItem { pub name: String, @@ -46,6 +57,16 @@ pub(crate) struct SelectionItem { pub disabled_reason: Option, } +/// Construction-time configuration for [`ListSelectionView`]. +/// +/// This config is consumed once by [`ListSelectionView::new`]. After +/// construction, mutable interaction state (filtering, scrolling, and selected +/// row) lives on the view itself. +/// +/// `col_width_mode` controls column width mode in selection lists: +/// `AutoVisible` (default) measures only rows visible in the viewport +/// `AutoAllRows` measures all rows to ensure stable column widths as the user scrolls +/// `Fixed` used a fixed 30/70 split between columns pub(crate) struct SelectionViewParams { pub title: Option, pub subtitle: Option, @@ -54,6 +75,7 @@ pub(crate) struct SelectionViewParams { pub items: Vec, pub is_searchable: bool, pub search_placeholder: Option, + pub col_width_mode: ColumnWidthMode, pub header: Box, pub initial_selected_idx: Option, } @@ -68,12 +90,18 @@ impl Default for SelectionViewParams { items: Vec::new(), is_searchable: false, search_placeholder: None, + col_width_mode: ColumnWidthMode::AutoVisible, header: Box::new(()), initial_selected_idx: None, } } } +/// Runtime state for rendering and interacting with a list-based selection popup. +/// +/// This type is the single authority for filtered index mapping between +/// visible rows and source items and for preserving selection while filters +/// change. pub(crate) struct ListSelectionView { footer_note: Option>, footer_hint: Option>, @@ -84,6 +112,7 @@ pub(crate) struct ListSelectionView { is_searchable: bool, search_query: String, search_placeholder: Option, + col_width_mode: ColumnWidthMode, filtered_indices: Vec, last_selected_actual_idx: Option, header: Box, @@ -91,6 +120,13 @@ pub(crate) struct ListSelectionView { } impl ListSelectionView { + /// Create a selection popup view with filtering, scrolling, and callbacks wired. + /// + /// The constructor normalizes header/title composition and immediately + /// applies filtering so `ScrollState` starts in a valid visible range. + /// When search is enabled, rows without `search_value` will disappear as + /// soon as the query is non-empty, which can look like dropped data unless + /// callers intentionally populate that field. pub fn new(params: SelectionViewParams, app_event_tx: AppEventSender) -> Self { let mut header = params.header; if params.title.is_some() || params.subtitle.is_some() { @@ -116,6 +152,7 @@ impl ListSelectionView { } else { None }, + col_width_mode: params.col_width_mode, filtered_indices: Vec::new(), last_selected_actual_idx: None, header, @@ -434,12 +471,27 @@ impl Renderable for ListSelectionView { // Build the same display rows used by the renderer so wrapping math matches. let rows = self.build_rows(); let rows_width = Self::rows_width(width); - let rows_height = measure_rows_height( - &rows, - &self.state, - MAX_POPUP_ROWS, - rows_width.saturating_add(1), - ); + let rows_height = match self.col_width_mode { + ColumnWidthMode::AutoVisible => measure_rows_height( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ), + ColumnWidthMode::AutoAllRows => measure_rows_height_stable_col_widths( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ), + ColumnWidthMode::Fixed => measure_rows_height_with_col_width_mode( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ColumnWidthMode::Fixed, + ), + }; // Subtract 4 for the padding on the left and right of the header. let mut height = self.header.desired_height(width.saturating_sub(4)); @@ -483,12 +535,27 @@ impl Renderable for ListSelectionView { .desired_height(outer_content_area.width.saturating_sub(4)); let rows = self.build_rows(); let rows_width = Self::rows_width(outer_content_area.width); - let rows_height = measure_rows_height( - &rows, - &self.state, - MAX_POPUP_ROWS, - rows_width.saturating_add(1), - ); + let rows_height = match self.col_width_mode { + ColumnWidthMode::AutoVisible => measure_rows_height( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ), + ColumnWidthMode::AutoAllRows => measure_rows_height_stable_col_widths( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ), + ColumnWidthMode::Fixed => measure_rows_height_with_col_width_mode( + &rows, + &self.state, + MAX_POPUP_ROWS, + rows_width.saturating_add(1), + ColumnWidthMode::Fixed, + ), + }; let [header_area, _, search_area, list_area] = Layout::vertical([ Constraint::Max(header_height), Constraint::Max(1), @@ -529,14 +596,33 @@ impl Renderable for ListSelectionView { width: rows_width.max(1), height: list_area.height, }; - render_rows( - render_area, - buf, - &rows, - &self.state, - render_area.height as usize, - "no matches", - ); + match self.col_width_mode { + ColumnWidthMode::AutoVisible => render_rows( + render_area, + buf, + &rows, + &self.state, + render_area.height as usize, + "no matches", + ), + ColumnWidthMode::AutoAllRows => render_rows_stable_col_widths( + render_area, + buf, + &rows, + &self.state, + render_area.height as usize, + "no matches", + ), + ColumnWidthMode::Fixed => render_rows_with_col_width_mode( + render_area, + buf, + &rows, + &self.state, + render_area.height as usize, + "no matches", + ColumnWidthMode::Fixed, + ), + } } if footer_area.height > 0 { @@ -585,7 +671,9 @@ mod tests { use super::*; use crate::app_event::AppEvent; use crate::bottom_pane::popup_consts::standard_popup_hint_line; + use crossterm::event::KeyCode; use insta::assert_snapshot; + use pretty_assertions::assert_eq; use ratatui::layout::Rect; use tokio::sync::mpsc::unbounded_channel; @@ -647,6 +735,55 @@ mod tests { lines.join("\n") } + fn description_col(rendered: &str, item_marker: &str, description: &str) -> usize { + let line = rendered + .lines() + .find(|line| line.contains(item_marker) && line.contains(description)) + .expect("expected rendered line to contain row marker and description"); + line.find(description) + .expect("expected rendered line to contain description") + } + + fn make_scrolling_width_items() -> Vec { + let mut items: Vec = (1..=8) + .map(|idx| SelectionItem { + name: format!("Item {idx}"), + description: Some(format!("desc {idx}")), + dismiss_on_select: true, + ..Default::default() + }) + .collect(); + items.push(SelectionItem { + name: "Item 9 with an intentionally much longer name".to_string(), + description: Some("desc 9".to_string()), + dismiss_on_select: true, + ..Default::default() + }); + items + } + + fn render_before_after_scroll_snapshot(col_width_mode: ColumnWidthMode, width: u16) -> String { + let (tx_raw, _rx) = unbounded_channel::(); + let tx = AppEventSender::new(tx_raw); + let mut view = ListSelectionView::new( + SelectionViewParams { + title: Some("Debug".to_string()), + items: make_scrolling_width_items(), + col_width_mode, + ..Default::default() + }, + tx, + ); + + let before_scroll = render_lines_with_width(&view, width); + for _ in 0..8 { + view.handle_key_event(KeyEvent::from(KeyCode::Down)); + } + let after_scroll = render_lines_with_width(&view, width); + + format!("before scroll:\n{before_scroll}\n\nafter scroll:\n{after_scroll}") + } + #[test] fn renders_blank_line_between_title_and_items_without_subtitle() { let view = make_selection_view(None); @@ -921,4 +1058,96 @@ mod tests { render_lines_with_width(&view, 24) ); } + + #[test] + fn snapshot_auto_visible_col_width_mode_scroll_behavior() { + assert_snapshot!( + "list_selection_col_width_mode_auto_visible_scroll", + render_before_after_scroll_snapshot(ColumnWidthMode::AutoVisible, 96) + ); + } + + #[test] + fn snapshot_auto_all_rows_col_width_mode_scroll_behavior() { + assert_snapshot!( + "list_selection_col_width_mode_auto_all_rows_scroll", + render_before_after_scroll_snapshot(ColumnWidthMode::AutoAllRows, 96) + ); + } + + #[test] + fn snapshot_fixed_col_width_mode_scroll_behavior() { + assert_snapshot!( + "list_selection_col_width_mode_fixed_scroll", + render_before_after_scroll_snapshot(ColumnWidthMode::Fixed, 96) + ); + } + + #[test] + fn auto_all_rows_col_width_does_not_shift_when_scrolling() { + let (tx_raw, _rx) = unbounded_channel::(); + let tx = AppEventSender::new(tx_raw); + + let mut view = ListSelectionView::new( + SelectionViewParams { + title: Some("Debug".to_string()), + items: make_scrolling_width_items(), + col_width_mode: ColumnWidthMode::AutoAllRows, + ..Default::default() + }, + tx, + ); + + let before_scroll = render_lines_with_width(&view, 96); + for _ in 0..8 { + view.handle_key_event(KeyEvent::from(KeyCode::Down)); + } + let after_scroll = render_lines_with_width(&view, 96); + + assert!( + after_scroll.contains("9. Item 9 with an intentionally much longer name"), + "expected the scrolled view to include the longer row:\n{after_scroll}" + ); + + let before_col = description_col(&before_scroll, "8. Item 8", "desc 8"); + let after_col = description_col(&after_scroll, "8. Item 8", "desc 8"); + assert_eq!( + before_col, after_col, + "description column changed across scroll:\nbefore:\n{before_scroll}\nafter:\n{after_scroll}" + ); + } + + #[test] + fn fixed_col_width_is_30_70_and_does_not_shift_when_scrolling() { + let (tx_raw, _rx) = unbounded_channel::(); + let tx = AppEventSender::new(tx_raw); + let width = 96; + let mut view = ListSelectionView::new( + SelectionViewParams { + title: Some("Debug".to_string()), + items: make_scrolling_width_items(), + col_width_mode: ColumnWidthMode::Fixed, + ..Default::default() + }, + tx, + ); + + let before_scroll = render_lines_with_width(&view, width); + let before_col = description_col(&before_scroll, "8. Item 8", "desc 8"); + let expected_desc_col = ((width.saturating_sub(2) as usize) * 3) / 10; + assert_eq!( + before_col, expected_desc_col, + "fixed mode should place description column at a 30/70 split:\n{before_scroll}" + ); + + for _ in 0..8 { + view.handle_key_event(KeyEvent::from(KeyCode::Down)); + } + let after_scroll = render_lines_with_width(&view, width); + let after_col = description_col(&after_scroll, "8. Item 8", "desc 8"); + assert_eq!( + before_col, after_col, + "fixed description column changed across scroll:\nbefore:\n{before_scroll}\nafter:\n{after_scroll}" + ); + } } diff --git a/codex-rs/tui/src/bottom_pane/mod.rs b/codex-rs/tui/src/bottom_pane/mod.rs index cd5e2adb80..b251beb80e 100644 --- a/codex-rs/tui/src/bottom_pane/mod.rs +++ b/codex-rs/tui/src/bottom_pane/mod.rs @@ -65,6 +65,7 @@ mod skill_popup; mod skills_toggle_view; mod slash_commands; pub(crate) use footer::CollaborationModeIndicator; +pub(crate) use list_selection_view::ColumnWidthMode; pub(crate) use list_selection_view::SelectionViewParams; mod feedback_view; pub(crate) use feedback_view::FeedbackAudience; diff --git a/codex-rs/tui/src/bottom_pane/request_user_input/mod.rs b/codex-rs/tui/src/bottom_pane/request_user_input/mod.rs index 59807a8a4c..9542772446 100644 --- a/codex-rs/tui/src/bottom_pane/request_user_input/mod.rs +++ b/codex-rs/tui/src/bottom_pane/request_user_input/mod.rs @@ -427,8 +427,8 @@ impl RequestUserInputOverlay { if self.selected_option_index().is_some() && !notes_visible { tips.push(FooterTip::highlighted("tab to add notes")); } - if self.selected_option_index().is_some() && notes_visible && self.focus_is_notes() { - tips.push(FooterTip::new("tab to clear notes")); + if self.selected_option_index().is_some() && notes_visible { + tips.push(FooterTip::new("tab or esc to clear notes")); } } @@ -449,7 +449,9 @@ impl RequestUserInputOverlay { tips.push(FooterTip::new("ctrl + n next question")); } } - tips.push(FooterTip::new("esc to interrupt")); + if !(self.has_options() && notes_visible) { + tips.push(FooterTip::new("esc to interrupt")); + } tips } @@ -663,6 +665,23 @@ impl RequestUserInputOverlay { self.sync_composer_placeholder(); } + fn clear_notes_and_focus_options(&mut self) { + if !self.has_options() { + return; + } + if let Some(answer) = self.current_answer_mut() { + answer.draft = ComposerDraft::default(); + answer.answer_committed = false; + answer.notes_visible = false; + } + self.pending_submission_draft = None; + self.composer + .set_text_content(String::new(), Vec::new(), Vec::new()); + self.composer.move_cursor_to_end(); + self.focus = Focus::Options; + self.sync_composer_placeholder(); + } + /// Ensure there is a selection before allowing notes entry. fn ensure_selected_for_notes(&mut self) { if let Some(answer) = self.current_answer_mut() { @@ -976,6 +995,10 @@ impl BottomPaneView for RequestUserInputOverlay { } if matches!(key_event.code, KeyCode::Esc) { + if self.has_options() && self.notes_ui_visible() { + self.clear_notes_and_focus_options(); + return; + } // TODO: Emit interrupted request_user_input results (including committed answers) // once core supports persisting them reliably without follow-up turn issues. self.app_event_tx.send(AppEvent::CodexOp(Op::Interrupt)); @@ -1093,16 +1116,7 @@ impl BottomPaneView for RequestUserInputOverlay { Focus::Notes => { let notes_empty = self.composer.current_text_with_pending().trim().is_empty(); if self.has_options() && matches!(key_event.code, KeyCode::Tab) { - if let Some(answer) = self.current_answer_mut() { - answer.draft = ComposerDraft::default(); - answer.answer_committed = false; - answer.notes_visible = false; - } - self.composer - .set_text_content(String::new(), Vec::new(), Vec::new()); - self.composer.move_cursor_to_end(); - self.focus = Focus::Options; - self.sync_composer_placeholder(); + self.clear_notes_and_focus_options(); return; } if self.has_options() && matches!(key_event.code, KeyCode::Backspace) && notes_empty @@ -1753,7 +1767,7 @@ mod tests { } #[test] - fn esc_in_notes_mode_interrupts() { + fn esc_in_notes_mode_clears_notes_and_hides_ui() { let (tx, mut rx) = test_sender(); let mut overlay = RequestUserInputOverlay::new( request_event("turn-1", vec![question_with_options("q1", "Pick one")]), @@ -1769,12 +1783,19 @@ mod tests { overlay.handle_key_event(KeyEvent::from(KeyCode::Tab)); overlay.handle_key_event(KeyEvent::from(KeyCode::Esc)); - assert_eq!(overlay.done, true); - expect_interrupt_only(&mut rx); + let answer = overlay.current_answer().expect("answer missing"); + assert_eq!(overlay.done, false); + assert!(matches!(overlay.focus, Focus::Options)); + assert_eq!(overlay.notes_ui_visible(), false); + assert_eq!(overlay.composer.current_text_with_pending(), ""); + assert_eq!(answer.draft.text, ""); + assert_eq!(answer.options_state.selected_idx, Some(0)); + assert_eq!(answer.answer_committed, false); + assert!(rx.try_recv().is_err()); } #[test] - fn esc_in_notes_mode_interrupts_with_notes_visible() { + fn esc_in_notes_mode_with_text_clears_notes_and_hides_ui() { let (tx, mut rx) = test_sender(); let mut overlay = RequestUserInputOverlay::new( request_event("turn-1", vec![question_with_options("q1", "Pick one")]), @@ -1791,8 +1812,15 @@ mod tests { overlay.handle_key_event(KeyEvent::from(KeyCode::Char('a'))); overlay.handle_key_event(KeyEvent::from(KeyCode::Esc)); - assert_eq!(overlay.done, true); - expect_interrupt_only(&mut rx); + let answer = overlay.current_answer().expect("answer missing"); + assert_eq!(overlay.done, false); + assert!(matches!(overlay.focus, Focus::Options)); + assert_eq!(overlay.notes_ui_visible(), false); + assert_eq!(overlay.composer.current_text_with_pending(), ""); + assert_eq!(answer.draft.text, ""); + assert_eq!(answer.options_state.selected_idx, Some(0)); + assert_eq!(answer.answer_committed, false); + assert!(rx.try_recv().is_err()); } #[test] diff --git a/codex-rs/tui/src/bottom_pane/request_user_input/snapshots/codex_tui__bottom_pane__request_user_input__tests__request_user_input_options_notes_visible.snap b/codex-rs/tui/src/bottom_pane/request_user_input/snapshots/codex_tui__bottom_pane__request_user_input__tests__request_user_input_options_notes_visible.snap index d9d219b627..a4540a2b26 100644 --- a/codex-rs/tui/src/bottom_pane/request_user_input/snapshots/codex_tui__bottom_pane__request_user_input__tests__request_user_input_options_notes_visible.snap +++ b/codex-rs/tui/src/bottom_pane/request_user_input/snapshots/codex_tui__bottom_pane__request_user_input__tests__request_user_input_options_notes_visible.snap @@ -1,5 +1,6 @@ --- source: tui/src/bottom_pane/request_user_input/mod.rs +assertion_line: 2321 expression: "render_snapshot(&overlay, area)" --- @@ -16,4 +17,4 @@ expression: "render_snapshot(&overlay, area)" - tab to clear notes | enter to submit answer | esc to interrupt + tab or esc to clear notes | enter to submit answer diff --git a/codex-rs/tui/src/bottom_pane/selection_popup_common.rs b/codex-rs/tui/src/bottom_pane/selection_popup_common.rs index 89ef8b3b3f..f8e1ad8902 100644 --- a/codex-rs/tui/src/bottom_pane/selection_popup_common.rs +++ b/codex-rs/tui/src/bottom_pane/selection_popup_common.rs @@ -19,7 +19,11 @@ use crate::style::user_message_style; use super::scroll_state::ScrollState; -/// A generic representation of a display row for selection popups. +/// Render-ready representation of one row in a selection popup. +/// +/// This type contains presentation-focused fields that are intentionally more +/// concrete than source domain models. `match_indices` are character offsets +/// into `name`, and `wrap_indent` is interpreted in terminal cell columns. #[derive(Default)] pub(crate) struct GenericDisplayRow { pub name: String, @@ -31,6 +35,25 @@ pub(crate) struct GenericDisplayRow { pub wrap_indent: Option, // optional indent for wrapped lines } +/// Controls how selection rows choose the split between left/right name/description columns. +/// +/// Callers should use the same mode for both measurement and rendering, or the +/// popup can reserve the wrong number of lines and clip content. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +#[cfg_attr(not(test), allow(dead_code))] +pub(crate) enum ColumnWidthMode { + /// Derive column placement from only the visible viewport rows. + #[default] + AutoVisible, + /// Derive column placement from all rows so scrolling does not shift columns. + AutoAllRows, + /// Use a fixed two-column split: 30% left (name), 70% right (description). + Fixed, +} + +const FIXED_LEFT_COLUMN_NUMERATOR: usize = 3; +const FIXED_LEFT_COLUMN_DENOMINATOR: usize = 10; + const MENU_SURFACE_INSET_V: u16 = 1; const MENU_SURFACE_INSET_H: u16 = 2; @@ -50,7 +73,8 @@ pub(crate) const fn menu_surface_padding_height() -> u16 { /// Paint the shared menu background and return the inset content area. /// /// This keeps the surface treatment consistent across selection-style overlays -/// (for example `/model`, approvals, and request-user-input). +/// (for example `/model`, approvals, and request-user-input). Callers should +/// render all inner content in the returned rect, not the original area. pub(crate) fn render_menu_surface(area: Rect, buf: &mut Buffer) -> Rect { if area.is_empty() { return area; @@ -61,6 +85,10 @@ pub(crate) fn render_menu_surface(area: Rect, buf: &mut Buffer) -> Rect { menu_surface_inset(area) } +/// Wrap a styled line while preserving span styles. +/// +/// The function clamps `width` to at least one terminal cell so callers can use +/// it safely with narrow layouts. pub(crate) fn wrap_styled_line<'a>(line: &'a Line<'a>, width: u16) -> Vec> { use crate::wrapping::RtOptions; use crate::wrapping::word_wrap_line; @@ -143,34 +171,60 @@ fn truncate_line_with_ellipsis_if_overflow(line: Line<'static>, max_width: usize Line::from(spans) } -/// Compute a shared description-column start based on the widest visible name -/// plus two spaces of padding. Ensures at least one column is left for the -/// description. +/// Computes the shared start column used for descriptions in selection rows. +/// The column is derived from the widest row name plus two spaces of padding +/// while always leaving at least one terminal cell for description content. +/// [`ColumnWidthMode::AutoAllRows`] computes width across the full dataset so +/// the description column does not shift as the user scrolls. fn compute_desc_col( rows_all: &[GenericDisplayRow], start_idx: usize, visible_items: usize, content_width: u16, + col_width_mode: ColumnWidthMode, ) -> usize { - let visible_range = start_idx..(start_idx + visible_items); - let max_name_width = rows_all - .iter() - .enumerate() - .filter(|(i, _)| visible_range.contains(i)) - .map(|(_, r)| { - let mut spans: Vec = vec![r.name.clone().into()]; - if r.disabled_reason.is_some() { - spans.push(" (disabled)".dim()); - } - Line::from(spans).width() - }) - .max() - .unwrap_or(0); - let mut desc_col = max_name_width.saturating_add(2); - if (desc_col as u16) >= content_width { - desc_col = content_width.saturating_sub(1) as usize; + if content_width <= 1 { + return 0; + } + + let max_desc_col = content_width.saturating_sub(1) as usize; + match col_width_mode { + ColumnWidthMode::Fixed => ((content_width as usize * FIXED_LEFT_COLUMN_NUMERATOR) + / FIXED_LEFT_COLUMN_DENOMINATOR) + .clamp(1, max_desc_col), + ColumnWidthMode::AutoVisible | ColumnWidthMode::AutoAllRows => { + let max_name_width = match col_width_mode { + ColumnWidthMode::AutoVisible => rows_all + .iter() + .enumerate() + .skip(start_idx) + .take(visible_items) + .map(|(_, row)| { + let mut spans: Vec = vec![row.name.clone().into()]; + if row.disabled_reason.is_some() { + spans.push(" (disabled)".dim()); + } + Line::from(spans).width() + }) + .max() + .unwrap_or(0), + ColumnWidthMode::AutoAllRows => rows_all + .iter() + .map(|row| { + let mut spans: Vec = vec![row.name.clone().into()]; + if row.disabled_reason.is_some() { + spans.push(" (disabled)".dim()); + } + Line::from(spans).width() + }) + .max() + .unwrap_or(0), + ColumnWidthMode::Fixed => 0, + }; + + max_name_width.saturating_add(2).min(max_desc_col) + } } - desc_col } /// Determine how many spaces to indent wrapped lines for a row. @@ -268,13 +322,14 @@ fn build_full_line(row: &GenericDisplayRow, desc_col: usize) -> Line<'static> { /// Render a list of rows using the provided ScrollState, with shared styling /// and behavior for selection popups. -pub(crate) fn render_rows( +fn render_rows_inner( area: Rect, buf: &mut Buffer, rows_all: &[GenericDisplayRow], state: &ScrollState, max_results: usize, empty_message: &str, + col_width_mode: ColumnWidthMode, ) { if rows_all.is_empty() { if area.height > 0 { @@ -301,7 +356,13 @@ pub(crate) fn render_rows( } } - let desc_col = compute_desc_col(rows_all, start_idx, visible_items, area.width); + let desc_col = compute_desc_col( + rows_all, + start_idx, + visible_items, + area.width, + col_width_mode, + ); // Render items, wrapping descriptions and aligning wrapped lines under the // shared description column. Stop when we run out of vertical space. @@ -358,7 +419,88 @@ pub(crate) fn render_rows( } } +/// Render a list of rows using the provided ScrollState, with shared styling +/// and behavior for selection popups. +/// Description alignment is computed from visible rows only, which allows the +/// layout to adapt tightly to the current viewport. +/// +/// This function should be paired with [`measure_rows_height`] when reserving +/// space; pairing it with a different measurement mode can cause clipping. +pub(crate) fn render_rows( + area: Rect, + buf: &mut Buffer, + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + empty_message: &str, +) { + render_rows_inner( + area, + buf, + rows_all, + state, + max_results, + empty_message, + ColumnWidthMode::AutoVisible, + ); +} + +/// Render a list of rows using the provided ScrollState, with shared styling +/// and behavior for selection popups. +/// This mode keeps column placement stable while scrolling by sizing the +/// description column against the full dataset. +/// +/// This function should be paired with +/// [`measure_rows_height_stable_col_widths`] so reserved and rendered heights +/// stay in sync. +pub(crate) fn render_rows_stable_col_widths( + area: Rect, + buf: &mut Buffer, + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + empty_message: &str, +) { + render_rows_inner( + area, + buf, + rows_all, + state, + max_results, + empty_message, + ColumnWidthMode::AutoAllRows, + ); +} + +/// Render a list of rows using the provided ScrollState and explicit +/// [`ColumnWidthMode`] behavior. +/// +/// This is the low-level entry point for callers that need to thread a mode +/// through higher-level configuration. +pub(crate) fn render_rows_with_col_width_mode( + area: Rect, + buf: &mut Buffer, + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + empty_message: &str, + col_width_mode: ColumnWidthMode, +) { + render_rows_inner( + area, + buf, + rows_all, + state, + max_results, + empty_message, + col_width_mode, + ); +} + /// Render rows as a single line each (no wrapping), truncating overflow with an ellipsis. +/// +/// This path always uses viewport-local width alignment and is best for dense +/// list UIs where multi-line descriptions would add too much vertical churn. pub(crate) fn render_rows_single_line( area: Rect, buf: &mut Buffer, @@ -390,7 +532,13 @@ pub(crate) fn render_rows_single_line( } } - let desc_col = compute_desc_col(rows_all, start_idx, visible_items, area.width); + let desc_col = compute_desc_col( + rows_all, + start_idx, + visible_items, + area.width, + ColumnWidthMode::AutoVisible, + ); let mut cur_y = area.y; for (i, row) in rows_all @@ -433,11 +581,62 @@ pub(crate) fn render_rows_single_line( /// items from `rows_all` given the current scroll/selection state and the /// available `width`. Accounts for description wrapping and alignment so the /// caller can allocate sufficient vertical space. +/// +/// This function matches [`render_rows`] semantics (`AutoVisible` column +/// sizing). Mixing it with stable or fixed render modes can under- or +/// over-estimate required height. pub(crate) fn measure_rows_height( rows_all: &[GenericDisplayRow], state: &ScrollState, max_results: usize, width: u16, +) -> u16 { + measure_rows_height_inner( + rows_all, + state, + max_results, + width, + ColumnWidthMode::AutoVisible, + ) +} + +/// Measures selection-row height while using full-dataset column alignment. +/// This should be paired with [`render_rows_stable_col_widths`] so layout +/// reservation matches rendering behavior. +pub(crate) fn measure_rows_height_stable_col_widths( + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + width: u16, +) -> u16 { + measure_rows_height_inner( + rows_all, + state, + max_results, + width, + ColumnWidthMode::AutoAllRows, + ) +} + +/// Measure selection-row height using explicit [`ColumnWidthMode`] behavior. +/// +/// This is the low-level companion to [`render_rows_with_col_width_mode`]. +pub(crate) fn measure_rows_height_with_col_width_mode( + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + width: u16, + col_width_mode: ColumnWidthMode, +) -> u16 { + measure_rows_height_inner(rows_all, state, max_results, width, col_width_mode) +} + +fn measure_rows_height_inner( + rows_all: &[GenericDisplayRow], + state: &ScrollState, + max_results: usize, + width: u16, + col_width_mode: ColumnWidthMode, ) -> u16 { if rows_all.is_empty() { return 1; // placeholder "no matches" line @@ -458,7 +657,13 @@ pub(crate) fn measure_rows_height( } } - let desc_col = compute_desc_col(rows_all, start_idx, visible_items, content_width); + let desc_col = compute_desc_col( + rows_all, + start_idx, + visible_items, + content_width, + col_width_mode, + ); use crate::wrapping::RtOptions; use crate::wrapping::word_wrap_line; diff --git a/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_all_rows_scroll.snap b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_all_rows_scroll.snap new file mode 100644 index 0000000000..6aaf439a9f --- /dev/null +++ b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_all_rows_scroll.snap @@ -0,0 +1,31 @@ +--- +source: tui/src/bottom_pane/list_selection_view.rs +assertion_line: 1054 +expression: "render_before_after_scroll_snapshot(ColumnWidthMode::AutoAllRows, 96)" +--- +before scroll: + + Debug + +β€Ί 1. Item 1 desc 1 + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 + + +after scroll: + + Debug + + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 +β€Ί 9. Item 9 with an intentionally much longer name desc 9 diff --git a/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_visible_scroll.snap b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_visible_scroll.snap new file mode 100644 index 0000000000..6875fb5433 --- /dev/null +++ b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_auto_visible_scroll.snap @@ -0,0 +1,31 @@ +--- +source: tui/src/bottom_pane/list_selection_view.rs +assertion_line: 1046 +expression: "render_before_after_scroll_snapshot(ColumnWidthMode::AutoVisible, 96)" +--- +before scroll: + + Debug + +β€Ί 1. Item 1 desc 1 + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 + + +after scroll: + + Debug + + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 +β€Ί 9. Item 9 with an intentionally much longer name desc 9 diff --git a/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_fixed_scroll.snap b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_fixed_scroll.snap new file mode 100644 index 0000000000..4672ab7f27 --- /dev/null +++ b/codex-rs/tui/src/bottom_pane/snapshots/codex_tui__bottom_pane__list_selection_view__tests__list_selection_col_width_mode_fixed_scroll.snap @@ -0,0 +1,31 @@ +--- +source: tui/src/bottom_pane/list_selection_view.rs +assertion_line: 1062 +expression: "render_before_after_scroll_snapshot(ColumnWidthMode::Fixed, 96)" +--- +before scroll: + + Debug + +β€Ί 1. Item 1 desc 1 + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 + + +after scroll: + + Debug + + 2. Item 2 desc 2 + 3. Item 3 desc 3 + 4. Item 4 desc 4 + 5. Item 5 desc 5 + 6. Item 6 desc 6 + 7. Item 7 desc 7 + 8. Item 8 desc 8 +β€Ί 9. Item 9 with an intent… desc 9 diff --git a/codex-rs/tui/src/chatwidget.rs b/codex-rs/tui/src/chatwidget.rs index a952b3354c..f29403f68c 100644 --- a/codex-rs/tui/src/chatwidget.rs +++ b/codex-rs/tui/src/chatwidget.rs @@ -144,6 +144,7 @@ use crate::bottom_pane::BottomPane; use crate::bottom_pane::BottomPaneParams; use crate::bottom_pane::CancellationEvent; use crate::bottom_pane::CollaborationModeIndicator; +use crate::bottom_pane::ColumnWidthMode; use crate::bottom_pane::DOUBLE_PRESS_QUIT_SHORTCUT_ENABLED; use crate::bottom_pane::ExperimentalFeatureItem; use crate::bottom_pane::ExperimentalFeaturesView; @@ -196,6 +197,9 @@ mod skills; use self::skills::collect_tool_mentions; use self::skills::find_app_mentions; use self::skills::find_skill_mentions_with_tool_mentions; +use crate::streaming::chunking::AdaptiveChunkingPolicy; +use crate::streaming::commit_tick::CommitTickScope; +use crate::streaming::commit_tick::run_commit_tick; use crate::streaming::controller::PlanStreamController; use crate::streaming::controller::StreamController; @@ -485,6 +489,7 @@ pub(crate) struct ChatWidget { rate_limit_warnings: RateLimitWarningState, rate_limit_switch_prompt: RateLimitSwitchPromptState, rate_limit_poller: Option>, + adaptive_chunking: AdaptiveChunkingPolicy, // Stream lifecycle controller stream_controller: Option, // Stream lifecycle controller for proposed plan output. @@ -769,6 +774,7 @@ impl ChatWidget { { self.add_boxed_history(cell); } + self.adaptive_chunking.reset(); } /// Update the status indicator header and details. @@ -943,6 +949,7 @@ impl ChatWidget { && controller.push(&delta) { self.app_event_tx.send(AppEvent::StartCommitAnimation); + self.run_catch_up_commit_tick(); } self.request_redraw(); } @@ -1020,6 +1027,7 @@ impl ChatWidget { self.saw_plan_item_this_turn = false; self.plan_delta_buffer.clear(); self.plan_item_active = false; + self.adaptive_chunking.reset(); self.plan_stream_controller = None; self.otel_manager.reset_runtime_metrics(); self.bottom_pane.clear_quit_shortcut_hint(); @@ -1286,7 +1294,9 @@ impl ChatWidget { self.last_unified_wait = None; self.unified_exec_wait_streak = None; self.clear_unified_exec_processes(); + self.adaptive_chunking.reset(); self.stream_controller = None; + self.plan_stream_controller = None; self.maybe_show_pending_rate_limit_prompt(); } @@ -1838,30 +1848,41 @@ impl ChatWidget { self.set_status(message, additional_details); } - /// Periodic tick to commit at most one queued line to history with a small delay, - /// animating the output. + /// Periodic tick for stream commits. In smooth mode this preserves one-line pacing, while + /// catch-up mode drains larger batches to reduce queue lag. pub(crate) fn on_commit_tick(&mut self) { - let mut has_controller = false; - let mut all_idle = true; - if let Some(controller) = self.stream_controller.as_mut() { - has_controller = true; - let (cell, is_idle) = controller.on_commit_tick(); - if let Some(cell) = cell { - self.bottom_pane.hide_status_indicator(); - self.add_boxed_history(cell); - } - all_idle &= is_idle; + self.run_commit_tick(); + } + + /// Runs a regular periodic commit tick. + fn run_commit_tick(&mut self) { + self.run_commit_tick_with_scope(CommitTickScope::AnyMode); + } + + /// Runs an opportunistic commit tick only if catch-up mode is active. + fn run_catch_up_commit_tick(&mut self) { + self.run_commit_tick_with_scope(CommitTickScope::CatchUpOnly); + } + + /// Runs a commit tick for the current stream queue snapshot. + /// + /// `scope` controls whether this call may commit in smooth mode or only when catch-up + /// is currently active. + fn run_commit_tick_with_scope(&mut self, scope: CommitTickScope) { + let now = Instant::now(); + let outcome = run_commit_tick( + &mut self.adaptive_chunking, + self.stream_controller.as_mut(), + self.plan_stream_controller.as_mut(), + scope, + now, + ); + for cell in outcome.cells { + self.bottom_pane.hide_status_indicator(); + self.add_boxed_history(cell); } - if let Some(controller) = self.plan_stream_controller.as_mut() { - has_controller = true; - let (cell, is_idle) = controller.on_commit_tick(); - if let Some(cell) = cell { - self.bottom_pane.hide_status_indicator(); - self.add_boxed_history(cell); - } - all_idle &= is_idle; - } - if has_controller && all_idle { + + if outcome.has_controller && outcome.all_idle { self.app_event_tx.send(AppEvent::StopCommitAnimation); } } @@ -1930,6 +1951,7 @@ impl ChatWidget { && controller.push(&delta) { self.app_event_tx.send(AppEvent::StartCommitAnimation); + self.run_catch_up_commit_tick(); } self.request_redraw(); } @@ -2268,6 +2290,7 @@ impl ChatWidget { rate_limit_warnings: RateLimitWarningState::default(), rate_limit_switch_prompt: RateLimitSwitchPromptState::default(), rate_limit_poller: None, + adaptive_chunking: AdaptiveChunkingPolicy::default(), stream_controller: None, plan_stream_controller: None, running_commands: HashMap::new(), @@ -2413,6 +2436,7 @@ impl ChatWidget { rate_limit_warnings: RateLimitWarningState::default(), rate_limit_switch_prompt: RateLimitSwitchPromptState::default(), rate_limit_poller: None, + adaptive_chunking: AdaptiveChunkingPolicy::default(), stream_controller: None, plan_stream_controller: None, running_commands: HashMap::new(), @@ -2547,6 +2571,7 @@ impl ChatWidget { rate_limit_warnings: RateLimitWarningState::default(), rate_limit_switch_prompt: RateLimitSwitchPromptState::default(), rate_limit_poller: None, + adaptive_chunking: AdaptiveChunkingPolicy::default(), stream_controller: None, plan_stream_controller: None, running_commands: HashMap::new(), @@ -2981,6 +3006,9 @@ impl ChatWidget { SlashCommand::Status => { self.add_status_output(); } + SlashCommand::DebugConfig => { + self.add_debug_config_output(); + } SlashCommand::Ps => { self.add_ps_output(); } @@ -3526,6 +3554,7 @@ impl ChatWidget { EventMsg::McpListToolsResponse(ev) => self.on_list_mcp_tools(ev), EventMsg::ListCustomPromptsResponse(ev) => self.on_list_custom_prompts(ev), EventMsg::ListSkillsResponse(ev) => self.on_list_skills(ev), + EventMsg::ListRemoteSkillsResponse(_) | EventMsg::RemoteSkillDownloaded(_) => {} EventMsg::SkillsUpdateAvailable => { self.submit_op(Op::ListSkills { cwds: Vec::new(), @@ -3755,6 +3784,10 @@ impl ChatWidget { )); } + pub(crate) fn add_debug_config_output(&mut self) { + self.add_to_history(crate::debug_config::new_debug_config_output(&self.config)); + } + pub(crate) fn add_ps_output(&mut self) { let processes = self .unified_exec_processes @@ -5472,6 +5505,7 @@ impl ChatWidget { fn personality_label(personality: Personality) -> &'static str { match personality { + Personality::None => "None", Personality::Friendly => "Friendly", Personality::Pragmatic => "Pragmatic", } @@ -5479,6 +5513,7 @@ impl ChatWidget { fn personality_description(personality: Personality) -> &'static str { match personality { + Personality::None => "No personality instructions.", Personality::Friendly => "Warm, collaborative, and helpful.", Personality::Pragmatic => "Concise, task-focused, and direct.", } @@ -5717,6 +5752,7 @@ impl ChatWidget { items, is_searchable: true, search_placeholder: Some("Type to search apps".to_string()), + col_width_mode: ColumnWidthMode::AutoAllRows, ..Default::default() }); } diff --git a/codex-rs/tui/src/chatwidget/tests.rs b/codex-rs/tui/src/chatwidget/tests.rs index d080140a32..dba8bf82e1 100644 --- a/codex-rs/tui/src/chatwidget/tests.rs +++ b/codex-rs/tui/src/chatwidget/tests.rs @@ -852,6 +852,7 @@ async fn make_chatwidget_manual( rate_limit_warnings: RateLimitWarningState::default(), rate_limit_switch_prompt: RateLimitSwitchPromptState::default(), rate_limit_poller: None, + adaptive_chunking: crate::streaming::chunking::AdaptiveChunkingPolicy::default(), stream_controller: None, plan_stream_controller: None, running_commands: HashMap::new(), diff --git a/codex-rs/tui/src/debug_config.rs b/codex-rs/tui/src/debug_config.rs new file mode 100644 index 0000000000..4202264fb2 --- /dev/null +++ b/codex-rs/tui/src/debug_config.rs @@ -0,0 +1,338 @@ +use crate::history_cell::PlainHistoryCell; +use codex_app_server_protocol::ConfigLayerSource; +use codex_core::config::Config; +use codex_core::config_loader::ConfigLayerStack; +use codex_core::config_loader::ConfigLayerStackOrdering; +use codex_core::config_loader::RequirementSource; +use codex_core::config_loader::ResidencyRequirement; +use codex_core::config_loader::SandboxModeRequirement; +use ratatui::style::Stylize; +use ratatui::text::Line; + +pub(crate) fn new_debug_config_output(config: &Config) -> PlainHistoryCell { + PlainHistoryCell::new(render_debug_config_lines(&config.config_layer_stack)) +} + +fn render_debug_config_lines(stack: &ConfigLayerStack) -> Vec> { + let mut lines = vec!["/debug-config".magenta().into(), "".into()]; + + lines.push( + "Config layer stack (lowest precedence first):" + .bold() + .into(), + ); + let layers = stack.get_layers(ConfigLayerStackOrdering::LowestPrecedenceFirst, true); + if layers.is_empty() { + lines.push(" ".dim().into()); + } else { + for (index, layer) in layers.iter().enumerate() { + let source = format_config_layer_source(&layer.name); + let status = if layer.is_disabled() { + "disabled" + } else { + "enabled" + }; + lines.push(format!(" {}. {source} ({status})", index + 1).into()); + if let Some(reason) = &layer.disabled_reason { + lines.push(format!(" reason: {reason}").dim().into()); + } + } + } + + let requirements = stack.requirements(); + let requirements_toml = stack.requirements_toml(); + + lines.push("".into()); + lines.push("Requirements:".bold().into()); + let mut requirement_lines = Vec::new(); + + if let Some(policies) = requirements_toml.allowed_approval_policies.as_ref() { + let value = join_or_empty(policies.iter().map(ToString::to_string).collect::>()); + requirement_lines.push(requirement_line( + "allowed_approval_policies", + value, + requirements.approval_policy.source.as_ref(), + )); + } + + if let Some(modes) = requirements_toml.allowed_sandbox_modes.as_ref() { + let value = join_or_empty( + modes + .iter() + .copied() + .map(format_sandbox_mode_requirement) + .collect::>(), + ); + requirement_lines.push(requirement_line( + "allowed_sandbox_modes", + value, + requirements.sandbox_policy.source.as_ref(), + )); + } + + if let Some(servers) = requirements_toml.mcp_servers.as_ref() { + let value = join_or_empty(servers.keys().cloned().collect::>()); + requirement_lines.push(requirement_line( + "mcp_servers", + value, + requirements + .mcp_servers + .as_ref() + .map(|sourced| &sourced.source), + )); + } + + // TODO(gt): Expand this debug output with detailed skills and rules display. + if requirements_toml.rules.is_some() { + requirement_lines.push(requirement_line( + "rules", + "configured".to_string(), + requirements.exec_policy_source(), + )); + } + + if let Some(residency) = requirements_toml.enforce_residency { + requirement_lines.push(requirement_line( + "enforce_residency", + format_residency_requirement(residency), + requirements.enforce_residency.source.as_ref(), + )); + } + + if requirement_lines.is_empty() { + lines.push(" ".dim().into()); + } else { + lines.extend(requirement_lines); + } + + lines +} + +fn requirement_line( + name: &str, + value: String, + source: Option<&RequirementSource>, +) -> Line<'static> { + let source = source + .map(ToString::to_string) + .unwrap_or_else(|| "".to_string()); + format!(" - {name}: {value} (source: {source})").into() +} + +fn join_or_empty(values: Vec) -> String { + if values.is_empty() { + "".to_string() + } else { + values.join(", ") + } +} + +fn format_config_layer_source(source: &ConfigLayerSource) -> String { + match source { + ConfigLayerSource::Mdm { domain, key } => { + format!("mdm ({domain}:{key})") + } + ConfigLayerSource::System { file } => { + format!("system ({})", file.as_path().display()) + } + ConfigLayerSource::User { file } => { + format!("user ({})", file.as_path().display()) + } + ConfigLayerSource::Project { dot_codex_folder } => { + format!( + "project ({}/config.toml)", + dot_codex_folder.as_path().display() + ) + } + ConfigLayerSource::SessionFlags => "session-flags".to_string(), + ConfigLayerSource::LegacyManagedConfigTomlFromFile { file } => { + format!("legacy managed_config.toml ({})", file.as_path().display()) + } + ConfigLayerSource::LegacyManagedConfigTomlFromMdm => { + "legacy managed_config.toml (mdm)".to_string() + } + } +} + +fn format_sandbox_mode_requirement(mode: SandboxModeRequirement) -> String { + match mode { + SandboxModeRequirement::ReadOnly => "read-only".to_string(), + SandboxModeRequirement::WorkspaceWrite => "workspace-write".to_string(), + SandboxModeRequirement::DangerFullAccess => "danger-full-access".to_string(), + SandboxModeRequirement::ExternalSandbox => "external-sandbox".to_string(), + } +} + +fn format_residency_requirement(requirement: ResidencyRequirement) -> String { + match requirement { + ResidencyRequirement::Us => "us".to_string(), + } +} + +#[cfg(test)] +mod tests { + use super::render_debug_config_lines; + use codex_app_server_protocol::ConfigLayerSource; + use codex_core::config::Constrained; + use codex_core::config_loader::ConfigLayerEntry; + use codex_core::config_loader::ConfigLayerStack; + use codex_core::config_loader::ConfigRequirements; + use codex_core::config_loader::ConfigRequirementsToml; + use codex_core::config_loader::ConstrainedWithSource; + use codex_core::config_loader::McpServerIdentity; + use codex_core::config_loader::McpServerRequirement; + use codex_core::config_loader::RequirementSource; + use codex_core::config_loader::ResidencyRequirement; + use codex_core::config_loader::SandboxModeRequirement; + use codex_core::config_loader::Sourced; + use codex_core::protocol::AskForApproval; + use codex_core::protocol::SandboxPolicy; + use codex_utils_absolute_path::AbsolutePathBuf; + use ratatui::text::Line; + use std::collections::BTreeMap; + use toml::Value as TomlValue; + + fn empty_toml_table() -> TomlValue { + TomlValue::Table(toml::map::Map::new()) + } + + fn absolute_path(path: &str) -> AbsolutePathBuf { + AbsolutePathBuf::from_absolute_path(path).expect("absolute path") + } + + fn render_to_text(lines: &[Line<'static>]) -> String { + lines + .iter() + .map(|line| { + line.spans + .iter() + .map(|span| span.content.as_ref()) + .collect::() + }) + .collect::>() + .join("\n") + } + + #[test] + fn debug_config_output_lists_all_layers_including_disabled() { + let system_file = if cfg!(windows) { + absolute_path("C:\\etc\\codex\\config.toml") + } else { + absolute_path("/etc/codex/config.toml") + }; + let project_folder = if cfg!(windows) { + absolute_path("C:\\repo\\.codex") + } else { + absolute_path("/repo/.codex") + }; + + let layers = vec![ + ConfigLayerEntry::new( + ConfigLayerSource::System { file: system_file }, + empty_toml_table(), + ), + ConfigLayerEntry::new_disabled( + ConfigLayerSource::Project { + dot_codex_folder: project_folder, + }, + empty_toml_table(), + "project is untrusted", + ), + ]; + let stack = ConfigLayerStack::new( + layers, + ConfigRequirements::default(), + ConfigRequirementsToml::default(), + ) + .expect("config layer stack"); + + let rendered = render_to_text(&render_debug_config_lines(&stack)); + assert!(rendered.contains("(enabled)")); + assert!(rendered.contains("(disabled)")); + assert!(rendered.contains("reason: project is untrusted")); + assert!(rendered.contains("Requirements:")); + assert!(rendered.contains(" ")); + } + + #[test] + fn debug_config_output_lists_requirement_sources() { + let requirements_file = if cfg!(windows) { + absolute_path("C:\\etc\\codex\\requirements.toml") + } else { + absolute_path("/etc/codex/requirements.toml") + }; + let mut requirements = ConfigRequirements::default(); + requirements.approval_policy = ConstrainedWithSource::new( + Constrained::allow_any(AskForApproval::OnRequest), + Some(RequirementSource::CloudRequirements), + ); + requirements.sandbox_policy = ConstrainedWithSource::new( + Constrained::allow_any(SandboxPolicy::ReadOnly), + Some(RequirementSource::SystemRequirementsToml { + file: requirements_file.clone(), + }), + ); + requirements.mcp_servers = Some(Sourced::new( + BTreeMap::from([( + "docs".to_string(), + McpServerRequirement { + identity: McpServerIdentity::Command { + command: "codex-mcp".to_string(), + }, + }, + )]), + RequirementSource::LegacyManagedConfigTomlFromMdm, + )); + requirements.enforce_residency = ConstrainedWithSource::new( + Constrained::allow_any(Some(ResidencyRequirement::Us)), + Some(RequirementSource::CloudRequirements), + ); + + let requirements_toml = ConfigRequirementsToml { + allowed_approval_policies: Some(vec![AskForApproval::OnRequest]), + allowed_sandbox_modes: Some(vec![SandboxModeRequirement::ReadOnly]), + mcp_servers: Some(BTreeMap::from([( + "docs".to_string(), + McpServerRequirement { + identity: McpServerIdentity::Command { + command: "codex-mcp".to_string(), + }, + }, + )])), + rules: None, + enforce_residency: Some(ResidencyRequirement::Us), + }; + + let user_file = if cfg!(windows) { + absolute_path("C:\\users\\alice\\.codex\\config.toml") + } else { + absolute_path("/home/alice/.codex/config.toml") + }; + let stack = ConfigLayerStack::new( + vec![ConfigLayerEntry::new( + ConfigLayerSource::User { file: user_file }, + empty_toml_table(), + )], + requirements, + requirements_toml, + ) + .expect("config layer stack"); + + let rendered = render_to_text(&render_debug_config_lines(&stack)); + assert!( + rendered.contains("allowed_approval_policies: on-request (source: cloud requirements)") + ); + assert!( + rendered.contains( + format!( + "allowed_sandbox_modes: read-only (source: {})", + requirements_file.as_path().display() + ) + .as_str(), + ) + ); + assert!(rendered.contains("mcp_servers: docs (source: MDM managed_config.toml (legacy))")); + assert!(rendered.contains("enforce_residency: us (source: cloud requirements)")); + assert!(!rendered.contains(" - rules:")); + } +} diff --git a/codex-rs/tui/src/history_cell.rs b/codex-rs/tui/src/history_cell.rs index 49ccc481bf..b85ca8f6bb 100644 --- a/codex-rs/tui/src/history_cell.rs +++ b/codex-rs/tui/src/history_cell.rs @@ -2213,6 +2213,14 @@ fn runtime_metrics_label(summary: RuntimeMetricsSummary) -> Option { summary.websocket_events.count )); } + if summary.responses_api_overhead_ms > 0 { + let duration = format_duration_ms(summary.responses_api_overhead_ms); + parts.push(format!("Responses API overhead: {duration}")); + } + if summary.responses_api_inference_time_ms > 0 { + let duration = format_duration_ms(summary.responses_api_inference_time_ms); + parts.push(format!("Responses API inference: {duration}")); + } if parts.is_empty() { None } else { @@ -2381,9 +2389,11 @@ mod tests { count: 4, duration_ms: 1_200, }, + responses_api_overhead_ms: 650, + responses_api_inference_time_ms: 1_940, }; let cell = FinalMessageSeparator::new(Some(12), Some(summary)); - let rendered = render_lines(&cell.display_lines(200)); + let rendered = render_lines(&cell.display_lines(300)); assert_eq!(rendered.len(), 1); assert!(!rendered[0].contains("Worked for")); @@ -2392,6 +2402,8 @@ mod tests { assert!(rendered[0].contains("WebSocket: 1 events send (700ms)")); assert!(rendered[0].contains("Streams: 6 events (900ms)")); assert!(rendered[0].contains("4 events received (1.2s)")); + assert!(rendered[0].contains("Responses API overhead: 650ms")); + assert!(rendered[0].contains("Responses API inference: 1.9s")); } #[test] diff --git a/codex-rs/tui/src/lib.rs b/codex-rs/tui/src/lib.rs index 2e5aee5c38..2dbc1fcaf9 100644 --- a/codex-rs/tui/src/lib.rs +++ b/codex-rs/tui/src/lib.rs @@ -67,6 +67,7 @@ mod collaboration_modes; mod color; pub mod custom_terminal; mod cwd_prompt; +mod debug_config; mod diff_render; mod exec_cell; mod exec_command; diff --git a/codex-rs/tui/src/pager_overlay.rs b/codex-rs/tui/src/pager_overlay.rs index 5f3f79e320..58f15fa7f3 100644 --- a/codex-rs/tui/src/pager_overlay.rs +++ b/codex-rs/tui/src/pager_overlay.rs @@ -17,7 +17,6 @@ use std::io::Result; use std::sync::Arc; -use std::time::Duration; use crate::chatwidget::ActiveCellTranscriptKey; use crate::history_cell::HistoryCell; @@ -299,7 +298,7 @@ impl PagerView { } } tui.frame_requester() - .schedule_frame_in(Duration::from_millis(16)); + .schedule_frame_in(crate::tui::TARGET_FRAME_INTERVAL); Ok(()) } diff --git a/codex-rs/tui/src/slash_command.rs b/codex-rs/tui/src/slash_command.rs index a71b1ddbe5..e4ac5dcacf 100644 --- a/codex-rs/tui/src/slash_command.rs +++ b/codex-rs/tui/src/slash_command.rs @@ -33,6 +33,7 @@ pub enum SlashCommand { Diff, Mention, Status, + DebugConfig, Mcp, Apps, Logout, @@ -63,6 +64,7 @@ impl SlashCommand { SlashCommand::Mention => "mention a file", SlashCommand::Skills => "use skills to improve how Codex performs specific tasks", SlashCommand::Status => "show current session configuration and token usage", + SlashCommand::DebugConfig => "show config layers and requirement sources for debugging", SlashCommand::Ps => "list background terminals", SlashCommand::Model => "choose what model and reasoning effort to use", SlashCommand::Personality => "choose a communication style for Codex", @@ -118,6 +120,7 @@ impl SlashCommand { | SlashCommand::Mention | SlashCommand::Skills | SlashCommand::Status + | SlashCommand::DebugConfig | SlashCommand::Ps | SlashCommand::Mcp | SlashCommand::Apps diff --git a/codex-rs/tui/src/streaming/chunking.rs b/codex-rs/tui/src/streaming/chunking.rs new file mode 100644 index 0000000000..b653f51ecb --- /dev/null +++ b/codex-rs/tui/src/streaming/chunking.rs @@ -0,0 +1,439 @@ +//! Adaptive stream chunking policy for commit animation ticks. +//! +//! This policy preserves the baseline user experience while adapting to bursty +//! stream input. In [`ChunkingMode::Smooth`], one queued line is drained per +//! baseline commit tick. When queue pressure rises, it switches to +//! [`ChunkingMode::CatchUp`] and drains queued backlog immediately so display +//! lag converges as quickly as possible. +//! +//! The policy is source-agnostic: it depends only on queue depth and queue +//! age from [`QueueSnapshot`]. It does not branch on source identity or explicit +//! throughput targets. +//! +//! # Mental model +//! +//! Think of this as a two-gear system: +//! +//! - [`ChunkingMode::Smooth`]: steady baseline display pacing. +//! - [`ChunkingMode::CatchUp`]: full queue draining while backlog exists. +//! +//! The transition logic intentionally uses hysteresis: +//! +//! - enter catch-up on higher-pressure thresholds +//! - exit catch-up on lower-pressure thresholds, held for [`EXIT_HOLD`] +//! - after exit, suppress immediate re-entry for [`REENTER_CATCH_UP_HOLD`] +//! unless backlog is severe +//! +//! This avoids rapid gear-flapping near threshold boundaries. +//! +//! # Policy flow +//! +//! On each decision tick, [`AdaptiveChunkingPolicy::decide`] does: +//! +//! 1. If queue is empty, reset to [`ChunkingMode::Smooth`]. +//! 2. If currently smooth, call [`AdaptiveChunkingPolicy::maybe_enter_catch_up`]. +//! 3. If currently catch-up, call [`AdaptiveChunkingPolicy::maybe_exit_catch_up`]. +//! 4. Build [`DrainPlan`] (`Single` for smooth, `Batch(queued_lines)` for catch-up). +//! +//! # Concrete examples +//! +//! With current defaults: +//! +//! - `Smooth` drains one line per commit tick. +//! - `CatchUp` drains all currently queued lines in a tick. +//! +//! # Tuning guide (in code terms) +//! +//! Prefer tuning in this order so causes remain clear: +//! +//! 1. enter/exit thresholds: [`ENTER_QUEUE_DEPTH_LINES`], [`ENTER_OLDEST_AGE`], +//! [`EXIT_QUEUE_DEPTH_LINES`], [`EXIT_OLDEST_AGE`] +//! 2. hysteresis windows: [`EXIT_HOLD`], [`REENTER_CATCH_UP_HOLD`] +//! 3. severe gates: [`SEVERE_QUEUE_DEPTH_LINES`], [`SEVERE_OLDEST_AGE`] +//! +//! Symptom-oriented adjustments: +//! +//! - lag starts too late: lower enter thresholds +//! - frequent smooth/catch-up chatter: increase hold windows, or tighten exit +//! thresholds +//! - catch-up re-enters too eagerly after exit: increase re-entry hold +//! +//! # Responsibilities +//! +//! - track mode and hysteresis state +//! - produce deterministic [`ChunkingDecision`] values from queue snapshots +//! - preserve queue order by draining from queue head only +//! +//! # Non-responsibilities +//! +//! - scheduling commit ticks +//! - reordering stream lines +//! - transport/source-specific semantics +//! +//! Markdown docs remain supplemental: +//! +//! - `docs/tui-stream-chunking-review.md` +//! - `docs/tui-stream-chunking-tuning.md` +//! - `docs/tui-stream-chunking-validation.md` + +use std::time::Duration; +use std::time::Instant; + +/// Queue-depth threshold that allows entering catch-up mode. +/// +/// Crossing this threshold alone is sufficient to leave smooth mode. +const ENTER_QUEUE_DEPTH_LINES: usize = 8; + +/// Oldest-line age threshold that allows entering catch-up mode. +/// +/// Crossing this threshold alone is sufficient to leave smooth mode. +const ENTER_OLDEST_AGE: Duration = Duration::from_millis(120); + +/// Queue-depth threshold used when evaluating catch-up exit hysteresis. +/// +/// Depth must be at or below this value before exit hold timing can begin. +const EXIT_QUEUE_DEPTH_LINES: usize = 2; + +/// Oldest-line age threshold used when evaluating catch-up exit hysteresis. +/// +/// Age must be at or below this value before exit hold timing can begin. +const EXIT_OLDEST_AGE: Duration = Duration::from_millis(40); + +/// Minimum duration queue pressure must stay below exit thresholds to leave catch-up mode. +const EXIT_HOLD: Duration = Duration::from_millis(250); + +/// Cooldown window after a catch-up exit that suppresses immediate re-entry. +/// +/// Severe backlog still bypasses this hold to avoid unbounded queue-age growth. +const REENTER_CATCH_UP_HOLD: Duration = Duration::from_millis(250); + +/// Queue-depth cutoff that marks backlog as severe for faster convergence. +/// +/// This threshold is used to bypass re-entry hold after a recent catch-up exit. +const SEVERE_QUEUE_DEPTH_LINES: usize = 64; + +/// Oldest-line age cutoff that marks backlog as severe for faster convergence. +const SEVERE_OLDEST_AGE: Duration = Duration::from_millis(300); + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) enum ChunkingMode { + /// Drain one line per baseline commit tick. + #[default] + Smooth, + /// Drain multiple lines per tick according to queue pressure. + CatchUp, +} + +/// Captures queue pressure inputs used by adaptive chunking decisions. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct QueueSnapshot { + /// Number of queued stream lines waiting to be displayed. + pub(crate) queued_lines: usize, + /// Age of the oldest queued line at decision time. + pub(crate) oldest_age: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum DrainPlan { + /// Emit exactly one queued line. + Single, + /// Emit up to `usize` queued lines. + Batch(usize), +} + +/// Represents one policy decision for a specific queue snapshot. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct ChunkingDecision { + /// Mode after applying hysteresis transitions for this decision. + pub(crate) mode: ChunkingMode, + /// Whether this decision transitioned from `Smooth` into `CatchUp`. + pub(crate) entered_catch_up: bool, + /// Drain plan to execute for the current commit tick. + pub(crate) drain_plan: DrainPlan, +} + +/// Maintains adaptive chunking mode and hysteresis state across ticks. +#[derive(Debug, Default)] +pub(crate) struct AdaptiveChunkingPolicy { + mode: ChunkingMode, + below_exit_threshold_since: Option, + last_catch_up_exit_at: Option, +} + +impl AdaptiveChunkingPolicy { + /// Returns the policy mode used by the most recent decision. + pub(crate) fn mode(&self) -> ChunkingMode { + self.mode + } + + /// Resets state to baseline smooth mode. + pub(crate) fn reset(&mut self) { + self.mode = ChunkingMode::Smooth; + self.below_exit_threshold_since = None; + self.last_catch_up_exit_at = None; + } + + /// Computes a drain decision from the current queue snapshot. + /// + /// The decision is deterministic for a given `(mode, snapshot, now)` triple. Callers should + /// avoid inventing synthetic snapshots; stale queue age data can cause premature catch-up exits. + pub(crate) fn decide(&mut self, snapshot: QueueSnapshot, now: Instant) -> ChunkingDecision { + if snapshot.queued_lines == 0 { + self.note_catch_up_exit(now); + self.mode = ChunkingMode::Smooth; + self.below_exit_threshold_since = None; + return ChunkingDecision { + mode: self.mode, + entered_catch_up: false, + drain_plan: DrainPlan::Single, + }; + } + + let entered_catch_up = match self.mode { + ChunkingMode::Smooth => self.maybe_enter_catch_up(snapshot, now), + ChunkingMode::CatchUp => { + self.maybe_exit_catch_up(snapshot, now); + false + } + }; + + let drain_plan = match self.mode { + ChunkingMode::Smooth => DrainPlan::Single, + ChunkingMode::CatchUp => DrainPlan::Batch(snapshot.queued_lines.max(1)), + }; + + ChunkingDecision { + mode: self.mode, + entered_catch_up, + drain_plan, + } + } + + /// Switches from `Smooth` to `CatchUp` when enter thresholds are crossed. + /// + /// Returns `true` only on the transition tick so callers can emit one-shot + /// transition observability. + fn maybe_enter_catch_up(&mut self, snapshot: QueueSnapshot, now: Instant) -> bool { + if !should_enter_catch_up(snapshot) { + return false; + } + if self.reentry_hold_active(now) && !is_severe_backlog(snapshot) { + return false; + } + self.mode = ChunkingMode::CatchUp; + self.below_exit_threshold_since = None; + self.last_catch_up_exit_at = None; + true + } + + /// Applies exit hysteresis while in `CatchUp` mode. + /// + /// The policy requires queue pressure to stay below exit thresholds for the + /// full `EXIT_HOLD` window before returning to `Smooth`. + fn maybe_exit_catch_up(&mut self, snapshot: QueueSnapshot, now: Instant) { + if !should_exit_catch_up(snapshot) { + self.below_exit_threshold_since = None; + return; + } + + match self.below_exit_threshold_since { + Some(since) if now.saturating_duration_since(since) >= EXIT_HOLD => { + self.mode = ChunkingMode::Smooth; + self.below_exit_threshold_since = None; + self.last_catch_up_exit_at = Some(now); + } + Some(_) => {} + None => { + self.below_exit_threshold_since = Some(now); + } + } + } + + fn note_catch_up_exit(&mut self, now: Instant) { + if self.mode == ChunkingMode::CatchUp { + self.last_catch_up_exit_at = Some(now); + } + } + + fn reentry_hold_active(&self, now: Instant) -> bool { + self.last_catch_up_exit_at + .is_some_and(|exit| now.saturating_duration_since(exit) < REENTER_CATCH_UP_HOLD) + } +} + +/// Returns whether current queue pressure warrants entering catch-up mode. +/// +/// Either depth or age pressure is sufficient to trigger catch-up. +fn should_enter_catch_up(snapshot: QueueSnapshot) -> bool { + snapshot.queued_lines >= ENTER_QUEUE_DEPTH_LINES + || snapshot + .oldest_age + .is_some_and(|oldest| oldest >= ENTER_OLDEST_AGE) +} + +/// Returns whether queue pressure is low enough to begin exit hysteresis. +/// +/// Both depth and age must be below thresholds; this prevents oscillation when +/// one signal is still under load. +fn should_exit_catch_up(snapshot: QueueSnapshot) -> bool { + snapshot.queued_lines <= EXIT_QUEUE_DEPTH_LINES + && snapshot + .oldest_age + .is_some_and(|oldest| oldest <= EXIT_OLDEST_AGE) +} + +/// Returns whether backlog is severe enough to use a faster catch-up target. +/// +/// Severe pressure bypasses re-entry hold to avoid queue-age growth after a +/// recent catch-up exit. +fn is_severe_backlog(snapshot: QueueSnapshot) -> bool { + snapshot.queued_lines >= SEVERE_QUEUE_DEPTH_LINES + || snapshot + .oldest_age + .is_some_and(|oldest| oldest >= SEVERE_OLDEST_AGE) +} + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + + fn snapshot(queued_lines: usize, oldest_age_ms: u64) -> QueueSnapshot { + QueueSnapshot { + queued_lines, + oldest_age: Some(Duration::from_millis(oldest_age_ms)), + } + } + + #[test] + fn smooth_mode_is_default() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + + let decision = policy.decide(snapshot(1, 10), now); + assert_eq!(decision.mode, ChunkingMode::Smooth); + assert_eq!(decision.entered_catch_up, false); + assert_eq!(decision.drain_plan, DrainPlan::Single); + } + + #[test] + fn enters_catch_up_on_depth_threshold() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + + let decision = policy.decide(snapshot(8, 10), now); + assert_eq!(decision.mode, ChunkingMode::CatchUp); + assert_eq!(decision.entered_catch_up, true); + assert_eq!(decision.drain_plan, DrainPlan::Batch(8)); + } + + #[test] + fn enters_catch_up_on_age_threshold() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + + let decision = policy.decide(snapshot(2, 120), now); + assert_eq!(decision.mode, ChunkingMode::CatchUp); + assert_eq!(decision.entered_catch_up, true); + assert_eq!(decision.drain_plan, DrainPlan::Batch(2)); + } + + #[test] + fn severe_backlog_uses_faster_paced_batches() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + let _ = policy.decide(snapshot(9, 10), now); + + let decision = policy.decide(snapshot(64, 10), now + Duration::from_millis(5)); + assert_eq!(decision.mode, ChunkingMode::CatchUp); + assert_eq!(decision.drain_plan, DrainPlan::Batch(64)); + } + + #[test] + fn catch_up_batch_drains_current_backlog() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + let decision = policy.decide(snapshot(512, 400), now); + assert_eq!(decision.mode, ChunkingMode::CatchUp); + assert_eq!(decision.drain_plan, DrainPlan::Batch(512)); + } + + #[test] + fn exits_catch_up_after_hysteresis_hold() { + let mut policy = AdaptiveChunkingPolicy::default(); + let t0 = Instant::now(); + + let _ = policy.decide(snapshot(9, 10), t0); + assert_eq!(policy.mode(), ChunkingMode::CatchUp); + + let pre_hold = policy.decide(snapshot(2, 40), t0 + Duration::from_millis(200)); + assert_eq!(pre_hold.mode, ChunkingMode::CatchUp); + + let post_hold = policy.decide(snapshot(2, 40), t0 + Duration::from_millis(460)); + assert_eq!(post_hold.mode, ChunkingMode::Smooth); + assert_eq!(post_hold.drain_plan, DrainPlan::Single); + } + + #[test] + fn drops_back_to_smooth_when_idle() { + let mut policy = AdaptiveChunkingPolicy::default(); + let now = Instant::now(); + let _ = policy.decide(snapshot(9, 10), now); + assert_eq!(policy.mode(), ChunkingMode::CatchUp); + + let decision = policy.decide( + QueueSnapshot { + queued_lines: 0, + oldest_age: None, + }, + now + Duration::from_millis(20), + ); + assert_eq!(decision.mode, ChunkingMode::Smooth); + assert_eq!(decision.drain_plan, DrainPlan::Single); + } + + #[test] + fn holds_reentry_after_catch_up_exit() { + let mut policy = AdaptiveChunkingPolicy::default(); + let t0 = Instant::now(); + + let entered = policy.decide(snapshot(8, 20), t0); + assert_eq!(entered.mode, ChunkingMode::CatchUp); + + let drained = policy.decide( + QueueSnapshot { + queued_lines: 0, + oldest_age: None, + }, + t0 + Duration::from_millis(20), + ); + assert_eq!(drained.mode, ChunkingMode::Smooth); + + let held = policy.decide(snapshot(8, 20), t0 + Duration::from_millis(120)); + assert_eq!(held.mode, ChunkingMode::Smooth); + assert_eq!(held.drain_plan, DrainPlan::Single); + + let reentered = policy.decide(snapshot(8, 20), t0 + Duration::from_millis(320)); + assert_eq!(reentered.mode, ChunkingMode::CatchUp); + assert_eq!(reentered.drain_plan, DrainPlan::Batch(8)); + } + + #[test] + fn severe_backlog_can_reenter_during_hold() { + let mut policy = AdaptiveChunkingPolicy::default(); + let t0 = Instant::now(); + + let _ = policy.decide(snapshot(8, 20), t0); + let _ = policy.decide( + QueueSnapshot { + queued_lines: 0, + oldest_age: None, + }, + t0 + Duration::from_millis(20), + ); + + let severe = policy.decide(snapshot(64, 20), t0 + Duration::from_millis(120)); + assert_eq!(severe.mode, ChunkingMode::CatchUp); + assert_eq!(severe.drain_plan, DrainPlan::Batch(64)); + } +} diff --git a/codex-rs/tui/src/streaming/commit_tick.rs b/codex-rs/tui/src/streaming/commit_tick.rs new file mode 100644 index 0000000000..bda63bccf6 --- /dev/null +++ b/codex-rs/tui/src/streaming/commit_tick.rs @@ -0,0 +1,224 @@ +//! Orchestrates commit-tick drains across streaming controllers. +//! +//! This module bridges queue-based chunking policy (`chunking`) with the concrete stream +//! controllers (`controller`). Callers provide the current controllers and tick scope; the module +//! computes queue pressure, selects a drain plan, applies it, and returns emitted history cells. +//! +//! The module preserves ordering by draining only from controller queue heads. It does not schedule +//! ticks and it does not mutate UI state directly; callers remain responsible for animation events +//! and history insertion side effects. +//! +//! The main flow is: +//! [`run_commit_tick`] -> [`stream_queue_snapshot`] -> [`QueueSnapshot`] -> +//! [`resolve_chunking_plan`] -> [`ChunkingDecision`]/[`DrainPlan`] -> +//! [`apply_commit_tick_plan`] -> [`CommitTickOutput`]. + +use std::time::Duration; +use std::time::Instant; + +use crate::history_cell::HistoryCell; + +use super::chunking::AdaptiveChunkingPolicy; +use super::chunking::ChunkingDecision; +use super::chunking::ChunkingMode; +use super::chunking::DrainPlan; +use super::chunking::QueueSnapshot; +use super::controller::PlanStreamController; +use super::controller::StreamController; + +/// Describes whether a commit tick may run in all modes or only in catch-up mode. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum CommitTickScope { + /// Always run the tick, regardless of current chunking mode. + AnyMode, + /// Run model transitions and policy updates, but commit lines only in `CatchUp`. + CatchUpOnly, +} + +/// Describes what a single commit tick produced. +pub(crate) struct CommitTickOutput { + /// Cells produced by drained stream lines during this tick. + pub(crate) cells: Vec>, + /// Whether at least one stream controller was present for this tick. + pub(crate) has_controller: bool, + /// Whether all present controllers were idle after this tick. + pub(crate) all_idle: bool, +} + +impl Default for CommitTickOutput { + /// Creates an output that represents "no commit performed". + /// + /// This is used when a tick is intentionally suppressed, for example when + /// the scope is [`CommitTickScope::CatchUpOnly`] and policy is not in catch-up mode. + fn default() -> Self { + Self { + cells: Vec::new(), + has_controller: false, + all_idle: true, + } + } +} + +/// Runs one commit tick against the provided stream controllers. +/// +/// This function collects a [`QueueSnapshot`], asks [`AdaptiveChunkingPolicy`] for a +/// [`ChunkingDecision`], and then applies the resulting [`DrainPlan`] to both controllers. +/// If callers pass stale controller references (for example, references not tied to the +/// current turn), queue age can be misread and the policy may stay in catch-up longer +/// than expected. +pub(crate) fn run_commit_tick( + policy: &mut AdaptiveChunkingPolicy, + stream_controller: Option<&mut StreamController>, + plan_stream_controller: Option<&mut PlanStreamController>, + scope: CommitTickScope, + now: Instant, +) -> CommitTickOutput { + let snapshot = stream_queue_snapshot( + stream_controller.as_deref(), + plan_stream_controller.as_deref(), + now, + ); + let decision = resolve_chunking_plan(policy, snapshot, now); + if scope == CommitTickScope::CatchUpOnly && decision.mode != ChunkingMode::CatchUp { + return CommitTickOutput::default(); + } + + let output = apply_commit_tick_plan( + decision.drain_plan, + stream_controller, + plan_stream_controller, + ); + tracing::trace!( + mode = ?decision.mode, + queued_lines = snapshot.queued_lines, + oldest_queued_age_ms = snapshot.oldest_age.map(|age| age.as_millis() as u64), + drain_plan = ?decision.drain_plan, + has_controller = output.has_controller, + all_idle = output.all_idle, + "stream chunking commit tick" + ); + output +} + +/// Builds the combined queue-pressure snapshot consumed by chunking policy. +/// +/// The snapshot sums queue depth across controllers and keeps the maximum oldest age +/// so policy decisions reflect the most delayed queued line currently visible. +fn stream_queue_snapshot( + stream_controller: Option<&StreamController>, + plan_stream_controller: Option<&PlanStreamController>, + now: Instant, +) -> QueueSnapshot { + let mut queued_lines = 0usize; + let mut oldest_age: Option = None; + + if let Some(controller) = stream_controller { + queued_lines += controller.queued_lines(); + oldest_age = max_duration(oldest_age, controller.oldest_queued_age(now)); + } + if let Some(controller) = plan_stream_controller { + queued_lines += controller.queued_lines(); + oldest_age = max_duration(oldest_age, controller.oldest_queued_age(now)); + } + + QueueSnapshot { + queued_lines, + oldest_age, + } +} + +/// Computes one policy decision and emits a trace log on mode transitions. +/// +/// This keeps policy transition logging in one place so callers can rely on +/// [`run_commit_tick`] to provide consistent observability. +fn resolve_chunking_plan( + policy: &mut AdaptiveChunkingPolicy, + snapshot: QueueSnapshot, + now: Instant, +) -> ChunkingDecision { + let prior_mode = policy.mode(); + let decision = policy.decide(snapshot, now); + if decision.mode != prior_mode { + tracing::trace!( + prior_mode = ?prior_mode, + new_mode = ?decision.mode, + queued_lines = snapshot.queued_lines, + oldest_queued_age_ms = snapshot.oldest_age.map(|age| age.as_millis() as u64), + entered_catch_up = decision.entered_catch_up, + "stream chunking mode transition" + ); + } + decision +} + +/// Applies a [`DrainPlan`] to all available stream controllers. +/// +/// The returned [`CommitTickOutput`] reports emitted cells and whether all +/// present controllers are idle after draining. +fn apply_commit_tick_plan( + drain_plan: DrainPlan, + stream_controller: Option<&mut StreamController>, + plan_stream_controller: Option<&mut PlanStreamController>, +) -> CommitTickOutput { + let mut output = CommitTickOutput::default(); + + if let Some(controller) = stream_controller { + output.has_controller = true; + let (cell, is_idle) = drain_stream_controller(controller, drain_plan); + if let Some(cell) = cell { + output.cells.push(cell); + } + output.all_idle &= is_idle; + } + if let Some(controller) = plan_stream_controller { + output.has_controller = true; + let (cell, is_idle) = drain_plan_stream_controller(controller, drain_plan); + if let Some(cell) = cell { + output.cells.push(cell); + } + output.all_idle &= is_idle; + } + + output +} + +/// Applies one drain step to the main stream controller. +/// +/// [`DrainPlan::Single`] maps to one-line drain; [`DrainPlan::Batch`] maps to +/// multi-line drain (including instant catch-up when policy requests the full +/// queued backlog). +fn drain_stream_controller( + controller: &mut StreamController, + drain_plan: DrainPlan, +) -> (Option>, bool) { + match drain_plan { + DrainPlan::Single => controller.on_commit_tick(), + DrainPlan::Batch(max_lines) => controller.on_commit_tick_batch(max_lines), + } +} + +/// Applies one drain step to the plan stream controller. +/// +/// This mirrors [`drain_stream_controller`] so both controller types follow the +/// same chunking policy decisions. +fn drain_plan_stream_controller( + controller: &mut PlanStreamController, + drain_plan: DrainPlan, +) -> (Option>, bool) { + match drain_plan { + DrainPlan::Single => controller.on_commit_tick(), + DrainPlan::Batch(max_lines) => controller.on_commit_tick_batch(max_lines), + } +} + +/// Returns the greater of two optional durations. +/// +/// This helper preserves whichever side is present when only one duration exists. +fn max_duration(lhs: Option, rhs: Option) -> Option { + match (lhs, rhs) { + (Some(left), Some(right)) => Some(left.max(right)), + (Some(left), None) => Some(left), + (None, Some(right)) => Some(right), + (None, None) => None, + } +} diff --git a/codex-rs/tui/src/streaming/controller.rs b/codex-rs/tui/src/streaming/controller.rs index 462962e980..6117485adf 100644 --- a/codex-rs/tui/src/streaming/controller.rs +++ b/codex-rs/tui/src/streaming/controller.rs @@ -4,6 +4,8 @@ use crate::render::line_utils::prefix_lines; use crate::style::proposed_plan_style; use ratatui::prelude::Stylize; use ratatui::text::Line; +use std::time::Duration; +use std::time::Instant; use super::StreamState; @@ -71,6 +73,28 @@ impl StreamController { (self.emit(step), self.state.is_idle()) } + /// Step animation: commit at most `max_lines` queued lines. + /// + /// This is intended for adaptive catch-up drains. Callers should keep `max_lines` bounded; a + /// very large value can collapse perceived animation into a single jump. + pub(crate) fn on_commit_tick_batch( + &mut self, + max_lines: usize, + ) -> (Option>, bool) { + let step = self.state.drain_n(max_lines.max(1)); + (self.emit(step), self.state.is_idle()) + } + + /// Returns the current number of queued lines waiting to be displayed. + pub(crate) fn queued_lines(&self) -> usize { + self.state.queued_len() + } + + /// Returns the age of the oldest queued line. + pub(crate) fn oldest_queued_age(&self, now: Instant) -> Option { + self.state.oldest_queued_age(now) + } + fn emit(&mut self, lines: Vec>) -> Option> { if lines.is_empty() { return None; @@ -142,6 +166,28 @@ impl PlanStreamController { (self.emit(step, false), self.state.is_idle()) } + /// Step animation: commit at most `max_lines` queued lines. + /// + /// This is intended for adaptive catch-up drains. Callers should keep `max_lines` bounded; a + /// very large value can collapse perceived animation into a single jump. + pub(crate) fn on_commit_tick_batch( + &mut self, + max_lines: usize, + ) -> (Option>, bool) { + let step = self.state.drain_n(max_lines.max(1)); + (self.emit(step, false), self.state.is_idle()) + } + + /// Returns the current number of queued plan lines waiting to be displayed. + pub(crate) fn queued_lines(&self) -> usize { + self.state.queued_len() + } + + /// Returns the age of the oldest queued plan line. + pub(crate) fn oldest_queued_age(&self, now: Instant) -> Option { + self.state.oldest_queued_age(now) + } + fn emit( &mut self, lines: Vec>, diff --git a/codex-rs/tui/src/streaming/mod.rs b/codex-rs/tui/src/streaming/mod.rs index fa00702834..c783f27ae9 100644 --- a/codex-rs/tui/src/streaming/mod.rs +++ b/codex-rs/tui/src/streaming/mod.rs @@ -1,17 +1,39 @@ +//! Streaming primitives used by the TUI transcript pipeline. +//! +//! `StreamState` owns newline-gated markdown collection and a FIFO queue of committed render lines. +//! Higher-level modules build on top of this state: +//! - `controller` adapts queued lines into `HistoryCell` emission rules for message and plan streams. +//! - `chunking` computes adaptive drain plans from queue pressure. +//! - `commit_tick` binds policy decisions to concrete controller drains. +//! +//! The key invariant is queue ordering. All drains pop from the front, and enqueue records an +//! arrival timestamp so policy code can reason about oldest queued age without peeking into text. + use std::collections::VecDeque; +use std::time::Duration; +use std::time::Instant; use ratatui::text::Line; use crate::markdown_stream::MarkdownStreamCollector; +pub(crate) mod chunking; +pub(crate) mod commit_tick; pub(crate) mod controller; +struct QueuedLine { + line: Line<'static>, + enqueued_at: Instant, +} + +/// Holds in-flight markdown stream state and queued committed lines. pub(crate) struct StreamState { pub(crate) collector: MarkdownStreamCollector, - queued_lines: VecDeque>, + queued_lines: VecDeque, pub(crate) has_seen_delta: bool, } impl StreamState { + /// Creates an empty stream state with an optional target wrap width. pub(crate) fn new(width: Option) -> Self { Self { collector: MarkdownStreamCollector::new(width), @@ -19,21 +41,75 @@ impl StreamState { has_seen_delta: false, } } + /// Resets collector and queue state for the next stream lifecycle. pub(crate) fn clear(&mut self) { self.collector.clear(); self.queued_lines.clear(); self.has_seen_delta = false; } + /// Drains one queued line from the front of the queue. pub(crate) fn step(&mut self) -> Vec> { - self.queued_lines.pop_front().into_iter().collect() + self.queued_lines + .pop_front() + .map(|queued| queued.line) + .into_iter() + .collect() } + /// Drains up to `max_lines` queued lines from the front of the queue. + /// + /// Callers that pass very large values still get bounded behavior because this method clamps to + /// the currently available queue length. + pub(crate) fn drain_n(&mut self, max_lines: usize) -> Vec> { + let end = max_lines.min(self.queued_lines.len()); + self.queued_lines + .drain(..end) + .map(|queued| queued.line) + .collect() + } + /// Drains all queued lines from the front of the queue. pub(crate) fn drain_all(&mut self) -> Vec> { - self.queued_lines.drain(..).collect() + self.queued_lines + .drain(..) + .map(|queued| queued.line) + .collect() } + /// Returns whether no lines are queued for commit. pub(crate) fn is_idle(&self) -> bool { self.queued_lines.is_empty() } + /// Returns the current queue depth. + pub(crate) fn queued_len(&self) -> usize { + self.queued_lines.len() + } + /// Returns the age of the oldest queued line. + pub(crate) fn oldest_queued_age(&self, now: Instant) -> Option { + self.queued_lines + .front() + .map(|queued| now.saturating_duration_since(queued.enqueued_at)) + } + /// Appends committed lines to the queue with a shared enqueue timestamp. pub(crate) fn enqueue(&mut self, lines: Vec>) { - self.queued_lines.extend(lines); + let now = Instant::now(); + self.queued_lines + .extend(lines.into_iter().map(|line| QueuedLine { + line, + enqueued_at: now, + })); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + + #[test] + fn drain_n_clamps_to_available_lines() { + let mut state = StreamState::new(None); + state.enqueue(vec![Line::from("one")]); + + let drained = state.drain_n(8); + assert_eq!(drained, vec![Line::from("one")]); + assert!(state.is_idle()); } } diff --git a/codex-rs/tui/src/tui.rs b/codex-rs/tui/src/tui.rs index 761fa83627..a77785c1b8 100644 --- a/codex-rs/tui/src/tui.rs +++ b/codex-rs/tui/src/tui.rs @@ -10,6 +10,7 @@ use std::pin::Pin; use std::sync::Arc; use std::sync::atomic::AtomicBool; use std::sync::atomic::Ordering; +use std::time::Duration; use crossterm::Command; use crossterm::SynchronizedUpdate; @@ -52,6 +53,9 @@ mod frame_requester; #[cfg(unix)] mod job_control; +/// Target frame interval for UI redraw scheduling. +pub(crate) const TARGET_FRAME_INTERVAL: Duration = frame_rate_limiter::MIN_FRAME_INTERVAL; + /// A type alias for the terminal type used in this application pub type Terminal = CustomTerminal>; diff --git a/codex-rs/tui/src/tui/frame_rate_limiter.rs b/codex-rs/tui/src/tui/frame_rate_limiter.rs index 56dd752e64..6f4ad8e747 100644 --- a/codex-rs/tui/src/tui/frame_rate_limiter.rs +++ b/codex-rs/tui/src/tui/frame_rate_limiter.rs @@ -1,7 +1,7 @@ //! Limits how frequently frame draw notifications may be emitted. //! //! Widgets sometimes call `FrameRequester::schedule_frame()` more frequently than a user can -//! perceive. This limiter clamps draw notifications to a maximum of 60 FPS to avoid wasted work. +//! perceive. This limiter clamps draw notifications to a maximum of 120 FPS to avoid wasted work. //! //! This is intentionally a small, pure helper so it can be unit-tested in isolation and used by //! the async frame scheduler without adding complexity to the app/event loop. @@ -9,8 +9,8 @@ use std::time::Duration; use std::time::Instant; -/// A 60 FPS minimum frame interval (β‰ˆ16.67ms). -pub(super) const MIN_FRAME_INTERVAL: Duration = Duration::from_nanos(16_666_667); +/// A 120 FPS minimum frame interval (β‰ˆ8.33ms). +pub(super) const MIN_FRAME_INTERVAL: Duration = Duration::from_nanos(8_333_334); /// Remembers the most recent emitted draw, allowing deadlines to be clamped forward. #[derive(Debug, Default)] diff --git a/codex-rs/tui/src/tui/frame_requester.rs b/codex-rs/tui/src/tui/frame_requester.rs index 88db9d71f3..d7e54d82cc 100644 --- a/codex-rs/tui/src/tui/frame_requester.rs +++ b/codex-rs/tui/src/tui/frame_requester.rs @@ -71,7 +71,7 @@ impl FrameRequester { /// /// This type is internal to `FrameRequester` and is spawned as a task to handle scheduling logic. /// -/// To avoid wasted redraw work, draw notifications are clamped to a maximum of 60 FPS (see +/// To avoid wasted redraw work, draw notifications are clamped to a maximum of 120 FPS (see /// [`FrameRateLimiter`]). struct FrameScheduler { receiver: mpsc::UnboundedReceiver, @@ -232,7 +232,7 @@ mod tests { } #[tokio::test(flavor = "current_thread", start_paused = true)] - async fn test_limits_draw_notifications_to_60fps() { + async fn test_limits_draw_notifications_to_120fps() { let (draw_tx, mut draw_rx) = broadcast::channel(16); let requester = FrameRequester::new(draw_tx); @@ -250,7 +250,7 @@ mod tests { let early = draw_rx.recv().timeout(Duration::from_millis(1)).await; assert!( early.is_err(), - "draw fired too early; expected max 60fps (min interval {MIN_FRAME_INTERVAL:?})" + "draw fired too early; expected max 120fps (min interval {MIN_FRAME_INTERVAL:?})" ); time::advance(MIN_FRAME_INTERVAL).await; @@ -278,11 +278,11 @@ mod tests { requester.schedule_frame_in(Duration::from_millis(1)); - time::advance(Duration::from_millis(10)).await; + time::advance(MIN_FRAME_INTERVAL / 2).await; let too_early = draw_rx.recv().timeout(Duration::from_millis(1)).await; assert!( too_early.is_err(), - "draw fired too early; expected max 60fps (min interval {MIN_FRAME_INTERVAL:?})" + "draw fired too early; expected max 120fps (min interval {MIN_FRAME_INTERVAL:?})" ); time::advance(MIN_FRAME_INTERVAL).await; diff --git a/codex-rs/windows-sandbox-rs/src/acl.rs b/codex-rs/windows-sandbox-rs/src/acl.rs index 7dd37a9452..0018856a44 100644 --- a/codex-rs/windows-sandbox-rs/src/acl.rs +++ b/codex-rs/windows-sandbox-rs/src/acl.rs @@ -233,7 +233,9 @@ pub unsafe fn dacl_has_write_deny_for_sid(p_dacl: *mut ACL, psid: *mut c_void) - | FILE_APPEND_DATA | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES - | GENERIC_WRITE_MASK; + | GENERIC_WRITE_MASK + | DELETE + | FILE_DELETE_CHILD; for i in 0..info.AceCount { let mut p_ace: *mut c_void = std::ptr::null_mut(); if GetAce(p_dacl as *const ACL, i, &mut p_ace) == 0 { @@ -477,7 +479,9 @@ pub unsafe fn add_deny_write_ace(path: &Path, psid: *mut c_void) -> Result | FILE_APPEND_DATA | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES - | GENERIC_WRITE_MASK; + | GENERIC_WRITE_MASK + | DELETE + | FILE_DELETE_CHILD; explicit.grfAccessMode = DENY_ACCESS; explicit.grfInheritance = CONTAINER_INHERIT_ACE | OBJECT_INHERIT_ACE; explicit.Trustee = trustee; diff --git a/codex-rs/windows-sandbox-rs/src/audit.rs b/codex-rs/windows-sandbox-rs/src/audit.rs index 2e6059e819..1af9f1b919 100644 --- a/codex-rs/windows-sandbox-rs/src/audit.rs +++ b/codex-rs/windows-sandbox-rs/src/audit.rs @@ -3,6 +3,7 @@ use crate::acl::path_mask_allows; use crate::cap::cap_sid_file; use crate::cap::load_or_create_cap_sids; use crate::logging::{debug_log, log_note}; +use crate::path_normalization::canonical_path_key; use crate::policy::SandboxPolicy; use crate::token::convert_string_sid_to_sid; use crate::token::world_sid; @@ -31,11 +32,6 @@ const SKIP_DIR_SUFFIXES: &[&str] = &[ "/programdata", ]; -pub(crate) fn normalize_path_key(p: &Path) -> String { - let n = dunce::canonicalize(p).unwrap_or_else(|_| p.to_path_buf()); - n.to_string_lossy().replace('\\', "/").to_ascii_lowercase() -} - fn unique_push(set: &mut HashSet, out: &mut Vec, p: PathBuf) { if let Ok(abs) = p.canonicalize() { if set.insert(abs.clone()) { @@ -131,7 +127,7 @@ pub fn audit_everyone_writable( checked += 1; let has = check_world_writable(&p); if has { - let key = normalize_path_key(&p); + let key = canonical_path_key(&p); if seen.insert(key) { flagged.push(p); } @@ -149,7 +145,7 @@ pub fn audit_everyone_writable( checked += 1; let has_root = check_world_writable(&root); if has_root { - let key = normalize_path_key(&root); + let key = canonical_path_key(&root); if seen.insert(key) { flagged.push(root.clone()); } @@ -181,7 +177,7 @@ pub fn audit_everyone_writable( checked += 1; let has_child = check_world_writable(&p); if has_child { - let key = normalize_path_key(&p); + let key = canonical_path_key(&p); if seen.insert(key) { flagged.push(p); } diff --git a/codex-rs/windows-sandbox-rs/src/cap.rs b/codex-rs/windows-sandbox-rs/src/cap.rs index 5100684965..a699d81d20 100644 --- a/codex-rs/windows-sandbox-rs/src/cap.rs +++ b/codex-rs/windows-sandbox-rs/src/cap.rs @@ -5,14 +5,23 @@ use rand::RngCore; use rand::SeedableRng; use serde::Deserialize; use serde::Serialize; +use std::collections::HashMap; use std::fs; use std::path::Path; use std::path::PathBuf; +use crate::path_normalization::canonical_path_key; #[derive(Serialize, Deserialize, Clone, Debug)] pub struct CapSids { pub workspace: String, pub readonly: String, + /// Per-workspace capability SIDs keyed by canonicalized CWD string. + /// + /// This is used to isolate workspaces from other workspace sandbox writes and to + /// apply per-workspace denies (e.g. protect `CWD/.codex`) + /// without permanently affecting other workspaces. + #[serde(default)] + pub workspace_by_cwd: HashMap, } pub fn cap_sid_file(codex_home: &Path) -> PathBuf { @@ -30,8 +39,7 @@ fn make_random_cap_sid_string() -> String { fn persist_caps(path: &Path, caps: &CapSids) -> Result<()> { if let Some(dir) = path.parent() { - fs::create_dir_all(dir) - .with_context(|| format!("create cap sid dir {}", dir.display()))?; + fs::create_dir_all(dir).with_context(|| format!("create cap sid dir {}", dir.display()))?; } let json = serde_json::to_string(caps)?; fs::write(path, json).with_context(|| format!("write cap sid file {}", path.display()))?; @@ -52,6 +60,7 @@ pub fn load_or_create_cap_sids(codex_home: &Path) -> Result { let caps = CapSids { workspace: t.to_string(), readonly: make_random_cap_sid_string(), + workspace_by_cwd: HashMap::new(), }; persist_caps(&path, &caps)?; return Ok(caps); @@ -60,7 +69,53 @@ pub fn load_or_create_cap_sids(codex_home: &Path) -> Result { let caps = CapSids { workspace: make_random_cap_sid_string(), readonly: make_random_cap_sid_string(), + workspace_by_cwd: HashMap::new(), }; persist_caps(&path, &caps)?; Ok(caps) } + +/// Returns the workspace-specific capability SID for `cwd`, creating and persisting it if missing. +pub fn workspace_cap_sid_for_cwd(codex_home: &Path, cwd: &Path) -> Result { + let path = cap_sid_file(codex_home); + let mut caps = load_or_create_cap_sids(codex_home)?; + let key = canonical_path_key(cwd); + if let Some(sid) = caps.workspace_by_cwd.get(&key) { + return Ok(sid.clone()); + } + let sid = make_random_cap_sid_string(); + caps.workspace_by_cwd.insert(key, sid.clone()); + persist_caps(&path, &caps)?; + Ok(sid) +} + +#[cfg(test)] +mod tests { + use super::load_or_create_cap_sids; + use super::workspace_cap_sid_for_cwd; + use pretty_assertions::assert_eq; + use std::path::PathBuf; + + #[test] + fn equivalent_cwd_spellings_share_workspace_sid_key() { + let temp = tempfile::tempdir().expect("tempdir"); + let codex_home = temp.path().join("codex-home"); + std::fs::create_dir_all(&codex_home).expect("create codex home"); + + let workspace = temp.path().join("WorkspaceRoot"); + std::fs::create_dir_all(&workspace).expect("create workspace root"); + + let canonical = dunce::canonicalize(&workspace).expect("canonical workspace root"); + let alt_spelling = PathBuf::from(canonical.to_string_lossy().replace('\\', "/").to_ascii_uppercase()); + + let first_sid = + workspace_cap_sid_for_cwd(&codex_home, canonical.as_path()).expect("first sid"); + let second_sid = + workspace_cap_sid_for_cwd(&codex_home, alt_spelling.as_path()).expect("second sid"); + + assert_eq!(first_sid, second_sid); + + let caps = load_or_create_cap_sids(&codex_home).expect("load caps"); + assert_eq!(caps.workspace_by_cwd.len(), 1); + } +} diff --git a/codex-rs/windows-sandbox-rs/src/command_runner_win.rs b/codex-rs/windows-sandbox-rs/src/command_runner_win.rs index a016c00c75..9ff61de932 100644 --- a/codex-rs/windows-sandbox-rs/src/command_runner_win.rs +++ b/codex-rs/windows-sandbox-rs/src/command_runner_win.rs @@ -5,8 +5,8 @@ use anyhow::Result; use codex_windows_sandbox::allow_null_device; use codex_windows_sandbox::convert_string_sid_to_sid; use codex_windows_sandbox::create_process_as_user; -use codex_windows_sandbox::create_readonly_token_with_cap_from; -use codex_windows_sandbox::create_workspace_write_token_with_cap_from; +use codex_windows_sandbox::create_readonly_token_with_caps_from; +use codex_windows_sandbox::create_workspace_write_token_with_caps_from; use codex_windows_sandbox::get_current_token_for_restriction; use codex_windows_sandbox::hide_current_user_profile_dir; use codex_windows_sandbox::log_note; @@ -20,7 +20,9 @@ use std::path::Path; use std::path::PathBuf; use windows_sys::Win32::Foundation::CloseHandle; use windows_sys::Win32::Foundation::GetLastError; +use windows_sys::Win32::Foundation::LocalFree; use windows_sys::Win32::Foundation::HANDLE; +use windows_sys::Win32::Foundation::HLOCAL; use windows_sys::Win32::Storage::FileSystem::CreateFileW; use windows_sys::Win32::Storage::FileSystem::FILE_GENERIC_READ; use windows_sys::Win32::Storage::FileSystem::FILE_GENERIC_WRITE; @@ -48,7 +50,7 @@ struct RunnerRequest { codex_home: PathBuf, // Real user's CODEX_HOME for shared data (caps, config). real_codex_home: PathBuf, - cap_sid: String, + cap_sids: Vec, command: Vec, cwd: PathBuf, env_map: HashMap, @@ -112,27 +114,43 @@ pub fn main() -> Result<()> { ); let policy = parse_policy(&req.policy_json_or_preset).context("parse policy_json_or_preset")?; - let psid_cap: *mut c_void = unsafe { convert_string_sid_to_sid(&req.cap_sid).unwrap() }; + let mut cap_psids: Vec<*mut c_void> = Vec::new(); + for sid in &req.cap_sids { + let Some(psid) = (unsafe { convert_string_sid_to_sid(sid) }) else { + anyhow::bail!("ConvertStringSidToSidW failed for capability SID"); + }; + cap_psids.push(psid); + } + if cap_psids.is_empty() { + anyhow::bail!("runner: empty capability SID list"); + } // Create restricted token from current process token. let base = unsafe { get_current_token_for_restriction()? }; - let token_res: Result<(HANDLE, *mut c_void)> = unsafe { + let token_res: Result = unsafe { match &policy { - SandboxPolicy::ReadOnly => create_readonly_token_with_cap_from(base, psid_cap), + SandboxPolicy::ReadOnly => create_readonly_token_with_caps_from(base, &cap_psids), SandboxPolicy::WorkspaceWrite { .. } => { - create_workspace_write_token_with_cap_from(base, psid_cap) + create_workspace_write_token_with_caps_from(base, &cap_psids) } SandboxPolicy::DangerFullAccess | SandboxPolicy::ExternalSandbox { .. } => { unreachable!() } } }; - let (h_token, psid_to_use) = token_res?; + let h_token = token_res?; unsafe { CloseHandle(base); } unsafe { - allow_null_device(psid_to_use); + for psid in &cap_psids { + allow_null_device(*psid); + } + for psid in cap_psids { + if !psid.is_null() { + LocalFree(psid as HLOCAL); + } + } } // Open named pipes for stdio. diff --git a/codex-rs/windows-sandbox-rs/src/elevated_impl.rs b/codex-rs/windows-sandbox-rs/src/elevated_impl.rs index 7dfcd6e0ae..ecd5732c09 100644 --- a/codex-rs/windows-sandbox-rs/src/elevated_impl.rs +++ b/codex-rs/windows-sandbox-rs/src/elevated_impl.rs @@ -196,7 +196,7 @@ mod windows_impl { codex_home: PathBuf, // Real user's CODEX_HOME for shared data (caps, config). real_codex_home: PathBuf, - cap_sid: String, + cap_sids: Vec, request_file: Option, command: Vec, cwd: PathBuf, @@ -239,14 +239,17 @@ mod windows_impl { anyhow::bail!("DangerFullAccess and ExternalSandbox are not supported for sandboxing") } let caps = load_or_create_cap_sids(codex_home)?; - let (psid_to_use, cap_sid_str) = match &policy { + let (psid_to_use, cap_sids) = match &policy { SandboxPolicy::ReadOnly => ( unsafe { convert_string_sid_to_sid(&caps.readonly).unwrap() }, - caps.readonly.clone(), + vec![caps.readonly.clone()], ), SandboxPolicy::WorkspaceWrite { .. } => ( unsafe { convert_string_sid_to_sid(&caps.workspace).unwrap() }, - caps.workspace.clone(), + vec![ + caps.workspace.clone(), + crate::cap::workspace_cap_sid_for_cwd(codex_home, cwd)?, + ], ), SandboxPolicy::DangerFullAccess | SandboxPolicy::ExternalSandbox { .. } => { unreachable!("DangerFullAccess handled above") @@ -294,7 +297,7 @@ mod windows_impl { sandbox_policy_cwd: sandbox_policy_cwd.to_path_buf(), codex_home: sandbox_base.clone(), real_codex_home: codex_home.to_path_buf(), - cap_sid: cap_sid_str.clone(), + cap_sids: cap_sids.clone(), request_file: Some(req_file.clone()), command: command.clone(), cwd: cwd.to_path_buf(), diff --git a/codex-rs/windows-sandbox-rs/src/lib.rs b/codex-rs/windows-sandbox-rs/src/lib.rs index fc062d6fd5..b938a86e6b 100644 --- a/codex-rs/windows-sandbox-rs/src/lib.rs +++ b/codex-rs/windows-sandbox-rs/src/lib.rs @@ -5,8 +5,21 @@ macro_rules! windows_modules { } windows_modules!( - acl, allow, audit, cap, dpapi, env, hide_users, identity, logging, policy, process, token, - winutil + acl, + allow, + audit, + cap, + dpapi, + env, + hide_users, + identity, + logging, + path_normalization, + policy, + process, + token, + winutil, + workspace_acl ); #[cfg(target_os = "windows")] @@ -19,6 +32,8 @@ mod elevated_impl; #[cfg(target_os = "windows")] mod setup_error; +#[cfg(target_os = "windows")] +pub use acl::add_deny_write_ace; #[cfg(target_os = "windows")] pub use acl::allow_null_device; #[cfg(target_os = "windows")] @@ -36,6 +51,8 @@ pub use audit::apply_world_writable_scan_and_denies; #[cfg(target_os = "windows")] pub use cap::load_or_create_cap_sids; #[cfg(target_os = "windows")] +pub use cap::workspace_cap_sid_for_cwd; +#[cfg(target_os = "windows")] pub use dpapi::protect as dpapi_protect; #[cfg(target_os = "windows")] pub use dpapi::unprotect as dpapi_unprotect; @@ -54,6 +71,8 @@ pub use logging::log_note; #[cfg(target_os = "windows")] pub use logging::LOG_FILE_NAME; #[cfg(target_os = "windows")] +pub use path_normalization::canonicalize_path; +#[cfg(target_os = "windows")] pub use policy::parse_policy; #[cfg(target_os = "windows")] pub use policy::SandboxPolicy; @@ -88,7 +107,9 @@ pub use token::convert_string_sid_to_sid; #[cfg(target_os = "windows")] pub use token::create_readonly_token_with_cap_from; #[cfg(target_os = "windows")] -pub use token::create_workspace_write_token_with_cap_from; +pub use token::create_readonly_token_with_caps_from; +#[cfg(target_os = "windows")] +pub use token::create_workspace_write_token_with_caps_from; #[cfg(target_os = "windows")] pub use token::get_current_token_for_restriction; #[cfg(target_os = "windows")] @@ -99,6 +120,10 @@ pub use windows_impl::CaptureResult; pub use winutil::string_from_sid_bytes; #[cfg(target_os = "windows")] pub use winutil::to_wide; +#[cfg(target_os = "windows")] +pub use workspace_acl::is_command_cwd_root; +#[cfg(target_os = "windows")] +pub use workspace_acl::protect_workspace_codex_dir; #[cfg(not(target_os = "windows"))] pub use stub::apply_world_writable_scan_and_denies; @@ -116,6 +141,7 @@ mod windows_impl { use super::allow::compute_allow_paths; use super::allow::AllowDenyPaths; use super::cap::load_or_create_cap_sids; + use super::cap::workspace_cap_sid_for_cwd; use super::env::apply_no_network_to_env; use super::env::ensure_non_interactive_pager; use super::env::normalize_null_device_env; @@ -123,13 +149,17 @@ mod windows_impl { use super::logging::log_failure; use super::logging::log_start; use super::logging::log_success; + use super::path_normalization::canonicalize_path; use super::policy::parse_policy; use super::policy::SandboxPolicy; use super::process::make_env_block; use super::token::convert_string_sid_to_sid; + use super::token::create_workspace_write_token_with_caps_from; use super::winutil::format_last_error; use super::winutil::quote_windows_arg; use super::winutil::to_wide; + use super::workspace_acl::is_command_cwd_root; + use super::workspace_acl::protect_workspace_codex_dir; use anyhow::Result; use std::collections::HashMap; use std::ffi::c_void; @@ -229,15 +259,25 @@ mod windows_impl { anyhow::bail!("DangerFullAccess and ExternalSandbox are not supported for sandboxing") } let caps = load_or_create_cap_sids(codex_home)?; - let (h_token, psid_to_use): (HANDLE, *mut c_void) = unsafe { + let (h_token, psid_generic, psid_workspace): (HANDLE, *mut c_void, Option<*mut c_void>) = unsafe { match &policy { SandboxPolicy::ReadOnly => { let psid = convert_string_sid_to_sid(&caps.readonly).unwrap(); - super::token::create_readonly_token_with_cap(psid)? + let (h, _) = super::token::create_readonly_token_with_cap(psid)?; + (h, psid, None) } SandboxPolicy::WorkspaceWrite { .. } => { - let psid = convert_string_sid_to_sid(&caps.workspace).unwrap(); - super::token::create_workspace_write_token_with_cap(psid)? + let psid_generic = convert_string_sid_to_sid(&caps.workspace).unwrap(); + let ws_sid = workspace_cap_sid_for_cwd(codex_home, cwd)?; + let psid_workspace = convert_string_sid_to_sid(&ws_sid).unwrap(); + let base = super::token::get_current_token_for_restriction()?; + let h_res = create_workspace_write_token_with_caps_from( + base, + &[psid_generic, psid_workspace], + ); + windows_sys::Win32::Foundation::CloseHandle(base); + let h = h_res?; + (h, psid_generic, Some(psid_workspace)) } SandboxPolicy::DangerFullAccess | SandboxPolicy::ExternalSandbox { .. } => { unreachable!("DangerFullAccess handled above") @@ -261,29 +301,39 @@ mod windows_impl { let persist_aces = is_workspace_write; let AllowDenyPaths { allow, deny } = compute_allow_paths(&policy, sandbox_policy_cwd, ¤t_dir, &env_map); + let canonical_cwd = canonicalize_path(¤t_dir); let mut guards: Vec<(PathBuf, *mut c_void)> = Vec::new(); unsafe { for p in &allow { - if let Ok(added) = add_allow_ace(p, psid_to_use) { + let psid = if is_workspace_write && is_command_cwd_root(p, &canonical_cwd) { + psid_workspace.unwrap_or(psid_generic) + } else { + psid_generic + }; + if let Ok(added) = add_allow_ace(p, psid) { if added { if persist_aces { if p.is_dir() { // best-effort seeding omitted intentionally } } else { - guards.push((p.clone(), psid_to_use)); + guards.push((p.clone(), psid)); } } } } for p in &deny { - if let Ok(added) = add_deny_write_ace(p, psid_to_use) { + if let Ok(added) = add_deny_write_ace(p, psid_generic) { if added && !persist_aces { - guards.push((p.clone(), psid_to_use)); + guards.push((p.clone(), psid_generic)); } } } - allow_null_device(psid_to_use); + allow_null_device(psid_generic); + if let Some(psid) = psid_workspace { + allow_null_device(psid); + let _ = protect_workspace_codex_dir(¤t_dir, psid); + } } let (stdin_pair, stdout_pair, stderr_pair) = unsafe { setup_stdio_pipes()? }; diff --git a/codex-rs/windows-sandbox-rs/src/path_normalization.rs b/codex-rs/windows-sandbox-rs/src/path_normalization.rs new file mode 100644 index 0000000000..fe6a932306 --- /dev/null +++ b/codex-rs/windows-sandbox-rs/src/path_normalization.rs @@ -0,0 +1,28 @@ +use std::path::Path; +use std::path::PathBuf; + +pub fn canonicalize_path(path: &Path) -> PathBuf { + dunce::canonicalize(path).unwrap_or_else(|_| path.to_path_buf()) +} + +pub fn canonical_path_key(path: &Path) -> String { + canonicalize_path(path) + .to_string_lossy() + .replace('\\', "/") + .to_ascii_lowercase() +} + +#[cfg(test)] +mod tests { + use super::canonical_path_key; + use pretty_assertions::assert_eq; + use std::path::Path; + + #[test] + fn canonical_path_key_normalizes_case_and_separators() { + let windows_style = Path::new(r"C:\Users\Dev\Repo"); + let slash_style = Path::new("c:/users/dev/repo"); + + assert_eq!(canonical_path_key(windows_style), canonical_path_key(slash_style)); + } +} diff --git a/codex-rs/windows-sandbox-rs/src/setup_main_win.rs b/codex-rs/windows-sandbox-rs/src/setup_main_win.rs index 6868028782..28c94a769c 100644 --- a/codex-rs/windows-sandbox-rs/src/setup_main_win.rs +++ b/codex-rs/windows-sandbox-rs/src/setup_main_win.rs @@ -6,18 +6,22 @@ use anyhow::Context; use anyhow::Result; use base64::engine::general_purpose::STANDARD as BASE64; use base64::Engine; +use codex_windows_sandbox::canonicalize_path; use codex_windows_sandbox::convert_string_sid_to_sid; use codex_windows_sandbox::ensure_allow_mask_aces_with_inheritance; use codex_windows_sandbox::ensure_allow_write_aces; use codex_windows_sandbox::extract_setup_failure; use codex_windows_sandbox::hide_newly_created_users; +use codex_windows_sandbox::is_command_cwd_root; use codex_windows_sandbox::load_or_create_cap_sids; use codex_windows_sandbox::log_note; use codex_windows_sandbox::path_mask_allows; +use codex_windows_sandbox::protect_workspace_codex_dir; use codex_windows_sandbox::sandbox_dir; use codex_windows_sandbox::sandbox_secrets_dir; use codex_windows_sandbox::string_from_sid_bytes; use codex_windows_sandbox::to_wide; +use codex_windows_sandbox::workspace_cap_sid_for_cwd; use codex_windows_sandbox::write_setup_error_report; use codex_windows_sandbox::SetupErrorCode; use codex_windows_sandbox::SetupErrorReport; @@ -75,6 +79,7 @@ struct Payload { offline_username: String, online_username: String, codex_home: PathBuf, + command_cwd: PathBuf, read_roots: Vec, write_roots: Vec, real_user: String, @@ -560,6 +565,11 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( )) })? }; + let workspace_sid_str = workspace_cap_sid_for_cwd(&payload.codex_home, &payload.command_cwd)?; + let workspace_psid = unsafe { + convert_string_sid_to_sid(&workspace_sid_str) + .ok_or_else(|| anyhow::anyhow!("convert workspace capability SID failed"))? + }; let mut refresh_errors: Vec = Vec::new(); if !refresh_only { let firewall_result = firewall::ensure_offline_outbound_block(&offline_sid_str, log); @@ -609,12 +619,12 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( let cap_sid_str = caps.workspace.clone(); let sandbox_group_sid_str = string_from_sid_bytes(&sandbox_group_sid).map_err(anyhow::Error::msg)?; - let sid_strings = vec![sandbox_group_sid_str, cap_sid_str]; let write_mask = FILE_GENERIC_READ | FILE_GENERIC_WRITE | FILE_GENERIC_EXECUTE | DELETE | FILE_DELETE_CHILD; let mut grant_tasks: Vec = Vec::new(); let mut seen_write_roots: HashSet = HashSet::new(); + let canonical_command_cwd = canonicalize_path(&payload.command_cwd); for root in &payload.write_roots { if !seen_write_roots.insert(root.clone()) { @@ -628,7 +638,21 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( continue; } let mut need_grant = false; - for (label, psid) in [("sandbox_group", sandbox_group_psid), ("cap", cap_psid)] { + let is_command_cwd = is_command_cwd_root(root, &canonical_command_cwd); + let cap_label = if is_command_cwd { + "workspace_cap" + } else { + "cap" + }; + let cap_psid_for_root = if is_command_cwd { + workspace_psid + } else { + cap_psid + }; + for (label, psid) in [ + ("sandbox_group", sandbox_group_psid), + (cap_label, cap_psid_for_root), + ] { let has = match path_mask_allows(root, &[psid], write_mask, true) { Ok(h) => h, Err(e) => { @@ -667,7 +691,12 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( let (tx, rx) = mpsc::channel::<(PathBuf, Result)>(); std::thread::scope(|scope| { for root in grant_tasks { - let sid_strings = sid_strings.clone(); + let is_command_cwd = is_command_cwd_root(&root, &canonical_command_cwd); + let sid_strings = if is_command_cwd { + vec![sandbox_group_sid_str.clone(), workspace_sid_str.clone()] + } else { + vec![sandbox_group_sid_str.clone(), cap_sid_str.clone()] + }; let tx = tx.clone(); scope.spawn(move || { // Convert SID strings to psids locally in this thread. @@ -758,6 +787,31 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( let _ = std::fs::remove_file(&legacy_users); } } + + // Protect the current workspace's `.codex` directory from tampering (write/delete) by using a + // workspace-specific capability SID. If `.codex` doesn't exist yet, skip it (it will be picked + // up on the next refresh). + match unsafe { protect_workspace_codex_dir(&payload.command_cwd, workspace_psid) } { + Ok(true) => { + let cwd_codex = payload.command_cwd.join(".codex"); + log_line( + log, + &format!( + "applied deny ACE to protect workspace .codex {}", + cwd_codex.display() + ), + )?; + } + Ok(false) => {} + Err(err) => { + let cwd_codex = payload.command_cwd.join(".codex"); + refresh_errors.push(format!("deny ACE failed on {}: {err}", cwd_codex.display())); + log_line( + log, + &format!("deny ACE failed on {}: {err}", cwd_codex.display()), + )?; + } + } unsafe { if !sandbox_group_psid.is_null() { LocalFree(sandbox_group_psid as HLOCAL); @@ -765,6 +819,9 @@ fn run_setup_full(payload: &Payload, log: &mut File, sbx_dir: &Path) -> Result<( if !cap_psid.is_null() { LocalFree(cap_psid as HLOCAL); } + if !workspace_psid.is_null() { + LocalFree(workspace_psid as HLOCAL); + } } if refresh_only && !refresh_errors.is_empty() { log_line( diff --git a/codex-rs/windows-sandbox-rs/src/setup_orchestrator.rs b/codex-rs/windows-sandbox-rs/src/setup_orchestrator.rs index 78e6d566fb..0e43d1ba4b 100644 --- a/codex-rs/windows-sandbox-rs/src/setup_orchestrator.rs +++ b/codex-rs/windows-sandbox-rs/src/setup_orchestrator.rs @@ -12,6 +12,7 @@ use std::process::Stdio; use crate::allow::compute_allow_paths; use crate::allow::AllowDenyPaths; use crate::logging::log_note; +use crate::path_normalization::canonical_path_key; use crate::policy::SandboxPolicy; use crate::setup_error::clear_setup_error_report; use crate::setup_error::failure; @@ -82,6 +83,7 @@ pub fn run_setup_refresh( offline_username: OFFLINE_USERNAME.to_string(), online_username: ONLINE_USERNAME.to_string(), codex_home: codex_home.to_path_buf(), + command_cwd: command_cwd.to_path_buf(), read_roots, write_roots, real_user: std::env::var("USERNAME").unwrap_or_else(|_| "Administrators".to_string()), @@ -259,6 +261,7 @@ struct ElevationPayload { offline_username: String, online_username: String, codex_home: PathBuf, + command_cwd: PathBuf, read_roots: Vec, write_roots: Vec, real_user: String, @@ -475,6 +478,7 @@ pub fn run_elevated_setup( offline_username: OFFLINE_USERNAME.to_string(), online_username: ONLINE_USERNAME.to_string(), codex_home: codex_home.to_path_buf(), + command_cwd: command_cwd.to_path_buf(), read_roots, write_roots, real_user: std::env::var("USERNAME").unwrap_or_else(|_| "Administrators".to_string()), @@ -517,14 +521,14 @@ fn build_payload_roots( fn filter_sensitive_write_roots(mut roots: Vec, codex_home: &Path) -> Vec { // Never grant capability write access to CODEX_HOME or anything under CODEX_HOME/.sandbox. // These locations contain sandbox control/state and must remain tamper-resistant. - let codex_home_key = crate::audit::normalize_path_key(codex_home); - let sbx_dir_key = crate::audit::normalize_path_key(&sandbox_dir(codex_home)); + let codex_home_key = canonical_path_key(codex_home); + let sbx_dir_key = canonical_path_key(&sandbox_dir(codex_home)); let sbx_dir_prefix = format!("{}/", sbx_dir_key.trim_end_matches('/')); - let secrets_dir_key = crate::audit::normalize_path_key(&sandbox_secrets_dir(codex_home)); + let secrets_dir_key = canonical_path_key(&sandbox_secrets_dir(codex_home)); let secrets_dir_prefix = format!("{}/", secrets_dir_key.trim_end_matches('/')); roots.retain(|root| { - let key = crate::audit::normalize_path_key(root); + let key = canonical_path_key(root); key != codex_home_key && key != sbx_dir_key && !key.starts_with(&sbx_dir_prefix) diff --git a/codex-rs/windows-sandbox-rs/src/token.rs b/codex-rs/windows-sandbox-rs/src/token.rs index fc0e840eb5..0cc9f59da4 100644 --- a/codex-rs/windows-sandbox-rs/src/token.rs +++ b/codex-rs/windows-sandbox-rs/src/token.rs @@ -282,17 +282,6 @@ unsafe fn enable_single_privilege(h_token: HANDLE, name: &str) -> Result<()> { Ok(()) } -/// # Safety -/// Caller must close the returned token handle. -pub unsafe fn create_workspace_write_token_with_cap( - psid_capability: *mut c_void, -) -> Result<(HANDLE, *mut c_void)> { - let base = get_current_token_for_restriction()?; - let res = create_workspace_write_token_with_cap_from(base, psid_capability); - CloseHandle(base); - res -} - /// # Safety /// Caller must close the returned token handle. pub unsafe fn create_readonly_token_with_cap( @@ -306,61 +295,63 @@ pub unsafe fn create_readonly_token_with_cap( /// # Safety /// Caller must close the returned token handle; base_token must be a valid primary token. -pub unsafe fn create_workspace_write_token_with_cap_from( - base_token: HANDLE, - psid_capability: *mut c_void, -) -> Result<(HANDLE, *mut c_void)> { - let mut logon_sid_bytes = get_logon_sid_bytes(base_token)?; - let psid_logon = logon_sid_bytes.as_mut_ptr() as *mut c_void; - let mut everyone = world_sid()?; - let psid_everyone = everyone.as_mut_ptr() as *mut c_void; - let mut entries: [SID_AND_ATTRIBUTES; 3] = std::mem::zeroed(); - // Exact set and order: Capability, Logon, Everyone - entries[0].Sid = psid_capability; - entries[0].Attributes = 0; - entries[1].Sid = psid_logon; - entries[1].Attributes = 0; - entries[2].Sid = psid_everyone; - entries[2].Attributes = 0; - let mut new_token: HANDLE = 0; - let flags = DISABLE_MAX_PRIVILEGE | LUA_TOKEN | WRITE_RESTRICTED; - let ok = CreateRestrictedToken( - base_token, - flags, - 0, - std::ptr::null(), - 0, - std::ptr::null(), - 3, - entries.as_mut_ptr(), - &mut new_token, - ); - if ok == 0 { - return Err(anyhow!("CreateRestrictedToken failed: {}", GetLastError())); - } - set_default_dacl(new_token, &[psid_logon, psid_everyone, psid_capability])?; - enable_single_privilege(new_token, "SeChangeNotifyPrivilege")?; - Ok((new_token, psid_capability)) -} - /// # Safety /// Caller must close the returned token handle; base_token must be a valid primary token. pub unsafe fn create_readonly_token_with_cap_from( base_token: HANDLE, psid_capability: *mut c_void, ) -> Result<(HANDLE, *mut c_void)> { + let new_token = create_token_with_caps_from(base_token, &[psid_capability])?; + Ok((new_token, psid_capability)) +} + +/// Create a restricted token that includes all provided capability SIDs. +/// +/// # Safety +/// Caller must close the returned token handle; base_token must be a valid primary token. +pub unsafe fn create_workspace_write_token_with_caps_from( + base_token: HANDLE, + psid_capabilities: &[*mut c_void], +) -> Result { + create_token_with_caps_from(base_token, psid_capabilities) +} + +/// Create a restricted token that includes all provided capability SIDs. +/// +/// # Safety +/// Caller must close the returned token handle; base_token must be a valid primary token. +pub unsafe fn create_readonly_token_with_caps_from( + base_token: HANDLE, + psid_capabilities: &[*mut c_void], +) -> Result { + create_token_with_caps_from(base_token, psid_capabilities) +} + +unsafe fn create_token_with_caps_from( + base_token: HANDLE, + psid_capabilities: &[*mut c_void], +) -> Result { + if psid_capabilities.is_empty() { + return Err(anyhow!("no capability SIDs provided")); + } let mut logon_sid_bytes = get_logon_sid_bytes(base_token)?; let psid_logon = logon_sid_bytes.as_mut_ptr() as *mut c_void; let mut everyone = world_sid()?; let psid_everyone = everyone.as_mut_ptr() as *mut c_void; - let mut entries: [SID_AND_ATTRIBUTES; 3] = std::mem::zeroed(); - // Exact set and order: Capability, Logon, Everyone - entries[0].Sid = psid_capability; - entries[0].Attributes = 0; - entries[1].Sid = psid_logon; - entries[1].Attributes = 0; - entries[2].Sid = psid_everyone; - entries[2].Attributes = 0; + + // Exact order: Capabilities..., Logon, Everyone + let mut entries: Vec = + vec![std::mem::zeroed(); psid_capabilities.len() + 2]; + for (i, psid) in psid_capabilities.iter().enumerate() { + entries[i].Sid = *psid; + entries[i].Attributes = 0; + } + let logon_idx = psid_capabilities.len(); + entries[logon_idx].Sid = psid_logon; + entries[logon_idx].Attributes = 0; + entries[logon_idx + 1].Sid = psid_everyone; + entries[logon_idx + 1].Attributes = 0; + let mut new_token: HANDLE = 0; let flags = DISABLE_MAX_PRIVILEGE | LUA_TOKEN | WRITE_RESTRICTED; let ok = CreateRestrictedToken( @@ -370,14 +361,20 @@ pub unsafe fn create_readonly_token_with_cap_from( std::ptr::null(), 0, std::ptr::null(), - 3, + entries.len() as u32, entries.as_mut_ptr(), &mut new_token, ); if ok == 0 { return Err(anyhow!("CreateRestrictedToken failed: {}", GetLastError())); } - set_default_dacl(new_token, &[psid_logon, psid_everyone, psid_capability])?; + + let mut dacl_sids: Vec<*mut c_void> = Vec::with_capacity(psid_capabilities.len() + 2); + dacl_sids.push(psid_logon); + dacl_sids.push(psid_everyone); + dacl_sids.extend_from_slice(psid_capabilities); + set_default_dacl(new_token, &dacl_sids)?; + enable_single_privilege(new_token, "SeChangeNotifyPrivilege")?; - Ok((new_token, psid_capability)) + Ok(new_token) } diff --git a/codex-rs/windows-sandbox-rs/src/workspace_acl.rs b/codex-rs/windows-sandbox-rs/src/workspace_acl.rs new file mode 100644 index 0000000000..b34a7191ad --- /dev/null +++ b/codex-rs/windows-sandbox-rs/src/workspace_acl.rs @@ -0,0 +1,20 @@ +use crate::acl::add_deny_write_ace; +use crate::path_normalization::canonicalize_path; +use anyhow::Result; +use std::ffi::c_void; +use std::path::Path; + +pub fn is_command_cwd_root(root: &Path, canonical_command_cwd: &Path) -> bool { + canonicalize_path(root) == canonical_command_cwd +} + +/// # Safety +/// Caller must ensure `psid` is a valid SID pointer. +pub unsafe fn protect_workspace_codex_dir(cwd: &Path, psid: *mut c_void) -> Result { + let cwd_codex = cwd.join(".codex"); + if cwd_codex.is_dir() { + add_deny_write_ace(&cwd_codex, psid) + } else { + Ok(false) + } +} diff --git a/docs/tui-request-user-input.md b/docs/tui-request-user-input.md index f8eb7aeb73..8ca6f5369b 100644 --- a/docs/tui-request-user-input.md +++ b/docs/tui-request-user-input.md @@ -30,7 +30,9 @@ friction for freeform input. - Enter advances to the next question. - Enter on the last question submits all answers. - PageUp/PageDown navigate across questions (when multiple are present). -- Esc interrupts the run. +- Esc interrupts the run in option selection mode. +- When notes are open for an option question, Tab or Esc clears notes and returns + to option selection. ## Layout priorities diff --git a/docs/tui-stream-chunking-review.md b/docs/tui-stream-chunking-review.md new file mode 100644 index 0000000000..3722492ddf --- /dev/null +++ b/docs/tui-stream-chunking-review.md @@ -0,0 +1,124 @@ +# TUI Stream Chunking + +This document explains how stream chunking in the TUI works and why it is +implemented this way. + +## Problem + +Streaming output can arrive faster than a one-line-per-tick animation can show +it. If commit speed stays fixed while arrival speed spikes, queued lines grow +and visible output lags behind received output. + +## Design goals + +- Preserve existing baseline behavior under normal load. +- Reduce display lag when backlog builds. +- Keep output order stable. +- Avoid abrupt single-frame flushes that look jumpy. +- Keep policy transport-agnostic and based only on queue state. + +## Non-goals + +- The policy does not schedule animation ticks. +- The policy does not depend on upstream source identity. +- The policy does not reorder queued output. + +## Where the logic lives + +- `codex-rs/tui/src/streaming/chunking.rs` + - Adaptive policy, mode transitions, and drain-plan selection. +- `codex-rs/tui/src/streaming/commit_tick.rs` + - Orchestration for each commit tick: snapshot, decide, drain, trace. +- `codex-rs/tui/src/streaming/controller.rs` + - Queue/drain primitives used by commit-tick orchestration. +- `codex-rs/tui/src/chatwidget.rs` + - Integration point that invokes commit-tick orchestration and handles UI + lifecycle events. + +## Runtime flow + +On each commit tick: + +1. Build a queue snapshot across active controllers. + - `queued_lines`: total queued lines. + - `oldest_age`: max age of the oldest queued line across controllers. +2. Ask adaptive policy for a decision. + - Output: current mode and a drain plan. +3. Apply drain plan to each controller. +4. Emit drained `HistoryCell`s for insertion by the caller. +5. Emit trace logs for observability. + +In `CatchUpOnly` scope, policy state still advances, but draining is skipped +unless mode is currently `CatchUp`. + +## Modes and transitions + +Two modes are used: + +- `Smooth` + - Baseline behavior: one line drained per baseline commit tick. + - Baseline tick interval currently comes from + `tui/src/app.rs:COMMIT_ANIMATION_TICK` (~8.3ms, ~120fps). +- `CatchUp` + - Drain current queued backlog per tick via `Batch(queued_lines)`. + +Entry and exit use hysteresis: + +- Enter `CatchUp` when queue depth or queue age exceeds enter thresholds. +- Exit requires both depth and age to be below exit thresholds for a hold + window (`EXIT_HOLD`). + +This prevents oscillation when load hovers near thresholds. + +## Current experimental tuning values + +These are the current values in `streaming/chunking.rs` plus the baseline +commit tick in `tui/src/app.rs`. They are +experimental and may change as we gather more trace data. + +- Baseline commit tick: `~8.3ms` (`COMMIT_ANIMATION_TICK` in `app.rs`) +- Enter catch-up: + - `queued_lines >= 8` OR `oldest_age >= 120ms` +- Exit catch-up eligibility: + - `queued_lines <= 2` AND `oldest_age <= 40ms` +- Exit hold (`CatchUp -> Smooth`): `250ms` +- Re-entry hold after catch-up exit: `250ms` +- Severe backlog thresholds: + - `queued_lines >= 64` OR `oldest_age >= 300ms` + +## Drain planning + +In `Smooth`, plan is always `Single`. + +In `CatchUp`, plan is `Batch(queued_lines)`, which drains the currently queued +backlog for immediate convergence. + +## Why this design + +This keeps normal animation semantics intact, while making backlog behavior +adaptive: + +- Under normal load, behavior stays familiar and stable. +- Under pressure, queue age is reduced quickly without sacrificing ordering. +- Hysteresis avoids rapid mode flapping. + +## Invariants + +- Queue order is preserved. +- Empty queue resets policy back to `Smooth`. +- `CatchUp` exits only after sustained low pressure. +- Catch-up drains are immediate while in `CatchUp`. + +## Observability + +Trace events are emitted from commit-tick orchestration: + +- `stream chunking commit tick` + - `mode`, `queued_lines`, `oldest_queued_age_ms`, `drain_plan`, + `has_controller`, `all_idle` +- `stream chunking mode transition` + - `prior_mode`, `new_mode`, `queued_lines`, `oldest_queued_age_ms`, + `entered_catch_up` + +These events are intended to explain display lag by showing queue pressure, +selected drain behavior, and mode transitions over time. diff --git a/docs/tui-stream-chunking-tuning.md b/docs/tui-stream-chunking-tuning.md new file mode 100644 index 0000000000..d9a2ea5e21 --- /dev/null +++ b/docs/tui-stream-chunking-tuning.md @@ -0,0 +1,98 @@ +# TUI Stream Chunking Tuning Guide + +This document explains how to tune adaptive stream chunking constants without +changing the underlying policy shape. + +## Scope + +Use this guide when adjusting queue-pressure thresholds and hysteresis windows in +`codex-rs/tui/src/streaming/chunking.rs`, and baseline commit cadence in +`codex-rs/tui/src/app.rs`. + +This guide is about tuning behavior, not redesigning the policy. + +## Before tuning + +- Keep the baseline behavior intact: + - `Smooth` mode drains one line per baseline tick. + - `CatchUp` mode drains queued backlog immediately. +- Capture trace logs with: + - `codex_tui::streaming::commit_tick` +- Evaluate on sustained, bursty, and mixed-output prompts. + +See `docs/tui-stream-chunking-validation.md` for the measurement process. + +## Tuning goals + +Tune for all three goals together: + +- low visible lag under bursty output +- low mode flapping (`Smooth <-> CatchUp` chatter) +- stable catch-up entry/exit behavior under mixed workloads + +## Constants and what they control + +### Baseline commit cadence + +- `COMMIT_ANIMATION_TICK` (`tui/src/app.rs`) + - Lower values increase smooth-mode update cadence and reduce steady-state lag. + - Higher values increase smoothing and can increase perceived lag. + - This should usually move after chunking thresholds/holds are in a good range. + +### Enter/exit thresholds + +- `ENTER_QUEUE_DEPTH_LINES`, `ENTER_OLDEST_AGE` + - Lower values enter catch-up earlier (less lag, more mode switching risk). + - Higher values enter later (more lag tolerance, fewer mode switches). +- `EXIT_QUEUE_DEPTH_LINES`, `EXIT_OLDEST_AGE` + - Lower values keep catch-up active longer. + - Higher values allow earlier exit and may increase re-entry churn. + +### Hysteresis holds + +- `EXIT_HOLD` + - Longer hold reduces flip-flop exits when pressure is noisy. + - Too long can keep catch-up active after pressure has cleared. +- `REENTER_CATCH_UP_HOLD` + - Longer hold suppresses rapid re-entry after exit. + - Too long can delay needed catch-up for near-term bursts. + - Severe backlog bypasses this hold by design. + +### Severe-backlog gates + +- `SEVERE_QUEUE_DEPTH_LINES`, `SEVERE_OLDEST_AGE` + - Lower values bypass re-entry hold earlier. + - Higher values reserve hold bypass for only extreme pressure. + +## Recommended tuning order + +Tune in this order to keep cause/effect clear: + +1. Entry/exit thresholds (`ENTER_*`, `EXIT_*`) +2. Hold windows (`EXIT_HOLD`, `REENTER_CATCH_UP_HOLD`) +3. Severe gates (`SEVERE_*`) +4. Baseline cadence (`COMMIT_ANIMATION_TICK`) + +Change one logical group at a time and re-measure before the next group. + +## Symptom-driven adjustments + +- Too much lag before catch-up starts: + - lower `ENTER_QUEUE_DEPTH_LINES` and/or `ENTER_OLDEST_AGE` +- Frequent `Smooth -> CatchUp -> Smooth` chatter: + - increase `EXIT_HOLD` + - increase `REENTER_CATCH_UP_HOLD` + - tighten exit thresholds (lower `EXIT_*`) +- Catch-up engages too often for short bursts: + - increase `ENTER_QUEUE_DEPTH_LINES` and/or `ENTER_OLDEST_AGE` + - increase `REENTER_CATCH_UP_HOLD` +- Catch-up engages too late: + - lower `ENTER_QUEUE_DEPTH_LINES` and/or `ENTER_OLDEST_AGE` + - lower severe gates (`SEVERE_*`) to bypass re-entry hold sooner + +## Validation checklist after each tuning pass + +- `cargo test -p codex-tui` passes. +- Trace window shows bounded queue-age behavior. +- Mode transitions are not concentrated in repeated short-interval cycles. +- Catch-up clears backlog quickly once mode enters `CatchUp`. diff --git a/docs/tui-stream-chunking-validation.md b/docs/tui-stream-chunking-validation.md new file mode 100644 index 0000000000..26bd949da6 --- /dev/null +++ b/docs/tui-stream-chunking-validation.md @@ -0,0 +1,105 @@ +# TUI Stream Chunking Validation Process + +This document records the process used to validate adaptive stream chunking +and anti-flap behavior. + +## Scope + +The goal is to verify two properties from runtime traces: + +- display lag is reduced when queue pressure rises +- mode transitions remain stable instead of rapidly flapping + +## Trace targets + +Chunking observability is emitted by: + +- `codex_tui::streaming::commit_tick` + +Two trace messages are used: + +- `stream chunking commit tick` +- `stream chunking mode transition` + +## Runtime command + +Run Codex with chunking traces enabled: + +```bash +RUST_LOG='codex_tui::streaming::commit_tick=trace,codex_tui=info,codex_core=info,codex_rmcp_client=info' \ + just codex --enable=responses_websockets +``` + +## Log capture process + +1. Record the current size of `~/.codex/log/codex-tui.log` as a start offset. +2. Run an interactive prompt that produces sustained streamed output. +3. Stop the run. +4. Parse only log bytes written after the recorded offset. + +This avoids mixing earlier sessions with the current measurement window. + +## Metrics reviewed + +For each measured window: + +- `commit_ticks` +- `mode_transitions` +- `smooth_ticks` +- `catchup_ticks` +- drain-plan distribution (`Single`, `Batch(n)`) +- queue depth (`max`, `p95`, `p99`) +- oldest queued age (`max`, `p95`, `p99`) +- rapid re-entry count: + - number of `Smooth -> CatchUp` transitions within 1 second of a + `CatchUp -> Smooth` transition + +## Interpretation + +- Healthy behavior: + - queue age remains bounded while backlog is drained + - transition count is low relative to total ticks + - rapid re-entry events are infrequent and localized to burst boundaries +- Regressed behavior: + - repeated short-interval mode toggles across an extended window + - persistent queue-age growth while in smooth mode + - long catch-up runs without backlog reduction + +## Experiment history + +This section captures the major tuning passes so future work can build on +what has already been tried. + +- Baseline + - One-line smooth draining with a 50ms commit tick. + - This preserved familiar pacing but could feel laggy under sustained + backlog. +- Pass 1: instant catch-up, baseline tick unchanged + - Kept smooth-mode semantics but made catch-up drain the full queued + backlog each catch-up tick. + - Result: queue lag dropped faster, but perceived motion could still feel + stepped because smooth-mode cadence remained coarse. +- Pass 2: faster baseline tick (25ms) + - Improved smooth-mode cadence and reduced visible stepping. + - Result: better, but still not aligned with draw cadence. +- Pass 3: frame-aligned baseline tick (~16.7ms) + - Set baseline commit cadence to approximately 60fps. + - Result: smoother perceived progression while retaining hysteresis and + fast backlog convergence. +- Pass 4: higher frame-aligned baseline tick (~8.3ms) + - Set baseline commit cadence to approximately 120fps. + - Result: further reduced smooth-mode stepping while preserving the same + adaptive catch-up policy shape. + +Current state combines: + +- instant catch-up draining in `CatchUp` +- hysteresis for mode-entry/exit stability +- frame-aligned smooth-mode commit cadence (~8.3ms) + +## Notes + +- Validation is source-agnostic and does not rely on naming any specific + upstream provider. +- This process intentionally preserves existing baseline smooth behavior and + focuses on burst/backlog handling behavior.