diff --git a/20260416_wasm_appserver.md b/20260416_wasm_appserver.md new file mode 100644 index 0000000000..6d8e537b94 --- /dev/null +++ b/20260416_wasm_appserver.md @@ -0,0 +1,427 @@ +# WASM Embedded App-Server Design + +## TL;DR + +The current browser harness wraps a single user submission by creating a thread, calling `thread.submit(Op::UserTurn { ... })`, and draining `thread.next_event()` until the turn completes. That path proved that the core harness can run in the browser, but it leaves the browser with a custom control surface that diverges from native app-server behavior and currently resets session state after every turn. + +This document proposes switching the browser integration to an embedded in-process app-server runtime. The browser would keep a long-lived app-server instance in wasm, communicate with it through the existing app-server request and event model, and reuse the native app-server boundary instead of maintaining a browser-specific harness API. + +## Objective + +Use the existing app-server boundary as the primary control plane for the browser runtime so that: + +- browser and native clients speak the same conceptual protocol +- multi-turn behavior matches native behavior more closely +- debugging and event inspection reuse existing app-server semantics +- future features such as thread lifecycle, steering, approvals, and richer tooling do not require a second browser-specific orchestration layer + +## Background + +### Current Browser Design + +The current browser runtime is centered on `BrowserCodex` in `codex-rs/wasm-harness/src/browser.rs`. + +Its main flow is: + +1. `BrowserCodex.submit_turn(prompt, on_event)` creates a fresh `LocalSet`. +2. It either reuses or creates a `BrowserSession`. +3. It submits a turn directly to core with: + + ```rust + session.thread.submit(Op::UserTurn { ... }).await + ``` + +4. It loops on `session.thread.next_event().await` and forwards events to JS. +5. When the turn finishes, it clears `self.session`. + +This is a thin wrapper around the core thread API. It is effectively a `UserSubmit`-style interface: the browser gives the harness a prompt, the harness starts one turn, then pushes turn events back to JS. + +The relevant behavior today is: + +- `submit_turn()` scopes execution to a per-turn `LocalSet` +- the browser wrapper owns a `BrowserSession { config, thread, session_configured }` +- the wrapper emits raw core events directly to JS +- the wrapper explicitly drops the session after each turn + +### Why The Current Design Is Not Enough + +The current design was useful as a proof of feasibility, but it creates several product and maintenance problems. + +First, it is not actually aligned with the native client boundary. Native clients talk to app-server through requests, notifications, and streamed server events. The browser currently bypasses that layer and talks directly to `CodexThread`. + +Second, it currently loses continuity across turns. Because `submit_turn()` uses a fresh `LocalSet` and clears `self.session` after each submission, the browser prototype starts a new harness session for every turn. That is why follow-up prompts appear to lack memory of earlier prompts. + +Third, it forces the browser to maintain a separate orchestration contract. Any feature added at the app-server boundary has to be re-exposed or re-invented in the browser wrapper. + +Fourth, it weakens debugging. Native app-server already has a request model, notification model, server-request model, and thread/turn lifecycle semantics. The browser wrapper currently exposes only a narrower direct-thread view. + +## Why Switch To An Embedded App-Server + +The app-server boundary already solves the problems the browser needs: + +- request/response for typed client operations +- notifications for fire-and-forget client messages +- streamed server notifications for turn progress +- server requests for approvals and similar interactive flows +- explicit thread lifecycle APIs +- a stable place to add future browser features without widening the direct core API + +The repository already includes an in-process embedding path in `codex-rs/app-server/src/in_process.rs`. That runtime preserves app-server semantics while replacing stdio/websocket transport with in-memory channels. + +That makes it a much better browser boundary than `BrowserCodex.submit_turn(...)`. + +## Goals + +- Preserve the existing app-server request and event model in the browser. +- Keep a long-lived runtime alive across multiple browser turns. +- Support multi-turn conversations without rebuilding browser-specific thread state each turn. +- Reuse existing app-server features such as `thread/start`, `turn/start`, `turn/interrupt`, and server-driven approval requests. +- Keep the wasm integration transport-local and in-process. The browser should not need to run a real socket server to use app-server semantics. + +## Non-Goals + +- This document does not require full native parity for browser tools, filesystem access, or sandboxing. +- This document does not solve browser persistence by itself. State DB and rollout persistence still need separate wasm-capable implementations. +- This document does not require reusing `run_main_with_transport(...)` directly. The proposal reuses the app-server boundary, not that exact native entrypoint. + +## Proposal + +### High-Level Design + +Replace the current direct-thread browser wrapper with a wasm-facing wrapper around an embedded in-process app-server runtime. + +This refactor should be a replacement, not an addition. We should remove the existing `BrowserCodex` implementation as part of the cleanup and move the current prototype onto the new app-server-based path. + +The new stack would look like: + +```text +JS UI + -> wasm wrapper + -> in-process app-server runtime + -> MessageProcessor + -> ThreadManager / Codex core +``` + +Instead of calling `thread.submit(Op::UserTurn { ... })` directly, the browser would: + +- start one in-process app-server runtime for the browser session +- send app-server client requests into that runtime +- consume app-server server notifications and server requests from that runtime + +### Browser Boundary + +The wasm wrapper should expose an API shaped like the app-server protocol rather than the current `submit_turn(prompt, on_event)` helper. + +A minimal JS-facing API is: + +- `start(options) -> handle` +- `request(request) -> Promise` +- `notify(notification) -> Promise` or `void` +- `nextEvent() -> Promise` +- `respondToServerRequest(requestId, result) -> Promise` +- `failServerRequest(requestId, error) -> Promise` +- `shutdown() -> Promise` + +This mirrors the capabilities already present on `InProcessClientHandle`: + +- `request(...)` +- `notify(...)` +- `next_event()` +- `respond_to_server_request(...)` +- `fail_server_request(...)` +- `shutdown()` + +### Runtime Lifecycle + +The browser should create one long-lived in-process app-server runtime and keep it alive until the browser session is reset or closed. + +Expected lifecycle: + +1. Browser constructs wasm wrapper. +2. Wrapper builds `InProcessStartArgs`. +3. Wrapper calls `codex_app_server::in_process::start(...)`. +4. Wrapper keeps the returned handle alive across turns. +5. JS sends requests such as `thread/start` and `turn/start`. +6. JS drains streamed server events with `nextEvent()`. +7. On shutdown or reset, wrapper calls `shutdown()`. + +This is the key change from the current design. The runtime is session-scoped, not turn-scoped. + +### Message Flow + +For a typical first turn: + +1. JS calls `request(thread/start { ... })`. +2. The app-server returns a thread id. +3. JS calls `request(turn/start { threadId, input, ... })`. +4. The app-server returns an in-progress turn response. +5. JS repeatedly calls `nextEvent()` or receives pushed events. +6. The runtime emits `turn/started`, item deltas, tool events, and `turn/completed`. + +For later turns: + +1. JS reuses the existing thread id. +2. JS calls `request(turn/start { threadId, input, ... })` again. +3. The same app-server runtime and the same underlying thread/session continue processing. + +This aligns browser behavior with native behavior and removes the current per-turn session reset. + +### Event Model + +The browser should consume app-server events, not raw core `EventMsg` values. + +That gives the browser: + +- a stable protocol-shaped event stream +- server notifications for turn lifecycle and content updates +- server requests for approvals and other interactive flows +- lag/backpressure signals already defined by the in-process embedding + +It also gives us a cleaner debugging surface because the browser can log: + +- outgoing client requests +- client notifications +- incoming server notifications +- incoming server requests +- responses to server requests + +### Why Use `in_process` Instead Of `run_main_with_transport` + +`run_main_with_transport(...)` is not the right wasm entrypoint. + +It hardcodes: + +- stdio or websocket transport startup +- native signal handling +- native logging and DB startup assumptions + +By contrast, `in_process.rs` already does the important part we want: + +- run `MessageProcessor` +- keep the app-server request/notification/event contract +- replace transport with in-memory channels + +So the design should reuse app-server semantics through `in_process`, not try to reuse the native transport bootstrap unchanged. + +## Detailed Design + +### 1. Replace `BrowserCodex` With A WASM App-Server Wrapper + +Add a new browser-facing wrapper that owns: + +- the long-lived in-process app-server handle +- browser-specific runtime services such as the code executor bridge +- browser session options needed to build config and initialize params + +This wrapper replaces `BrowserCodex` as the main orchestration entrypoint. + +As part of this refactor we should: + +- remove the existing direct-thread `BrowserCodex` implementation +- remove `BrowserSession` as a browser-specific wrapper around `CodexThread` +- update the current browser prototype to construct and use the new app-server-based wrapper instead + +We do not want to maintain two browser orchestration paths. The prototype should become the first consumer of the new design. + +### 2. Request Serialization + +The wrapper should accept JSON or `JsValue` payloads that correspond to app-server requests and notifications. + +At the wasm boundary: + +- JS passes a request object +- wasm deserializes into `ClientRequest` or `ClientNotification` +- `in_process` handles the request +- wasm serializes responses and events back to JS + +This keeps the browser API close to the existing protocol and avoids introducing a second custom Rust-to-JS command language. + +### 3. Thread Ownership + +The browser should treat thread ids as app-server resources, not as direct `CodexThread` handles. + +That means: + +- creating threads through `thread/start` +- reading state through `thread/read` and related APIs +- starting turns through `turn/start` +- steering or interrupting turns through existing turn APIs + +This is an important layering choice. The browser should stop owning direct thread runtime objects. + +### 4. Approval And Server Request Handling + +The browser must support app-server initiated requests back to the client. + +Examples include: + +- tool approval requests +- user input requests +- future browser-specific interactive flows + +When `nextEvent()` yields a `ServerRequest`, JS must either: + +- answer with `respondToServerRequest(...)` +- or reject with `failServerRequest(...)` + +This is a capability the current direct-thread wrapper does not model cleanly. + +### 5. Debugging + +The browser wrapper should log the app-server boundary directly. + +Recommended browser-side logging: + +- every outgoing request with method and id +- every request result +- every incoming server notification +- every incoming server request +- every reply to a server request +- backpressure or lag markers + +This is a better debugging surface than ad hoc logging around direct `submit()` and `next_event()` calls because it reflects the real control plane used by native clients. + +## Storage And Persistence + +This proposal improves the control plane, but it does not by itself provide browser persistence. + +There are two separate persistence problems: + +1. Core state DB + Core currently initializes session storage internally. On `wasm32`, the current state DB bridge is stubbed and returns `None`. + +2. Rollout persistence + The current wasm rollout recorder is also stubbed. Recording is effectively a no-op and history reload is unavailable. + +As a result, the embedded app-server design gives us: + +- live multi-turn continuity within a running browser session + +But it does not yet give us: + +- browser reload/resume +- durable thread metadata +- durable rollout history + +Those require follow-on work below the `in_process` layer. + +## Migration Plan + +### Milestone 1: Replace The Prototype Runtime With `in_process` + +Build the wasm wrapper on top of `codex_app_server::in_process` and migrate the existing prototype to use it. + +Scope: + +- long-lived runtime +- request/notification/event API exposed to JS +- thread and turn flow through app-server methods +- browser code executor still supplied through existing wasm/runtime hooks +- remove the direct `BrowserCodex` / `BrowserSession` path from `wasm-harness` + +Success criteria: + +- multi-turn browser session works without resetting runtime state between turns +- browser logs show app-server request and event traffic +- browser no longer calls `thread.submit(Op::UserTurn { ... })` directly +- the existing browser prototype runs on the app-server-based implementation +- there is only one supported browser runtime path in the codebase + +### Milestone 2: Align With `codex-app-server-client` + +Decide whether the wasm wrapper should sit directly on `in_process` or on a thinner variant of `codex-app-server-client`. + +The client facade is appealing because it already wraps: + +- in-process runtime startup +- event forwarding +- server-request resolution helpers +- convergence with the remote app-server client shape + +This may reduce custom browser-side orchestration code. + +### Milestone 3: Browser Persistence + +Add real wasm-backed persistence for: + +- rollout history +- thread metadata +- any state DB-backed browser features we want to preserve across reloads + +This likely requires explicit storage seams below the app-server wrapper. + +## Alternatives Considered + +### Keep The Current `BrowserCodex` Model And Preserve Session State + +We could keep the current wrapper and only stop clearing `self.session`. + +That would likely fix the immediate memory bug, but it would not fix the larger architectural problem: + +- the browser would still use a custom direct-thread boundary +- app-server APIs would still need browser-specific re-exposure +- debugging would still happen at a less stable abstraction level + +This is a tactical fix, not the design we want to converge on. + +### Reuse `run_main_with_transport(...)` Directly + +We could try to add a custom wasm transport and plug it into `run_main_with_transport(...)`. + +This is not attractive because the function currently bundles: + +- transport startup +- native shutdown handling +- logging and telemetry bootstrap +- native transport assumptions + +`in_process` is already a better split for embedding. + +## Risks + +- The browser protocol surface becomes closer to app-server, which may require slightly more client-side plumbing than the current `submit_turn(prompt)` helper. +- Some app-server internals still assume native runtime pieces and may need additional injection points for wasm. +- Storage is still unresolved. This design fixes control-plane drift first, not persistence. +- If we expose raw JSON-RPC too literally to JS, the browser API may become awkward. We should keep the protocol shape while still providing small ergonomic helpers. + +## Open Questions + +1. Should the wasm wrapper expose raw JSON-RPC payloads, typed helper methods, or both? +2. Should the browser wrap `in_process` directly, or should it reuse a slimmer `codex-app-server-client` facade? +3. Which app-server methods do we want to support in browser v1 beyond `thread/start` and `turn/start`? +4. Do we want server events delivered by pull (`nextEvent`) only, or also by JS callback subscription? +5. What is the right storage abstraction for browser-backed rollout and state DB persistence? + +## Recommended Decisions For Implementation Start + +To begin implementation, we should make the following decisions explicit. + +- Browser API surface + Recommendation: + Expose a thin typed wrapper over the app-server protocol rather than raw JSON-RPC only. The JS API should provide ergonomic helpers such as `startThread`, `startTurn`, `interruptTurn`, `nextEvent`, and `respondToServerRequest`, while staying close to app-server request and event types under the hood. + +- `in_process` vs `codex-app-server-client` + Recommendation: + Start directly on `codex_app_server::in_process`. It is the simpler runtime dependency and gives us direct control in wasm. We should borrow the client facade's worker and event-forwarding patterns where useful, but avoid adding a second abstraction layer unless the wasm wrapper clearly grows into it. + +- Session lifecycle + Recommendation: + Create one long-lived embedded app-server runtime per browser session and keep it alive until explicit reset or shutdown. Runtime lifetime should be browser-session scoped, not turn scoped. Reset should happen only on explicit session teardown, configuration changes that require rebuild, or API key changes. + +- Event delivery model + Recommendation: + Use callback subscription as the primary browser-facing event model, while optionally retaining a polling escape hatch for tests or simple consumers. Internally we can still drain `next_event()`, but most browser UI code is simpler if events are pushed into JS callbacks. + +- Server request handling + Recommendation: + Treat server requests as first-class and require explicit client handling. The wrapper should surface every `ServerRequest` to JS and require the browser to answer or reject it. For unsupported request types in browser v1, reject them explicitly with a clear error rather than letting turns hang. + +- v1 app-server method set + Recommendation: + Support a narrow but complete thread and turn slice in browser v1: `thread/start`, `thread/read`, `turn/start`, `turn/interrupt`, and `turn/steer`. That is enough for a real multi-turn conversational product and debugging workflow without taking on the full app-server surface immediately. + +## Recommendation + +Move the browser runtime to an embedded in-process app-server boundary. + +That keeps the browser on the same architectural path as native clients, fixes the current per-turn session model, improves debugging, and creates a better long-term seam for browser-specific runtime and storage work. diff --git a/codex-rs/Cargo.lock b/codex-rs/Cargo.lock index 902c534da5..614816f8f4 100644 --- a/codex-rs/Cargo.lock +++ b/codex-rs/Cargo.lock @@ -2913,6 +2913,7 @@ name = "codex-wasm-harness" version = "0.0.0" dependencies = [ "async-trait", + "codex-app-server-protocol", "codex-code-mode", "codex-core", "codex-exec-server", diff --git a/codex-rs/core/src/codex.rs b/codex-rs/core/src/codex.rs index 04f05139a6..2b35a6bd84 100644 --- a/codex-rs/core/src/codex.rs +++ b/codex-rs/core/src/codex.rs @@ -3234,6 +3234,12 @@ impl Session { "Overwriting existing pending elicitation for server_name: {server_name}, request_id: {request_id}" ); } + #[cfg(not(target_arch = "wasm32"))] + let id = match request_id.clone() { + RequestId::String(value) => codex_protocol::mcp::RequestId::String(value.to_string()), + RequestId::Number(value) => codex_protocol::mcp::RequestId::Integer(value), + }; + #[cfg(target_arch = "wasm32")] let id = request_id.clone(); let event = EventMsg::ElicitationRequest(ElicitationRequestEvent { turn_id: params.turn_id, @@ -4644,6 +4650,8 @@ mod handlers { use codex_protocol::request_user_input::RequestUserInputResponse; use crate::context_manager::is_user_turn_boundary; + #[cfg(not(target_arch = "wasm32"))] + use crate::mcp_types::RequestId; use codex_protocol::config_types::CollaborationMode; use codex_protocol::config_types::ModeKind; use codex_protocol::config_types::Settings; @@ -4836,6 +4844,12 @@ mod handlers { content, meta, }; + #[cfg(not(target_arch = "wasm32"))] + let request_id = match request_id { + ProtocolRequestId::String(value) => RequestId::String(value.into()), + ProtocolRequestId::Integer(value) => RequestId::Number(value), + }; + #[cfg(target_arch = "wasm32")] let request_id = request_id; if let Err(err) = sess .resolve_elicitation(server_name, request_id, response) diff --git a/codex-rs/core/src/config/network_proxy_spec.rs b/codex-rs/core/src/config/network_proxy_spec.rs index 59142c5e0f..2d36d7dfd4 100644 --- a/codex-rs/core/src/config/network_proxy_spec.rs +++ b/codex-rs/core/src/config/network_proxy_spec.rs @@ -137,7 +137,14 @@ impl NetworkProxySpec { None => builder.policy_decider(|_request| async { // In restricted sandbox modes, allowlist misses should ask for // explicit network approval instead of hard-denying. - Ok(NetworkDecision::ask("not_allowed")) + #[cfg(not(target_arch = "wasm32"))] + { + NetworkDecision::ask("not_allowed") + } + #[cfg(target_arch = "wasm32")] + { + Ok(NetworkDecision::ask("not_allowed")) + } }), }; } diff --git a/codex-rs/core/src/mcp_tool_call.rs b/codex-rs/core/src/mcp_tool_call.rs index 16c5e1194c..a8946ee545 100644 --- a/codex-rs/core/src/mcp_tool_call.rs +++ b/codex-rs/core/src/mcp_tool_call.rs @@ -794,7 +794,7 @@ async fn maybe_request_mcp_tool_approval( mcp_tool_approval_question_text(question.question, monitor_reason.as_deref()); if tool_call_mcp_elicitation_enabled { let request_id = - RequestId::String(format!("{MCP_TOOL_APPROVAL_QUESTION_ID_PREFIX}_{call_id}")); + RequestId::String(format!("{MCP_TOOL_APPROVAL_QUESTION_ID_PREFIX}_{call_id}").into()); let params = build_mcp_tool_approval_elicitation_request( sess.as_ref(), turn_context.as_ref(), diff --git a/codex-rs/core/src/tools/network_approval.rs b/codex-rs/core/src/tools/network_approval.rs index 27f19e677a..dadee60e50 100644 --- a/codex-rs/core/src/tools/network_approval.rs +++ b/codex-rs/core/src/tools/network_approval.rs @@ -545,11 +545,11 @@ pub(crate) fn build_network_policy_decider( let network_policy_decider_session = Arc::clone(&network_policy_decider_session); async move { let Some(session) = network_policy_decider_session.read().await.upgrade() else { - return Ok(NetworkDecision::ask("not_allowed")); + return NetworkDecision::ask("not_allowed"); }; - Ok(network_approval + network_approval .handle_inline_policy_request(session, request) - .await) + .await } }) } diff --git a/codex-rs/wasm-harness/Cargo.toml b/codex-rs/wasm-harness/Cargo.toml index d1bfefd7c6..208e15608b 100644 --- a/codex-rs/wasm-harness/Cargo.toml +++ b/codex-rs/wasm-harness/Cargo.toml @@ -14,6 +14,7 @@ workspace = true [dependencies] async-trait = { workspace = true } +codex-app-server-protocol = { workspace = true } codex-code-mode = { workspace = true } codex-core = { workspace = true } codex-exec-server = { workspace = true } @@ -22,7 +23,7 @@ codex-protocol = { workspace = true } js-sys = { workspace = true } serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } -tokio = { workspace = true, features = ["rt"] } +tokio = { workspace = true, features = ["rt", "sync"] } wasm-bindgen = { workspace = true } wasm-bindgen-futures = { workspace = true } web-sys = { workspace = true, features = [ diff --git a/codex-rs/wasm-harness/README.md b/codex-rs/wasm-harness/README.md index 82970f51f1..be8acd4483 100644 --- a/codex-rs/wasm-harness/README.md +++ b/codex-rs/wasm-harness/README.md @@ -4,9 +4,11 @@ This crate is the first browser-facing seam for a Codex harness prototype. It exposes one browser-facing layer: -- `BrowserCodex`: a `wasm_bindgen` adapter that starts a real - `codex-core::CodexThread`, calls `CodexThread::submit(Op::UserTurn { ... })`, - and streams real Codex protocol events back to JavaScript. +- `BrowserAppServer`: a `wasm_bindgen` adapter that exposes an app-server-shaped + browser boundary. It keeps a live `codex-core::CodexThread` behind a + request/event interface, accepts app-server request envelopes such as + `thread/start` and `turn/start`, and streams notifications plus raw core + events back to JavaScript. The demo page passes the user's API key into the WASM facade so Rust can make browser `fetch` requests to the Responses API through the wasm-compatible @@ -17,9 +19,12 @@ The API key field is for local prototype use only: it stores the key in browser integration should use a proxy or an ephemeral-token flow instead of persisting long-lived API keys in the page origin. -The remaining work is to replace the current browser-only host shims with more -complete browser implementations for persistence, richer tools, and other -host-heavy services. +This replaces the earlier `BrowserCodex.submit_turn(...)` prototype. The +browser example now uses the same high-level control plane as native Codex: +create or reuse a thread, start turns against it, and listen for async events. +The remaining work is to swap the direct core bridge under this wrapper for a +deeper reuse of the in-process app-server runtime once the storage/runtime +dependencies are wasm-compatible. ## Library Boundary @@ -27,7 +32,7 @@ The intended downstream integration point is the crate itself. Downstream webapps can: - depend on `codex-wasm-harness` from a Git branch or local path; -- construct `BrowserCodex` from JavaScript or wrap the crate with their own +- construct `BrowserAppServer` from JavaScript or wrap the crate with their own browser bindings; - provide session-scoped prompt inputs such as `cwd`, developer instructions, and user/project-doc instructions; @@ -45,22 +50,39 @@ The intended boundary is: Example: ```js -const codex = new BrowserCodex(apiKey); -codex.setSessionOptions({ +const app = new BrowserAppServer(apiKey); +app.setSessionOptions({ cwd: "/workspace", instructions: { developer: "Code runs inside AppKernel. OPFS and network APIs are available.", user: "# AGENTS.md instructions for /workspace\n\n\n...\n", }, }); +app.setEventHandler((event) => console.log(event)); + +const thread = await app.request({ + method: "thread/start", + id: 1, + params: {}, +}); + +await app.request({ + method: "turn/start", + id: 2, + params: { + threadId: thread.thread.id, + input: [{ type: "text", text: "Say hello." }], + }, +}); ``` ## Current Limitations -This is a minimal browser port of the Codex turn loop, not full desktop Codex. +This is a minimal browser port of the Codex app-server boundary, not full +desktop Codex. -- It uses the real `CodexThread` turn path, but many host-heavy services are - still compiled into degraded wasm implementations. +- It keeps real Codex threads alive across turns, but many host-heavy services + are still compiled into degraded wasm implementations. - It currently relies on a single injected browser code executor for code mode. - Native shell, PTY, MCP, plugin runtime, and filesystem-backed persistence are not available in the browser prototype. diff --git a/codex-rs/wasm-harness/examples/browser/index.html b/codex-rs/wasm-harness/examples/browser/index.html index 9ea5d8970b..e51b178399 100644 --- a/codex-rs/wasm-harness/examples/browser/index.html +++ b/codex-rs/wasm-harness/examples/browser/index.html @@ -202,9 +202,11 @@