codex/file-streaming-api.md

App Server File Streaming API

Status

Implemented in app-server and exec-server.

The existing buffered fs/readFile and fs/writeFile methods remain unchanged.

Goals

• Read and write files without buffering the entire file in memory.
• Provide backpressure through bounded request/response chunks.
• Allow positional reads.
• Support cancellation and deterministic cleanup.
• Match the existing whole-file write behavior, including preserving the identity and metadata of existing destination files.

Read API

fs/readFile/open

Opens a file for positional reads.

{
  "handleId": "read-1",
  "path": "/absolute/path/to/file"
}

Response:

{
  "maxChunkBytes": 262144
}

fs/readFile/read

Reads at most maxBytes starting at the absolute byte offset. If maxBytes is omitted or exceeds maxChunkBytes, the server uses maxChunkBytes. Clients may use maxChunkBytes to size a reusable read buffer.

{
  "handleId": "read-1",
  "offset": 0,
  "maxBytes": 65536
}

Response:

{
  "dataBase64": "SGVsbG8=",
  "eof": false
}

eof means no more bytes were available at the time of the read. A later read at the same offset may return data if the file grows.

fs/readFile/stat

Returns metadata for the open file.

{
  "handleId": "read-1"
}

Response:

{
  "sizeBytes": 1234,
  "createdAtMs": 1730910000000,
  "modifiedAtMs": 1730910000000
}

The metadata applies to the opened file handle, so the operation remains valid if the path is renamed or replaced after it is opened.

fs/readFile/close

Closes the read handle.

{
  "handleId": "read-1"
}

Response:

{}

Write API

fs/writeFile/open

Creates the destination if needed or truncates the existing file. Existing file identity and metadata, including hardlinks and platform access controls, are preserved in the same way as fs/writeFile.

{
  "handleId": "write-1",
  "path": "/absolute/path/to/file"
}

Response:

{
  "maxChunkBytes": 262144
}

fs/writeFile/write

Decodes and appends the complete chunk to the open destination file. A successful response acknowledges the entire decoded chunk; partial success is not exposed.

{
  "handleId": "write-1",
  "dataBase64": "SGVsbG8="
}

Response:

{}

fs/writeFile/close

Closes the write handle.

{
  "handleId": "write-1"
}

Response:

{}

Shared Semantics

• Handle IDs are client-supplied strings scoped to one connection.
• Opening a duplicate active handle ID returns INVALID_REQUEST.
• Operations for one handle are serialized. Different handles may run concurrently.
• Reads are positional and do not maintain a server-side cursor.
• Writes are sequential appends.
• Each read or write transfers at most the maxChunkBytes returned by its corresponding open operation.
• maxChunkBytes is currently 262144 bytes.
• Backpressure comes from awaiting bounded read and write responses. Clients should use a bounded pipeline of up to two read requests to hide transport round-trip latency without accumulating unbounded response data.
• Close operations are idempotent. App-server close requests bypass queued operations and cancel an active open or chunk operation. A close received while open is pending prevents that handle from becoming live. Exec-server handles bounded file RPCs in request order, so close takes effect before the next file operation.
• Closing a connection closes all of its handles.
• Any filesystem or I/O error closes the affected handle. Protocol errors such as an unknown handle do not affect other handles.
• Streamed writes are visible as they complete. Opening truncates an existing destination, and an I/O error, cancellation, or disconnect may leave a partially written file.
• Errors use normal JSON-RPC error responses.
• App-server operations target the app-server host filesystem. Exec-server exposes the same pull-based handle operations for remote filesystem clients.
• Streaming through the platform sandbox helper is not supported because the helper is one-shot and cannot retain open-file identity across requests.