Commit Graph
17 Commits
Author SHA1 Message Date
joeytrasatti-openai 85c6da1c79 Add persisted sections for organizing threads (#35722)
## What changed

- Replace the `isPinned` thread metadata and filters with an optional persisted
  `section` and `sectionId`.
- Add the paginated `threadSection/list` app-server method so clients can
  discover sections even when they contain no threads.
- Seed a stable `Pinned` section, validate section assignments, and support
  filtering for a specific section or for unsectioned threads.

## Testing

- Cover section protocol serialization, listing and pagination, metadata
  updates, filtering, persistence, migration compatibility, and operation
  without SQLite state.

GitOrigin-RevId: 7972b5471d29317b9387bfd90aa9f573f691ad4c
2026-07-28 05:26:16 +00:00
joeytrasatti-openai 400ee190c3 Add persisted thread pinning to the app server (#34840)
## What changed

- Add `isPinned` to thread responses and allow `thread/metadata/update` to pin or unpin stored threads.
- Add an `isPinned` filter to `thread/list`, including cursor-based pagination and combinations with relationship filters.
- Persist pin state in SQLite without modifying rollout files, default existing threads to unpinned, and preserve pins through reconciliation and archive transitions.

## Testing

- Cover protocol serialization, database migration and indexing, pin updates, filtered pagination, rollout reconciliation, and archive/unarchive behavior.

GitOrigin-RevId: c2eacabd6dae465e2ae6ce8a169e00740960cdb7
2026-07-22 22:54:28 +00:00
Brent Traut 5a208c1fc3 Persist names for paginated threads (#34229)
## Why

Paginated threads need explicit user-facing names to remain distinct from
derived titles and previews without requiring rollout metadata writes.

## What changed

- Add a nullable `name` column to persisted thread metadata and use it as the
  canonical name for paginated threads.
- Route paginated name updates through the state database while retaining the
  name index as best-effort compatibility data and preserving legacy thread
  behavior.
- Return persisted names when reading, listing, searching, and resuming
  paginated threads, and include them in state-database search matching.

## Testing

- Cover paginated name updates, reads, list/search results, and metadata-only
  resume behavior, including stale or unavailable compatibility data.

GitOrigin-RevId: 5ac4dfa2a83682b43d59ecc87e0f515580164dca
2026-07-19 21:13:13 +00:00
Owen Lin 5267e805fb feat(app-server): add history_mode to thread (#29927)
## Description

This PR adds a new `historyMode = "legacy" | "paginated"` to `Thread`.
This will be stored in `SessionMeta` in the JSONL rollout file and as a
new column in the SQLite thread_metadata table, and exposed on
`thread/start` and on the `Thread` object in app-server.

## What changed

- Added canonical `ThreadHistoryMode` with `legacy` and `paginated`,
defaulting old and new SessionMeta to `legacy`.
- Carried `history_mode` through core session config, ThreadStore stored
metadata, local/in-memory stores, rollout metadata extraction, and the
existing SQLite `threads` table.
- Added experimental `historyMode` to app-server v2 `Thread` and
`thread/start`.
- Made paginated stored threads metadata-discoverable but unsupported
for legacy full-history reads, `load_history`, live resume, and create
paths.
- Regenerated app-server schema fixtures and added
protocol/state/thread-store/app-server coverage for persistence and
fail-closed behavior.

## Compatibility floor
Because users may be running various versions of Codex binaries on the
same machine (TUI, Codex App, etc.), we will need to establish a
compatibility floor for upcoming paginated threads, which will change
how thread storage reads and writes work.

The overall plan here:
```
Release N:
- Add historyMode to SessionMeta / Thread / SQLite metadata.
- Teach binaries to understand paginated threads.
- If a binary sees `historyMode="paginated"` but does not support the paginated contract, it refuses to resume/mutate the thread.
- Default remains `"legacy"`.

Release N+1:
- First-party clients start opting into paginated threads where appropriate.
- Internal dogfood / staged rollout.
- Measure old-client usage and paginated-thread unsupported errors.

Release N+2:
- Only after Release N+ is overwhelmingly deployed, make paginated the default.
- Accept that a small tail of N-1-or-older binaries may not understand paginated threads.
```

The important behavior change is fail-closed handling for a binary that
encounters a persisted `paginated` thread before it knows how to fully
support paginated history. In app-server, if a thread is `paginated`, we
will:

- allow metadata-only discovery paths like `thread/list` and
`thread/read(includeTurns=false)`, so clients can still see the thread
and inspect its `historyMode`
- reject legacy full-history/live-thread paths like
`thread/read(includeTurns=true)` and `thread/resume` with an unsupported
JSON-RPC error
- avoid silently treating an unknown or future `historyMode` as `legacy`

Under the hood, the ThreadStore layer also rejects legacy operations
that would need to load or replay the full thread history for a
paginated thread. That gives us the behavior we want for Release N:
future paginated threads are visible, but this binary fails closed
instead of trying to operate on them as if they were legacy threads.
2026-06-26 09:12:42 -07:00
Jeremy Rose 7dc7096ae1 [codex] Restore thread recency with compatible migration history (#28671)
## Summary

- Revert #28655, restoring the thread `recencyAt` behavior introduced by
#27910.
- Move `threads_recency_at` to migration 0039 so it no longer collides
with `external_agent_config_imports` at version 0038.
- Repair databases that already applied the recency migration as version
38 by moving the matching migration-history row to version 39 before
SQLx validation. The current version-38 migration can then apply
normally.

## Validation

- `just test -p codex-state
migrations::tests::repairs_recency_migration_that_was_applied_as_version_38`
- `just test -p codex-state -p codex-rollout -p codex-thread-store -p
codex-app-server-protocol -p codex-tui`: 3,439 passed; six TUI tests
could not open the machine's existing read-only incident database at
`~/.codex/sqlite/state_5.sqlite`.
- `just fix -p codex-state`
- `just fmt`
- Verified that state migration versions are unique.
2026-06-17 18:52:18 +00:00
pakrym-oai cb15c64760 Revert thread recencyAt for sidebar ordering (#28655)
## Why

Revert #27910 to remove the newly introduced thread `recencyAt`
persistence and API behavior from `main`.

## What changed

This reverts commit `fac3158c2a783095768076489815f361fa9b0db4`,
including the state migration, thread-store propagation, app-server API
surface, generated schemas, and related tests.

## Validation

Not run before opening; relying on CI for the initial fast signal.
2026-06-16 21:39:30 -07:00
Jeremy Rose fac3158c2a Add thread recencyAt for sidebar ordering (#27910)
## Summary

Add a server-owned `recencyAt` timestamp and `recency_at` thread-list
sort key for product recency ordering while preserving the existing
meaning of `updatedAt` as the latest persisted thread mutation.

This is the server-side alternative to #27697. Rather than narrowing
`updatedAt`, clients can sort the sidebar by `recency_at` and continue
treating `updatedAt` as mutation time.

Paired Codex Apps PR:
[openai/openai#1024599](https://github.com/openai/openai/pull/1024599)

## Contract

- `recencyAt` initializes when a thread is created.
- A turn start advances `recencyAt` monotonically.
- Commentary, agent output, tool results, token/accounting updates, turn
completion, archive, unarchive, resume, and generic metadata writes do
not advance it.
- `updatedAt` retains its existing behavior and continues to advance for
persisted thread mutations.
- Current servers populate `recencyAt`; the response field is optional
in generated TypeScript so clients connected to older servers can fall
back to `updatedAt`.
- Filesystem-only fallback uses existing updated/mtime ordering when
SQLite is unavailable.

## Persistence and compatibility

Migration 0038 adds second- and millisecond-precision recency columns,
backfills them from the existing updated timestamp, creates list
indexes, and includes an insert trigger so older binaries writing to a
migrated database seed recency without causing later mutations to
advance it.

Generic metadata upserts preserve existing recency values. Turn-start
updates use a dedicated monotonic touch, and process-local allocation
keeps millisecond cursor values unique. State DB list, search, read,
filtered-list repair, rollout fallback propagation, and app-server
conversions all carry the new field.

## API

`Thread` responses include:

```ts
recencyAt?: number
```

`thread/list` and `thread/search` accept:

```json
{ "sortKey": "recency_at" }
```

Generated TypeScript and JSON schemas are included.

## Validation

- `just test -p codex-state` — 146 passed
- `just test -p codex-rollout` — 69 passed
- `just test -p codex-thread-store` — 81 passed
- `just test -p codex-app-server-protocol` — 231 passed
- Focused app-server list ordering, response mapping, archive/unarchive,
and resume lifecycle tests passed
- Scoped `just fix` for state, rollout, thread-store,
app-server-protocol, and app-server
- `just fmt`
- `git diff --check`
- Independent correctness, simplicity, elegance, security, and
test-quality reviews; actionable ordering, lifecycle, query-projection,
and timestamp-uniqueness findings were addressed
2026-06-16 17:06:22 -07:00
Boyang Niu e127a0cf99 Add extra config to StoredThread, leave empty for now (#27092) 2026-06-09 00:38:35 +00:00
jif-oai a8a6071279 Read compressed rollouts and materialize before append (#25087)
## Why

Local rollout compression needs a cold `.jsonl.zst` representation
without letting compressed physical paths leak into append-mode writers.
The unsafe case is resume or metadata update code successfully reading a
compressed rollout and then appending raw JSONL bytes to the zstd file.

This PR folds the former #25088 materialization slice into the
read-support PR so the reader changes and append-safety invariant land
together.

## What Changed

- Teach rollout readers, discovery, listing, search, and ID lookup to
understand compressed `.jsonl.zst` rollouts.
- Keep `.jsonl` as the logical/stored rollout path while allowing read
paths to open either plain or compressed storage.
- Materialize compressed rollouts back to plain `.jsonl` before
append-mode writes, including resume and direct metadata append paths.
- Preserve compressed-file permissions when materializing back to plain
JSONL.
- Refresh thread-store resolved rollout paths after compatibility
metadata writes so reconciliation follows the materialized file.
- Avoid treating transient compression temp files as real rollout lookup
results.

## Remaining Stack

#25089 remains the separate worker PR. It is based directly on this PR
and stays behind the disabled `local_thread_store_compression` feature
flag.

The worker still has a broader coordination question: a resume or
metadata update can race with background compression while a plain file
is being replaced by `.jsonl.zst`. This PR handles the read and
materialize-before-append primitives; it does not make the worker
production-ready.

## Validation

- `just test -p codex-rollout`
- `just test -p codex-thread-store`
- `just fix -p codex-rollout`
- `just fix -p codex-thread-store`
- `just bazel-lock-check`
2026-06-01 15:14:19 +02:00
Owen Lin cf0911076f store and expose parent_thread_id on Threads (#25113)
## Why

This PR
https://github.com/openai/codex/pull/24161#discussion_r3325692763
revealed a subagent data modeling issue, where we overloaded
`forked_from_id` to also mean `parent_thread_id`. That's incorrect since
guardian and review subagents can be a subagent and NOT fork the main
thread's history.

The solution here is to explicitly store a new `parent_thread_id` on
`SessionMeta`, alongside `forked_from_id` which already exists. While
we're at it, also expose it in the app-server protocol on the `Thread`
object.

A thread->subagent relationship and a fork of thread history are
orthogonal concepts.

## What Changed

- Added top-level `parent_thread_id` persistence on `SessionMeta` and
runtime/session plumbing through `SessionConfiguredEvent`,
`CodexSpawnArgs`, `SessionConfiguration`, `ThreadConfigSnapshot`,
`TurnContext`, and `ModelClient`.
- Made turn metadata, request headers, analytics, and subagent-start
events read the separate runtime/top-level parent field instead of
deriving general parent lineage from `SessionSource` or
`forked_from_thread_id`.
- Passed parent lineage separately at delegated subagent, review,
guardian, agent-job, and multi-agent spawn construction sites;
copied-history fork lineage remains derived only from `InitialHistory`.
- Persisted and exposed parent lineage through rollout/thread-store
projections and app-server v2 `Thread.parentThreadId`.
- Updated app-server README text and regenerated app-server schema
fixtures for the additive `parentThreadId` response field.
2026-06-01 04:33:20 +00:00
Michael Bolin a1ecf0cf1c thread-store: store permission profiles (#23165)
## Why

`SandboxPolicy` is the legacy compatibility shape, but
`codex-thread-store` still exposed it through `StoredThread`,
`ThreadMetadataPatch`, and live metadata sync. That kept thread-store
consumers tied to the legacy representation and meant richer permission
profile data could not round-trip through thread metadata or cold
rollout reconciliation.

## What Changed

- Replaced thread-store `sandbox_policy` API fields with canonical
`PermissionProfile` fields.
- Persist new permission-profile metadata as canonical JSON in the
existing SQLite metadata slot while continuing to read older legacy
sandbox policy values.
- Updated local, in-memory, live metadata sync, and rollout extraction
paths to propagate `TurnContextItem::permission_profile()`.
- Re-materialize legacy permission metadata against the final rollout
cwd when rollout-derived metadata replaces stale SQLite summaries.
- Updated affected app-server and core test constructors to build
`PermissionProfile` values directly.

## Test Plan

- `cargo test -p codex-state`
- `cargo test -p codex-thread-store`
- `cargo test -p codex-app-server
summary_from_stored_thread_preserves_millisecond_precision --lib`
- `cargo test -p codex-core realtime_context --lib`
2026-05-29 11:55:31 -07:00
Eric Traut f10ddc3f13 Use goal preview metadata for goal-first threads (#21981)
Fixes #20792

## Why

`/goal`-first threads are valid resumable threads, but they can be
missing from `codex resume` and app recents because discovery depends on
metadata derived from a normal first user message.

PR #21489 attempted to fix this by using the goal objective as
`first_user_message`. Review feedback pointed out that
`first_user_message` does more than provide visible text today: it gates
listing, supplies preview text, and participates in deciding whether a
later title should surface as a distinct thread name. Reusing it for the
goal objective could leave a `/goal`-first thread with
`first_user_message=<goal>` and `title=<later prompt>`, even though the
goal should only provide the initial visible preview.

This PR follows that feedback by and keeps the `first_user_message` as
is but introduces a new `preview` field to separate concerns. The
`preview` field is populated from the first user message or the goal
objective. We can extend it in the future to include other sources.

## What Changed

- Added internal thread `preview` metadata in `codex-state`, including a
SQLite migration that backfills from `first_user_message` and from
existing `thread_goals` objectives when needed.
- Treated `ThreadGoalUpdated` as preview-bearing metadata so goal-first
threads can be listed and searched without mutating
`first_user_message`.
- Updated rollout listing, state queries, thread-store conversion, and
app-server mapping to use preview metadata while continuing to expose
the existing public `preview` field.
- Preserved title/name distinctness behavior around literal
`first_user_message`, so a later normal prompt after `/goal` does not
surface as a separate name just because the goal supplied the initial
preview.
- Preserved compatibility for older/internal metadata writes by deriving
preview from `first_user_message` when explicit preview metadata is
absent.

## Verification

- Manually verified that a thread that starts with a `/goal <objective>`
shows up in the resume picker.
2026-05-11 10:12:46 -07:00
rhan-oai b3d4f1a9f0 [codex-analytics] rework thread_source for thread analytics (#20949)
## Summary
- make `thread_source` an explicit optional thread-level field on
`thread/start`, `thread/fork`, and returned thread payloads
- persist `thread_source` in rollout/session metadata so resumed live
threads retain the original value
- replace the old best-effort `session_source` -> `thread_source`
mapping with an explicit caller-supplied analytics classification

## Why
Before this change, analytics `thread_source` was populated by a
best-effort mapping from `session_source`. `session_source` describes
the runtime/client surface, not the actual thread-level origin, so that
projection was not accurate enough to distinguish cases such as `user`,
`subagent`, `memory_consolidation`, and future thread origins reliably.

Making `thread_source` explicit keeps one thread-level analytics field
while letting callers provide the real classification directly instead
of recovering it indirectly from `session_source`.

## Impact
For new analytics events, `thread_source` now reflects the explicit
thread-level classification supplied by the caller rather than an
inferred value derived from `session_source`. Existing protocol fields
remain optional; callers that omit `threadSource` now produce `null`
instead of a best-effort inferred value.

## Validation
- `just write-app-server-schema`
- `cargo test -p codex-analytics -p codex-core -p
codex-app-server-protocol --no-run`
- `cargo test -p codex-app-server-protocol
generated_ts_optional_nullable_fields_only_in_params`
- `cargo test -p codex-analytics
thread_initialized_event_serializes_expected_shape`
- `cargo test -p codex-core
resume_stopped_thread_from_rollout_preserves_thread_source`
2026-05-06 02:12:31 +00:00
Tom fe05acad23 Make thread store process-scoped (#19474)
- Build one app-server process ThreadStore from startup config and share
it with ThreadManager and CodexMessageProcessor.
- Remove per-thread/fork store reconstruction so effective thread config
cannot switch the persistence backend.
- Add params to ThreadStore create/resume for specifying thread
metadata, since otherwise the metadata from store creation would be used
(incorrectly).
2026-04-30 21:24:59 -07:00
Tom 127be0612c [codex] Migrate thread turns list to thread store (#19280)
- migrate `thread/turns/list` to ThreadStore. Uses ThreadStore for most
data now but merges in the in-memory state from thread manager
- keep v2 `thread/list` pathless-store friendly by converting
`StoredThread` directly to API `Thread`
- add regression coverage for pathless store history/listing
2026-04-30 14:16:42 -07:00
Tom fad3d0f1d0 codex: route thread/read persistence through thread store (#18352)
Summary
- replace the thread/read persisted-load helper with
ThreadStore::read_thread
- move SQLite/rollout summary, name, fork metadata, and history loading
for persisted reads into LocalThreadStore
- leave getConversationSummary unchanged for a later PR

Context
- Replaces closed stacked PR #18232 after PR #18231 merged and its base
branch was deleted.
2026-04-17 10:31:30 -07:00
Tom 50d3128269 Migrate archive/unarchive to local ThreadStore (#17892)
# Summary
- implement local ThreadStore archive/unarchive operations
- implement local ThreadStore read_thread operation
- break up the various ThreadStore local method implementations into
separate files
- migrate app-server archive/unarchive and core archive fixture to use
ThreadStore (but not all read operations yet!)
- use the ThreadStore's read operation as a proxy check for thread
persistence/existence in the app server code
- move all other filesystem operations related to archive (path
validation etc) into the local thread store.

# Tests
- add dedicated local store archive/unarchive tests
2026-04-15 20:48:09 +00:00