## Why
A database can open successfully while containing corruption that startup previously missed. Detecting it before runtime initialization completes allows recoverable databases to be rebuilt while preserving the damaged files.
## What changed
- Run `PRAGMA quick_check(1)` when opening writable databases, with a 100 ms scan budget. Share attempts across cloned `SqliteConfig` instances by file identity so repeated opens skip validation and replacement files are checked again.
- Back up and rebuild runtime databases that support recovery. Report corruption without rebuilding thread history or other databases without an automatic recovery policy. Treat interrupted or locked checks as incomplete rather than confirmed corruption.
- Collect startup backups across recovery retries and include their locations in the app server warning. Explain that saved conversations can restore the thread list and history, while some database-only metadata may be unavailable.
- Record confirmed corruption in `codex.sqlite.corruption.count`.
## Testing
Add tests for recovery policies, backup preservation, validation caching and concurrency, cancelled and interrupted checks, and locked databases. Add app server coverage for recovery warnings and saved conversation restoration, a TUI warning snapshot, and message board post preservation.
GitOrigin-RevId: 43e26328888e0301def642d69bd324acee867566
## Why
Deleting logs leaves free SQLite pages available for reuse without shrinking the database file. Reclaim excess free space while preserving a reserve for future writes.
## What changed
- Start a background incremental-vacuum worker with `StateRuntime`, enabling reclamation only for the logs database. Use a per-home file lock to select one worker and wait for it to finish during shutdown.
- Reclaim pages when at least 64 MiB and 25% of the database are free, preserving a reserve of at least 16 MiB or 10% of the database.
- Use short transactions with page and cooperative time budgets, defer work on contention, and reduce batch sizes with exponential backoff when interrupted passes make no progress.
- Record reclamation counts, durations, and reclaimed pages in telemetry.
## Testing
Add a regression test that interrupts a vacuum commit, verifies foreground writes can resume without data loss, then confirms a resumed pass shrinks the database and passes `PRAGMA integrity_check`.
GitOrigin-RevId: 8421f87b0aeef6540eda5690c84216906673c72e
## Why
SQLite write failures can leave persisted logs incomplete, while log snapshots taken before collection miss errors raised during that collection.
## What changed
- Report SQLite flush failures directly to the in-memory log buffer without re-entering tracing, using an error category and batch size instead of raw error details.
- Track whether the log writer has lost a batch and make app-server diagnostic uploads use the memory buffer when SQLite logs are incomplete.
- Refresh log bytes after flushing and querying SQLite so the same submission includes collection failures, while preserving captured metadata and thread identity.
## Testing
Add regression tests for failure reporting without recursion, subsequent write recovery, metadata preservation, and healthy and failing SQLite collection. Add an upload integration test that captures attachments locally and checks failure diagnostics, memory-buffer fallback, and omission of raw SQLite error details.
GitOrigin-RevId: 51b0c4b69c977d9f4e74f2f245a9fd90fdb13761
## Why
SQLite log persistence needs visibility into batch size, write latency,
failures, and entries dropped before they reach the writer. Exporter
diagnostics must not feed back into the SQLite log sink and keep metric
exports active indefinitely.
## What changed
- Record write count, duration, batch bytes, entry count, largest-entry size,
and queue drops with bounded outcome tags.
- Add explicit byte histogram boundaries for log batch metrics.
- Filter OTLP transport noise and the benign unsolicited HTTP/2 PING warning
from SQLite log persistence to prevent export cycles while retaining other
HTTP/2 warnings.
## Testing
- Cover metric values, tags, queue-drop reasons, and histogram boundaries.
- Verify successful and failed OTLP HTTP and gRPC exports become idle instead
of generating recurring SQLite log-write metrics.
GitOrigin-RevId: a63224ae15945fe84adc977a8b7d55bb47a4de53
## What changed
- Add a storage-neutral `QueueStore` interface backed by a dedicated SQLite
database for ordered, thread-scoped user submissions.
- Support enqueueing, paginated listing, editing, deletion, and atomic
reordering, with a limit of 100 pending items per thread.
- Remove queued submissions when their thread is deleted.
## Testing
- Cover concurrent inserts, ordering and pagination, edits, deletion,
cross-thread isolation, queue limits, and thread cleanup.
GitOrigin-RevId: 0a8e95ac71137afc6f3a039d4b5fc8603110c891
## Why
Paginated thread history needs its own SQLite database to avoid adding lock
contention to the main state store.
## What changed
- Add the `thread_history_1.sqlite` path and migration scaffolding.
- Create tables and pagination indexes for projected turns and items, plus a
per-thread projection checkpoint.
- Register the database with runtime diagnostics, Bazel inputs, and database
telemetry.
GitOrigin-RevId: d194310835f1df2a2a29c7827d77ca37eabbd0a3
## Summary
Generated memory rows and their stage-one/stage-two job state currently
live in `state_5.sqlite` alongside thread metadata. That makes memory
cleanup and regeneration share the main state schema even though those
rows are memory-pipeline data and can be rebuilt independently from the
durable thread records.
This PR moves the memory-owned tables into a dedicated
`memories_1.sqlite` runtime database while keeping thread metadata in
`state_5.sqlite`.
## Changes
- Adds a separate memories DB runtime, migrator, path helpers, telemetry
kind, and Bazel compile data for `state/memory_migrations`.
- Introduces `MemoryStore` behind `StateRuntime::memories()` and moves
memory table/job operations onto that store.
- Drops the old memory tables from the state DB and recreates their
schema in `state/memory_migrations/0001_memories.sql`.
- Updates memory startup, citation usage tracking, rollout pollution
handling, `debug clear-memories`, and app-server `memory/reset` to
operate through the memories DB.
- Preserves cross-DB behavior by hydrating thread metadata from the
state DB when selecting visible memory outputs and checking stage-one
staleness.
## Verification
- Added/updated `codex-state` tests for deleted-thread memory visibility
and already-polluted phase-two enqueue behavior.
- Updated `debug clear-memories`, app-server `memory/reset`, and
memories startup tests to seed and assert memory rows through
`memories_1.sqlite`.
## Why
Thread goals are moving toward extension-owned runtime behavior, but
their persisted state was still stored in the shared state database.
This makes the goal store harder to isolate and keeps future storage
splits tied to ad hoc runtime plumbing.
This PR gives goals their own SQLite database while keeping the existing
`StateRuntime` entry point. The goal is to make this the pattern for
adding more dedicated runtime databases later.
This also reduce load on existing DB and reduce contention
## Limitation
Thread preview from goal is not supported anymore. I'm looking into this
[EDIT]: solved
## What changed
- Added a dedicated `goals_1.sqlite` database with its own
`goals_migrations` directory.
- Moved `thread_goals` creation into the goals DB migration set.
- Dropped the old `thread_goals` table from the main state DB with a
normal state migration. There is intentionally no backfill for existing
goal rows.
- Changed `GoalStore` to be backed only by the goals DB pool.
- Removed the old goal-write side effect that filled empty
`threads.preview` values from the goal objective.
- Added shared runtime DB path metadata so startup, telemetry, `codex
doctor`, and repair handling can include future DBs without bespoke path
lists.
- Updated Bazel compile data so the new goals migration directory is
available to `sqlx::migrate!`.
## Verification
- `cargo check --tests -p codex-state -p codex-cli -p codex-core -p
codex-app-server`
- `just fix -p codex-state`
- `just fix -p codex-cli`
- `just fix -p codex-app-server`
## Summary
- add SQLite init, backfill-gate, and fallback telemetry without
introducing a cross-cutting state-db access wrapper
- install one process-scoped telemetry sink after OTEL startup and let
low-level state/rollout paths emit through it directly
- add process-start metrics for the process owners that initialize
SQLite
---------
Co-authored-by: Owen Lin <owen@openai.com>
Dmitriy Kovalenko