## Why
SQLite log persistence needs visibility into batch size, write latency,
failures, and entries dropped before they reach the writer. Exporter
diagnostics must not feed back into the SQLite log sink and keep metric
exports active indefinitely.
## What changed
- Record write count, duration, batch bytes, entry count, largest-entry size,
and queue drops with bounded outcome tags.
- Add explicit byte histogram boundaries for log batch metrics.
- Filter OTLP transport noise and the benign unsolicited HTTP/2 PING warning
from SQLite log persistence to prevent export cycles while retaining other
HTTP/2 warnings.
## Testing
- Cover metric values, tags, queue-drop reasons, and histogram boundaries.
- Verify successful and failed OTLP HTTP and gRPC exports become idle instead
of generating recurring SQLite log-write metrics.
GitOrigin-RevId: a63224ae15945fe84adc977a8b7d55bb47a4de53
## Why
Telemetry transport and storage can emit their own diagnostics, causing exports
or SQLite writes to recursively generate more telemetry.
## What changed
- Exclude `h2` spans from trace export so OTLP transport activity cannot trigger
additional OTLP exports.
- Exclude all `sqlx`, `sqlx_core`, and `sqlx_sqlite` diagnostics from the SQLite
log sink, including when no external subscriber filter is installed.
- Keep similarly named targets such as `sqlx_application` eligible for storage.
## Testing
- Extend the SQLite log-filter test to cover the broader SQLx exclusion and the
unrelated-target case.
GitOrigin-RevId: 13a832e1f228e9564a31cc08f00add286342bac1
## Why
Warnings emitted while querying SQLite or acquiring a connection must not be
written back through the same SQLite log sink.
## What changed
- Disable the `sqlx::query` and `sqlx::pool::acquire` targets for the SQLite
log filter.
- Keep warnings from other `sqlx` targets so useful diagnostics remain
available.
## Testing
Extend the log filter test to verify that query and pool-acquisition warnings
are dropped while another `sqlx` warning is retained.
GitOrigin-RevId: cec9d4e07d113de8eccec7445ede0ae68e21886d
## What changed
- Increase the default bounded log queue capacity from 512 to 2,048 entries.
- Raise the default insert batch size from 128 to 512 entries.
- Extend the default periodic flush interval from 2 to 10 seconds.
GitOrigin-RevId: e96be0996152b4a2a84d88c23d36959436685682
## Why
Per-chunk streaming traces can flood the bounded SQLite log queue.
## What changed
- Persist `codex_tui::streaming::controller` and
`codex_tui::streaming::table_holdback` events at `DEBUG` and above.
- Keep `TRACE` events enabled for other TUI streaming targets.
## Testing
Extended the SQLite log filter test to cover dropped streaming traces, retained
debug diagnostics, and unaffected streaming targets.
GitOrigin-RevId: e7eb999a0770eba6cce5f389005896e64928f02b
## Why
High-volume request and streamed-response payloads can overwhelm the SQLite log database and diagnostic ring buffer used for reports.
## What changed
- Limit HTTP transport, SSE, and WebSocket diagnostics to `DEBUG` in persistent log sinks while leaving unrelated trace diagnostics available.
- Log known unhandled response events and delta events at `TRACE`, and surface unexpected event kinds at `DEBUG` without including their payloads.
- Report structured parse-error metadata instead of logging an unparseable SSE payload.
## Testing
- Cover filtering for transport, SSE, WebSocket, and unrelated trace records in both report and SQLite log sinks.
- Cover unknown and unsupported delta response events.
GitOrigin-RevId: 6d9121e093ddadf6834da394df544d9c09d8aeb9
## What changed
- Persist `rmcp` and `codex_rmcp_client` events only at `INFO` level or above.
- Cover nested targets from both namespaces in the SQLite log filter test.
GitOrigin-RevId: dfeeec302a9562f533a09a735b392d202c1f7801
## Why
Codex and SQLite data can use separate home directories, but state consumers
could reconstruct database paths from the Codex home instead of consistently
using the resolved SQLite configuration.
## What changed
- Pass `SqliteConfig` through the core, rollout, state runtime, and thread store
instead of passing a directory and rebuilding the configuration downstream.
- Use that shared configuration for state, logs, memories, goals, and paginated
thread-history database access, including integrity checks and cleanup.
- Reject state database handles whose SQLite configuration does not match the
requesting store.
## Testing
Add coverage with separate Codex and SQLite homes that verifies startup
backfill, thread listing, and paginated history all use the configured SQLite
directory.
GitOrigin-RevId: 1de1cdd1d6ff1d70bbb6c360c8352e6543fb8ebf
## Why
The post-sampling trace calculated an estimated token count even when only the
always-on diagnostic and state log sinks were listening, adding unnecessary work
after every sample.
## What changed
- Move the estimate to a dedicated trace target and calculate it only when that
event is enabled.
- Disable the target for the always-on diagnostic and state log sinks while
keeping it available to explicitly configured subscribers.
## Testing
- Verify that the combined always-on subscribers do not enable the estimate
event.
GitOrigin-RevId: ea572584597dda313e7719897d0ab1fdf7362745
## What changed
- Emit `responsesapi.websocket_timing` payloads as opt-in trace events with request context, while excluding them from diagnostic uploads and persisted logs by default.
- Preserve fractional-millisecond TBT values in telemetry histograms and runtime summaries, rounding only when formatting the TUI label.
## Testing
- Cover timing-log filtering, fractional TBT collection and display, and existing whole-millisecond duration behavior.
GitOrigin-RevId: 5f0a1c60c237abed17cc8b2d569ca77fbbc6ec41
## Why
The SQLite feedback log currently persists Hyper's TRACE, DEBUG, and
INFO connection-pool bookkeeping. Healthy connects, checkouts, reuse,
and timer activity generated thousands of small rows during active
sessions without adding useful feedback context.
## What changed
- Set the `hyper_util` target prefix to WARN for the SQLite feedback-log
layer.
- Continue persisting Hyper warnings and errors.
- Leave other tracing and OpenTelemetry subscribers unchanged.
This removes routine dependency chatter while preserving actionable
failures.
Related to #28224.
## Why
The SQLite feedback log enables TRACE by default. `rmcp::service`
renders complete MCP service events at TRACE, including large tool
catalogs and other protocol payloads. In a busy local sample, this was
the largest retained source and individual rows reached about 2 MiB.
These rows can quickly consume the per-thread retention budget, displace
smaller diagnostics, and add unnecessary SQLite insert-and-prune work.
## What changed
- Persist `rmcp::service` at INFO and above in the SQLite feedback log.
- Keep service initialization, shutdown, warnings, and errors available
for feedback.
- Leave other tracing and OpenTelemetry subscribers unchanged.
This is intentionally target-specific. Broader persistent-log policy
changes can remain separate.
Related to #28224.
## Why
The 0.142.0 persistent-log filter disables target=log, but bridged log
records are filtered using their original dependency target before
tracing-log emits them as target=log. This allowed high-volume
dependency TRACE events to keep reaching SQLite.
This is a follow-up to #28224.
## What changed
- Reject bridged target=log events inside the SQLite sink before
formatting or queueing them.
## Why
The local SQLite log sink currently enables TRACE for every target. This
persists high-volume dependency logs bridged through `target=log` and
duplicates OpenTelemetry mirror events in `codex_otel.log_only` and
`codex_otel.trace_safe`.
These records rapidly consume the per-partition log budget and cause
unnecessary SQLite insert-and-prune churn.
## What changed
- Keep TRACE persistence for other targets.
- Exclude bridged `target=log` events from the SQLite sink.
- Exclude the two `codex_otel` mirror targets from the SQLite sink.
- Share the same filter between app-server and TUI.
Remote OpenTelemetry export and metrics are unchanged.
## Summary
- skip `opentelemetry_sdk` DEBUG and TRACE events before formatting or
queueing them for the SQLite log sink
- preserve INFO, WARN, and ERROR events from the SDK, along with TRACE
events from application targets
- add a persistence-level regression test for the target and level
policy
## Why
OpenTelemetry's batch log processor emits internal
`BatchLogProcessor.ExportingDueToTimer` meta-events every second per
Codex process. In measured high-fanout `logs_2.sqlite` databases,
low-level `opentelemetry_sdk` events accounted for over 30% of retained
rows (30-60% on the machines of people I asked to check).
Persisting this SDK bookkeeping across many processes adds substantial
write volume and contention without representing application activity.
## Validation
- `just test -p codex-state` (132/132 tests passed, plus bench smoke)
- `just fix -p codex-state`
- `just fmt`
## Why
The log DB writer batches tracing events before inserting them into
SQLite, but `tokio::time::interval` produces an immediate first tick.
That meant the inserter could flush the first accepted log entry before
`batch_size` was reached, making
`configured_batch_size_flushes_without_explicit_flush` timing-sensitive
in CI.
## What Changed
- Consume the interval's startup tick before entering the inserter loop,
so interval flushing starts after the configured delay.
- Remove the test's startup sleep, which was masking the race instead of
proving the batch-size behavior.
## Validation
- `cargo test -p codex-state`
- `cargo test -p codex-state
configured_batch_size_flushes_without_explicit_flush` passed 3
consecutive focused runs
- PR checks passed across `rust-ci`, Bazel, `ci`, `sdk`, `cargo-deny`,
Codespell, blob-size policy, and CLA
## Why
This prepares feedback log capture for a future remote app-server hook
sink without changing the current local SQLite upload path. The
important boundary is now intentionally small: a log sink is a tracing
`Layer` that can also flush entries it has accepted.
That keeps the existing SQLite implementation simple while giving the
upcoming gRPC sink a place to fit beside it. SQLite and gRPC have
different worker/write semantics, so this PR avoids introducing a shared
buffered-sink abstraction and instead lets each `LogWriter` own the
buffering mechanics it needs.
## What Changed
- Added `LogSinkQueueConfig` with the existing local defaults: queue
capacity `512`, batch size `128`, and flush interval `2s`.
- Added `LogDbLayer::start_with_config(...)` while preserving
`LogDbLayer::start(...)` and `log_db::start(...)` defaults.
- Introduced the `LogWriter` trait as the minimal shared interface:
`tracing_subscriber::Layer` plus `flush()`.
- Made `LogDbLayer` implement `LogWriter`.
- Kept tracing event formatting inside `LogDbLayer`; it still creates
one `LogEntry` per tracing event before queueing it for SQLite.
- Kept normal event capture best-effort and non-blocking via bounded
`try_send`.
## Behavior Notes
This does not change the SQLite schema, retention behavior,
`/feedback/upload`, or Sentry upload behavior. Normal log events still
drop when the queue is full; explicit `flush()` still waits for queue
capacity and receiver processing before returning.
## Verification
- `cargo test -p codex-state log_db`
- `cargo test -p codex-state`
- `just fix -p codex-state`
The added tests cover configured batch-size flushing, configured
interval flushing, queue-full drops, and the flush barrier semantics.
## Summary
- store a pre-rendered `feedback_log_body` in SQLite so `/feedback`
exports keep span prefixes and structured event fields
- render SQLite feedback exports with timestamps and level prefixes to
match the old in-memory feedback formatter, while preserving existing
trailing newlines
- count `feedback_log_body` in the SQLite retention budget so structured
or span-prefixed rows still prune correctly
- bound `/feedback` row loading in SQL with the retention estimate, then
apply exact whole-line truncation in Rust so uploads stay capped without
splitting lines
## Details
- add a `feedback_log_body` column to `logs` and backfill it from
`message` for existing rows
- capture span names plus formatted span and event fields at write time,
since SQLite does not retain enough structure to reconstruct the old
formatter later
- keep SQLite feedback queries scoped to the requested thread plus
same-process threadless rows
- restore a SQL-side cumulative `estimated_bytes` cap for feedback
export queries so over-retained partitions do not load every matching
row before truncation
- add focused formatting coverage for exported feedback lines and parity
coverage against `tracing_subscriber`
## Testing
- cargo test -p codex-state
- just fix -p codex-state
- just fmt
codex author: `codex resume 019ca1b0-0ecc-78b1-85eb-6befdd7e4f1f`
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- reduce the SQLite-backed log retention window from 90 days to 10 days
## Testing
- just fmt
- cargo test -p codex-state
Co-authored-by: Codex <noreply@openai.com>
## Summary
- move sqlite log reads and writes onto a dedicated `logs_1.sqlite`
database to reduce lock contention with the main state DB
- add a dedicated logs migrator and route `codex-state-logs` to the new
database path
- leave the old `logs` table in the existing state DB untouched for now
## Testing
- just fmt
- cargo test -p codex-state
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- keep the SQLite schema unchanged (no migrations)
- add timestamps to SQLite-backed `/feedback` log exports
- keep the existing SQL-side byte cap behavior and newline handling
- document the remaining fidelity gap (span prefixes + structured
fields) with TODOs
## Details
- update `query_feedback_logs` to format each exported line as:
- `YYYY-MM-DDTHH:MM:SS.ffffffZ {level} {message}`
- continue scoping rows to requested-thread + same-process threadless
logs
- continue capping in SQL before returning rows
- keep the existing fallback behavior unchanged when SQLite returns no
rows
- update parity tests to normalize away the new timestamp prefix while
we still only store `message`
## Follow-up
- TODO already in code: persist enough span/event metadata in SQLite to
reproduce span prefixes and structured fields in `/feedback` exports
## Testing
- `cargo test -p codex-state`
- `just fmt`
---------
Co-authored-by: Codex <noreply@openai.com>
## Summary
- write app-server SQLite logs at TRACE level when SQLite is enabled
- source app-server `/feedback` log attachments from SQLite for the
requested thread when available
- flush buffered SQLite log writes before `/feedback` queries them so
newly emitted events are not lost behind the async inserter
- include same-process threadless SQLite rows in those `/feedback` logs
so the attachment matches the process-wide feedback buffer more closely
- keep the existing in-memory ring buffer fallback unchanged, including
when the SQLite query returns no rows
## Details
- add a byte-bounded `query_feedback_logs` helper in `codex-state` so
`/feedback` does not fetch all rows before truncating
- scope SQLite feedback logs to the requested thread plus threadless
rows from the same `process_uuid`
- format exported SQLite feedback lines with the log level prefix to
better match the in-memory feedback formatter
- add an explicit `LogDbLayer::flush()` control path and await it in
app-server before querying SQLite for feedback logs
- pass optional SQLite log bytes through `codex-feedback` as the
`codex-logs.log` attachment override
- leave TUI behavior unchanged apart from the updated `upload_feedback`
call signature
- add regression coverage for:
- newest-within-budget ordering
- excluding oversized newest rows
- including same-process threadless rows
- keeping the newest suffix across mixed thread and threadless rows
- matching the feedback formatter shape aside from span prefixes
- falling back to the in-memory snapshot when SQLite returns no logs
- flushing buffered SQLite rows before querying
## Follow-up
- SQLite feedback exports still do not reproduce span prefixes like
`feedback-thread{thread_id=...}:`; there is a `TODO(ccunningham)` in
`codex-rs/state/src/log_db.rs` for that follow-up.
## Testing
- `cd codex-rs && cargo test -p codex-state`
- `cd codex-rs && cargo test -p codex-app-server`
- `cd codex-rs && just fmt`
## Summary
This PR is the first slice of the per-session `/feedback` logging work:
it adds a process-unique identifier to SQLite log rows.
It does **not** change `/feedback` sourcing behavior yet.
## Changes
- Add migration `0009_logs_process_id.sql` to extend `logs` with:
- `process_uuid TEXT`
- `idx_logs_process_uuid` index
- Extend state log models:
- `LogEntry.process_uuid: Option<String>`
- `LogRow.process_uuid: Option<String>`
- Stamp each log row with a stable per-process UUID in the sqlite log
layer:
- generated once per process as `pid:<pid>:<uuid>`
- Update sqlite log insert/query paths to persist and read
`process_uuid`:
- `INSERT INTO logs (..., process_uuid, ...)`
- `SELECT ..., process_uuid, ... FROM logs`
## Why
App-server runs many sessions in one process. This change provides a
process-scoping primitive we need for follow-up `/feedback` work, so
threadless/process-level logs can be associated with the emitting
process without mixing across processes.
## Non-goals in this PR
- No `/feedback` transport/source changes
- No attachment size changes
- No sqlite retention/trim policy changes
## Testing
- `just fmt`
- CI will run the full checks
Alexi Christakis