Commit Graph
20 Commits
Author SHA1 Message Date
ioanbudea-OAI ad5e6e57a1 Bind memory consolidation permissions to the memory root (#59507)
## What changed

Configure managed memory consolidation agents with a `PermissionProfile` directly and explicitly bind both configuration and permission workspace roots to the memory directory. Keep network access disabled and exclude temporary directories from writable roots. Preserve disabled and externally managed parent permission profiles.

## Testing

Extend regression coverage to check that inherited workspace roots remain readable but are not writable, protected metadata stays read-only whether or not it exists, and incompatible permission constraints prevent agent configuration. Verify effective filesystem enforcement, network policy, and disabled approvals across parent permission profiles.

GitOrigin-RevId: b1ae115cade059da9e5fb83c218a1d89185b4a70
2026-11-04 22:43:34 +00:00
mandal-OAI 02bc25c89f Configure memory consolidation service tiers independently (#57568)
## Why

Memory consolidation inherits the session's service tier. Give background consolidation its own setting, defaulting to Standard.

## What changed

Add `memories.consolidation_service_tier` with a default of `"default"` and apply it to the consolidation agent. Normalize recognized tier aliases, including `"fast"` to `"priority"`, and expose the setting in the configuration schema.

## Testing

Add coverage for configuration parsing and alias normalization, Standard consolidation despite an Ultrafast session tier, and an explicit `"flex"` consolidation override.

GitOrigin-RevId: 58031ad56196152d212531d56e7fe9813f83317e
2026-10-27 21:51:26 +00:00
jif 90894a780d Prevent memory imports from racing with consolidation (#56462)
## Why

Concurrent imports and consolidation can modify the same memory workspace, leaving a partial import across memory versions or interfering with consolidation snapshots and baselines.

## What changed

- Add a cross-process workspace lock shared by memory imports and consolidation.
- Acquire locks for every import destination before modifying any workspace. Return a retryable busy error if a destination is locked, and skip consolidation when its workspace is busy.
- Hold locks through import completion even if the caller drops the future, and through consolidation agent termination and baseline finalization.

## Testing

Add an app-server integration test that blocks an active consolidation, verifies import rejection without partial writes or snapshot changes, then retries successfully and confirms imported resources remain visible to the next consolidation.

GitOrigin-RevId: c4ce945f916dcb4d75a3190ea869751fac51de83
2026-10-23 12:23:10 +00:00
jif 6500c1f844 Record memory storage size after successful consolidation (#45956)
## What changed

Emit a `codex.memory.storage_bytes` histogram after successful phase-two consolidation, including runs with no workspace changes. Sum regular-file sizes recursively from filesystem metadata, excluding `.git` entries and symbolic links. Stop the phase-two duration timer before measuring storage.

## Testing

Add coverage for both memory versions, nested files, UTF-8 byte counts, Git metadata exclusion, and Unix symbolic links, including broken links and directory loops. Extend the workspace baseline reset test to verify the remaining storage size.

GitOrigin-RevId: 9170c34d656728897790decd2e2cabb38a81836f
2026-09-16 14:49:24 +00:00
jif 553df1c691 Add dedicated memory v2 consolidation and read prompts (#43813)
## What changed

- Consolidate v2 rollout summaries into `memory_summary.md` without generating `raw_memories.md` or requiring `MEMORY.md`. Validate the summary's required sections and size below 10,000 UTF-8 bytes.
- Add v2 read instructions for selective history retrieval, evidence-grounded preferences, citations, and explicit memory edits. Split injected instructions into fragments to preserve the complete summary within fragment byte limits.
- Record memory citation usage in the store selected by `memories.version`.

## Testing

Add coverage for v2 consolidation without a handbook, summary validation, version-isolated memory reads, and resetting both memory versions while preserving threads.

GitOrigin-RevId: 1d895fb4a23a973f1a45ba03be07a1f480c10227
2026-09-08 12:58:11 +00:00
jif 3f76e88a48 Add configurable memory versions with isolated storage (#43797)
## What changed

- Add `memories.version` with `v1` as the default and `v2` as an opt-in selection.
- Route memory generation, summaries, and retrieval tools through the selected version. Store `v2` artifacts in `memories_v2` and lazily create a separate SQLite database, keeping jobs and outputs isolated while sharing the thread catalog.
- Preserve the initial memory version across extension config updates so summaries and retrieval tools use the same namespace.
- Clear both versions on memory reset and remove thread memory from both stores on thread deletion. Recognize both artifact roots in shell usage telemetry.

## Testing

Add regression coverage for version defaults and validation, independent job claims and outputs, deletion and reset across versions, and shell usage classification for both memory roots.

GitOrigin-RevId: aa799bf87ab6ec10c8f5668213931e57547d080b
2026-09-08 10:55:48 +00:00
jif 7c6eb0eef1 Scope stop hooks for memory consolidation (#40587)
## Why

Background memory consolidation must honor managed completion policies without
running project completion checks or sending user-facing notifications.

## What changed

- Add a memory-consolidation stop target that selects managed policy hooks and
  executor-scoped cleanup while excluding user, project, session, and plugin
  stop hooks.
- End memory consolidation with an error when a managed stop hook rejects it,
  and propagate that turn error into agent status.
- Disable legacy completion notifications for the background memory worker.

## Testing

- Cover hook selection, executor cleanup, managed rejection, agent error
  status, and notification suppression.

GitOrigin-RevId: be350679ddcde2dde8a408b0972f08facff01a37
2026-08-25 10:29:26 +00:00
jif a04940cb12 Reject symbolic links in memory workspaces (#39205)
## What changed

- Reject a symbolic link used as the memory root.
- Recursively remove symbolic links before preparing or seeding the memory workspace.
- Remove and reject symbolic links created during consolidation, including when the worker fails, so they cannot affect files outside the workspace.

## Testing

Add Unix startup tests covering pre-existing extension links and links created by both successful and failed consolidation workers.

GitOrigin-RevId: f96e03812773f7beaa16455dd77370ee47d8a71f
2026-08-18 14:14:57 +00:00
jif 5ad367fb95 Wait for memory consolidation shutdown before finishing jobs (#36140)
## Why

A phase-two memory job could release its lease while its consolidation agent was
still shutting down, allowing another worker to race the existing agent.

## What changed

- Await consolidation-agent shutdown before validating artifacts and completing
  the job.
- Keep the lease until it expires if shutdown fails, and remove the agent from
  the thread manager only after shutdown succeeds.

## Testing

Update startup tests to wait for the phase-two job lifecycle to finish and verify
that the consolidation thread has been removed.

GitOrigin-RevId: 7ac3a7369529822ac95fd2ee9dcb27d39c191d20
2026-07-30 11:52:38 +00:00
jif 687f05cb94 Remove CSV-backed agent jobs (#34413)
## What changed

- Remove the `spawn_agents_on_csv` and `report_agent_job_result` tools and their agent-job runtime and state models.
- Drop the legacy `agent_jobs` and `agent_job_items` tables during state database migration.
- Keep `features.enable_fanout` and `agents.job_max_runtime_seconds` accepted as no-op compatibility settings while omitting them from the generated configuration schema.

## Testing

- Verify upgrades remove both legacy agent-job tables.
- Verify the removed feature and configuration keys still parse without taking effect.

GitOrigin-RevId: 8cc3337da78c67162229f02f40a747f503542646
2026-07-20 21:00:17 +00:00
Eric Traut 2edad72de3 Track prompt cache write token usage (#33454)
## What changed

- Parse `cache_write_tokens` from response input token details and carry it through token usage aggregation.
- Expose `cache_write_input_tokens` in protocol, app-server, exec, and TypeScript SDK usage events, defaulting it to zero for compatibility with older payloads.
- Report cache-write usage in analytics, metrics, traces, and rollout data alongside existing cache-read usage.

## Testing

- Add coverage for response parsing, usage event output, aggregation, analytics, metrics, and telemetry fields.

GitOrigin-RevId: 634e4fbe3086a8923c0ee13b622573e7d9d221bd
2026-07-16 00:22:53 +00:00
jif 54b8f112a3 Preserve parent sandbox enforcement for memory consolidation (#32441)
## What changed

- Pass the parent turn's effective permission profile to the memory consolidation agent, including thread-level permission and legacy sandbox overrides.
- Preserve disabled and externally enforced permission profiles instead of replacing them with a managed sandbox.
- Continue restricting consolidation to the memory root without network access when the parent uses Codex-managed permissions.

## Testing

- Add coverage for disabled, external, and managed parent permission profiles.

GitOrigin-RevId: 9ca3be0e41dc14d858053f6f70e33f4ae7c578e1
2026-07-11 15:20:49 +00:00
jif 601b61a83f Rebind memory consolidation workspace roots (#32197)
## Why

The memory consolidation agent changes its working directory to the memory
root, but applying its sandbox policy directly to the permissions object can
leave workspace roots inherited from the parent configuration.

## What changed

Apply the consolidation sandbox policy through `Config` so its workspace roots
are synchronized with the memory root. Add a test covering the working
directory, workspace roots, and effective legacy sandbox policy.

GitOrigin-RevId: dc6e75aef1ffa84a2f66fec6ed3a6fedcf9aa672
2026-07-10 15:44:56 +00:00
jif 52c9605dc7 Validate memory consolidation artifacts before succeeding (#32193)
## Why

A completed Phase 2 agent run does not guarantee that its required outputs were
created correctly. Treating a clean workspace as success can also preserve a
state where those outputs are missing.

## What changed

- Require `MEMORY.md` to be a file and `memory_summary.md` to start with `v1`
  before marking consolidation successful.
- Fail completed runs with invalid artifacts without resetting the workspace
  baseline, allowing the job to be retried.
- Run consolidation for a clean workspace when its required artifacts are
  invalid instead of taking the no-change success path.

## Testing

Added coverage for rejecting an invalid summary and retrying a clean workspace
whose consolidation artifacts are missing.

GitOrigin-RevId: ac57b2ba9d062c5203ca51afd716795a91814bb3
2026-07-10 15:40:35 +00:00
Celia Chen 51fc4b0559 feat: use provider defaults for memory models (#27129)
## Why

Memory startup used hardcoded OpenAI model slugs for extraction and
consolidation. That works for the default OpenAI-compatible path, but
provider-specific backends can require different model identifiers. In
particular, Amazon Bedrock should use its Bedrock model ID for these
background memory requests instead of the OpenAI `gpt-5.4-mini` /
`gpt-5.4` slugs.

## What Changed

- Added provider-owned preferred memory model methods alongside
`approval_review_preferred_model`.
- Updated memory extraction and consolidation to resolve their default
model through the active `ModelProvider`.
- Added Amazon Bedrock overrides so both memory stages use
`openai.gpt-5.4` through Bedrock’s provider-specific model ID.
- Kept explicit `memories.extract_model` and
`memories.consolidation_model` config overrides taking precedence.
- Added startup coverage for default OpenAI and Bedrock memory model
selection.

#closes #26288
2026-06-09 23:49:09 +00:00
jif-oai aad59a0916 Move memory state to a dedicated SQLite DB (#24591)
## Summary

Generated memory rows and their stage-one/stage-two job state currently
live in `state_5.sqlite` alongside thread metadata. That makes memory
cleanup and regeneration share the main state schema even though those
rows are memory-pipeline data and can be rebuilt independently from the
durable thread records.

This PR moves the memory-owned tables into a dedicated
`memories_1.sqlite` runtime database while keeping thread metadata in
`state_5.sqlite`.

## Changes

- Adds a separate memories DB runtime, migrator, path helpers, telemetry
kind, and Bazel compile data for `state/memory_migrations`.
- Introduces `MemoryStore` behind `StateRuntime::memories()` and moves
memory table/job operations onto that store.
- Drops the old memory tables from the state DB and recreates their
schema in `state/memory_migrations/0001_memories.sql`.
- Updates memory startup, citation usage tracking, rollout pollution
handling, `debug clear-memories`, and app-server `memory/reset` to
operate through the memories DB.
- Preserves cross-DB behavior by hydrating thread metadata from the
state DB when selecting visible memory outputs and checking stage-one
staleness.

## Verification

- Added/updated `codex-state` tests for deleted-thread memory visibility
and already-polluted phase-two enqueue behavior.
- Updated `debug clear-memories`, app-server `memory/reset`, and
memories startup tests to seed and assert memory rows through
`memories_1.sqlite`.
2026-05-26 20:07:25 +02:00
jif-oai 5b7d6f5c4f feat: house-keeping memories 3 (#20005)
Move stuff in memories, no behavioural change expected
2026-04-28 18:13:35 +02:00
jif-oai 21e19912e0 feat: house-keeping memories 2 (#20000)
Just move metrics in a dedicated file
2026-04-28 17:26:44 +02:00
jif-oaiandCodex a9e5c34083 feat: trigger memories from user turns with cooldown (#19970)
## Why

Memory startup was tied to thread lifecycle events such as create, load,
and fork. That can run memory work before a thread receives real user
input, and it makes startup cost scale with thread management instead of
actual turns. Moving the trigger to `thread/sendInput` keeps memory
startup aligned with the first real user turn and lets it use the
current thread config at turn time.

The idea is to prevent ghost cost due to pre-warm triggered by the app

Turn-based startup can also make global phase-2 consolidation easier to
request repeatedly, so this adds a success cooldown and tightens the
default startup scan window.

## What Changed

- Start `codex_memories_write::start_memories_startup_task` after a
non-empty `thread/sendInput` turn is submitted, instead of from thread
create/load/fork paths:
https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/app-server/src/codex_message_processor.rs#L6477-L6487
- Expose `CodexThread::config()` so app-server can pass the live config
into memory startup at turn time.
- Add a six-hour successful-run cooldown for global phase-2
consolidation via `SkippedCooldown`:
https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/state/src/runtime/memories.rs#L963-L966
- Reduce memory startup defaults to at most 2 rollouts over 10 days:
https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/config/src/types.rs#L31-L34

## Verification

Updated the memory runtime coverage around phase-2 reclaim behavior,
including `phase2_global_lock_respects_success_cooldown`.

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-28 16:23:13 +02:00
jif-oaiandCodex 431ebeaef7 feat: split memories part 2 (#19860)
Keep extracting memories out of core and moving the write trigger in the
app-server
This is temporary and it should move at the client level as a follow-up
This makes core fully independant from `codex-memories-write`

---------

Co-authored-by: Codex <noreply@openai.com>
2026-04-28 13:03:28 +02:00