mirror of
https://github.com/openai/codex.git
synced 2026-11-02 19:43:24 +00:00
e8fc5ca09f114d1c02ccd9e871b9bdbac0fe1f1e
7479
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Michael Bolin
|
e8fc5ca09f | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
ee4c217a44 |
Make sandbox read access configurable with ReadOnlyAccess
## What
This change introduces a new `ReadOnlyAccess` model and threads it through
sandbox policy consumers so read access is explicit instead of implicit.
- Added `ReadOnlyAccess` to protocol:
- `Restricted { include_platform_defaults, readable_roots }`
- `FullAccess`
- Changed `SandboxPolicy` shape:
- `ReadOnly` is now `ReadOnly { access: ReadOnlyAccess }`
- `WorkspaceWrite` now carries `read_only_access: ReadOnlyAccess`
- Kept existing behavior for now by defaulting to `ReadOnlyAccess::FullAccess`
in constructors and current config/app-server mappings.
- Added helper methods to compute effective readable roots (including optional
platform defaults + cwd) and to detect full read access.
- Updated seatbelt policy generation to honor restricted read roots by emitting
scoped `(allow file-read* ...)` entries when full read access is not granted.
- Updated Linux backends (`bwrap`, legacy landlock path) to fail closed with an
explicit `UnsupportedOperation` when restricted read access is requested but
not yet implemented there.
- Updated Windows sandbox backends (standard, elevated, and runner paths) to
fail closed in the same way for restricted read access.
- Updated all call sites/tests/pattern matches for the new structured variants
and regenerated app-server protocol schema/types.
## Why
The previous `SandboxPolicy::ReadOnly` implied full-disk read access and left
no way to express a narrower read surface.
This refactor establishes the policy model needed to support user-configurable
read restrictions in a follow-up without changing current runtime behavior.
It also ensures we do not silently ignore future restricted-read policies on
platform backends that do not support them yet. Failing closed keeps sandbox
semantics predictable and avoids accidental over-permission.
## Compatibility and rollout notes
- Existing behavior is preserved by default (`FullAccess`).
- Existing config/app-server flows continue to serialize/deserialize cleanly.
- New schema artifacts are included to keep generated protocol outputs in sync.
## Validation
- `just fmt`
- `just fix -p codex-protocol -p codex-core -p codex-linux-sandbox -p codex-windows-sandbox -p codex-app-server-protocol`
- `cargo check -p codex-windows-sandbox`
- Targeted crate/test runs were executed during development for protocol/core/
sandbox-related crates.
|
||
|
Michael Bolin
|
b313ed13e3 |
Merge 884e0a0cb7 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
884e0a0cb7 |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
f4aea826f6 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
a944b6044d |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
b68a84ee8e |
Remove deterministic_process_ids feature to avoid duplicate codex-core builds (#11393)
## Why `codex-core` enabled `deterministic_process_ids` through a self dev-dependency. That forced a second feature-resolved build of the same crate, which increased compile time and test latency. ## What Changed - Removed the `deterministic_process_ids` feature from `codex-rs/core/Cargo.toml`. - Removed the self dev-dependency on `codex-core` that enabled that feature. - Removed the Bazel `deterministic_process_ids` crate feature for `codex-core`. - Added a test-only `AtomicBool` override in unified exec process-id allocation. - Added a test-support setter for that override and re-exported it from `codex-core`. - Enabled deterministic process IDs in integration tests via `core_test_support` ctor. ## Behavior - Production behavior remains random process IDs. - Unit tests remain deterministic via `cfg(test)`. - Integration tests remain deterministic via explicit test-support initialization. ## Validation - `just fmt` - `cargo test -p codex-core unified_exec::` - `cargo test -p codex-core --test all unified_exec -- --test-threads=1` - `cargo tree -p codex-core -e features` (verified the removed feature path) |
||
|
Michael Bolin
|
52cce2a428 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
727ea4b547 |
Make sandbox read access configurable with ReadOnlyAccess
## What
This change introduces a new `ReadOnlyAccess` model and threads it through
sandbox policy consumers so read access is explicit instead of implicit.
- Added `ReadOnlyAccess` to protocol:
- `Restricted { include_platform_defaults, readable_roots }`
- `FullAccess`
- Changed `SandboxPolicy` shape:
- `ReadOnly` is now `ReadOnly { access: ReadOnlyAccess }`
- `WorkspaceWrite` now carries `read_only_access: ReadOnlyAccess`
- Kept existing behavior for now by defaulting to `ReadOnlyAccess::FullAccess`
in constructors and current config/app-server mappings.
- Added helper methods to compute effective readable roots (including optional
platform defaults + cwd) and to detect full read access.
- Updated seatbelt policy generation to honor restricted read roots by emitting
scoped `(allow file-read* ...)` entries when full read access is not granted.
- Updated Linux backends (`bwrap`, legacy landlock path) to fail closed with an
explicit `UnsupportedOperation` when restricted read access is requested but
not yet implemented there.
- Updated Windows sandbox backends (standard, elevated, and runner paths) to
fail closed in the same way for restricted read access.
- Updated all call sites/tests/pattern matches for the new structured variants
and regenerated app-server protocol schema/types.
## Why
The previous `SandboxPolicy::ReadOnly` implied full-disk read access and left
no way to express a narrower read surface.
This refactor establishes the policy model needed to support user-configurable
read restrictions in a follow-up without changing current runtime behavior.
It also ensures we do not silently ignore future restricted-read policies on
platform backends that do not support them yet. Failing closed keeps sandbox
semantics predictable and avoids accidental over-permission.
## Compatibility and rollout notes
- Existing behavior is preserved by default (`FullAccess`).
- Existing config/app-server flows continue to serialize/deserialize cleanly.
- New schema artifacts are included to keep generated protocol outputs in sync.
## Validation
- `just fmt`
- `just fix -p codex-protocol -p codex-core -p codex-linux-sandbox -p codex-windows-sandbox -p codex-app-server-protocol`
- `cargo check -p codex-windows-sandbox`
- Targeted crate/test runs were executed during development for protocol/core/
sandbox-related crates.
|
||
|
Michael Bolin
|
71bd4e5bd9 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
31eddcc7db |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
43977a9f39 |
Merge 5338d44cc1 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
5338d44cc1 |
Make sandbox read access configurable with ReadOnlyAccess
## What
This change introduces a new `ReadOnlyAccess` model and threads it through
sandbox policy consumers so read access is explicit instead of implicit.
- Added `ReadOnlyAccess` to protocol:
- `Restricted { include_platform_defaults, readable_roots }`
- `FullAccess`
- Changed `SandboxPolicy` shape:
- `ReadOnly` is now `ReadOnly { access: ReadOnlyAccess }`
- `WorkspaceWrite` now carries `read_only_access: ReadOnlyAccess`
- Kept existing behavior for now by defaulting to `ReadOnlyAccess::FullAccess`
in constructors and current config/app-server mappings.
- Added helper methods to compute effective readable roots (including optional
platform defaults + cwd) and to detect full read access.
- Updated seatbelt policy generation to honor restricted read roots by emitting
scoped `(allow file-read* ...)` entries when full read access is not granted.
- Updated Linux backends (`bwrap`, legacy landlock path) to fail closed with an
explicit `UnsupportedOperation` when restricted read access is requested but
not yet implemented there.
- Updated Windows sandbox backends (standard, elevated, and runner paths) to
fail closed in the same way for restricted read access.
- Updated all call sites/tests/pattern matches for the new structured variants
and regenerated app-server protocol schema/types.
## Why
The previous `SandboxPolicy::ReadOnly` implied full-disk read access and left
no way to express a narrower read surface.
This refactor establishes the policy model needed to support user-configurable
read restrictions in a follow-up without changing current runtime behavior.
It also ensures we do not silently ignore future restricted-read policies on
platform backends that do not support them yet. Failing closed keeps sandbox
semantics predictable and avoids accidental over-permission.
## Compatibility and rollout notes
- Existing behavior is preserved by default (`FullAccess`).
- Existing config/app-server flows continue to serialize/deserialize cleanly.
- New schema artifacts are included to keep generated protocol outputs in sync.
## Validation
- `just fmt`
- `just fix -p codex-protocol -p codex-core -p codex-linux-sandbox -p codex-windows-sandbox -p codex-app-server-protocol`
- `cargo check -p codex-windows-sandbox`
- Targeted crate/test runs were executed during development for protocol/core/
sandbox-related crates.
|
||
|
Charley Cunningham
|
8b46c0ce00 |
tui: queue non-pending rollback trims in app-event order (#11373)
## Summary
This PR fixes TUI transcript-sync behavior for
`EventMsg::ThreadRolledBack` and makes rollback application order
deterministic.
Previously, rollback handling depended on `pending_rollback`:
- if `pending_rollback` was set (local backtrack), TUI trimmed correctly
- otherwise, replayed/external rollbacks were either ignored or could be
applied at the wrong time relative to queued transcript inserts
This change keeps the local backtrack path intact and routes non-pending
rollbacks through the app event queue so rollback trims are applied in
FIFO order with transcript cell inserts.
## What changed
- Added/used `trim_transcript_cells_drop_last_n_user_turns(...)` for
rollback-by-`num_turns` semantics.
- Renamed rollback app event:
- `AppEvent::ApplyReplayedThreadRollback` ->
`AppEvent::ApplyThreadRollback`
- Replay path (`ChatWidget`) now emits `ApplyThreadRollback`.
- Live non-pending rollback path (`App::handle_backtrack_event`) now
emits `ApplyThreadRollback` instead of trimming immediately.
- App-level event handler applies `ApplyThreadRollback` after queued
`InsertHistoryCell` events and schedules redraw only when a trim
occurred.
- When a trim occurs with an overlay open, TUI now syncs transcript
overlay committed cells, clamps backtrack preview selection, and clears
stale `deferred_history_lines` so closed overlays do not re-append
rolled-back lines.
- Clarified inline comments around the `pending_rollback` branch so
future readers can reason about why there are two paths.
## Why queueing matters
During resume/replay, transcript cells are populated via queued
`InsertHistoryCell` app events. If a rollback is applied immediately
outside that queue, it can run against an incomplete transcript and
under-trim. Queueing non-pending rollbacks ensures consistent ordering
and correct final transcript state.
## Behavior by rollback source
- `pending_rollback = Some(...)` (local backtrack requested by this
TUI):
- use `finish_pending_backtrack()` and the stored selection boundary
- `pending_rollback = None` (replay/external/non-local rollback):
- enqueue `AppEvent::ApplyThreadRollback { num_turns }` and trim in
app-event order
## Tests
Added/updated tests covering ordering and semantics:
-
`app_backtrack::tests::trim_drop_last_n_user_turns_applies_rollback_semantics`
- `app_backtrack::tests::trim_drop_last_n_user_turns_allows_overflow`
- `app::tests::replayed_initial_messages_apply_rollback_in_queue_order`
-
`app::tests::live_rollback_during_replay_is_applied_in_app_event_order`
-
`app::tests::queued_rollback_syncs_overlay_and_clears_deferred_history`
- `chatwidget::tests::replayed_thread_rollback_emits_ordered_app_event`
Validation run:
- `just fmt`
- `cargo test -p codex-tui`
|
||
|
Michael Bolin
|
b44ad66833 |
Merge 6311cd8b59 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
6311cd8b59 |
Remove deterministic_process_ids feature to avoid duplicate codex-core builds
## Why `codex-core` enabled `deterministic_process_ids` through a self dev-dependency. That forced a second feature-resolved build of the same crate, which increased compile time and test latency. ## What Changed - Removed the `deterministic_process_ids` feature from `codex-rs/core/Cargo.toml`. - Removed the self dev-dependency on `codex-core` that enabled that feature. - Removed the Bazel `deterministic_process_ids` crate feature for `codex-core`. - Added a test-only `AtomicBool` override in unified exec process-id allocation. - Added a test-support setter for that override and re-exported it from `codex-core`. - Enabled deterministic process IDs in integration tests via `core_test_support` ctor. ## Behavior - Production behavior remains random process IDs. - Unit tests remain deterministic via `cfg(test)`. - Integration tests remain deterministic via explicit test-support initialization. ## Validation - `just fmt` - `cargo test -p codex-core unified_exec::` - `cargo test -p codex-core --test all unified_exec -- --test-threads=1` - `cargo tree -p codex-core -e features` (verified the removed feature path) |
||
|
pakrym-oai
|
c68999ee6d |
Prefer websocket transport when model opts in (#11386)
Summary - add a `prefer_websockets` field to `ModelInfo`, defaulting to `false` in all fixtures and constructors - wire the new flag into websocket selection so models that opt in always use websocket transport even when the feature gate is off Testing - Not run (not requested) |
||
|
pakrym-oai
|
bfd4e2112c |
Disable very flaky tests (#11394)
Collected from last 20 builds of main in https://github.com/openai/codex/commits/main/. |
||
|
Michael Bolin
|
d5d84256a8 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
0da458e025 |
Remove deterministic_process_ids feature to avoid duplicate codex-core builds
## Why `codex-core` enabled `deterministic_process_ids` through a self dev-dependency. That forced a second feature-resolved build of the same crate, which increased compile time and test latency. ## What Changed - Removed the `deterministic_process_ids` feature from `codex-rs/core/Cargo.toml`. - Removed the self dev-dependency on `codex-core` that enabled that feature. - Removed the Bazel `deterministic_process_ids` crate feature for `codex-core`. - Added a test-only `AtomicBool` override in unified exec process-id allocation. - Added a test-support setter for that override and re-exported it from `codex-core`. - Enabled deterministic process IDs in integration tests via `core_test_support` ctor. ## Behavior - Production behavior remains random process IDs. - Unit tests remain deterministic via `cfg(test)`. - Integration tests remain deterministic via explicit test-support initialization. ## Validation - `just fmt` - `cargo test -p codex-core unified_exec::` - `cargo test -p codex-core --test all unified_exec -- --test-threads=1` - `cargo tree -p codex-core -e features` (verified the removed feature path) |
||
|
Michael Bolin
|
6f4f1ac440 |
Merge b8acede333 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
b8acede333 |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
e223f9c1c2 |
Merge fd2bca17dd into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
fd2bca17dd |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
31bfcd56cf | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
41c7eb1caf |
Extract codex-config from codex-core
## Why
`codex-core` has accumulated config loading, requirements parsing, constraint logic, and config-layer state handling in one large crate. This refactor pulls that cohesive subsystem into a dedicated crate so we can reduce the compile/test surface area of `codex-core` and make future config work more isolated.
This is part of the broader goal of right-sizing crates to reduce monolithic rebuild cost and improve incremental development speed.
## What Changed
### New crate
- Added a new workspace crate: `codex-rs/config` (`codex-config`)
- Added workspace wiring in `codex-rs/Cargo.toml`
- Added dependency from `codex-core` to `codex-config`
### Moved config internals from `core` to `config`
Moved these modules into `codex-config`:
- `core/src/config/constraint.rs` -> `config/src/constraint.rs`
- `core/src/config_loader/cloud_requirements.rs` -> `config/src/config_loader/cloud_requirements.rs`
- `core/src/config_loader/config_requirements.rs` -> `config/src/config_loader/config_requirements.rs`
- `core/src/config_loader/fingerprint.rs` -> `config/src/config_loader/fingerprint.rs`
- `core/src/config_loader/merge.rs` -> `config/src/config_loader/merge.rs`
- `core/src/config_loader/overrides.rs` -> `config/src/config_loader/overrides.rs`
- `core/src/config_loader/requirements_exec_policy.rs` -> `config/src/config_loader/requirements_exec_policy.rs`
- `core/src/config_loader/state.rs` -> `config/src/config_loader/state.rs`
### Removed shim modules in `core`
After the move, the temporary one-line `pub use` shim files under `core/src/config_loader/` were deleted so history is a clean move/delete rather than introducing extra permanent forwarding modules.
`core/src/config_loader/mod.rs` now imports/re-exports directly from `codex_config`, including direct use of `build_cli_overrides_layer` and test-only access to `version_for_toml`.
### Follow-on fixes for direct imports
- Updated `core/src/config_loader/macos.rs` to use `super::{ConfigRequirementsToml, ConfigRequirementsWithSources, RequirementSource}`.
- Updated `core/src/config_loader/tests.rs` imports to reference `crate::config_loader` re-exports and `codex_config` exec-policy TOML types.
## Behavior and API Notes
- Config behavior is intended to be unchanged.
- `codex-core` continues to expose the same config-loader-facing API surface to its internal callers via `core/src/config_loader/mod.rs` re-exports.
- The main functional change is crate ownership and dependency direction, not config semantics.
## Validation
Ran:
- `cargo test -p codex-config`
- `cargo test -p codex-core --no-run`
- `cargo test -p codex-core config_loader::tests::load_requirements_toml_produces_expected_constraints`
- `cargo test -p codex-core config_loader::tests::requirements_exec_policy_tests::parses_single_prefix_rule_from_raw_toml`
- `just fmt`
- `just fix -p codex-config -p codex-core`
- `just fix -p codex-core`
All listed commands completed successfully in this workspace environment.
|
||
|
Michael Bolin
|
c1aa935f8c |
Merge 9fe4ec9567 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
9fe4ec9567 |
Make sandbox read access configurable with ReadOnlyAccess
## What
This change introduces a new `ReadOnlyAccess` model and threads it through
sandbox policy consumers so read access is explicit instead of implicit.
- Added `ReadOnlyAccess` to protocol:
- `Restricted { include_platform_defaults, readable_roots }`
- `FullAccess`
- Changed `SandboxPolicy` shape:
- `ReadOnly` is now `ReadOnly { access: ReadOnlyAccess }`
- `WorkspaceWrite` now carries `read_only_access: ReadOnlyAccess`
- Kept existing behavior for now by defaulting to `ReadOnlyAccess::FullAccess`
in constructors and current config/app-server mappings.
- Added helper methods to compute effective readable roots (including optional
platform defaults + cwd) and to detect full read access.
- Updated seatbelt policy generation to honor restricted read roots by emitting
scoped `(allow file-read* ...)` entries when full read access is not granted.
- Updated Linux backends (`bwrap`, legacy landlock path) to fail closed with an
explicit `UnsupportedOperation` when restricted read access is requested but
not yet implemented there.
- Updated Windows sandbox backends (standard, elevated, and runner paths) to
fail closed in the same way for restricted read access.
- Updated all call sites/tests/pattern matches for the new structured variants
and regenerated app-server protocol schema/types.
## Why
The previous `SandboxPolicy::ReadOnly` implied full-disk read access and left
no way to express a narrower read surface.
This refactor establishes the policy model needed to support user-configurable
read restrictions in a follow-up without changing current runtime behavior.
It also ensures we do not silently ignore future restricted-read policies on
platform backends that do not support them yet. Failing closed keeps sandbox
semantics predictable and avoids accidental over-permission.
## Compatibility and rollout notes
- Existing behavior is preserved by default (`FullAccess`).
- Existing config/app-server flows continue to serialize/deserialize cleanly.
- New schema artifacts are included to keep generated protocol outputs in sync.
## Validation
- `just fmt`
- `just fix -p codex-protocol -p codex-core -p codex-linux-sandbox -p codex-windows-sandbox -p codex-app-server-protocol`
- `cargo check -p codex-windows-sandbox`
- Targeted crate/test runs were executed during development for protocol/core/
sandbox-related crates.
|
||
|
|
f101300dba |
Update models.json (#11376)
Automated update of models.json. --------- Co-authored-by: aibrahim-oai <219906144+aibrahim-oai@users.noreply.github.com> Co-authored-by: sayan-oai <sayan@openai.com> |
||
|
Michael Bolin
|
d44f4205fb |
chore: rename codex-command to codex-shell-command (#11378)
This addresses some post-merge feedback on https://github.com/openai/codex/pull/11361: - crate rename - reuse `detect_shell_type()` utility |
||
|
jif-oai
|
87bbfc50a1 |
feat: prevent double backfill (#11377)
## Summary Add a DB-backed lease to prevent duplicate `.sqlite` backfill workers from running concurrently. ### What changed - Added StateRuntime::try_claim_backfill(lease_seconds) that atomically claims backfill only when: - backfill is not complete, and - no fresh running worker currently owns it. - Updated backfill_sessions to use the claim API and exit early when another worker already holds the lease. - Added runtime tests covering: - singleton claim behavior, - stale lease takeover, - claim blocked after complete. - Set backfill lease to 900s in production and 1s in tests. ### Why This avoids duplicate backfill work and reduces backfill status churn under concurrent startup, while preserving current best-effort fallback behavior. |
||
|
Michael Bolin
|
c863478f4f | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
9feb54cbc6 |
chore: rename codex-command to codex-shell-command
This addresses some post-merge feedback on https://github.com/openai/codex/pull/11361: - crate rename - reuse `detect_shell_type()` utility |
||
|
jif-oai
|
674799d356 | feat: mem v2 - PR6 (consolidation) (#11374) | ||
|
Michael Bolin
|
8b807735e5 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
708927300f |
chore: rename codex-command to codex-shell-command
This addresses some post-merge feedback on https://github.com/openai/codex/pull/11361: - crate rename - reuse `detect_shell_type()` utility |
||
|
jif-oai
|
2c9be54c9a | feat: mem v2 - PR5 (#11372) | ||
|
Josh McKinney
|
34fb4b6e63 |
ci: fall back to local Bazel on forks without BuildBuddy key (#11359)
## Summary
- detect whether BUILDBUDDY_API_KEY is present in Bazel CI
- keep existing remote BuildBuddy path when key is available
- add a local fallback path for fork PRs without secrets by clearing
remote cache/executor/BES endpoints
- document each fallback flag inline with links to Bazel docs
## Testing
- ruby -e 'require "yaml";
YAML.load_file(".github/workflows/bazel.yml"); puts "ok"'
- verified Bazel docs/flag references used in workflow comments
|
||
|
viyatb-oai
|
1d47927aa0 |
Enable SOCKS defaults for common local network proxy use cases (#11362)
## Summary - enable local-use defaults in network proxy settings: SOCKS5 on, SOCKS5 UDP on, upstream proxying on, and local binding on - add a regression test that asserts the full `NetworkProxySettings::default()` baseline - Fixed managed listener reservation behavior. Before: we always reserved a loopback SOCKS listener, even when enable_socks5 = false. Now: SOCKS listener is only reserved when SOCKS is enabled. - Fixed /debug-config env output for SOCKS-disabled sessions. ALL_PROXY now shows the HTTP proxy URL when SOCKS is disabled (instead of incorrectly showing socks5h://...). ## Validation - just fmt - cargo test -p codex-network-proxy - cargo clippy -p codex-network-proxy --all-targets |
||
|
jif-oai
|
623d3f4071 |
feat: mem v2 - PR4 (#11369)
# Memories migration plan (simplified global workflow) ## Target behavior - One shared memory root only: `~/.codex/memories/`. - No per-cwd memory buckets, no cwd hash handling. - Phase 1 candidate rules: - Not currently being processed unless the job lease is stale. - Rollout updated within the max-age window (currently 30 days). - Rollout idle for at least 12 hours (new constant). - Global cap: at most 64 stage-1 jobs in `running` state at any time (new invariant). - Stage-1 model output shape (new): - `rollout_slug` (accepted but ignored for now). - `rollout_summary`. - `raw_memory`. - Phase-1 artifacts written under the shared root: - `rollout_summaries/<thread_id>.md` for each rollout summary. - `raw_memories.md` containing appended/merged raw memory paragraphs. - Phase 2 runs one consolidation agent for the shared `memories/` directory. - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry. ## Current code map - Core startup pipeline: `core/src/memories/startup/mod.rs`. - Stage-1 request+parse: `core/src/memories/startup/extract.rs`, `core/src/memories/stage_one.rs`, templates in `core/templates/memories/`. - File materialization: `core/src/memories/storage.rs`, `core/src/memories/layout.rs`. - Scope routing (cwd/user): `core/src/memories/scope.rs`, `core/src/memories/startup/mod.rs`. - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`. ## PR plan ## PR 1: Correct phase-1 selection invariants (no behavior-breaking layout changes yet) - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in `core/src/memories/mod.rs`. - Thread this into `state::claim_stage1_jobs_for_startup(...)`. - Enforce idle-time filter in DB selection logic (not only in-memory filtering after `scan_limit`) so eligible threads are not starved by very recent threads. - Enforce global running cap of 64 at claim time in DB logic: - Count fresh `memory_stage1` running jobs. - Only allow new claims while count < cap. - Keep stale-lease takeover behavior intact. - Add/adjust tests in `state/src/runtime.rs`: - Idle filter inclusion/exclusion around 12h boundary. - Global running-cap guarantee. - Existing stale/fresh ownership behavior still passes. Acceptance criteria: - Startup never creates more than 64 fresh `memory_stage1` running jobs. - Threads updated <12h ago are skipped. - Threads older than 30d are skipped. ## PR 2: Stage-1 output contract + storage artifacts (forward-compatible) - Update parser/types to accept the new structured output while keeping backward compatibility: - Add `rollout_slug` (optional for now). - Add `rollout_summary`. - Keep alias support for legacy `summary` and `rawMemory` until prompt swap completes. - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to include the new keys. - Update prompt templates: - `core/templates/memories/stage_one_system.md`. - `core/templates/memories/stage_one_input.md`. - Replace storage model in `core/src/memories/storage.rs`: - Introduce `rollout_summaries/` directory writer (`<thread_id>.md` files). - Introduce `raw_memories.md` aggregator writer from DB rows. - Keep deterministic rebuild behavior from DB outputs so files can always be regenerated. - Update consolidation prompt template to reference `rollout_summaries/` + `raw_memories.md` inputs. Acceptance criteria: - Stage-1 accepts both old and new output keys during migration. - Phase-1 artifacts are generated in new format from DB state. - No dependence on per-thread files in `raw_memories/`. ## PR 3: Remove per-cwd memories and move to one global memory root - Simplify layout in `core/src/memories/layout.rs`: - Single root: `codex_home/memories`. - Remove cwd-hash bucket helpers and normalization logic used only for memory pathing. - Remove scope branching from startup phase-2 dispatch path: - No cwd/user mapping in `core/src/memories/startup/mod.rs`. - One target root for consolidation. - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd consolidation scope. - Keep one logical consolidation scope/job key (global/user) to avoid a risky schema rewrite in same PR. - Add one-time migration helper (core side) to preserve current shared memory output: - If `~/.codex/memories/user/memory` exists and new root is empty, move/copy contents into `~/.codex/memories`. - Leave old hashed cwd buckets untouched for now (safe/no-destructive migration). Acceptance criteria: - New runs only read/write `~/.codex/memories`. - No new cwd-scoped consolidation jobs are enqueued. - Existing user-shared memory content is preserved. ## PR 4: Phase-2 global lock simplification and cleanup - Replace multi-scope dispatch with a single global consolidation claim path: - Either reuse jobs table with one fixed key, or add a tiny dedicated lock helper; keep 1h lease. - Ensure at most one consolidation agent can run at once. - Keep heartbeat + stale lock recovery semantics in `core/src/memories/startup/watch.rs`. - Remove dead scope code and legacy constants no longer used. - Update tests: - One-agent-at-a-time behavior. - Lock expiry allows takeover after stale lease. Acceptance criteria: - Exactly one phase-2 consolidation agent can be active cluster-wide (per local DB). - Stale lock recovers automatically. ## PR 5: Final cleanup and docs - Remove legacy artifacts and references: - `raw_memories/` and `memory_summary.md` assumptions from prompts/comments/tests. - Scope constants for cwd memory pathing in core/state if fully unused. - Update docs under `docs/` for memory workflow and directory layout. - Add a brief operator note for rollout: compatibility window for old stage-1 JSON keys and when to remove aliases. Acceptance criteria: - Code and docs reflect only the simplified global workflow. - No stale references to per-cwd memory buckets. ## Notes on sequencing - PR 1 is safest first because it improves correctness without changing external artifact layout. - PR 2 keeps parser compatibility so prompt deployment can happen independently. - PR 3 and PR 4 split filesystem/scope simplification from locking simplification to reduce blast radius. - PR 5 is intentionally cleanup-only. |
||
|
Michael Bolin
|
d8f9bb65e2 |
# Split command parsing/safety out of codex-core into new codex-command (#11361)
`codex-core` had accumulated command parsing and command safety logic (`bash`, `powershell`, `parse_command`, and `command_safety`) that is logically cohesive but orthogonal to most core session/runtime logic. Keeping this code in `codex-core` made the crate increasingly monolithic and raised iteration cost for unrelated core changes. This change extracts that surface into a dedicated crate, `codex-command`, while preserving existing `codex_core::...` call sites via re-exports. ## Why this refactor During analysis, command parsing/safety stood out as a good first split because it has: - a clear domain boundary (shell parsing + safety classification) - relatively self-contained dependencies (notably `tree-sitter` / `tree-sitter-bash`) - a meaningful standalone test surface (`134` tests moved with the crate) - many downstream uses that benefit from independent compilation and caching The practical problem was build latency from a large `codex-core` compile/test graph. Clean-build timings before and after this split showed measurable wins: - `cargo check -p codex-core`: `57.08s` -> `53.54s` (~`6.2%` faster) - `cargo test -p codex-core --no-run`: `2m39.9s` -> `2m20s` (~`12.4%` faster) - `codex-core lib` compile unit: `57.18s` -> `49.67s` (~`13.1%` faster) - `codex-core lib(test)` compile unit: `60.87s` -> `53.21s` (~`12.6%` faster) This gives a concrete reduction in core build overhead without changing behavior. ## What changed ### New crate - Added `codex-rs/command` as workspace crate `codex-command`. - Added: - `command/src/lib.rs` - `command/src/bash.rs` - `command/src/powershell.rs` - `command/src/parse_command.rs` - `command/src/command_safety/*` - `command/src/shell_detect.rs` - `command/BUILD.bazel` ### Code moved out of `codex-core` - Moved modules from `core/src` into `command/src`: - `bash.rs` - `powershell.rs` - `parse_command.rs` - `command_safety/*` ### Dependency graph updates - Added workspace member/dependency entries for `codex-command` in `codex-rs/Cargo.toml`. - Added `codex-command` dependency to `codex-rs/core/Cargo.toml`. - Removed `tree-sitter` and `tree-sitter-bash` from `codex-core` direct deps (now owned by `codex-command`). ### API compatibility for callers To avoid immediate downstream churn, `codex-core` now re-exports the moved modules/functions: - `codex_command::bash` - `codex_command::powershell` - `codex_command::parse_command` - `codex_command::is_safe_command` - `codex_command::is_dangerous_command` This keeps existing `codex_core::...` paths working while enabling gradual migration to direct `codex-command` usage. ### Internal decoupling detail - Added `command::shell_detect` so moved `bash`/`powershell` logic no longer depends on core shell internals. - Adjusted PowerShell helper visibility in `codex-command` for existing core test usage (`UTF8` prefix helper + executable discovery functions). ## Validation - `just fmt` - `just fix -p codex-command -p codex-core` - `cargo test -p codex-command` (`134` passed) - `cargo test -p codex-core --no-run` - `cargo test -p codex-core shell_command_handler` ## Notes / follow-up This commit intentionally prioritizes boundary extraction and compatibility. A follow-up can migrate downstream crates to depend directly on `codex-command` (instead of through `codex-core` re-exports) to realize additional incremental build wins. |
||
|
|
3626399811 |
Update models.json (#11274)
Automated update of models.json. --------- Co-authored-by: aibrahim-oai <219906144+aibrahim-oai@users.noreply.github.com> Co-authored-by: Ahmed Ibrahim <aibrahim@openai.com> Co-authored-by: Sayan Sisodiya <sayan@openai.com> |
||
|
Michael Bolin
|
30406ad405 | merge commit for archive created by Sapling | ||
|
Michael Bolin
|
5d3e4bbf2c |
# Split command parsing/safety out of codex-core into new codex-command
`codex-core` had accumulated command parsing and command safety logic (`bash`, `powershell`, `parse_command`, and `command_safety`) that is logically cohesive but orthogonal to most core session/runtime logic. Keeping this code in `codex-core` made the crate increasingly monolithic and raised iteration cost for unrelated core changes. This change extracts that surface into a dedicated crate, `codex-command`, while preserving existing `codex_core::...` call sites via re-exports. ## Why this refactor During analysis, command parsing/safety stood out as a good first split because it has: - a clear domain boundary (shell parsing + safety classification) - relatively self-contained dependencies (notably `tree-sitter` / `tree-sitter-bash`) - a meaningful standalone test surface (`134` tests moved with the crate) - many downstream uses that benefit from independent compilation and caching The practical problem was build latency from a large `codex-core` compile/test graph. Clean-build timings before and after this split showed measurable wins: - `cargo check -p codex-core`: `57.08s` -> `53.54s` (~`6.2%` faster) - `cargo test -p codex-core --no-run`: `2m39.9s` -> `2m20s` (~`12.4%` faster) - `codex-core lib` compile unit: `57.18s` -> `49.67s` (~`13.1%` faster) - `codex-core lib(test)` compile unit: `60.87s` -> `53.21s` (~`12.6%` faster) This gives a concrete reduction in core build overhead without changing behavior. ## What changed ### New crate - Added `codex-rs/command` as workspace crate `codex-command`. - Added: - `command/src/lib.rs` - `command/src/bash.rs` - `command/src/powershell.rs` - `command/src/parse_command.rs` - `command/src/command_safety/*` - `command/src/shell_detect.rs` - `command/BUILD.bazel` ### Code moved out of `codex-core` - Moved modules from `core/src` into `command/src`: - `bash.rs` - `powershell.rs` - `parse_command.rs` - `command_safety/*` ### Dependency graph updates - Added workspace member/dependency entries for `codex-command` in `codex-rs/Cargo.toml`. - Added `codex-command` dependency to `codex-rs/core/Cargo.toml`. - Removed `tree-sitter` and `tree-sitter-bash` from `codex-core` direct deps (now owned by `codex-command`). ### API compatibility for callers To avoid immediate downstream churn, `codex-core` now re-exports the moved modules/functions: - `codex_command::bash` - `codex_command::powershell` - `codex_command::parse_command` - `codex_command::is_safe_command` - `codex_command::is_dangerous_command` This keeps existing `codex_core::...` paths working while enabling gradual migration to direct `codex-command` usage. ### Internal decoupling detail - Added `command::shell_detect` so moved `bash`/`powershell` logic no longer depends on core shell internals. - Adjusted PowerShell helper visibility in `codex-command` for existing core test usage (`UTF8` prefix helper + executable discovery functions). ## Validation - `just fmt` - `just fix -p codex-command -p codex-core` - `cargo test -p codex-command` (`134` passed) - `cargo test -p codex-core --no-run` - `cargo test -p codex-core shell_command_handler` ## Notes / follow-up This commit intentionally prioritizes boundary extraction and compatibility. A follow-up can migrate downstream crates to depend directly on `codex-command` (instead of through `codex-core` re-exports) to realize additional incremental build wins. |
||
|
jif-oai
|
3419660767 |
feat: mem v2 - PR3 (#11366)
# Memories migration plan (simplified global workflow) ## Target behavior - One shared memory root only: `~/.codex/memories/`. - No per-cwd memory buckets, no cwd hash handling. - Phase 1 candidate rules: - Not currently being processed unless the job lease is stale. - Rollout updated within the max-age window (currently 30 days). - Rollout idle for at least 12 hours (new constant). - Global cap: at most 64 stage-1 jobs in `running` state at any time (new invariant). - Stage-1 model output shape (new): - `rollout_slug` (accepted but ignored for now). - `rollout_summary`. - `raw_memory`. - Phase-1 artifacts written under the shared root: - `rollout_summaries/<thread_id>.md` for each rollout summary. - `raw_memories.md` containing appended/merged raw memory paragraphs. - Phase 2 runs one consolidation agent for the shared `memories/` directory. - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry. ## Current code map - Core startup pipeline: `core/src/memories/startup/mod.rs`. - Stage-1 request+parse: `core/src/memories/startup/extract.rs`, `core/src/memories/stage_one.rs`, templates in `core/templates/memories/`. - File materialization: `core/src/memories/storage.rs`, `core/src/memories/layout.rs`. - Scope routing (cwd/user): `core/src/memories/scope.rs`, `core/src/memories/startup/mod.rs`. - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`. ## PR plan ## PR 1: Correct phase-1 selection invariants (no behavior-breaking layout changes yet) - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in `core/src/memories/mod.rs`. - Thread this into `state::claim_stage1_jobs_for_startup(...)`. - Enforce idle-time filter in DB selection logic (not only in-memory filtering after `scan_limit`) so eligible threads are not starved by very recent threads. - Enforce global running cap of 64 at claim time in DB logic: - Count fresh `memory_stage1` running jobs. - Only allow new claims while count < cap. - Keep stale-lease takeover behavior intact. - Add/adjust tests in `state/src/runtime.rs`: - Idle filter inclusion/exclusion around 12h boundary. - Global running-cap guarantee. - Existing stale/fresh ownership behavior still passes. Acceptance criteria: - Startup never creates more than 64 fresh `memory_stage1` running jobs. - Threads updated <12h ago are skipped. - Threads older than 30d are skipped. ## PR 2: Stage-1 output contract + storage artifacts (forward-compatible) - Update parser/types to accept the new structured output while keeping backward compatibility: - Add `rollout_slug` (optional for now). - Add `rollout_summary`. - Keep alias support for legacy `summary` and `rawMemory` until prompt swap completes. - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to include the new keys. - Update prompt templates: - `core/templates/memories/stage_one_system.md`. - `core/templates/memories/stage_one_input.md`. - Replace storage model in `core/src/memories/storage.rs`: - Introduce `rollout_summaries/` directory writer (`<thread_id>.md` files). - Introduce `raw_memories.md` aggregator writer from DB rows. - Keep deterministic rebuild behavior from DB outputs so files can always be regenerated. - Update consolidation prompt template to reference `rollout_summaries/` + `raw_memories.md` inputs. Acceptance criteria: - Stage-1 accepts both old and new output keys during migration. - Phase-1 artifacts are generated in new format from DB state. - No dependence on per-thread files in `raw_memories/`. ## PR 3: Remove per-cwd memories and move to one global memory root - Simplify layout in `core/src/memories/layout.rs`: - Single root: `codex_home/memories`. - Remove cwd-hash bucket helpers and normalization logic used only for memory pathing. - Remove scope branching from startup phase-2 dispatch path: - No cwd/user mapping in `core/src/memories/startup/mod.rs`. - One target root for consolidation. - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd consolidation scope. - Keep one logical consolidation scope/job key (global/user) to avoid a risky schema rewrite in same PR. - Add one-time migration helper (core side) to preserve current shared memory output: - If `~/.codex/memories/user/memory` exists and new root is empty, move/copy contents into `~/.codex/memories`. - Leave old hashed cwd buckets untouched for now (safe/no-destructive migration). Acceptance criteria: - New runs only read/write `~/.codex/memories`. - No new cwd-scoped consolidation jobs are enqueued. - Existing user-shared memory content is preserved. ## PR 4: Phase-2 global lock simplification and cleanup - Replace multi-scope dispatch with a single global consolidation claim path: - Either reuse jobs table with one fixed key, or add a tiny dedicated lock helper; keep 1h lease. - Ensure at most one consolidation agent can run at once. - Keep heartbeat + stale lock recovery semantics in `core/src/memories/startup/watch.rs`. - Remove dead scope code and legacy constants no longer used. - Update tests: - One-agent-at-a-time behavior. - Lock expiry allows takeover after stale lease. Acceptance criteria: - Exactly one phase-2 consolidation agent can be active cluster-wide (per local DB). - Stale lock recovers automatically. ## PR 5: Final cleanup and docs - Remove legacy artifacts and references: - `raw_memories/` and `memory_summary.md` assumptions from prompts/comments/tests. - Scope constants for cwd memory pathing in core/state if fully unused. - Update docs under `docs/` for memory workflow and directory layout. - Add a brief operator note for rollout: compatibility window for old stage-1 JSON keys and when to remove aliases. Acceptance criteria: - Code and docs reflect only the simplified global workflow. - No stale references to per-cwd memory buckets. ## Notes on sequencing - PR 1 is safest first because it improves correctness without changing external artifact layout. - PR 2 keeps parser compatibility so prompt deployment can happen independently. - PR 3 and PR 4 split filesystem/scope simplification from locking simplification to reduce blast radius. - PR 5 is intentionally cleanup-only. |
||
|
jif-oai
|
0229dc5ccf |
feat: mem v2 - PR2 (#11365)
# Memories migration plan (simplified global workflow) ## Target behavior - One shared memory root only: `~/.codex/memories/`. - No per-cwd memory buckets, no cwd hash handling. - Phase 1 candidate rules: - Not currently being processed unless the job lease is stale. - Rollout updated within the max-age window (currently 30 days). - Rollout idle for at least 12 hours (new constant). - Global cap: at most 64 stage-1 jobs in `running` state at any time (new invariant). - Stage-1 model output shape (new): - `rollout_slug` (accepted but ignored for now). - `rollout_summary`. - `raw_memory`. - Phase-1 artifacts written under the shared root: - `rollout_summaries/<thread_id>.md` for each rollout summary. - `raw_memories.md` containing appended/merged raw memory paragraphs. - Phase 2 runs one consolidation agent for the shared `memories/` directory. - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry. ## Current code map - Core startup pipeline: `core/src/memories/startup/mod.rs`. - Stage-1 request+parse: `core/src/memories/startup/extract.rs`, `core/src/memories/stage_one.rs`, templates in `core/templates/memories/`. - File materialization: `core/src/memories/storage.rs`, `core/src/memories/layout.rs`. - Scope routing (cwd/user): `core/src/memories/scope.rs`, `core/src/memories/startup/mod.rs`. - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`. ## PR plan ## PR 1: Correct phase-1 selection invariants (no behavior-breaking layout changes yet) - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in `core/src/memories/mod.rs`. - Thread this into `state::claim_stage1_jobs_for_startup(...)`. - Enforce idle-time filter in DB selection logic (not only in-memory filtering after `scan_limit`) so eligible threads are not starved by very recent threads. - Enforce global running cap of 64 at claim time in DB logic: - Count fresh `memory_stage1` running jobs. - Only allow new claims while count < cap. - Keep stale-lease takeover behavior intact. - Add/adjust tests in `state/src/runtime.rs`: - Idle filter inclusion/exclusion around 12h boundary. - Global running-cap guarantee. - Existing stale/fresh ownership behavior still passes. Acceptance criteria: - Startup never creates more than 64 fresh `memory_stage1` running jobs. - Threads updated <12h ago are skipped. - Threads older than 30d are skipped. ## PR 2: Stage-1 output contract + storage artifacts (forward-compatible) - Update parser/types to accept the new structured output while keeping backward compatibility: - Add `rollout_slug` (optional for now). - Add `rollout_summary`. - Keep alias support for legacy `summary` and `rawMemory` until prompt swap completes. - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to include the new keys. - Update prompt templates: - `core/templates/memories/stage_one_system.md`. - `core/templates/memories/stage_one_input.md`. - Replace storage model in `core/src/memories/storage.rs`: - Introduce `rollout_summaries/` directory writer (`<thread_id>.md` files). - Introduce `raw_memories.md` aggregator writer from DB rows. - Keep deterministic rebuild behavior from DB outputs so files can always be regenerated. - Update consolidation prompt template to reference `rollout_summaries/` + `raw_memories.md` inputs. Acceptance criteria: - Stage-1 accepts both old and new output keys during migration. - Phase-1 artifacts are generated in new format from DB state. - No dependence on per-thread files in `raw_memories/`. ## PR 3: Remove per-cwd memories and move to one global memory root - Simplify layout in `core/src/memories/layout.rs`: - Single root: `codex_home/memories`. - Remove cwd-hash bucket helpers and normalization logic used only for memory pathing. - Remove scope branching from startup phase-2 dispatch path: - No cwd/user mapping in `core/src/memories/startup/mod.rs`. - One target root for consolidation. - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd consolidation scope. - Keep one logical consolidation scope/job key (global/user) to avoid a risky schema rewrite in same PR. - Add one-time migration helper (core side) to preserve current shared memory output: - If `~/.codex/memories/user/memory` exists and new root is empty, move/copy contents into `~/.codex/memories`. - Leave old hashed cwd buckets untouched for now (safe/no-destructive migration). Acceptance criteria: - New runs only read/write `~/.codex/memories`. - No new cwd-scoped consolidation jobs are enqueued. - Existing user-shared memory content is preserved. ## PR 4: Phase-2 global lock simplification and cleanup - Replace multi-scope dispatch with a single global consolidation claim path: - Either reuse jobs table with one fixed key, or add a tiny dedicated lock helper; keep 1h lease. - Ensure at most one consolidation agent can run at once. - Keep heartbeat + stale lock recovery semantics in `core/src/memories/startup/watch.rs`. - Remove dead scope code and legacy constants no longer used. - Update tests: - One-agent-at-a-time behavior. - Lock expiry allows takeover after stale lease. Acceptance criteria: - Exactly one phase-2 consolidation agent can be active cluster-wide (per local DB). - Stale lock recovers automatically. ## PR 5: Final cleanup and docs - Remove legacy artifacts and references: - `raw_memories/` and `memory_summary.md` assumptions from prompts/comments/tests. - Scope constants for cwd memory pathing in core/state if fully unused. - Update docs under `docs/` for memory workflow and directory layout. - Add a brief operator note for rollout: compatibility window for old stage-1 JSON keys and when to remove aliases. Acceptance criteria: - Code and docs reflect only the simplified global workflow. - No stale references to per-cwd memory buckets. ## Notes on sequencing - PR 1 is safest first because it improves correctness without changing external artifact layout. - PR 2 keeps parser compatibility so prompt deployment can happen independently. - PR 3 and PR 4 split filesystem/scope simplification from locking simplification to reduce blast radius. - PR 5 is intentionally cleanup-only. |
||
|
jif-oai
|
07da740c8a |
feat: mem v2 - PR1 (#11364)
# Memories migration plan (simplified global workflow) ## Target behavior - One shared memory root only: `~/.codex/memories/`. - No per-cwd memory buckets, no cwd hash handling. - Phase 1 candidate rules: - Not currently being processed unless the job lease is stale. - Rollout updated within the max-age window (currently 30 days). - Rollout idle for at least 12 hours (new constant). - Global cap: at most 64 stage-1 jobs in `running` state at any time (new invariant). - Stage-1 model output shape (new): - `rollout_slug` (accepted but ignored for now). - `rollout_summary`. - `raw_memory`. - Phase-1 artifacts written under the shared root: - `rollout_summaries/<thread_id>.md` for each rollout summary. - `raw_memories.md` containing appended/merged raw memory paragraphs. - Phase 2 runs one consolidation agent for the shared `memories/` directory. - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry. ## Current code map - Core startup pipeline: `core/src/memories/startup/mod.rs`. - Stage-1 request+parse: `core/src/memories/startup/extract.rs`, `core/src/memories/stage_one.rs`, templates in `core/templates/memories/`. - File materialization: `core/src/memories/storage.rs`, `core/src/memories/layout.rs`. - Scope routing (cwd/user): `core/src/memories/scope.rs`, `core/src/memories/startup/mod.rs`. - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`. ## PR plan ## PR 1: Correct phase-1 selection invariants (no behavior-breaking layout changes yet) - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in `core/src/memories/mod.rs`. - Thread this into `state::claim_stage1_jobs_for_startup(...)`. - Enforce idle-time filter in DB selection logic (not only in-memory filtering after `scan_limit`) so eligible threads are not starved by very recent threads. - Enforce global running cap of 64 at claim time in DB logic: - Count fresh `memory_stage1` running jobs. - Only allow new claims while count < cap. - Keep stale-lease takeover behavior intact. - Add/adjust tests in `state/src/runtime.rs`: - Idle filter inclusion/exclusion around 12h boundary. - Global running-cap guarantee. - Existing stale/fresh ownership behavior still passes. Acceptance criteria: - Startup never creates more than 64 fresh `memory_stage1` running jobs. - Threads updated <12h ago are skipped. - Threads older than 30d are skipped. ## PR 2: Stage-1 output contract + storage artifacts (forward-compatible) - Update parser/types to accept the new structured output while keeping backward compatibility: - Add `rollout_slug` (optional for now). - Add `rollout_summary`. - Keep alias support for legacy `summary` and `rawMemory` until prompt swap completes. - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to include the new keys. - Update prompt templates: - `core/templates/memories/stage_one_system.md`. - `core/templates/memories/stage_one_input.md`. - Replace storage model in `core/src/memories/storage.rs`: - Introduce `rollout_summaries/` directory writer (`<thread_id>.md` files). - Introduce `raw_memories.md` aggregator writer from DB rows. - Keep deterministic rebuild behavior from DB outputs so files can always be regenerated. - Update consolidation prompt template to reference `rollout_summaries/` + `raw_memories.md` inputs. Acceptance criteria: - Stage-1 accepts both old and new output keys during migration. - Phase-1 artifacts are generated in new format from DB state. - No dependence on per-thread files in `raw_memories/`. ## PR 3: Remove per-cwd memories and move to one global memory root - Simplify layout in `core/src/memories/layout.rs`: - Single root: `codex_home/memories`. - Remove cwd-hash bucket helpers and normalization logic used only for memory pathing. - Remove scope branching from startup phase-2 dispatch path: - No cwd/user mapping in `core/src/memories/startup/mod.rs`. - One target root for consolidation. - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd consolidation scope. - Keep one logical consolidation scope/job key (global/user) to avoid a risky schema rewrite in same PR. - Add one-time migration helper (core side) to preserve current shared memory output: - If `~/.codex/memories/user/memory` exists and new root is empty, move/copy contents into `~/.codex/memories`. - Leave old hashed cwd buckets untouched for now (safe/no-destructive migration). Acceptance criteria: - New runs only read/write `~/.codex/memories`. - No new cwd-scoped consolidation jobs are enqueued. - Existing user-shared memory content is preserved. ## PR 4: Phase-2 global lock simplification and cleanup - Replace multi-scope dispatch with a single global consolidation claim path: - Either reuse jobs table with one fixed key, or add a tiny dedicated lock helper; keep 1h lease. - Ensure at most one consolidation agent can run at once. - Keep heartbeat + stale lock recovery semantics in `core/src/memories/startup/watch.rs`. - Remove dead scope code and legacy constants no longer used. - Update tests: - One-agent-at-a-time behavior. - Lock expiry allows takeover after stale lease. Acceptance criteria: - Exactly one phase-2 consolidation agent can be active cluster-wide (per local DB). - Stale lock recovers automatically. ## PR 5: Final cleanup and docs - Remove legacy artifacts and references: - `raw_memories/` and `memory_summary.md` assumptions from prompts/comments/tests. - Scope constants for cwd memory pathing in core/state if fully unused. - Update docs under `docs/` for memory workflow and directory layout. - Add a brief operator note for rollout: compatibility window for old stage-1 JSON keys and when to remove aliases. Acceptance criteria: - Code and docs reflect only the simplified global workflow. - No stale references to per-cwd memory buckets. ## Notes on sequencing - PR 1 is safest first because it improves correctness without changing external artifact layout. - PR 2 keeps parser compatibility so prompt deployment can happen independently. - PR 3 and PR 4 split filesystem/scope simplification from locking simplification to reduce blast radius. - PR 5 is intentionally cleanup-only. |
||
|
Michael Bolin
|
af7d9fc7d9 |
Merge e1fa4b0f21 into sapling-pr-archive-bolinfest
|
||
|
Michael Bolin
|
e1fa4b0f21 |
# Split command parsing/safety out of codex-core into new codex-command
`codex-core` had accumulated command parsing and command safety logic (`bash`, `powershell`, `parse_command`, and `command_safety`) that is logically cohesive but orthogonal to most core session/runtime logic. Keeping this code in `codex-core` made the crate increasingly monolithic and raised iteration cost for unrelated core changes. This change extracts that surface into a dedicated crate, `codex-command`, while preserving existing `codex_core::...` call sites via re-exports. ## Why this refactor During analysis, command parsing/safety stood out as a good first split because it has: - a clear domain boundary (shell parsing + safety classification) - relatively self-contained dependencies (notably `tree-sitter` / `tree-sitter-bash`) - a meaningful standalone test surface (`134` tests moved with the crate) - many downstream uses that benefit from independent compilation and caching The practical problem was build latency from a large `codex-core` compile/test graph. Clean-build timings before and after this split showed measurable wins: - `cargo check -p codex-core`: `57.08s` -> `53.54s` (~`6.2%` faster) - `cargo test -p codex-core --no-run`: `2m39.9s` -> `2m20s` (~`12.4%` faster) - `codex-core lib` compile unit: `57.18s` -> `49.67s` (~`13.1%` faster) - `codex-core lib(test)` compile unit: `60.87s` -> `53.21s` (~`12.6%` faster) This gives a concrete reduction in core build overhead without changing behavior. ## What changed ### New crate - Added `codex-rs/command` as workspace crate `codex-command`. - Added: - `command/src/lib.rs` - `command/src/bash.rs` - `command/src/powershell.rs` - `command/src/parse_command.rs` - `command/src/command_safety/*` - `command/src/shell_detect.rs` - `command/BUILD.bazel` ### Code moved out of `codex-core` - Moved modules from `core/src` into `command/src`: - `bash.rs` - `powershell.rs` - `parse_command.rs` - `command_safety/*` ### Dependency graph updates - Added workspace member/dependency entries for `codex-command` in `codex-rs/Cargo.toml`. - Added `codex-command` dependency to `codex-rs/core/Cargo.toml`. - Removed `tree-sitter` and `tree-sitter-bash` from `codex-core` direct deps (now owned by `codex-command`). ### API compatibility for callers To avoid immediate downstream churn, `codex-core` now re-exports the moved modules/functions: - `codex_command::bash` - `codex_command::powershell` - `codex_command::parse_command` - `codex_command::is_safe_command` - `codex_command::is_dangerous_command` This keeps existing `codex_core::...` paths working while enabling gradual migration to direct `codex-command` usage. ### Internal decoupling detail - Added `command::shell_detect` so moved `bash`/`powershell` logic no longer depends on core shell internals. - Adjusted PowerShell helper visibility in `codex-command` for existing core test usage (`UTF8` prefix helper + executable discovery functions). ## Validation - `just fmt` - `just fix -p codex-command -p codex-core` - `cargo test -p codex-command` (`134` passed) - `cargo test -p codex-core --no-run` - `cargo test -p codex-core shell_command_handler` ## Notes / follow-up This commit intentionally prioritizes boundary extraction and compatibility. A follow-up can migrate downstream crates to depend directly on `codex-command` (instead of through `codex-core` re-exports) to realize additional incremental build wins. |
||
|
jif-oai
|
a6e9469fa4 | chore: unify memory job flow (#11334) |