This is a significant change to how layers of configuration are applied. In particular, the `ConfigLayerStack` now has two important fields:
- `layers: Vec<ConfigLayerEntry>`
- `requirements: ConfigRequirements`
We merge `TomlValue`s across the layers, but they are subject to `ConfigRequirements` before creating a `Config`.
This is a significant change to how layers of configuration are applied. In particular, the `ConfigLayerStack` now has two important fields:
- `layers: Vec<ConfigLayerEntry>`
- `requirements: ConfigRequirements`
We merge `TomlValue`s across the layers, but they are subject to `ConfigRequirements` before creating a `Config`.
This is a significant change to how layers of configuration are applied. In particular, the `ConfigLayerStack` now has two important fields:
- `layers: Vec<ConfigLayerEntry>`
- `requirements: ConfigRequirements`
We merge `TomlValue`s across the layers, but they are subject to `ConfigRequirements` before creating a `Config`.
Introduce `ConfigBuilder` as an alternative to our existing `Config`
constructors.
I noticed that the existing constructors,
`Config::load_with_cli_overrides()` and
`Config::load_with_cli_overrides_and_harness_overrides()`, did not take
`codex_home` as a parameter, which can be a problem.
Historically, when Codex was purely a CLI, we wanted to be extra sure
that the creation of `codex_home` was always done via
`find_codex_home()`, so we did not expose `codex_home` as a parameter
when creating `Config` in business logic. But in integration tests,
`codex_home` nearly always needs to be configured (as a temp directory),
which is why callers would have to go through
`Config::load_from_base_config_with_overrides()` instead.
Now that the Codex harness also functions as an app server, which could
conceivably load multiple threads where `codex_home` is parameterized
differently in each one, I think it makes sense to make this
configurable. Going to a builder pattern makes it more flexible to
ensure an arbitrary permutation of options can be set when constructing
a `Config` while using the appropriate defaults for the options that
aren't set explicitly.
Ultimately, I think this should make it possible for us to make
`Config::load_from_base_config_with_overrides()` private because all
integration tests should be able to leverage `ConfigBuilder` instead.
Though there could be edge cases, so I'll pursue that migration after we
get through the current config overhaul.
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/8235).
* #8237
* __->__ #8235
1. Remove PUBLIC skills and introduce SYSTEM skills embedded in the
binary and installed into $CODEX_HOME/skills/.system at startup.
2. Skills are now always enabled (feature flag removed).
3. Update skills/list to accept forceReload and plumb it through (not
used by clients yet).
# External (non-OpenAI) Pull Request Requirements
Before opening this Pull Request, please read the dedicated
"Contributing" markdown file or your PR may be closed:
https://github.com/openai/codex/blob/main/docs/contributing.md
If your PR conforms to our contribution guidelines, replace this text
with a detailed and high quality description of your changes.
Include a link to a bug report or enhancement request.
This PR does various types of cleanup before I can proceed with more
ambitious changes to config loading.
First, I noticed duplicated code across these two methods:
https://github.com/openai/codex/blob/774bd9e432fa2e0f4e059e97648cf92216912e19/codex-rs/core/src/config/mod.rs#L314-L324https://github.com/openai/codex/blob/774bd9e432fa2e0f4e059e97648cf92216912e19/codex-rs/core/src/config/mod.rs#L334-L344
This has now been consolidated in
`load_config_as_toml_with_cli_overrides()`.
Further, I noticed that `Config::load_with_cli_overrides()` took two
similar arguments:
https://github.com/openai/codex/blob/774bd9e432fa2e0f4e059e97648cf92216912e19/codex-rs/core/src/config/mod.rs#L308-L311
The difference between `cli_overrides` and `overrides` was not
immediately obvious to me. At first glance, it appears that one should
be able to be expressed in terms of the other, but it turns out that
some fields of `ConfigOverrides` (such as `cwd` and
`codex_linux_sandbox_exe`) are, by design, not configurable via a
`.toml` file or a command-line `--config` flag.
That said, I discovered that many callers of
`Config::load_with_cli_overrides()` were passing
`ConfigOverrides::default()` for `overrides`, so I created two separate
methods:
- `Config::load_with_cli_overrides(cli_overrides: Vec<(String,
TomlValue)>)`
- `Config::load_with_cli_overrides_and_harness_overrides(cli_overrides:
Vec<(String, TomlValue)>, harness_overrides: ConfigOverrides)`
The latter has a long name, as it is _not_ what should be used in the
common case, so the extra typing is designed to draw attention to this
fact. I tried to update the existing callsites to use the shorter name,
where possible.
Further, in the cases where `ConfigOverrides` is used, usually only a
limited subset of fields are actually set, so I updated the declarations
to leverage `..Default::default()` where possible.
# External (non-OpenAI) Pull Request Requirements
Before opening this Pull Request, please read the dedicated
"Contributing" markdown file or your PR may be closed:
https://github.com/openai/codex/blob/main/docs/contributing.md
If your PR conforms to our contribution guidelines, replace this text
with a detailed and high quality description of your changes.
Include a link to a bug report or enhancement request.
- Batch read ACL creation for online/offline sandbox user
- creates a new ACL helper process that is long-lived and runs in the
background
- uses a mutex so that only one helper process is running at a time.
### Summary
* Make `app_server.list_models` to be non-blocking and consumers (i.e.
extension) can manage the flow themselves.
* Force config to use remote models and therefore fetch codex-auto model
list.
We should not have any `PathBuf` fields in `ConfigToml` or any of the
transitive structs we include, as we should use `AbsolutePathBuf`
instead so that we do not have to keep track of the file from which
`ConfigToml` was loaded such that we need it to resolve relative paths
later when the values of `ConfigToml` are used.
I only found two instances of this: `experimental_instructions_file` and
`experimental_compact_prompt_file`. Incidentally, when these were
specified as relative paths, they were resolved against `cwd` rather
than `config.toml`'s parent, which seems wrong to me. I changed the
behavior so they are resolved against the parent folder of the
`config.toml` being parsed, which we get "for free" due to the
introduction of `AbsolutePathBufGuard ` in
https://github.com/openai/codex/pull/7796.
While it is not great to change the behavior of a released feature,
these fields are prefixed with `experimental_`, which I interpret to
mean we have the liberty to change the contract.
For reference:
- `experimental_instructions_file` was introduced in
https://github.com/openai/codex/pull/1803
- `experimental_compact_prompt_file` was introduced in
https://github.com/openai/codex/pull/5959
The problem is that the `tokio` task own an `Arc` reference of the
session and that this task only exit with the broadcast channel get
closed. But this never get closed if the session is not dropped. So it's
a snake biting his tail basically
The most notable result was that non of the `Drop` implementation were
triggered (temporary files, shell snapshots, session cleaning etc etc)
when closing the session (through a `/new` for example)
The fix is just to weaken the `Arc` and upgrade it on the fly
Michael Bolin