mirror of
https://github.com/openai/codex.git
synced 2026-11-20 22:40:37 +00:00
## Why When Codex uses a remote `ExecutorFileSystem`, every `get_metadata` call is an exec-server round trip. Upward discovery currently pays those round trips serially in two latency-sensitive places: - session startup, while locating the configured project root before loading `AGENTS.md`; and - Git-root discovery, which runs before per-turn Git diff enrichment. The goal is to remove the serial ancestor dependency without adding a new filesystem RPC, JSON-RPC batch method, Git executable dependency, or cache. ## Example Assume this layout, with `.git` as the configured project-root marker: ```text /workspace/repo/.git /workspace/repo/AGENTS.md /workspace/repo/crates/core/ <- cwd ``` The marker probes have this required precedence: ```text 1. /workspace/repo/crates/core/.git 2. /workspace/repo/crates/.git 3. /workspace/repo/.git 4. /workspace/.git 5. /.git ``` Previously, probe 2 was not sent until probe 1 returned, and probe 3 was not sent until probe 2 returned. With this change, the client lazily keeps up to eight ordinary `fs/getMetadata` requests in flight, but consumes their results in the order above. Codex must still learn that probes 1 and 2 are absent before accepting probe 3, so the nearest root always wins. Once probe 3 succeeds, the client has its answer and stops awaiting probes 4 and 5. Requests that were already sent may still finish on the worker. For the marker phase alone, with a 50 ms client-to-worker round trip and fast local metadata calls, finding the root at probe 3 changes from roughly three serialized round trips (150 ms) to one round trip plus worker processing. The later `AGENTS.md` candidate phase remains separate and ordered. Only after `/workspace/repo` is selected does `AGENTS.md` discovery check instruction candidates, in root-to-cwd order: ```text /workspace/repo/AGENTS.override.md /workspace/repo/AGENTS.md /workspace/repo/crates/AGENTS.override.md /workspace/repo/crates/AGENTS.md /workspace/repo/crates/core/AGENTS.override.md /workspace/repo/crates/core/AGENTS.md ``` The first configured candidate found in each directory wins. These checks remain ordered and no instruction candidate above `/workspace/repo` is issued. Git-root discovery uses the same bounded lookup with only `.git` as the marker. ## What changed - Added a client-side find-up helper that generates `ancestor x marker` probes lazily, nearest directory first and configured marker order within each directory. - Uses an ordered concurrency window of eight scalar metadata requests. This bounds executor load while preserving nearest-root and marker precedence. - Reuses the helper for both configured project-root discovery and remote Git-root discovery. - Keeps Git ancestor and marker construction in `AbsolutePathBuf`, converting only each complete `.git` probe to `PathUri`. This preserves native paths that require an opaque URI fallback, such as Windows namespace paths. - Preserves existing error behavior: `AGENTS.md` discovery propagates non-`NotFound` metadata errors, while Git discovery treats a failed marker probe as absent and continues upward. - Reads each discovered `AGENTS.md` directly instead of statting it a second time. No filesystem trait or exec-server protocol method is added. An empty `project_root_markers` list performs no ancestor-marker I/O and checks instruction candidates only in `cwd`. This change also deliberately does not cache roots across turns. ## Symlinks Upward traversal remains **lexical**. The helper does not canonicalize `cwd`; it appends marker names to the supplied path and walks that path's textual parents. The filesystem performs the actual metadata/read operation, and the current local and exec-server implementations follow live symlink targets. For example: ```text /tmp/pkg -> /workspace/repo/packages/pkg cwd = /tmp/pkg/src actual Git marker = /workspace/repo/.git ``` The lexical probes are `/tmp/pkg/src/.git`, `/tmp/pkg/.git`, `/tmp/.git`, and `/.git`. They do not jump from `/tmp/pkg` to the target's parent `/workspace/repo`, so this spelling of `cwd` does not discover `/workspace/repo/.git`. That is the existing behavior and is unchanged by this PR. Conversely, if `/tmp/repo -> /workspace/repo`, then probing `/tmp/repo/.git` follows the directory symlink and finds `/workspace/repo/.git`; the reported root remains the lexical path `/tmp/repo`. A live symlink used directly as `.git`, another configured marker, or `AGENTS.md` is also followed. A symlinked `AGENTS.md` is loaded when its target is a regular file, while a broken symlink behaves as `NotFound`.
122 lines
4.0 KiB
Rust
122 lines
4.0 KiB
Rust
use crate::ExecutorFileSystem;
|
|
use crate::FileSystemResult;
|
|
use crate::FileSystemSandboxContext;
|
|
use codex_utils_absolute_path::AbsolutePathBuf;
|
|
use codex_utils_path_uri::PathUri;
|
|
use futures::StreamExt;
|
|
use std::io;
|
|
|
|
const MAX_CONCURRENT_PROBES: usize = 8;
|
|
|
|
/// Controls how an upward marker search handles metadata errors other than `NotFound`.
|
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
|
pub enum FindUpErrorPolicy {
|
|
/// Return the first error in lexical search order.
|
|
Propagate,
|
|
/// Treat errors as missing markers and continue searching.
|
|
Ignore,
|
|
}
|
|
|
|
/// Finds the nearest ancestor containing one of the provided marker names.
|
|
///
|
|
/// Marker paths are probed in lexical order from `start` toward the filesystem root. A bounded
|
|
/// number of ordinary metadata calls are kept in flight so remote filesystems can pipeline them
|
|
/// without requiring a batch protocol operation.
|
|
pub async fn find_nearest_ancestor_with_markers(
|
|
file_system: &dyn ExecutorFileSystem,
|
|
start: &PathUri,
|
|
markers: Vec<String>,
|
|
error_policy: FindUpErrorPolicy,
|
|
sandbox: Option<&FileSystemSandboxContext>,
|
|
) -> FileSystemResult<Option<PathUri>> {
|
|
find_nearest_ancestor(
|
|
file_system,
|
|
start.clone(),
|
|
markers,
|
|
PathUri::parent,
|
|
|ancestor, marker| {
|
|
ancestor
|
|
.join(marker)
|
|
.map_err(|err| io::Error::new(io::ErrorKind::InvalidInput, err))
|
|
},
|
|
error_policy,
|
|
sandbox,
|
|
)
|
|
.await
|
|
}
|
|
|
|
/// Finds the nearest native ancestor containing one of the provided marker names.
|
|
///
|
|
/// Ancestors and marker paths remain native until each complete probe is converted to a URI. This
|
|
/// preserves paths that require an opaque [`PathUri`] fallback.
|
|
pub async fn find_nearest_native_ancestor_with_markers(
|
|
file_system: &dyn ExecutorFileSystem,
|
|
start: &AbsolutePathBuf,
|
|
markers: Vec<String>,
|
|
error_policy: FindUpErrorPolicy,
|
|
sandbox: Option<&FileSystemSandboxContext>,
|
|
) -> FileSystemResult<Option<AbsolutePathBuf>> {
|
|
find_nearest_ancestor(
|
|
file_system,
|
|
start.clone(),
|
|
markers,
|
|
AbsolutePathBuf::parent,
|
|
|ancestor, marker| Ok(PathUri::from_abs_path(&ancestor.join(marker))),
|
|
error_policy,
|
|
sandbox,
|
|
)
|
|
.await
|
|
}
|
|
|
|
async fn find_nearest_ancestor<P, Parent, MarkerPath>(
|
|
file_system: &dyn ExecutorFileSystem,
|
|
start: P,
|
|
markers: Vec<String>,
|
|
parent: Parent,
|
|
mut marker_path: MarkerPath,
|
|
error_policy: FindUpErrorPolicy,
|
|
sandbox: Option<&FileSystemSandboxContext>,
|
|
) -> FileSystemResult<Option<P>>
|
|
where
|
|
P: Clone + Send,
|
|
Parent: FnMut(&P) -> Option<P> + Send,
|
|
MarkerPath: FnMut(&P, &str) -> FileSystemResult<PathUri> + Send,
|
|
{
|
|
let mut ancestors = std::iter::successors(Some(start), parent);
|
|
let mut ancestor = ancestors.next();
|
|
let mut marker_index = 0;
|
|
let probes = std::iter::from_fn(move || {
|
|
let current_ancestor = ancestor.clone()?;
|
|
let marker = markers.get(marker_index)?;
|
|
let marker_path = marker_path(¤t_ancestor, marker);
|
|
|
|
marker_index += 1;
|
|
if marker_index == markers.len() {
|
|
marker_index = 0;
|
|
ancestor = ancestors.next();
|
|
}
|
|
|
|
Some((current_ancestor, marker_path))
|
|
});
|
|
let mut results = futures::stream::iter(probes)
|
|
.map(|(ancestor, marker_path)| async move {
|
|
let marker_path = marker_path?;
|
|
match file_system.get_metadata(&marker_path, sandbox).await {
|
|
Ok(_) => Ok(Some(ancestor)),
|
|
Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None),
|
|
Err(err) => match error_policy {
|
|
FindUpErrorPolicy::Propagate => Err(err),
|
|
FindUpErrorPolicy::Ignore => Ok(None),
|
|
},
|
|
}
|
|
})
|
|
.buffered(MAX_CONCURRENT_PROBES);
|
|
|
|
while let Some(result) = results.next().await {
|
|
if let Some(ancestor) = result? {
|
|
return Ok(Some(ancestor));
|
|
}
|
|
}
|
|
Ok(None)
|
|
}
|