Merge 4e902ba021 into sapling-pr-archive-bolinfest

This commit is contained in:
Michael Bolin
2026-04-27 11:15:58 -07:00
committed by GitHub
105 changed files with 4747 additions and 2555 deletions
+102
View File
@@ -0,0 +1,102 @@
---
name: codex-issue-digest
description: Run a GitHub issue digest for openai/codex by feature-area labels, all areas, and configurable time windows. Use when asked to summarize recent Codex bug reports or enhancement requests, especially for owner-specific labels such as tui, exec, app, or similar areas.
---
# Codex Issue Digest
## Objective
Produce a concise, insight-oriented digest of `openai/codex` issues for the requested feature-area labels over the previous 24 hours by default. Honor a different duration when the user asks for one, for example "past week" or "48 hours".
Include only issues that currently have `bug` or `enhancement` plus at least one requested owner label. If the user asks for all areas or all labels, collect `bug`/`enhancement` issues across all labels.
## Inputs
- Feature-area labels, for example `tui exec`
- `all areas` / `all labels` to scan all current feature labels
- Optional repo override, default `openai/codex`
- Optional time window, default previous 24 hours; examples: `48h`, `7d`, `1w`, `past week`
## Workflow
1. Run the collector from a current Codex repo checkout:
```bash
python3 .codex/skills/codex-issue-digest/scripts/collect_issue_digest.py --labels tui exec --window-hours 24
```
Use `--window "past week"` or `--window-hours 168` when the user asks for a non-default duration. Use `--all-labels` when the user says all areas or all labels.
2. Use the JSON as the source of truth. It includes new issues, new issue comments, new reactions/upvotes, current labels, current reaction counts, model-ready `summary_inputs`, and detailed `digest_rows`.
3. Start the report with `## Summary`, then `## Details`.
4. In `## Summary`, write skim-first headlines:
- Lead with the most important fact or judgment. Do not start with aggregate counts unless the aggregate itself is the story.
- Make the first 1-3 bullets answer "what should owners pay attention to right now?"
- Bold only the critical insight phrase in each high-priority bullet, for example `**GPT-5.5 context is the dominant pressure point**`.
- Keep summary bullets short enough to scan in about 20 seconds.
- Put broad stats near the end of the summary, after the owner-relevant takeaways.
- Say clearly when there is nothing significant to act on.
- Call out any areas or themes receiving lots of user attention.
- Cluster and name themes yourself from `summary_inputs`; the collector intentionally does not hard-code issue categories.
- Use a cluster only when the issues genuinely share the same product problem. If several issues merely share a broad platform or label, describe them individually.
- Do not omit a repeated theme just because its individual issues fall below the details table cutoff. Several similar reports should be called out as a repeated customer concern.
- For single-issue rows, summarize the concern directly instead of calling it a cluster.
- Use inline numbered issue links from each relevant row's `ref_markdown`.
5. In `## Details`, include a compact table only when useful:
- Prefer rows from `digest_rows`; include a `Refs` column using each row's `ref_markdown`.
- Keep the table short; omit low-signal rows when the summary already covers them.
- Use compact columns such as marker, area, type, description, interactions, and refs.
- The `Description` cell should be a short owner-readable phrase. Use row `description`, title, body excerpts, and recent comments, but do not mechanically copy the raw GitHub issue title when it contains incidental details.
- A clear quiet/no-concern sentence when there is no meaningful signal.
6. Use the JSON `attention_marker` exactly. It is empty for normal rows, `🔥` for elevated rows, and `🔥🔥` for very high-attention rows. The actual cutoffs are in `attention_thresholds`.
7. Use inline numbered references where a row or bullet points to issues, for example `Compaction bugs [1](https://github.com/openai/codex/issues/123), [2](https://github.com/openai/codex/issues/456)`. Do not add a separate footnotes section.
8. Label `interactions` as `Interactions`; it counts posts/comments/reactions during the requested window, not unique people.
9. Mention the collector `script_version`, repo checkout `git_head`, and time window in the digest footer or final line.
## Reaction Handling
The collector uses GitHub reactions endpoints, which include `created_at`, to count reactions created during the digest window for hydrated issues. It reports both in-window reaction counts and current reaction totals. Treat current reaction totals as standing engagement, and treat `new_reactions` / `new_upvotes` as windowed activity.
By default, the collector fetches issue comments with `since=<window start>` and caps the number of comment pages per issue. This keeps very long historical threads from dominating a digest run and focuses the report on recent posts. Use `--fetch-all-comments` only when exhaustive comment history is more important than runtime.
GitHub issue search is still seeded by issue `updated_at`, so a purely reaction-only issue may be missed if reactions do not bump `updated_at`. Covering every reaction-only case would require either a persisted snapshot store or a broader scan of labeled issues.
## Attention Markers
The collector scales attention markers by the requested time window. The baseline is 10 human user interactions for `🔥` and 20 for `🔥🔥` over 24 hours; longer or shorter windows scale those cutoffs linearly and round up. For example, a one-week report uses 70 and 140 interactions. Human user interactions are human-authored new issue posts, human-authored new comments, and human reactions created during the window, including upvotes. Bot posts and bot reactions are excluded. In prose, explain this as high user interaction rather than naming the emoji.
## Freshness
The automation should run from a repo checkout that contains this skill. For shared daily use, prefer one of these patterns:
- Run the automation in a checkout that is refreshed before the automation starts, for example with `git pull --ff-only`.
- If the automation cannot safely mutate the checkout, have it report the current `git_head` from the collector output so readers know which skill/script version produced the digest.
## Sample Owner Prompt
```text
Use $codex-issue-digest to run the Codex issue digest for labels tui and exec over the previous 24 hours.
```
```text
Use $codex-issue-digest to run the Codex issue digest for all areas over the past week.
```
## Validation
Dry run the collector against recent issues:
```bash
python3 .codex/skills/codex-issue-digest/scripts/collect_issue_digest.py --labels tui exec --window-hours 24
```
```bash
python3 .codex/skills/codex-issue-digest/scripts/collect_issue_digest.py --all-labels --window "past week" --limit-issues 10
```
Run the focused script tests:
```bash
pytest .codex/skills/codex-issue-digest/scripts/test_collect_issue_digest.py
```
@@ -0,0 +1,4 @@
interface:
display_name: "Codex Issue Digest"
short_description: "Summarize Codex issues by labels or all areas"
default_prompt: "Use $codex-issue-digest to run the Codex issue digest for labels tui and exec over the previous 24 hours."
@@ -0,0 +1,988 @@
#!/usr/bin/env python3
"""Collect recent openai/codex issue activity for owner-focused digests."""
import argparse
import json
import math
import re
import subprocess
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import quote
SCRIPT_VERSION = 2
QUALIFYING_KIND_LABELS = ("bug", "enhancement")
REACTION_KEYS = ("+1", "-1", "laugh", "hooray", "confused", "heart", "rocket", "eyes")
BASE_ATTENTION_WINDOW_HOURS = 24.0
ONE_ATTENTION_INTERACTION_THRESHOLD = 10
TWO_ATTENTION_INTERACTION_THRESHOLD = 20
ALL_LABEL_PHRASES = {"all", "all areas", "all labels", "all-areas", "all-labels", "*"}
class GhCommandError(RuntimeError):
pass
def parse_args():
parser = argparse.ArgumentParser(
description="Collect recent GitHub issue activity for a Codex owner digest."
)
parser.add_argument(
"--repo", default="openai/codex", help="OWNER/REPO, default openai/codex"
)
parser.add_argument(
"--labels",
nargs="+",
default=[],
help="Feature-area labels owned by the digest recipient, for example: tui exec",
)
parser.add_argument(
"--all-labels",
action="store_true",
help="Collect bug/enhancement issues across all feature-area labels",
)
parser.add_argument(
"--window",
help='Lookback duration such as "24h", "7d", "1w", or "past week"',
)
parser.add_argument(
"--window-hours", type=float, default=24.0, help="Lookback window"
)
parser.add_argument(
"--since", help="UTC ISO timestamp override for the window start"
)
parser.add_argument("--until", help="UTC ISO timestamp override for the window end")
parser.add_argument(
"--limit-issues",
type=int,
default=200,
help="Maximum candidate issues to hydrate after search",
)
parser.add_argument(
"--body-chars", type=int, default=1200, help="Issue body excerpt length"
)
parser.add_argument(
"--comment-chars", type=int, default=900, help="Comment excerpt length"
)
parser.add_argument(
"--max-comment-pages",
type=int,
default=3,
help=(
"Maximum pages of issue comments to hydrate per issue after applying the "
"window filter. Use 0 with --fetch-all-comments for no page cap."
),
)
parser.add_argument(
"--fetch-all-comments",
action="store_true",
help="Hydrate complete issue comment histories instead of only window-updated comments.",
)
return parser.parse_args()
def parse_timestamp(value, arg_name):
if value is None:
return None
normalized = value.strip()
if not normalized:
return None
if normalized.endswith("Z"):
normalized = f"{normalized[:-1]}+00:00"
try:
parsed = datetime.fromisoformat(normalized)
except ValueError as err:
raise ValueError(f"{arg_name} must be an ISO timestamp") from err
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed.astimezone(timezone.utc)
def format_timestamp(value):
return (
value.astimezone(timezone.utc)
.replace(microsecond=0)
.isoformat()
.replace("+00:00", "Z")
)
def resolve_window(args):
until = parse_timestamp(args.until, "--until") or datetime.now(timezone.utc)
since = parse_timestamp(args.since, "--since")
if since is None:
hours = parse_duration_hours(getattr(args, "window", None))
if hours is None:
hours = getattr(args, "window_hours", 24.0)
if hours <= 0:
raise ValueError("window duration must be > 0")
since = until - timedelta(hours=hours)
if since >= until:
raise ValueError("--since must be before --until")
return since, until
def parse_duration_hours(value):
if value is None:
return None
text = value.strip().casefold().replace("_", " ")
if not text:
return None
text = re.sub(r"^(past|last)\s+", "", text)
aliases = {
"day": 24.0,
"24h": 24.0,
"week": 168.0,
"7d": 168.0,
}
if text in aliases:
return aliases[text]
match = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(h|hr|hrs|hour|hours)", text)
if match:
return float(match.group(1))
match = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(d|day|days)", text)
if match:
return float(match.group(1)) * 24.0
match = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(w|week|weeks)", text)
if match:
return float(match.group(1)) * 168.0
raise ValueError(f"Unsupported duration: {value}")
def normalize_requested_labels(labels, all_labels=False):
out = []
seen = set()
for raw in labels:
for piece in raw.split(","):
label = piece.strip()
if not label:
continue
key = label.casefold()
if key not in seen:
out.append(label)
seen.add(key)
phrase = " ".join(label.casefold() for label in out)
if all_labels or phrase in ALL_LABEL_PHRASES:
return [], True
if not out:
raise ValueError(
"At least one feature-area label is required, or use --all-labels"
)
return out, False
def quote_label(label):
if re.fullmatch(r"[A-Za-z0-9_.:-]+", label):
return f"label:{label}"
escaped = label.replace('"', '\\"')
return f'label:"{escaped}"'
def build_search_queries(
repo, owner_labels, since, kind_labels=QUALIFYING_KIND_LABELS, all_labels=False
):
since_date = since.date().isoformat()
queries = []
if all_labels:
for kind_label in kind_labels:
queries.append(
" ".join(
[
f"repo:{repo}",
"is:issue",
f"updated:>={since_date}",
quote_label(kind_label),
]
)
)
return queries
for owner_label in owner_labels:
for kind_label in kind_labels:
queries.append(
" ".join(
[
f"repo:{repo}",
"is:issue",
f"updated:>={since_date}",
quote_label(owner_label),
quote_label(kind_label),
]
)
)
return queries
def _format_gh_error(cmd, err):
stdout = (err.stdout or "").strip()
stderr = (err.stderr or "").strip()
parts = [f"GitHub CLI command failed: {' '.join(cmd)}"]
if stdout:
parts.append(f"stdout: {stdout}")
if stderr:
parts.append(f"stderr: {stderr}")
return "\n".join(parts)
def gh_json(args):
cmd = ["gh", *args]
try:
proc = subprocess.run(cmd, check=True, capture_output=True, text=True)
except FileNotFoundError as err:
raise GhCommandError("`gh` command not found") from err
except subprocess.CalledProcessError as err:
raise GhCommandError(_format_gh_error(cmd, err)) from err
raw = proc.stdout.strip()
if not raw:
return None
try:
return json.loads(raw)
except json.JSONDecodeError as err:
raise GhCommandError(
f"Failed to parse JSON from gh output for {' '.join(args)}"
) from err
def gh_text(args):
cmd = ["gh", *args]
try:
proc = subprocess.run(cmd, check=True, capture_output=True, text=True)
except (FileNotFoundError, subprocess.CalledProcessError):
return ""
return proc.stdout.strip()
def git_head():
try:
proc = subprocess.run(
["git", "rev-parse", "--short=12", "HEAD"],
check=True,
capture_output=True,
text=True,
)
except (FileNotFoundError, subprocess.CalledProcessError):
return None
return proc.stdout.strip() or None
def skill_relative_path():
try:
return str(Path(__file__).resolve().relative_to(Path.cwd().resolve()))
except ValueError:
return str(Path(__file__).resolve())
def gh_api_list_paginated(endpoint, per_page=100, max_pages=None, with_metadata=False):
items = []
page = 1
truncated = False
while True:
sep = "&" if "?" in endpoint else "?"
page_endpoint = f"{endpoint}{sep}per_page={per_page}&page={page}"
payload = gh_json(["api", page_endpoint])
if payload is None:
break
if not isinstance(payload, list):
raise GhCommandError(f"Unexpected paginated payload from gh api {endpoint}")
items.extend(payload)
if len(payload) < per_page:
break
if max_pages is not None and page >= max_pages:
truncated = True
break
page += 1
if with_metadata:
return {
"items": items,
"truncated": truncated,
"pages": page,
"max_pages": max_pages,
}
return items
def search_issue_numbers(queries, limit):
numbers = {}
for query in queries:
page = 1
while True:
payload = gh_json(
[
"api",
"search/issues",
"-X",
"GET",
"-f",
f"q={query}",
"-f",
"per_page=100",
"-f",
f"page={page}",
]
)
if not isinstance(payload, dict):
raise GhCommandError("Unexpected payload from GitHub issue search")
items = payload.get("items") or []
if not isinstance(items, list):
raise GhCommandError("Expected search `items` to be a list")
for item in items:
if not isinstance(item, dict):
continue
number = item.get("number")
if isinstance(number, int):
numbers[number] = str(item.get("updated_at") or "")
if len(items) < 100 or len(numbers) >= limit:
break
page += 1
ordered = sorted(
numbers, key=lambda number: (numbers[number], number), reverse=True
)
return ordered[:limit]
def fetch_issue(repo, number):
payload = gh_json(["api", f"repos/{repo}/issues/{number}"])
if not isinstance(payload, dict):
raise GhCommandError(f"Unexpected issue payload for #{number}")
return payload
def fetch_comments(repo, number, since=None, max_pages=None):
endpoint = f"repos/{repo}/issues/{number}/comments"
if since is not None:
endpoint = f"{endpoint}?since={quote(format_timestamp(since), safe='')}"
return gh_api_list_paginated(
endpoint,
max_pages=max_pages,
with_metadata=True,
)
def fetch_reactions_for_item(endpoint, item):
if reaction_summary(item)["total"] <= 0:
return []
return gh_api_list_paginated(endpoint)
def fetch_comment_reactions(repo, comments):
reactions_by_comment_id = {}
for comment in comments:
comment_id = comment.get("id")
if comment_id in (None, ""):
continue
endpoint = f"repos/{repo}/issues/comments/{comment_id}/reactions"
reactions_by_comment_id[comment_id] = fetch_reactions_for_item(
endpoint, comment
)
return reactions_by_comment_id
def extract_login(user_obj):
if isinstance(user_obj, dict):
return str(user_obj.get("login") or "")
return ""
def is_bot_login(login):
return bool(login) and login.lower().endswith("[bot]")
def is_human_user(user_obj):
login = extract_login(user_obj)
return bool(login) and not is_bot_login(login)
def label_names(issue):
labels = []
for label in issue.get("labels") or []:
if isinstance(label, dict) and label.get("name"):
labels.append(str(label["name"]))
return sorted(labels, key=str.casefold)
def matching_labels(labels, requested):
labels_by_key = {label.casefold(): label for label in labels}
return [label for label in requested if label.casefold() in labels_by_key]
def area_labels(labels):
kind_keys = {label.casefold() for label in QUALIFYING_KIND_LABELS}
return [label for label in labels if label.casefold() not in kind_keys]
def attention_thresholds_for_window(window_hours):
if window_hours <= 0:
raise ValueError("window_hours must be > 0")
window_hours = round(window_hours, 6)
scale = window_hours / BASE_ATTENTION_WINDOW_HOURS
elevated = max(1, math.ceil(ONE_ATTENTION_INTERACTION_THRESHOLD * scale))
very_high = max(
elevated + 1, math.ceil(TWO_ATTENTION_INTERACTION_THRESHOLD * scale)
)
return {
"base_window_hours": BASE_ATTENTION_WINDOW_HOURS,
"window_hours": round(window_hours, 3),
"scale": round(scale, 3),
"elevated": elevated,
"very_high": very_high,
}
def attention_level_for(user_interactions, attention_thresholds=None):
thresholds = attention_thresholds or attention_thresholds_for_window(
BASE_ATTENTION_WINDOW_HOURS
)
if user_interactions >= thresholds["very_high"]:
return 2
if user_interactions >= thresholds["elevated"]:
return 1
return 0
def attention_marker_for(user_interactions, attention_thresholds=None):
return "🔥" * attention_level_for(user_interactions, attention_thresholds)
def reaction_summary(item):
reactions = item.get("reactions")
if not isinstance(reactions, dict):
return {"total": 0, "counts": {}}
counts = {}
for key in REACTION_KEYS:
value = reactions.get(key, 0)
if isinstance(value, int) and value:
counts[key] = value
total = reactions.get("total_count")
if not isinstance(total, int):
total = sum(counts.values())
return {"total": total, "counts": counts}
def reaction_event_summary(reactions, since, until):
counts = {}
total = 0
for reaction in reactions or []:
if not isinstance(reaction, dict):
continue
if not is_in_window(str(reaction.get("created_at") or ""), since, until):
continue
if not is_human_user(reaction.get("user")):
continue
content = str(reaction.get("content") or "")
if not content:
continue
counts[content] = counts.get(content, 0) + 1
total += 1
return {
"total": total,
"counts": counts,
"upvotes": counts.get("+1", 0),
}
def compact_text(value, limit):
text = re.sub(r"\s+", " ", str(value or "")).strip()
if limit <= 0:
return ""
if len(text) <= limit:
return text
return f"{text[: max(limit - 1, 0)].rstrip()}..."
def clean_title_for_description(title):
cleaned = re.sub(r"\s+", " ", str(title or "")).strip()
cleaned = re.sub(
r"^(codex(?: desktop| app|\.app| cli)?|desktop|windows codex app)\s*[:,-]\s*",
"",
cleaned,
flags=re.IGNORECASE,
)
cleaned = re.sub(r"^on windows,\s*", "Windows: ", cleaned, flags=re.IGNORECASE)
cleaned = cleaned.strip(" -:;")
return compact_text(cleaned, 80) or "Issue needs owner review"
def issue_description(issue):
return clean_title_for_description(issue.get("title"))
def is_in_window(timestamp, since, until):
parsed = parse_timestamp(timestamp, "timestamp")
if parsed is None:
return False
return since <= parsed < until
def summarize_comment(
comment, comment_chars, reaction_events=None, since=None, until=None
):
reactions = reaction_summary(comment)
new_reactions = (
reaction_event_summary(reaction_events, since, until)
if since is not None and until is not None
else {"total": 0, "counts": {}, "upvotes": 0}
)
human_user_interaction = is_human_user(comment.get("user"))
return {
"id": comment.get("id"),
"author": extract_login(comment.get("user")),
"author_association": str(comment.get("author_association") or ""),
"created_at": str(comment.get("created_at") or ""),
"updated_at": str(comment.get("updated_at") or ""),
"url": str(comment.get("html_url") or ""),
"human_user_interaction": human_user_interaction,
"reactions": reactions["counts"],
"reaction_total": reactions["total"],
"new_reactions": new_reactions["total"],
"new_upvotes": new_reactions["upvotes"],
"new_reaction_counts": new_reactions["counts"],
"body_excerpt": compact_text(comment.get("body"), comment_chars),
}
def summarize_issue(
issue,
comments,
requested_labels,
since,
until,
body_chars,
comment_chars,
issue_reaction_events=None,
comment_reactions_by_id=None,
all_labels=False,
comments_hydration=None,
attention_thresholds=None,
):
labels = label_names(issue)
labels_by_key = {label.casefold() for label in labels}
kind_labels = [
label for label in QUALIFYING_KIND_LABELS if label.casefold() in labels_by_key
]
if all_labels:
owner_labels = area_labels(labels) or ["unlabeled"]
else:
owner_labels = matching_labels(labels, requested_labels)
if not kind_labels or not owner_labels:
return None
updated_at = str(issue.get("updated_at") or "")
if not is_in_window(updated_at, since, until):
return None
new_issue = is_in_window(str(issue.get("created_at") or ""), since, until)
comment_reactions_by_id = comment_reactions_by_id or {}
new_comments = [
summarize_comment(
comment,
comment_chars,
reaction_events=comment_reactions_by_id.get(comment.get("id")),
since=since,
until=until,
)
for comment in comments
if is_in_window(str(comment.get("created_at") or ""), since, until)
]
new_comments.sort(key=lambda item: (item["created_at"], str(item["id"])))
issue_reactions = reaction_summary(issue)
issue_reaction_events_summary = reaction_event_summary(
issue_reaction_events, since, until
)
comment_reaction_events_summary = reaction_event_summary(
[
reaction
for reactions in comment_reactions_by_id.values()
for reaction in reactions
],
since,
until,
)
new_reactions = (
issue_reaction_events_summary["total"]
+ comment_reaction_events_summary["total"]
)
new_upvotes = (
issue_reaction_events_summary["upvotes"]
+ comment_reaction_events_summary["upvotes"]
)
all_comment_reaction_total = sum(
reaction_summary(comment)["total"] for comment in comments
)
new_comment_reaction_total = sum(
comment["reaction_total"] for comment in new_comments
)
new_issue_user_interaction = new_issue and is_human_user(issue.get("user"))
new_comment_user_interactions = sum(
1 for comment in new_comments if comment["human_user_interaction"]
)
user_interactions = (
int(new_issue_user_interaction) + new_comment_user_interactions + new_reactions
)
attention_level = attention_level_for(user_interactions, attention_thresholds)
attention_marker = attention_marker_for(user_interactions, attention_thresholds)
updated_without_visible_new_post = (
not new_issue and not new_comments and new_reactions == 0
)
engagement_score = (
len(new_comments) * 3
+ new_reactions
+ issue_reactions["total"]
+ new_comment_reaction_total
+ min(int(issue.get("comments") or len(comments) or 0), 10)
)
return {
"number": issue.get("number"),
"title": str(issue.get("title") or ""),
"description": issue_description(issue),
"url": str(issue.get("html_url") or ""),
"state": str(issue.get("state") or ""),
"author": extract_login(issue.get("user")),
"author_association": str(issue.get("author_association") or ""),
"created_at": str(issue.get("created_at") or ""),
"updated_at": updated_at,
"labels": labels,
"kind_labels": kind_labels,
"owner_labels": owner_labels,
"comments_total": int(issue.get("comments") or len(comments) or 0),
"comments_hydration": comments_hydration
or {
"fetched": len(comments),
"since": None,
"truncated": False,
"max_pages": None,
},
"issue_reactions": issue_reactions["counts"],
"issue_reaction_total": issue_reactions["total"],
"comment_reaction_total": all_comment_reaction_total,
"new_comment_reaction_total": new_comment_reaction_total,
"new_issue_reactions": issue_reaction_events_summary["total"],
"new_issue_upvotes": issue_reaction_events_summary["upvotes"],
"new_comment_reactions": comment_reaction_events_summary["total"],
"new_comment_upvotes": comment_reaction_events_summary["upvotes"],
"new_reactions": new_reactions,
"new_upvotes": new_upvotes,
"user_interactions": user_interactions,
"attention": attention_level > 0,
"attention_level": attention_level,
"attention_marker": attention_marker,
"engagement_score": engagement_score,
"activity": {
"new_issue": new_issue,
"new_comments": len(new_comments),
"new_human_comments": new_comment_user_interactions,
"new_reactions": new_reactions,
"new_upvotes": new_upvotes,
"updated_without_visible_new_post": updated_without_visible_new_post,
},
"body_excerpt": compact_text(issue.get("body"), body_chars),
"new_comments": new_comments,
}
def count_by_label(issues, labels):
out = {}
for label in labels:
matching = [issue for issue in issues if label in issue["owner_labels"]]
out[label] = {
"issues": len(matching),
"new_issues": sum(
1 for issue in matching if issue["activity"]["new_issue"]
),
"new_comments": sum(
issue["activity"]["new_comments"] for issue in matching
),
}
return out
def count_by_kind(issues):
out = {}
for kind in QUALIFYING_KIND_LABELS:
matching = [issue for issue in issues if kind in issue["kind_labels"]]
out[kind] = {
"issues": len(matching),
"new_issues": sum(
1 for issue in matching if issue["activity"]["new_issue"]
),
"new_comments": sum(
issue["activity"]["new_comments"] for issue in matching
),
}
return out
def hot_items(issues, limit=8):
ranked = sorted(
issues,
key=lambda issue: (
issue["attention"],
issue["attention_level"],
issue["user_interactions"],
issue["engagement_score"],
issue["activity"]["new_comments"],
issue["issue_reaction_total"] + issue["comment_reaction_total"],
issue["updated_at"],
),
reverse=True,
)
return [
{
"number": issue["number"],
"title": issue["title"],
"url": issue["url"],
"owner_labels": issue["owner_labels"],
"kind_labels": issue["kind_labels"],
"attention": issue["attention"],
"attention_level": issue["attention_level"],
"attention_marker": issue["attention_marker"],
"user_interactions": issue["user_interactions"],
"new_reactions": issue["new_reactions"],
"new_upvotes": issue["new_upvotes"],
"engagement_score": issue["engagement_score"],
"new_comments": issue["activity"]["new_comments"],
"reaction_total": issue["issue_reaction_total"]
+ issue["comment_reaction_total"],
}
for issue in ranked[:limit]
if issue["engagement_score"] > 0
]
def ranked_digest_issues(issues):
return sorted(
issues,
key=lambda issue: (
issue["attention"],
issue["attention_level"],
issue["user_interactions"],
issue["engagement_score"],
issue["activity"]["new_comments"],
issue["updated_at"],
),
reverse=True,
)
def digest_rows(issues, limit=10, ref_map=None):
ranked = ranked_digest_issues(issues)
if ref_map is None:
ref_map = {issue["number"]: ref for ref, issue in enumerate(ranked, start=1)}
rows = []
for issue in ranked[:limit]:
ref = ref_map[issue["number"]]
reaction_total = issue["issue_reaction_total"] + issue["comment_reaction_total"]
rows.append(
{
"ref": ref,
"ref_markdown": f"[{ref}]({issue['url']})",
"marker": issue["attention_marker"],
"attention_marker": issue["attention_marker"],
"number": issue["number"],
"description": issue["description"],
"title": issue["title"],
"url": issue["url"],
"area": ", ".join(issue["owner_labels"]),
"kind": ", ".join(issue["kind_labels"]),
"state": issue["state"],
"interactions": issue["user_interactions"],
"user_interactions": issue["user_interactions"],
"new_reactions": issue["new_reactions"],
"new_upvotes": issue["new_upvotes"],
"current_reactions": reaction_total,
}
)
return rows
def issue_ref_markdown(issue, ref_map):
ref = ref_map[issue["number"]]
return f"[{ref}]({issue['url']})"
def summary_inputs(issues, limit=80, ref_map=None):
ranked = ranked_digest_issues(issues)
if ref_map is None:
ref_map = {issue["number"]: ref for ref, issue in enumerate(ranked, start=1)}
rows = []
for issue in ranked[:limit]:
rows.append(
{
"ref": ref_map[issue["number"]],
"ref_markdown": issue_ref_markdown(issue, ref_map),
"number": issue["number"],
"title": issue["title"],
"description": issue["description"],
"url": issue["url"],
"labels": issue["labels"],
"owner_labels": issue["owner_labels"],
"kind_labels": issue["kind_labels"],
"state": issue.get("state", ""),
"attention_marker": issue.get("attention_marker", ""),
"interactions": issue["user_interactions"],
"new_comments": issue["activity"].get("new_comments", 0),
"new_reactions": issue.get("new_reactions", 0),
"new_upvotes": issue.get("new_upvotes", 0),
"current_reactions": issue.get("issue_reaction_total", 0)
+ issue.get("comment_reaction_total", 0),
}
)
return rows
def collect_digest(args):
since, until = resolve_window(args)
window_hours = (until - since).total_seconds() / 3600
attention_thresholds = attention_thresholds_for_window(window_hours)
requested_labels, all_labels = normalize_requested_labels(
args.labels, all_labels=args.all_labels
)
queries = build_search_queries(
args.repo, requested_labels, since, all_labels=all_labels
)
numbers = search_issue_numbers(queries, args.limit_issues)
gh_version_output = gh_text(["--version"])
issues = []
max_comment_pages = None if args.max_comment_pages <= 0 else args.max_comment_pages
for number in numbers:
issue = fetch_issue(args.repo, number)
comments_since = None if args.fetch_all_comments else since
comments_payload = fetch_comments(
args.repo,
number,
since=comments_since,
max_pages=max_comment_pages,
)
comments = comments_payload["items"]
issue_reaction_events = fetch_reactions_for_item(
f"repos/{args.repo}/issues/{number}/reactions", issue
)
comment_reactions_by_id = fetch_comment_reactions(args.repo, comments)
comments_hydration = {
"fetched": len(comments),
"total": int(issue.get("comments") or len(comments) or 0),
"since": format_timestamp(comments_since) if comments_since else None,
"truncated": comments_payload["truncated"],
"max_pages": comments_payload["max_pages"],
"fetch_all_comments": args.fetch_all_comments,
}
summary = summarize_issue(
issue,
comments,
requested_labels,
since,
until,
args.body_chars,
args.comment_chars,
issue_reaction_events=issue_reaction_events,
comment_reactions_by_id=comment_reactions_by_id,
all_labels=all_labels,
comments_hydration=comments_hydration,
attention_thresholds=attention_thresholds,
)
if summary is not None:
issues.append(summary)
issues.sort(
key=lambda issue: (issue["updated_at"], int(issue["number"] or 0)), reverse=True
)
totals = {
"candidate_issues": len(numbers),
"included_issues": len(issues),
"new_issues": sum(1 for issue in issues if issue["activity"]["new_issue"]),
"issues_with_new_comments": sum(
1 for issue in issues if issue["activity"]["new_comments"] > 0
),
"new_comments": sum(issue["activity"]["new_comments"] for issue in issues),
"comments_fetched": sum(
issue["comments_hydration"]["fetched"] for issue in issues
),
"issues_with_truncated_comment_hydration": sum(
1 for issue in issues if issue["comments_hydration"]["truncated"]
),
"updated_without_visible_new_post": sum(
1
for issue in issues
if issue["activity"]["updated_without_visible_new_post"]
),
"issue_reactions_current_total": sum(
issue["issue_reaction_total"] for issue in issues
),
"comment_reactions_current_total": sum(
issue["comment_reaction_total"] for issue in issues
),
"new_reactions": sum(issue["new_reactions"] for issue in issues),
"new_upvotes": sum(issue["new_upvotes"] for issue in issues),
"user_interactions": sum(issue["user_interactions"] for issue in issues),
}
ranked = ranked_digest_issues(issues)
ref_map = {issue["number"]: ref for ref, issue in enumerate(ranked, start=1)}
filter_label = "all" if all_labels else requested_labels
return {
"generated_at": format_timestamp(datetime.now(timezone.utc)),
"source": {
"repo": args.repo,
"skill": "codex-issue-digest",
"collector": skill_relative_path(),
"script_version": SCRIPT_VERSION,
"git_head": git_head(),
"gh_version": gh_version_output.splitlines()[0]
if gh_version_output
else None,
},
"window": {
"since": format_timestamp(since),
"until": format_timestamp(until),
"hours": round(window_hours, 3),
},
"attention_thresholds": attention_thresholds,
"filters": {
"owner_labels": filter_label,
"all_labels": all_labels,
"kind_labels": list(QUALIFYING_KIND_LABELS),
},
"collection_notes": [
"Issues are selected when they currently have bug or enhancement plus at least one requested owner label and were updated during the window.",
"By default, issue comments are fetched with since=window_start and a max page cap to avoid long historical threads; use --fetch-all-comments when exhaustive comment history is needed.",
"New issue comments are filtered by comment creation time within the window from the fetched comment set.",
"Reaction events are counted by GitHub reaction created_at timestamps for hydrated issues and fetched comments.",
"Current reaction totals are standing engagement signals; new_reactions and new_upvotes are windowed activity.",
"The collector does not assign semantic clusters; use summary_inputs as model-ready evidence for report-time clustering.",
"Pure reaction-only issues may be missed if GitHub issue search does not surface them via updated_at.",
"Issues updated during the window without a new issue body or new comment are retained because label/status edits can still be useful owner signals.",
],
"totals": totals,
"by_owner_label": count_by_label(
issues,
sorted(
{area for issue in issues for area in issue["owner_labels"]},
key=str.casefold,
)
if all_labels
else requested_labels,
),
"by_kind_label": count_by_kind(issues),
"hot_items": hot_items(issues),
"summary_inputs": summary_inputs(issues, ref_map=ref_map),
"digest_rows": digest_rows(issues, ref_map=ref_map),
"issues": issues,
}
def main():
args = parse_args()
try:
digest = collect_digest(args)
except (GhCommandError, RuntimeError, ValueError) as err:
sys.stderr.write(f"collect_issue_digest.py error: {err}\n")
return 1
sys.stdout.write(json.dumps(digest, indent=2, sort_keys=True) + "\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,614 @@
import importlib.util
from datetime import timezone
from pathlib import Path
MODULE_PATH = Path(__file__).with_name("collect_issue_digest.py")
MODULE_SPEC = importlib.util.spec_from_file_location(
"collect_issue_digest", MODULE_PATH
)
collect_issue_digest = importlib.util.module_from_spec(MODULE_SPEC)
assert MODULE_SPEC.loader is not None
MODULE_SPEC.loader.exec_module(collect_issue_digest)
def test_build_search_queries_uses_each_owner_and_kind_label():
since = collect_issue_digest.parse_timestamp("2026-04-25T12:34:56Z", "--since")
queries = collect_issue_digest.build_search_queries(
"openai/codex", ["tui", "exec"], since
)
assert queries == [
"repo:openai/codex is:issue updated:>=2026-04-25 label:tui label:bug",
"repo:openai/codex is:issue updated:>=2026-04-25 label:tui label:enhancement",
"repo:openai/codex is:issue updated:>=2026-04-25 label:exec label:bug",
"repo:openai/codex is:issue updated:>=2026-04-25 label:exec label:enhancement",
]
def test_build_search_queries_can_scan_all_labels():
since = collect_issue_digest.parse_timestamp("2026-04-25T12:34:56Z", "--since")
queries = collect_issue_digest.build_search_queries(
"openai/codex", [], since, all_labels=True
)
assert queries == [
"repo:openai/codex is:issue updated:>=2026-04-25 label:bug",
"repo:openai/codex is:issue updated:>=2026-04-25 label:enhancement",
]
def test_normalize_requested_labels_accepts_all_area_phrases():
assert collect_issue_digest.normalize_requested_labels(["all", "areas"]) == (
[],
True,
)
assert collect_issue_digest.normalize_requested_labels(["all-labels"]) == (
[],
True,
)
def test_summarize_issue_keeps_new_comments_and_reaction_signals():
since = collect_issue_digest.parse_timestamp("2026-04-25T00:00:00Z", "--since")
until = collect_issue_digest.parse_timestamp("2026-04-26T00:00:00Z", "--until")
issue = {
"number": 123,
"title": "TUI does not redraw",
"html_url": "https://github.com/openai/codex/issues/123",
"state": "open",
"created_at": "2026-04-24T20:00:00Z",
"updated_at": "2026-04-25T10:00:00Z",
"user": {"login": "alice"},
"author_association": "NONE",
"comments": 2,
"body": "The terminal freezes after resize.",
"labels": [{"name": "bug"}, {"name": "tui"}],
"reactions": {"total_count": 3, "+1": 2, "rocket": 1},
}
comments = [
{
"id": 1,
"created_at": "2026-04-25T11:00:00Z",
"updated_at": "2026-04-25T11:00:00Z",
"html_url": "https://github.com/openai/codex/issues/123#issuecomment-1",
"user": {"login": "bob"},
"author_association": "MEMBER",
"body": "I can reproduce this on main.",
"reactions": {"total_count": 4, "heart": 1, "+1": 3},
},
{
"id": 2,
"created_at": "2026-04-24T11:00:00Z",
"updated_at": "2026-04-24T11:00:00Z",
"html_url": "https://github.com/openai/codex/issues/123#issuecomment-2",
"user": {"login": "carol"},
"author_association": "NONE",
"body": "Older comment.",
"reactions": {"total_count": 1, "eyes": 1},
},
]
summary = collect_issue_digest.summarize_issue(
issue,
comments,
["tui", "exec"],
since,
until,
body_chars=200,
comment_chars=200,
)
assert summary == {
"number": 123,
"title": "TUI does not redraw",
"description": "TUI does not redraw",
"url": "https://github.com/openai/codex/issues/123",
"state": "open",
"author": "alice",
"author_association": "NONE",
"created_at": "2026-04-24T20:00:00Z",
"updated_at": "2026-04-25T10:00:00Z",
"labels": ["bug", "tui"],
"kind_labels": ["bug"],
"owner_labels": ["tui"],
"comments_total": 2,
"comments_hydration": {
"fetched": 2,
"since": None,
"truncated": False,
"max_pages": None,
},
"issue_reactions": {"+1": 2, "rocket": 1},
"issue_reaction_total": 3,
"comment_reaction_total": 5,
"new_comment_reaction_total": 4,
"new_issue_reactions": 0,
"new_issue_upvotes": 0,
"new_comment_reactions": 0,
"new_comment_upvotes": 0,
"new_reactions": 0,
"new_upvotes": 0,
"user_interactions": 1,
"attention": False,
"attention_level": 0,
"attention_marker": "",
"engagement_score": 12,
"activity": {
"new_issue": False,
"new_comments": 1,
"new_human_comments": 1,
"new_reactions": 0,
"new_upvotes": 0,
"updated_without_visible_new_post": False,
},
"body_excerpt": "The terminal freezes after resize.",
"new_comments": [
{
"id": 1,
"author": "bob",
"author_association": "MEMBER",
"created_at": "2026-04-25T11:00:00Z",
"updated_at": "2026-04-25T11:00:00Z",
"url": "https://github.com/openai/codex/issues/123#issuecomment-1",
"human_user_interaction": True,
"reactions": {"+1": 3, "heart": 1},
"reaction_total": 4,
"new_reactions": 0,
"new_upvotes": 0,
"new_reaction_counts": {},
"body_excerpt": "I can reproduce this on main.",
}
],
}
def test_summarize_issue_filters_non_owner_or_non_kind_labels():
since = collect_issue_digest.parse_timestamp("2026-04-25T00:00:00Z", "--since")
until = collect_issue_digest.parse_timestamp("2026-04-26T00:00:00Z", "--until")
base_issue = {
"number": 1,
"title": "Question",
"created_at": "2026-04-25T01:00:00Z",
"updated_at": "2026-04-25T01:00:00Z",
"labels": [{"name": "question"}, {"name": "tui"}],
}
assert (
collect_issue_digest.summarize_issue(
base_issue,
[],
["tui"],
since,
until,
body_chars=100,
comment_chars=100,
)
is None
)
issue_without_owner = dict(base_issue)
issue_without_owner["labels"] = [{"name": "bug"}, {"name": "app"}]
assert (
collect_issue_digest.summarize_issue(
issue_without_owner,
[],
["tui"],
since,
until,
body_chars=100,
comment_chars=100,
)
is None
)
def test_resolve_window_defaults_to_previous_hours():
class Args:
since = None
until = "2026-04-26T12:00:00Z"
window_hours = 24
since, until = collect_issue_digest.resolve_window(Args())
assert since.isoformat() == "2026-04-25T12:00:00+00:00"
assert until.tzinfo == timezone.utc
def test_parse_duration_hours_accepts_common_phrases():
assert collect_issue_digest.parse_duration_hours("past week") == 168
assert collect_issue_digest.parse_duration_hours("48h") == 48
assert collect_issue_digest.parse_duration_hours("2 days") == 48
assert collect_issue_digest.parse_duration_hours("1w") == 168
def test_attention_thresholds_scale_by_window_length():
one_day = collect_issue_digest.attention_thresholds_for_window(24)
assert one_day["elevated"] == 10
assert one_day["very_high"] == 20
half_day = collect_issue_digest.attention_thresholds_for_window(12)
assert half_day["elevated"] == 5
assert half_day["very_high"] == 10
week = collect_issue_digest.attention_thresholds_for_window(168)
assert week["elevated"] == 70
assert week["very_high"] == 140
assert collect_issue_digest.attention_marker_for(69, week) == ""
assert collect_issue_digest.attention_marker_for(107, week) == "🔥"
assert collect_issue_digest.attention_marker_for(140, week) == "🔥🔥"
def test_fetch_comments_uses_since_filter_and_page_cap(monkeypatch):
calls = []
def fake_gh_json(args):
calls.append(args)
return [{"id": idx} for idx in range(100)]
monkeypatch.setattr(collect_issue_digest, "gh_json", fake_gh_json)
since = collect_issue_digest.parse_timestamp("2026-04-25T00:00:00Z", "--since")
payload = collect_issue_digest.fetch_comments(
"openai/codex", 123, since=since, max_pages=1
)
assert len(payload["items"]) == 100
assert payload["truncated"] is True
assert payload["max_pages"] == 1
assert calls == [
[
"api",
"repos/openai/codex/issues/123/comments?since=2026-04-25T00%3A00%3A00Z&per_page=100&page=1",
]
]
def test_issue_description_prefers_title_over_body_noise():
issue = {
"title": "Codex.app GUI: MCP child processes not reaped after task completion",
"body": "A later crash mention should not override the title-level symptom.",
"labels": [{"name": "app"}, {"name": "bug"}],
}
description = collect_issue_digest.issue_description(issue)
assert "MCP child processes" in description
assert "crash" not in description.casefold()
def test_attention_markers_count_human_user_interactions():
since = collect_issue_digest.parse_timestamp("2026-04-25T00:00:00Z", "--since")
until = collect_issue_digest.parse_timestamp("2026-04-26T00:00:00Z", "--until")
issue = {
"number": 456,
"title": "Agent context is exploding",
"html_url": "https://github.com/openai/codex/issues/456",
"state": "open",
"created_at": "2026-04-25T01:00:00Z",
"updated_at": "2026-04-25T12:00:00Z",
"user": {"login": "alice"},
"labels": [{"name": "bug"}, {"name": "agent"}],
}
comments = [
{
"id": idx,
"created_at": "2026-04-25T02:00:00Z",
"updated_at": "2026-04-25T02:00:00Z",
"user": {"login": f"user-{idx}"},
"body": "same here",
}
for idx in range(9)
]
comments.append(
{
"id": 99,
"created_at": "2026-04-25T02:00:00Z",
"updated_at": "2026-04-25T02:00:00Z",
"user": {"login": "github-actions[bot]"},
"body": "duplicate bot note",
}
)
summary = collect_issue_digest.summarize_issue(
issue,
comments,
["agent"],
since,
until,
body_chars=100,
comment_chars=100,
)
assert summary["user_interactions"] == 10
assert summary["activity"]["new_human_comments"] == 9
assert summary["attention"] is True
assert summary["attention_level"] == 1
assert summary["attention_marker"] == "🔥"
issue["created_at"] = "2026-04-24T01:00:00Z"
comments.extend(
{
"id": idx,
"created_at": "2026-04-25T03:00:00Z",
"updated_at": "2026-04-25T03:00:00Z",
"user": {"login": f"extra-user-{idx}"},
"body": "also seeing this",
}
for idx in range(11)
)
summary = collect_issue_digest.summarize_issue(
issue,
comments,
["agent"],
since,
until,
body_chars=100,
comment_chars=100,
)
assert summary["user_interactions"] == 20
assert summary["attention_level"] == 2
assert summary["attention_marker"] == "🔥🔥"
def test_reactions_count_toward_attention_markers():
since = collect_issue_digest.parse_timestamp("2026-04-25T00:00:00Z", "--since")
until = collect_issue_digest.parse_timestamp("2026-04-26T00:00:00Z", "--until")
issue = {
"number": 789,
"title": "Support 1M token context",
"html_url": "https://github.com/openai/codex/issues/789",
"state": "open",
"created_at": "2026-04-24T01:00:00Z",
"updated_at": "2026-04-25T12:00:00Z",
"user": {"login": "alice"},
"labels": [{"name": "enhancement"}, {"name": "context"}],
"reactions": {"total_count": 20, "+1": 20},
}
comments = [
{
"id": 1,
"created_at": "2026-04-25T02:00:00Z",
"updated_at": "2026-04-25T02:00:00Z",
"user": {"login": "commenter"},
"body": "please",
"reactions": {"total_count": 2, "+1": 2},
}
]
issue_reactions = [
{
"content": "+1",
"created_at": "2026-04-25T03:00:00Z",
"user": {"login": f"reactor-{idx}"},
}
for idx in range(18)
]
comment_reactions_by_id = {
1: [
{
"content": "heart",
"created_at": "2026-04-25T04:00:00Z",
"user": {"login": "human-reactor"},
},
{
"content": "+1",
"created_at": "2026-04-25T04:00:00Z",
"user": {"login": "github-actions[bot]"},
},
]
}
summary = collect_issue_digest.summarize_issue(
issue,
comments,
["context"],
since,
until,
body_chars=100,
comment_chars=100,
issue_reaction_events=issue_reactions,
comment_reactions_by_id=comment_reactions_by_id,
)
assert summary["new_reactions"] == 19
assert summary["new_upvotes"] == 18
assert summary["user_interactions"] == 20
assert summary["attention_level"] == 2
assert summary["attention_marker"] == "🔥🔥"
assert summary["new_comments"][0]["new_reactions"] == 1
assert summary["new_comments"][0]["new_upvotes"] == 0
def test_digest_rows_are_table_ready_with_concise_descriptions():
rows = collect_issue_digest.digest_rows(
[
{
"number": 1,
"title": "Quiet bug",
"description": "Quiet bug",
"url": "https://github.com/openai/codex/issues/1",
"owner_labels": ["context"],
"kind_labels": ["bug"],
"state": "open",
"attention": False,
"attention_level": 0,
"attention_marker": "",
"user_interactions": 1,
"new_reactions": 0,
"new_upvotes": 0,
"engagement_score": 3,
"issue_reaction_total": 0,
"comment_reaction_total": 0,
"updated_at": "2026-04-25T01:00:00Z",
"activity": {
"new_issue": True,
"new_comments": 0,
"new_reactions": 0,
"updated_without_visible_new_post": False,
},
},
{
"number": 2,
"title": "Busy bug",
"description": "High-volume bug report",
"url": "https://github.com/openai/codex/issues/2",
"owner_labels": ["agent"],
"kind_labels": ["bug"],
"state": "open",
"attention": True,
"attention_level": 1,
"attention_marker": "🔥",
"user_interactions": 17,
"new_reactions": 3,
"new_upvotes": 2,
"engagement_score": 20,
"issue_reaction_total": 5,
"comment_reaction_total": 2,
"updated_at": "2026-04-25T02:00:00Z",
"activity": {
"new_issue": False,
"new_comments": 16,
"new_reactions": 3,
"updated_without_visible_new_post": False,
},
},
]
)
assert rows[0] == {
"ref": 1,
"ref_markdown": "[1](https://github.com/openai/codex/issues/2)",
"marker": "🔥",
"attention_marker": "🔥",
"number": 2,
"description": "High-volume bug report",
"title": "Busy bug",
"url": "https://github.com/openai/codex/issues/2",
"area": "agent",
"kind": "bug",
"state": "open",
"interactions": 17,
"user_interactions": 17,
"new_reactions": 3,
"new_upvotes": 2,
"current_reactions": 7,
}
def test_summary_inputs_are_model_ready_without_preclustering():
issues = [
{
"number": 20,
"title": "Windows app Browser Use external navigation fails",
"description": "Browser Use navigation or app-server failure",
"url": "https://github.com/openai/codex/issues/20",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"attention": False,
"attention_level": 0,
"attention_marker": "",
"user_interactions": 3,
"new_reactions": 1,
"engagement_score": 8,
"updated_at": "2026-04-25T04:00:00Z",
"activity": {"new_comments": 2},
},
{
"number": 21,
"title": "On Windows, cmake output waits until timeout",
"description": "Windows command timeout/capture problem",
"url": "https://github.com/openai/codex/issues/21",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"attention": False,
"attention_level": 0,
"attention_marker": "",
"user_interactions": 3,
"new_reactions": 0,
"engagement_score": 7,
"updated_at": "2026-04-25T03:00:00Z",
"activity": {"new_comments": 3},
},
{
"number": 22,
"title": "Windows computer use tool fails to click buttons",
"description": "Computer-use workflow failure",
"url": "https://github.com/openai/codex/issues/22",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"attention": False,
"attention_level": 0,
"attention_marker": "",
"user_interactions": 3,
"new_reactions": 0,
"engagement_score": 6,
"updated_at": "2026-04-25T02:00:00Z",
"activity": {"new_comments": 3},
},
]
rows = collect_issue_digest.summary_inputs(issues, ref_map={20: 1, 21: 2, 22: 3})
assert rows == [
{
"ref": 1,
"ref_markdown": "[1](https://github.com/openai/codex/issues/20)",
"number": 20,
"title": "Windows app Browser Use external navigation fails",
"description": "Browser Use navigation or app-server failure",
"url": "https://github.com/openai/codex/issues/20",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"state": "",
"attention_marker": "",
"interactions": 3,
"new_comments": 2,
"new_reactions": 1,
"new_upvotes": 0,
"current_reactions": 0,
},
{
"ref": 2,
"ref_markdown": "[2](https://github.com/openai/codex/issues/21)",
"number": 21,
"title": "On Windows, cmake output waits until timeout",
"description": "Windows command timeout/capture problem",
"url": "https://github.com/openai/codex/issues/21",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"state": "",
"attention_marker": "",
"interactions": 3,
"new_comments": 3,
"new_reactions": 0,
"new_upvotes": 0,
"current_reactions": 0,
},
{
"ref": 3,
"ref_markdown": "[3](https://github.com/openai/codex/issues/22)",
"number": 22,
"title": "Windows computer use tool fails to click buttons",
"description": "Computer-use workflow failure",
"url": "https://github.com/openai/codex/issues/22",
"labels": ["app", "bug"],
"owner_labels": ["app"],
"kind_labels": ["bug"],
"state": "",
"attention_marker": "",
"interactions": 3,
"new_comments": 3,
"new_reactions": 0,
"new_upvotes": 0,
"current_reactions": 0,
},
]
+3 -1
View File
@@ -33,7 +33,9 @@ runs:
run: Copy-Item (Get-Command dotslash).Source -Destination "$env:LOCALAPPDATA\Microsoft\WindowsApps\dotslash.exe"
- name: Set up Bazel
uses: bazelbuild/setup-bazelisk@b39c379c82683a5f25d34f0d062761f62693e0b2 # v3
# setup-bazelisk is archived; use the maintained successor and let it
# prefer the runner's pre-installed Bazelisk when available.
uses: bazel-contrib/setup-bazel@c5acdfb288317d0b5c0bbd7a396a3dc868bb0f86 # 0.19.0
- name: Configure Bazel repository cache
id: configure_bazel_repository_cache
+3 -1
View File
@@ -78,7 +78,9 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Set up Bazel
uses: bazelbuild/setup-bazelisk@b39c379c82683a5f25d34f0d062761f62693e0b2 # v3
uses: ./.github/actions/setup-bazel-ci
with:
target: ${{ matrix.target }}
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
+3 -1
View File
@@ -75,7 +75,9 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Set up Bazel
uses: bazelbuild/setup-bazelisk@b39c379c82683a5f25d34f0d062761f62693e0b2 # v3
uses: ./.github/actions/setup-bazel-ci
with:
target: ${{ matrix.target }}
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
+36
View File
@@ -2370,6 +2370,8 @@ dependencies = [
"codex-hooks",
"codex-login",
"codex-mcp",
"codex-memories-read",
"codex-memories-write",
"codex-model-provider",
"codex-model-provider-info",
"codex-models-manager",
@@ -2921,6 +2923,40 @@ dependencies = [
"wiremock",
]
[[package]]
name = "codex-memories-read"
version = "0.0.0"
dependencies = [
"codex-protocol",
"codex-shell-command",
"codex-utils-absolute-path",
"codex-utils-output-truncation",
"codex-utils-template",
"pretty_assertions",
"tempfile",
"tokio",
]
[[package]]
name = "codex-memories-write"
version = "0.0.0"
dependencies = [
"anyhow",
"chrono",
"codex-git-utils",
"codex-models-manager",
"codex-protocol",
"codex-state",
"codex-utils-absolute-path",
"codex-utils-output-truncation",
"codex-utils-template",
"pretty_assertions",
"tempfile",
"tokio",
"tracing",
"uuid",
]
[[package]]
name = "codex-model-provider"
version = "0.0.0"
+4
View File
@@ -46,6 +46,8 @@ members = [
"login",
"codex-mcp",
"mcp-server",
"memories/read",
"memories/write",
"model-provider-info",
"models-manager",
"network-proxy",
@@ -153,6 +155,8 @@ codex-keyring-store = { path = "keyring-store" }
codex-linux-sandbox = { path = "linux-sandbox" }
codex-lmstudio = { path = "lmstudio" }
codex-login = { path = "login" }
codex-memories-read = { path = "memories/read" }
codex-memories-write = { path = "memories/write" }
codex-mcp = { path = "codex-mcp" }
codex-mcp-server = { path = "mcp-server" }
codex-model-provider-info = { path = "model-provider-info" }
+132 -243
View File
@@ -7,6 +7,7 @@ use crate::error_code::INPUT_TOO_LARGE_ERROR_CODE;
use crate::error_code::INTERNAL_ERROR_CODE;
use crate::error_code::INVALID_PARAMS_ERROR_CODE;
use crate::error_code::INVALID_REQUEST_ERROR_CODE;
use crate::error_code::invalid_params;
use crate::fuzzy_file_search::FuzzyFileSearchSession;
use crate::fuzzy_file_search::run_fuzzy_file_search;
use crate::fuzzy_file_search::start_fuzzy_file_search_session;
@@ -1353,7 +1354,7 @@ impl CodexMessageProcessor {
self.config.cli_auth_credentials_store_mode,
) {
Ok(()) => {
self.auth_manager.reload();
self.auth_manager.reload().await;
Ok(())
}
Err(err) => Err(JSONRPCErrorError {
@@ -1504,7 +1505,7 @@ impl CodexMessageProcessor {
.await;
if success {
auth_manager.reload();
auth_manager.reload().await;
config_manager.replace_cloud_requirements_loader(
auth_manager.clone(),
chatgpt_base_url,
@@ -1612,7 +1613,7 @@ impl CodexMessageProcessor {
.await;
if success {
auth_manager.reload();
auth_manager.reload().await;
config_manager.replace_cloud_requirements_loader(
auth_manager.clone(),
chatgpt_base_url,
@@ -1748,7 +1749,7 @@ impl CodexMessageProcessor {
self.outgoing.send_error(request_id, error).await;
return;
}
self.auth_manager.reload();
self.auth_manager.reload().await;
self.config_manager.replace_cloud_requirements_loader(
self.auth_manager.clone(),
self.config.chatgpt_base_url.clone(),
@@ -6323,6 +6324,15 @@ impl CodexMessageProcessor {
self.outgoing.send_error(request_id, error).await;
}
async fn send_internal_error(&self, request_id: ConnectionRequestId, message: String) {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message,
data: None,
};
self.outgoing.send_error(request_id, error).await;
}
fn input_too_large_error(actual_chars: usize) -> JSONRPCErrorError {
JSONRPCErrorError {
code: INVALID_PARAMS_ERROR_CODE,
@@ -6345,41 +6355,6 @@ impl CodexMessageProcessor {
Ok(())
}
async fn send_internal_error(&self, request_id: ConnectionRequestId, message: String) {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message,
data: None,
};
self.outgoing.send_error(request_id, error).await;
}
async fn send_marketplace_error(
&self,
request_id: ConnectionRequestId,
err: MarketplaceError,
action: &str,
) {
match err {
MarketplaceError::MarketplaceNotFound { .. } => {
self.send_invalid_request_error(request_id, err.to_string())
.await;
}
MarketplaceError::Io { .. } => {
self.send_internal_error(request_id, format!("failed to {action}: {err}"))
.await;
}
MarketplaceError::InvalidMarketplaceFile { .. }
| MarketplaceError::PluginNotFound { .. }
| MarketplaceError::PluginNotAvailable { .. }
| MarketplaceError::PluginsDisabled
| MarketplaceError::InvalidPlugin(_) => {
self.send_invalid_request_error(request_id, err.to_string())
.await;
}
}
}
async fn wait_for_thread_shutdown(thread: &Arc<CodexThread>) -> ThreadShutdownResult {
match tokio::time::timeout(Duration::from_secs(10), thread.shutdown_and_wait()).await {
Ok(Ok(())) => ThreadShutdownResult::Complete,
@@ -6458,34 +6433,33 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: ThreadUnsubscribeParams,
) {
let thread_id = match ThreadId::from_string(&params.thread_id) {
Ok(id) => id,
Err(err) => {
self.send_invalid_request_error(request_id, format!("invalid thread id: {err}"))
.await;
return;
}
};
let result = self
.thread_unsubscribe_response(params, request_id.connection_id)
.await;
self.outgoing.send_result(request_id, result).await;
}
async fn thread_unsubscribe_response(
&self,
params: ThreadUnsubscribeParams,
connection_id: ConnectionId,
) -> Result<ThreadUnsubscribeResponse, JSONRPCErrorError> {
let thread_id = ThreadId::from_string(&params.thread_id)
.map_err(|err| invalid_request(format!("invalid thread id: {err}")))?;
if self.thread_manager.get_thread(thread_id).await.is_err() {
// Reconcile stale app-server bookkeeping when the thread has already been
// removed from the core manager. This keeps loaded-status/subscription state
// consistent with the source of truth before reporting NotLoaded.
self.finalize_thread_teardown(thread_id).await;
self.outgoing
.send_response(
request_id,
ThreadUnsubscribeResponse {
status: ThreadUnsubscribeStatus::NotLoaded,
},
)
.await;
return;
return Ok(ThreadUnsubscribeResponse {
status: ThreadUnsubscribeStatus::NotLoaded,
});
};
let was_subscribed = self
.thread_state_manager
.unsubscribe_connection_from_thread(thread_id, request_id.connection_id)
.unsubscribe_connection_from_thread(thread_id, connection_id)
.await;
let status = if was_subscribed {
@@ -6493,9 +6467,7 @@ impl CodexMessageProcessor {
} else {
ThreadUnsubscribeStatus::NotSubscribed
};
self.outgoing
.send_response(request_id, ThreadUnsubscribeResponse { status })
.await;
Ok(ThreadUnsubscribeResponse { status })
}
async fn prepare_thread_for_archive(&self, thread_id: ThreadId) {
@@ -6589,6 +6561,16 @@ impl CodexMessageProcessor {
config: Config,
environment_manager: Arc<EnvironmentManager>,
) {
let result = Self::apps_list_response(&outgoing, params, config, environment_manager).await;
outgoing.send_result(request_id, result).await;
}
async fn apps_list_response(
outgoing: &Arc<OutgoingMessageSender>,
params: AppsListParams,
config: Config,
environment_manager: Arc<EnvironmentManager>,
) -> Result<AppsListResponse, JSONRPCErrorError> {
let AppsListParams {
cursor,
limit,
@@ -6598,15 +6580,7 @@ impl CodexMessageProcessor {
let start = match cursor {
Some(cursor) => match cursor.parse::<usize>() {
Ok(idx) => idx,
Err(_) => {
let error = JSONRPCErrorError {
code: INVALID_REQUEST_ERROR_CODE,
message: format!("invalid cursor: {cursor}"),
data: None,
};
outgoing.send_error(request_id, error).await;
return;
}
Err(_) => return Err(invalid_request(format!("invalid cursor: {cursor}"))),
},
None => 0,
};
@@ -6660,7 +6634,7 @@ impl CodexMessageProcessor {
accessible_loaded,
all_loaded,
) {
apps_list_helpers::send_app_list_updated_notification(&outgoing, merged.clone())
apps_list_helpers::send_app_list_updated_notification(outgoing, merged.clone())
.await;
last_notified_apps = Some(merged);
}
@@ -6670,25 +6644,13 @@ impl CodexMessageProcessor {
let result = match tokio::time::timeout_at(app_list_deadline, rx.recv()).await {
Ok(Some(result)) => result,
Ok(None) => {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message: "failed to load app lists".to_string(),
data: None,
};
outgoing.send_error(request_id, error).await;
return;
return Err(internal_error("failed to load app lists"));
}
Err(_) => {
let timeout_seconds = APP_LIST_LOAD_TIMEOUT.as_secs();
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message: format!(
"timed out waiting for app lists after {timeout_seconds} seconds"
),
data: None,
};
outgoing.send_error(request_id, error).await;
return;
return Err(internal_error(format!(
"timed out waiting for app lists after {timeout_seconds} seconds"
)));
}
};
@@ -6698,26 +6660,14 @@ impl CodexMessageProcessor {
accessible_loaded = true;
}
AppListLoadResult::Accessible(Err(err)) => {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message: err,
data: None,
};
outgoing.send_error(request_id, error).await;
return;
return Err(internal_error(err));
}
AppListLoadResult::Directory(Ok(connectors)) => {
all_connectors = Some(connectors);
all_loaded = true;
}
AppListLoadResult::Directory(Err(err)) => {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message: err,
data: None,
};
outgoing.send_error(request_id, error).await;
return;
return Err(internal_error(err));
}
}
@@ -6747,27 +6697,26 @@ impl CodexMessageProcessor {
all_loaded,
) && last_notified_apps.as_ref() != Some(&merged)
{
apps_list_helpers::send_app_list_updated_notification(&outgoing, merged.clone())
apps_list_helpers::send_app_list_updated_notification(outgoing, merged.clone())
.await;
last_notified_apps = Some(merged.clone());
}
if accessible_loaded && all_loaded {
match apps_list_helpers::paginate_apps(merged.as_slice(), start, limit) {
Ok(response) => {
outgoing.send_response(request_id, response).await;
return;
}
Err(error) => {
outgoing.send_error(request_id, error).await;
return;
}
}
return apps_list_helpers::paginate_apps(merged.as_slice(), start, limit);
}
}
}
async fn skills_list(&self, request_id: ConnectionRequestId, params: SkillsListParams) {
let result = self.skills_list_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn skills_list_response(
&self,
params: SkillsListParams,
) -> Result<SkillsListResponse, JSONRPCErrorError> {
let SkillsListParams {
cwds,
force_reload,
@@ -6792,17 +6741,13 @@ impl CodexMessageProcessor {
let mut valid_extra_roots = Vec::new();
for root in entry.extra_user_roots {
let Ok(root) = AbsolutePathBuf::from_absolute_path_checked(root.as_path()) else {
self.send_invalid_request_error(
request_id,
format!(
let root =
AbsolutePathBuf::from_absolute_path_checked(root.as_path()).map_err(|_| {
invalid_request(format!(
"skills/list perCwdExtraUserRoots extraUserRoots paths must be absolute: {}",
root.display()
),
)
.await;
return;
};
))
})?;
valid_extra_roots.push(root);
}
extra_roots_by_cwd
@@ -6811,13 +6756,7 @@ impl CodexMessageProcessor {
.extend(valid_extra_roots);
}
let config = match self.load_latest_config(/*fallback_cwd*/ None).await {
Ok(config) => config,
Err(error) => {
self.outgoing.send_error(request_id, error).await;
return;
}
};
let config = self.load_latest_config(/*fallback_cwd*/ None).await?;
let auth = self.auth_manager.auth().await;
let workspace_codex_plugins_enabled = self
.workspace_codex_plugins_enabled(&config, auth.as_ref())
@@ -6896,9 +6835,7 @@ impl CodexMessageProcessor {
errors,
});
}
self.outgoing
.send_response(request_id, SkillsListResponse { data })
.await;
Ok(SkillsListResponse { data })
}
async fn marketplace_remove(
&self,
@@ -6911,27 +6848,16 @@ impl CodexMessageProcessor {
marketplace_name: params.marketplace_name,
},
)
.await;
match result {
Ok(outcome) => {
self.outgoing
.send_response(
request_id,
MarketplaceRemoveResponse {
marketplace_name: outcome.marketplace_name,
installed_root: outcome.removed_installed_root,
},
)
.await;
}
Err(MarketplaceRemoveError::InvalidRequest(message)) => {
self.send_invalid_request_error(request_id, message).await;
}
Err(MarketplaceRemoveError::Internal(message)) => {
self.send_internal_error(request_id, message).await;
}
}
.await
.map(|outcome| MarketplaceRemoveResponse {
marketplace_name: outcome.marketplace_name,
installed_root: outcome.removed_installed_root,
})
.map_err(|err| match err {
MarketplaceRemoveError::InvalidRequest(message) => invalid_request(message),
MarketplaceRemoveError::Internal(message) => internal_error(message),
});
self.outgoing.send_result(request_id, result).await;
}
async fn marketplace_upgrade(
@@ -6939,53 +6865,38 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: MarketplaceUpgradeParams,
) {
let config = match self.load_latest_config(/*fallback_cwd*/ None).await {
Ok(config) => config,
Err(err) => {
self.outgoing.send_error(request_id, err).await;
return;
}
};
let result = self.marketplace_upgrade_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn marketplace_upgrade_response(
&self,
params: MarketplaceUpgradeParams,
) -> Result<MarketplaceUpgradeResponse, JSONRPCErrorError> {
let config = self.load_latest_config(/*fallback_cwd*/ None).await?;
let plugins_manager = self.thread_manager.plugins_manager();
let MarketplaceUpgradeParams { marketplace_name } = params;
let result = tokio::task::spawn_blocking(move || {
let outcome = tokio::task::spawn_blocking(move || {
plugins_manager
.upgrade_configured_marketplaces_for_config(&config, marketplace_name.as_deref())
})
.await;
.await
.map_err(|err| internal_error(format!("failed to upgrade marketplaces: {err}")))?
.map_err(invalid_request)?;
match result {
Ok(Ok(outcome)) => {
self.outgoing
.send_response(
request_id,
MarketplaceUpgradeResponse {
selected_marketplaces: outcome.selected_marketplaces,
upgraded_roots: outcome.upgraded_roots,
errors: outcome
.errors
.into_iter()
.map(|err| MarketplaceUpgradeErrorInfo {
marketplace_name: err.marketplace_name,
message: err.message,
})
.collect(),
},
)
.await;
}
Ok(Err(message)) => {
self.send_invalid_request_error(request_id, message).await;
}
Err(err) => {
self.send_internal_error(
request_id,
format!("failed to upgrade marketplaces: {err}"),
)
.await;
}
}
Ok(MarketplaceUpgradeResponse {
selected_marketplaces: outcome.selected_marketplaces,
upgraded_roots: outcome.upgraded_roots,
errors: outcome
.errors
.into_iter()
.map(|err| MarketplaceUpgradeErrorInfo {
marketplace_name: err.marketplace_name,
message: err.message,
})
.collect(),
})
}
async fn marketplace_add(&self, request_id: ConnectionRequestId, params: MarketplaceAddParams) {
@@ -6997,28 +6908,17 @@ impl CodexMessageProcessor {
sparse_paths: params.sparse_paths.unwrap_or_default(),
},
)
.await;
match result {
Ok(outcome) => {
self.outgoing
.send_response(
request_id,
MarketplaceAddResponse {
marketplace_name: outcome.marketplace_name,
installed_root: outcome.installed_root,
already_added: outcome.already_added,
},
)
.await;
}
Err(MarketplaceAddError::InvalidRequest(message)) => {
self.send_invalid_request_error(request_id, message).await;
}
Err(MarketplaceAddError::Internal(message)) => {
self.send_internal_error(request_id, message).await;
}
}
.await
.map(|outcome| MarketplaceAddResponse {
marketplace_name: outcome.marketplace_name,
installed_root: outcome.installed_root,
already_added: outcome.already_added,
})
.map_err(|err| match err {
MarketplaceAddError::InvalidRequest(message) => invalid_request(message),
MarketplaceAddError::Internal(message) => internal_error(message),
});
self.outgoing.send_result(request_id, result).await;
}
async fn skills_config_write(
@@ -7026,6 +6926,14 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: SkillsConfigWriteParams,
) {
let result = self.skills_config_write_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn skills_config_write_response(
&self,
params: SkillsConfigWriteParams,
) -> Result<SkillsConfigWriteResponse, JSONRPCErrorError> {
let SkillsConfigWriteParams {
path,
name,
@@ -7040,43 +6948,24 @@ impl CodexMessageProcessor {
ConfigEdit::SetSkillConfigByName { name, enabled }
}
_ => {
let error = JSONRPCErrorError {
code: INVALID_PARAMS_ERROR_CODE,
message: "skills/config/write requires exactly one of path or name".to_string(),
data: None,
};
self.outgoing.send_error(request_id, error).await;
return;
return Err(invalid_params(
"skills/config/write requires exactly one of path or name",
));
}
};
let edits = vec![edit];
let result = ConfigEditsBuilder::new(&self.config.codex_home)
ConfigEditsBuilder::new(&self.config.codex_home)
.with_edits(edits)
.apply()
.await;
match result {
Ok(()) => {
.await
.map(|()| {
self.thread_manager.plugins_manager().clear_cache();
self.thread_manager.skills_manager().clear_cache();
self.outgoing
.send_response(
request_id,
SkillsConfigWriteResponse {
effective_enabled: enabled,
},
)
.await;
}
Err(err) => {
let error = JSONRPCErrorError {
code: INTERNAL_ERROR_CODE,
message: format!("failed to update skill settings: {err}"),
data: None,
};
self.outgoing.send_error(request_id, error).await;
}
}
SkillsConfigWriteResponse {
effective_enabled: enabled,
}
})
.map_err(|err| internal_error(format!("failed to update skill settings: {err}")))
}
async fn turn_start(
@@ -1,4 +1,6 @@
use super::*;
use crate::error_code::internal_error;
use crate::error_code::invalid_request;
use codex_app_server_protocol::PluginInstallPolicy;
impl CodexMessageProcessor {
@@ -7,46 +9,33 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: PluginListParams,
) {
let result = self.plugin_list_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn plugin_list_response(
&self,
params: PluginListParams,
) -> Result<PluginListResponse, JSONRPCErrorError> {
let plugins_manager = self.thread_manager.plugins_manager();
let PluginListParams { cwds } = params;
let roots = cwds.unwrap_or_default();
let config = match self.load_latest_config(/*fallback_cwd*/ None).await {
Ok(config) => config,
Err(err) => {
self.outgoing.send_error(request_id, err).await;
return;
}
let config = self.load_latest_config(/*fallback_cwd*/ None).await?;
let empty_response = || PluginListResponse {
marketplaces: Vec::new(),
marketplace_load_errors: Vec::new(),
featured_plugin_ids: Vec::new(),
};
if !config.features.enabled(Feature::Plugins) {
self.outgoing
.send_response(
request_id,
PluginListResponse {
marketplaces: Vec::new(),
marketplace_load_errors: Vec::new(),
featured_plugin_ids: Vec::new(),
},
)
.await;
return;
return Ok(empty_response());
}
let auth = self.auth_manager.auth().await;
if !self
.workspace_codex_plugins_enabled(&config, auth.as_ref())
.await
{
self.outgoing
.send_response(
request_id,
PluginListResponse {
marketplaces: Vec::new(),
marketplace_load_errors: Vec::new(),
featured_plugin_ids: Vec::new(),
},
)
.await;
return;
return Ok(empty_response());
}
plugins_manager.maybe_start_non_curated_plugin_cache_refresh(&roots);
@@ -100,18 +89,11 @@ impl CodexMessageProcessor {
.await
{
Ok(Ok(outcome)) => outcome,
Ok(Err(err)) => {
self.send_marketplace_error(request_id, err, "list marketplace plugins")
.await;
return;
}
Ok(Err(err)) => return Err(Self::marketplace_error(err, "list marketplace plugins")),
Err(err) => {
self.send_internal_error(
request_id,
format!("failed to list marketplace plugins: {err}"),
)
.await;
return;
return Err(internal_error(format!(
"failed to list marketplace plugins: {err}"
)));
}
};
@@ -174,16 +156,11 @@ impl CodexMessageProcessor {
Vec::new()
};
self.outgoing
.send_response(
request_id,
PluginListResponse {
marketplaces: data,
marketplace_load_errors,
featured_plugin_ids,
},
)
.await;
Ok(PluginListResponse {
marketplaces: data,
marketplace_load_errors,
featured_plugin_ids,
})
}
pub(super) async fn plugin_read(
@@ -191,6 +168,14 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: PluginReadParams,
) {
let result = self.plugin_read_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn plugin_read_response(
&self,
params: PluginReadParams,
) -> Result<PluginReadResponse, JSONRPCErrorError> {
let plugins_manager = self.thread_manager.plugins_manager();
let PluginReadParams {
marketplace_path,
@@ -201,30 +186,16 @@ impl CodexMessageProcessor {
(Some(marketplace_path), None) => Ok(marketplace_path),
(None, Some(remote_marketplace_name)) => Err(remote_marketplace_name),
(Some(_), Some(_)) | (None, None) => {
self.outgoing
.send_error(
request_id,
JSONRPCErrorError {
code: INVALID_REQUEST_ERROR_CODE,
message: "plugin/read requires exactly one of marketplacePath or remoteMarketplaceName".to_string(),
data: None,
},
)
.await;
return;
return Err(invalid_request(
"plugin/read requires exactly one of marketplacePath or remoteMarketplaceName",
));
}
};
let config_cwd = read_source.as_ref().ok().and_then(|marketplace_path| {
marketplace_path.as_path().parent().map(Path::to_path_buf)
});
let config = match self.load_latest_config(config_cwd).await {
Ok(config) => config,
Err(err) => {
self.outgoing.send_error(request_id, err).await;
return;
}
};
let config = self.load_latest_config(config_cwd).await?;
let plugin = match read_source {
Ok(marketplace_path) => {
@@ -232,17 +203,10 @@ impl CodexMessageProcessor {
plugin_name,
marketplace_path,
};
let outcome = match plugins_manager
let outcome = plugins_manager
.read_plugin_for_config(&config, &request)
.await
{
Ok(outcome) => outcome,
Err(err) => {
self.send_marketplace_error(request_id, err, "read plugin details")
.await;
return;
}
};
.map_err(|err| Self::marketplace_error(err, "read plugin details"))?;
let environment_manager = self.thread_manager.environment_manager();
let app_summaries = plugin_app_helpers::load_plugin_app_summaries(
&config,
@@ -287,19 +251,9 @@ impl CodexMessageProcessor {
if !config.features.enabled(Feature::Plugins)
|| !config.features.enabled(Feature::RemotePlugin)
{
self.outgoing
.send_error(
request_id,
JSONRPCErrorError {
code: INVALID_REQUEST_ERROR_CODE,
message: format!(
"remote plugin read is not enabled for marketplace {remote_marketplace_name}"
),
data: None,
},
)
.await;
return;
return Err(invalid_request(format!(
"remote plugin read is not enabled for marketplace {remote_marketplace_name}"
)));
}
let auth = self.auth_manager.auth().await;
let remote_plugin_service_config = RemotePluginServiceConfig {
@@ -310,36 +264,20 @@ impl CodexMessageProcessor {
.chars()
.all(|ch| ch.is_ascii_alphanumeric() || ch == '-' || ch == '_' || ch == '~')
{
self.send_invalid_request_error(
request_id,
"invalid remote plugin id: only ASCII letters, digits, `_`, `-`, and `~` are allowed"
.to_string(),
)
.await;
return;
return Err(invalid_request(
"invalid remote plugin id: only ASCII letters, digits, `_`, `-`, and `~` are allowed",
));
}
let remote_detail = match codex_core_plugins::remote::fetch_remote_plugin_detail(
let remote_detail = codex_core_plugins::remote::fetch_remote_plugin_detail(
&remote_plugin_service_config,
auth.as_ref(),
&remote_marketplace_name,
&plugin_name,
)
.await
{
Ok(remote_detail) => remote_detail,
Err(err) => {
self.outgoing
.send_error(
request_id,
remote_plugin_catalog_error_to_jsonrpc(
err,
"read remote plugin details",
),
)
.await;
return;
}
};
.map_err(|err| {
remote_plugin_catalog_error_to_jsonrpc(err, "read remote plugin details")
})?;
let plugin_apps = remote_detail
.app_ids
.iter()
@@ -357,9 +295,7 @@ impl CodexMessageProcessor {
}
};
self.outgoing
.send_response(request_id, PluginReadResponse { plugin })
.await;
Ok(PluginReadResponse { plugin })
}
pub(super) async fn plugin_install(
@@ -367,6 +303,14 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: PluginInstallParams,
) {
let result = self.plugin_install_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn plugin_install_response(
&self,
params: PluginInstallParams,
) -> Result<PluginInstallResponse, JSONRPCErrorError> {
let PluginInstallParams {
marketplace_path,
remote_marketplace_name,
@@ -375,44 +319,27 @@ impl CodexMessageProcessor {
let marketplace_path = match (marketplace_path, remote_marketplace_name) {
(Some(marketplace_path), None) => marketplace_path,
(None, Some(remote_marketplace_name)) => {
self.remote_plugin_install(request_id, remote_marketplace_name, plugin_name)
return self
.remote_plugin_install_response(remote_marketplace_name, plugin_name)
.await;
return;
}
(Some(_), Some(_)) | (None, None) => {
self.outgoing
.send_error(
request_id,
JSONRPCErrorError {
code: INVALID_REQUEST_ERROR_CODE,
message: "plugin/install requires exactly one of marketplacePath or remoteMarketplaceName".to_string(),
data: None,
},
)
.await;
return;
return Err(invalid_request(
"plugin/install requires exactly one of marketplacePath or remoteMarketplaceName",
));
}
};
let config_cwd = marketplace_path.as_path().parent().map(Path::to_path_buf);
let config = match self.load_latest_config(config_cwd.clone()).await {
Ok(config) => config,
Err(err) => {
self.outgoing.send_error(request_id, err).await;
return;
}
};
let config = self.load_latest_config(config_cwd.clone()).await?;
let auth = self.auth_manager.auth().await;
if !self
.workspace_codex_plugins_enabled(&config, auth.as_ref())
.await
{
self.send_invalid_request_error(
request_id,
"Codex plugins are disabled for this workspace".to_string(),
)
.await;
return;
return Err(invalid_request(
"Codex plugins are disabled for this workspace",
));
}
let plugins_manager = self.thread_manager.plugins_manager();
@@ -421,197 +348,103 @@ impl CodexMessageProcessor {
marketplace_path,
};
let install_result = plugins_manager.install_plugin(request).await;
match install_result {
Ok(result) => {
let config = match self.load_latest_config(config_cwd).await {
Ok(config) => config,
Err(err) => {
warn!(
"failed to reload config after plugin install, using current config: {err:?}"
);
config
}
};
self.clear_plugin_related_caches();
let plugin_mcp_servers =
load_plugin_mcp_servers(result.installed_path.as_path()).await;
if !plugin_mcp_servers.is_empty() {
if let Err(err) = self.queue_mcp_server_refresh_for_config(&config).await {
warn!(
plugin = result.plugin_id.as_key(),
"failed to queue MCP refresh after plugin install: {err:?}"
);
}
self.start_plugin_mcp_oauth_logins(&config, plugin_mcp_servers)
.await;
}
let plugin_apps = load_plugin_apps(result.installed_path.as_path()).await;
let auth = self.auth_manager.auth().await;
let apps_needing_auth = self
.plugin_apps_needing_auth_for_install(
&config,
auth.as_ref().is_some_and(CodexAuth::is_chatgpt_auth),
&result.plugin_id.as_key(),
&plugin_apps,
)
.await;
self.outgoing
.send_response(
request_id,
PluginInstallResponse {
auth_policy: result.auth_policy.into(),
apps_needing_auth,
},
)
.await;
}
Err(err) => {
if err.is_invalid_request() {
self.send_invalid_request_error(request_id, err.to_string())
.await;
return;
}
match err {
CorePluginInstallError::Marketplace(err) => {
self.send_marketplace_error(request_id, err, "install plugin")
.await;
}
CorePluginInstallError::Config(err) => {
self.send_internal_error(
request_id,
format!("failed to persist installed plugin config: {err}"),
)
.await;
}
CorePluginInstallError::Remote(err) => {
self.send_internal_error(
request_id,
format!("failed to enable remote plugin: {err}"),
)
.await;
}
CorePluginInstallError::Join(err) => {
self.send_internal_error(
request_id,
format!("failed to install plugin: {err}"),
)
.await;
}
CorePluginInstallError::Store(err) => {
self.send_internal_error(
request_id,
format!("failed to install plugin: {err}"),
)
.await;
}
}
}
}
}
async fn remote_plugin_install(
&self,
request_id: ConnectionRequestId,
remote_marketplace_name: String,
plugin_name: String,
) {
let config = match self.load_latest_config(/*fallback_cwd*/ None).await {
let result = plugins_manager
.install_plugin(request)
.await
.map_err(Self::plugin_install_error)?;
let config = match self.load_latest_config(config_cwd).await {
Ok(config) => config,
Err(err) => {
self.outgoing.send_error(request_id, err).await;
return;
warn!(
"failed to reload config after plugin install, using current config: {err:?}"
);
config
}
};
self.clear_plugin_related_caches();
let plugin_mcp_servers = load_plugin_mcp_servers(result.installed_path.as_path()).await;
if !plugin_mcp_servers.is_empty() {
if let Err(err) = self.queue_mcp_server_refresh_for_config(&config).await {
warn!(
plugin = result.plugin_id.as_key(),
"failed to queue MCP refresh after plugin install: {err:?}"
);
}
self.start_plugin_mcp_oauth_logins(&config, plugin_mcp_servers)
.await;
}
let plugin_apps = load_plugin_apps(result.installed_path.as_path()).await;
let auth = self.auth_manager.auth().await;
let apps_needing_auth = self
.plugin_apps_needing_auth_for_install(
&config,
auth.as_ref().is_some_and(CodexAuth::is_chatgpt_auth),
&result.plugin_id.as_key(),
&plugin_apps,
)
.await;
Ok(PluginInstallResponse {
auth_policy: result.auth_policy.into(),
apps_needing_auth,
})
}
async fn remote_plugin_install_response(
&self,
remote_marketplace_name: String,
plugin_name: String,
) -> Result<PluginInstallResponse, JSONRPCErrorError> {
let config = self.load_latest_config(/*fallback_cwd*/ None).await?;
if !config.features.enabled(Feature::Plugins)
|| !config.features.enabled(Feature::RemotePlugin)
{
self.outgoing
.send_error(
request_id,
JSONRPCErrorError {
code: INVALID_REQUEST_ERROR_CODE,
message: format!(
"remote plugin install is not enabled for marketplace {remote_marketplace_name}"
),
data: None,
},
)
.await;
return;
return Err(invalid_request(format!(
"remote plugin install is not enabled for marketplace {remote_marketplace_name}"
)));
}
if plugin_name.is_empty()
|| !plugin_name
.chars()
.all(|ch| ch.is_ascii_alphanumeric() || ch == '-' || ch == '_' || ch == '~')
{
self.send_invalid_request_error(
request_id,
"invalid remote plugin id: only ASCII letters, digits, `_`, `-`, and `~` are allowed"
.to_string(),
)
.await;
return;
return Err(invalid_request(
"invalid remote plugin id: only ASCII letters, digits, `_`, `-`, and `~` are allowed",
));
}
let auth = self.auth_manager.auth().await;
let remote_plugin_service_config = RemotePluginServiceConfig {
chatgpt_base_url: config.chatgpt_base_url.clone(),
};
let remote_detail = match codex_core_plugins::remote::fetch_remote_plugin_detail(
let remote_detail = codex_core_plugins::remote::fetch_remote_plugin_detail(
&remote_plugin_service_config,
auth.as_ref(),
&remote_marketplace_name,
&plugin_name,
)
.await
{
Ok(remote_detail) => remote_detail,
Err(err) => {
self.outgoing
.send_error(
request_id,
remote_plugin_catalog_error_to_jsonrpc(
err,
"read remote plugin details before install",
),
)
.await;
return;
}
};
.map_err(|err| {
remote_plugin_catalog_error_to_jsonrpc(err, "read remote plugin details before install")
})?;
if remote_detail.summary.install_policy == PluginInstallPolicy::NotAvailable {
self.send_invalid_request_error(
request_id,
format!("remote plugin {plugin_name} is not available for install"),
)
.await;
return;
return Err(invalid_request(format!(
"remote plugin {plugin_name} is not available for install"
)));
}
if let Err(err) = codex_core_plugins::remote::install_remote_plugin(
codex_core_plugins::remote::install_remote_plugin(
&remote_plugin_service_config,
auth.as_ref(),
&remote_marketplace_name,
&plugin_name,
)
.await
{
self.outgoing
.send_error(
request_id,
remote_plugin_catalog_error_to_jsonrpc(err, "install remote plugin"),
)
.await;
return;
}
.map_err(|err| remote_plugin_catalog_error_to_jsonrpc(err, "install remote plugin"))?;
self.clear_plugin_related_caches();
@@ -629,15 +462,10 @@ impl CodexMessageProcessor {
)
.await;
self.outgoing
.send_response(
request_id,
PluginInstallResponse {
auth_policy: remote_detail.summary.auth_policy,
apps_needing_auth,
},
)
.await;
Ok(PluginInstallResponse {
auth_policy: remote_detail.summary.auth_policy,
apps_needing_auth,
})
}
async fn plugin_apps_needing_auth_for_install(
@@ -709,59 +537,82 @@ impl CodexMessageProcessor {
request_id: ConnectionRequestId,
params: PluginUninstallParams,
) {
let result = self.plugin_uninstall_response(params).await;
self.outgoing.send_result(request_id, result).await;
}
async fn plugin_uninstall_response(
&self,
params: PluginUninstallParams,
) -> Result<PluginUninstallResponse, JSONRPCErrorError> {
let PluginUninstallParams { plugin_id } = params;
let plugins_manager = self.thread_manager.plugins_manager();
let uninstall_result = plugins_manager.uninstall_plugin(plugin_id).await;
plugins_manager
.uninstall_plugin(plugin_id)
.await
.map_err(Self::plugin_uninstall_error)?;
self.clear_plugin_related_caches();
Ok(PluginUninstallResponse {})
}
match uninstall_result {
Ok(()) => {
self.clear_plugin_related_caches();
self.outgoing
.send_response(request_id, PluginUninstallResponse {})
.await;
}
Err(err) => {
if err.is_invalid_request() {
self.send_invalid_request_error(request_id, err.to_string())
.await;
return;
}
fn plugin_install_error(err: CorePluginInstallError) -> JSONRPCErrorError {
if err.is_invalid_request() {
return invalid_request(err.to_string());
}
match err {
CorePluginUninstallError::Config(err) => {
self.send_internal_error(
request_id,
format!("failed to clear plugin config: {err}"),
)
.await;
}
CorePluginUninstallError::Remote(err) => {
self.send_internal_error(
request_id,
format!("failed to uninstall remote plugin: {err}"),
)
.await;
}
CorePluginUninstallError::Join(err) => {
self.send_internal_error(
request_id,
format!("failed to uninstall plugin: {err}"),
)
.await;
}
CorePluginUninstallError::Store(err) => {
self.send_internal_error(
request_id,
format!("failed to uninstall plugin: {err}"),
)
.await;
}
CorePluginUninstallError::InvalidPluginId(_) => {
unreachable!("invalid plugin ids are handled above");
}
}
match err {
CorePluginInstallError::Marketplace(err) => {
Self::marketplace_error(err, "install plugin")
}
CorePluginInstallError::Config(err) => {
internal_error(format!("failed to persist installed plugin config: {err}"))
}
CorePluginInstallError::Remote(err) => {
internal_error(format!("failed to enable remote plugin: {err}"))
}
CorePluginInstallError::Join(err) => {
internal_error(format!("failed to install plugin: {err}"))
}
CorePluginInstallError::Store(err) => {
internal_error(format!("failed to install plugin: {err}"))
}
}
}
fn plugin_uninstall_error(err: CorePluginUninstallError) -> JSONRPCErrorError {
if err.is_invalid_request() {
return invalid_request(err.to_string());
}
match err {
CorePluginUninstallError::Config(err) => {
internal_error(format!("failed to clear plugin config: {err}"))
}
CorePluginUninstallError::Remote(err) => {
internal_error(format!("failed to uninstall remote plugin: {err}"))
}
CorePluginUninstallError::Join(err) => {
internal_error(format!("failed to uninstall plugin: {err}"))
}
CorePluginUninstallError::Store(err) => {
internal_error(format!("failed to uninstall plugin: {err}"))
}
CorePluginUninstallError::InvalidPluginId(_) => {
unreachable!("invalid plugin ids are handled above");
}
}
}
fn marketplace_error(err: MarketplaceError, action: &str) -> JSONRPCErrorError {
match err {
MarketplaceError::MarketplaceNotFound { .. }
| MarketplaceError::InvalidMarketplaceFile { .. }
| MarketplaceError::PluginNotFound { .. }
| MarketplaceError::PluginNotAvailable { .. }
| MarketplaceError::PluginsDisabled
| MarketplaceError::InvalidPlugin(_) => invalid_request(err.to_string()),
MarketplaceError::Io { .. } => internal_error(format!("failed to {action}: {err}")),
}
}
}
+2 -1
View File
@@ -391,7 +391,8 @@ fn start_uninitialized(args: InProcessStartArgs) -> InProcessClientHandle {
let processor_outgoing = Arc::clone(&outgoing_message_sender);
let auth_manager =
AuthManager::shared_from_config(args.config.as_ref(), args.enable_codex_api_key_env);
AuthManager::shared_from_config(args.config.as_ref(), args.enable_codex_api_key_env)
.await;
let config_manager = ConfigManager::new(
args.config.codex_home.to_path_buf(),
args.cli_overrides,
+3 -3
View File
@@ -469,7 +469,7 @@ pub async fn run_main_with_transport_options(
config_manager
.replace_thread_config_loader(Arc::clone(&discovered_thread_config_loader));
let auth_manager =
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false).await;
config_manager.replace_cloud_requirements_loader(auth_manager, config.chatgpt_base_url);
}
Err(err) => {
@@ -631,7 +631,7 @@ pub async fn run_main_with_transport_options(
}
let auth_manager =
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false).await;
let remote_control_enabled = config.features.enabled(Feature::RemoteControl);
if transport_accept_handles.is_empty() && !remote_control_enabled {
@@ -712,7 +712,7 @@ pub async fn run_main_with_transport_options(
let outgoing_message_sender = Arc::new(OutgoingMessageSender::new(outgoing_tx));
let outbound_control_tx = outbound_control_tx;
let auth_manager =
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false).await;
let processor = Arc::new(MessageProcessor::new(MessageProcessorArgs {
outgoing: outgoing_message_sender,
arg0_paths,
@@ -127,7 +127,7 @@ impl TracingHarness {
let server = create_mock_responses_server_repeating_assistant("Done").await;
let codex_home = TempDir::new()?;
let config = Arc::new(build_test_config(codex_home.path(), &server.uri()).await?);
let (processor, outgoing_rx) = build_test_processor(config);
let (processor, outgoing_rx) = build_test_processor(config).await;
let tracing = init_test_tracing();
tracing.exporter.reset();
tracing::callsite::rebuild_interest_cache();
@@ -257,7 +257,7 @@ async fn build_test_config(codex_home: &Path, server_uri: &str) -> Result<Config
.await?)
}
fn build_test_processor(
async fn build_test_processor(
config: Arc<Config>,
) -> (
Arc<MessageProcessor>,
@@ -266,7 +266,7 @@ fn build_test_processor(
let (outgoing_tx, outgoing_rx) = mpsc::channel(16);
let outgoing = Arc::new(OutgoingMessageSender::new(outgoing_tx));
let auth_manager =
AuthManager::shared_from_config(config.as_ref(), /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config.as_ref(), /*enable_codex_api_key_env*/ false).await;
let config_manager = ConfigManager::new(
config.codex_home.to_path_buf(),
Vec::new(),
@@ -497,7 +497,8 @@ async fn remote_control_start_allows_missing_auth_when_enabled() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
let (transport_event_tx, _transport_event_rx) =
mpsc::channel::<TransportEvent>(CHANNEL_CAPACITY);
let shutdown_token = CancellationToken::new();
@@ -1085,7 +1086,8 @@ async fn remote_control_waits_for_account_id_before_enrolling() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
let expected_server_name = gethostname().to_string_lossy().trim().to_string();
let expected_enrollment = RemoteControlEnrollment {
account_id: "account_id".to_string(),
@@ -706,7 +706,7 @@ pub(crate) async fn load_remote_control_auth(
"remote control requires ChatGPT authentication",
));
}
auth_manager.reload();
auth_manager.reload().await;
reloaded = true;
continue;
};
@@ -714,7 +714,7 @@ pub(crate) async fn load_remote_control_auth(
break auth;
}
if auth.get_account_id().is_none() && !reloaded {
auth_manager.reload();
auth_manager.reload().await;
reloaded = true;
continue;
}
@@ -1090,7 +1090,8 @@ mod tests {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
let mut auth_recovery = auth_manager.unauthorized_recovery();
let mut enrollment = Some(RemoteControlEnrollment {
account_id: "account_id".to_string(),
@@ -1172,7 +1173,8 @@ mod tests {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
let mut auth_recovery = auth_manager.unauthorized_recovery();
let mut enrollment = None;
save_auth(
+1 -1
View File
@@ -21,7 +21,7 @@ pub(crate) async fn chatgpt_get_request_with_timeout<T: DeserializeOwned>(
) -> anyhow::Result<T> {
let chatgpt_base_url = &config.chatgpt_base_url;
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
let auth = auth_manager
.auth()
.await
+2 -2
View File
@@ -26,7 +26,7 @@ const DIRECTORY_CONNECTORS_TIMEOUT: Duration = Duration::from_secs(60);
async fn apps_enabled(config: &Config) -> bool {
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
let auth = auth_manager.auth().await;
config
.features
@@ -35,7 +35,7 @@ async fn apps_enabled(config: &Config) -> bool {
async fn connector_auth(config: &Config) -> anyhow::Result<CodexAuth> {
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
let auth = auth_manager
.auth()
.await
+3 -1
View File
@@ -362,7 +362,9 @@ pub async fn run_login_with_device_code_fallback_to_browser(
pub async fn run_login_status(cli_config_overrides: CliConfigOverrides) -> ! {
let config = load_config_or_exit(cli_config_overrides).await;
match CodexAuth::from_auth_storage(&config.codex_home, config.cli_auth_credentials_store_mode) {
match CodexAuth::from_auth_storage(&config.codex_home, config.cli_auth_credentials_store_mode)
.await
{
Ok(Some(auth)) => match auth.auth_mode() {
AuthMode::ApiKey => match auth.get_token() {
Ok(api_key) => {
+1 -1
View File
@@ -1384,7 +1384,7 @@ async fn run_debug_models_command(
.map_err(anyhow::Error::msg)?;
let config = Config::load_with_cli_overrides(cli_overrides).await?;
let auth_manager =
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ true);
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ true).await;
let models_manager =
build_models_manager(&config, auth_manager, CollaborationModesConfig::default());
models_manager
+92 -67
View File
@@ -727,7 +727,7 @@ pub fn cloud_requirements_loader(
})
}
pub fn cloud_requirements_loader_for_storage(
pub async fn cloud_requirements_loader_for_storage(
codex_home: PathBuf,
enable_codex_api_key_env: bool,
credentials_store_mode: AuthCredentialsStoreMode,
@@ -738,7 +738,8 @@ pub fn cloud_requirements_loader_for_storage(
enable_codex_api_key_env,
credentials_store_mode,
Some(chatgpt_base_url.clone()),
);
)
.await;
cloud_requirements_loader(auth_manager, chatgpt_base_url, codex_home)
}
@@ -853,7 +854,7 @@ mod tests {
Ok(())
}
fn auth_manager_with_api_key() -> Arc<AuthManager> {
async fn auth_manager_with_api_key() -> Arc<AuthManager> {
let tmp = tempdir().expect("tempdir");
let auth_json = json!({
"OPENAI_API_KEY": "sk-test-key",
@@ -861,15 +862,18 @@ mod tests {
"last_refresh": null,
});
write_auth_json(tmp.path(), auth_json).expect("write auth");
Arc::new(AuthManager::new(
tmp.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
))
Arc::new(
AuthManager::new(
tmp.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
)
}
fn auth_manager_with_plan_and_identity(
async fn auth_manager_with_plan_and_identity(
plan_type: &str,
chatgpt_user_id: Option<&str>,
account_id: Option<&str>,
@@ -886,12 +890,15 @@ mod tests {
),
)
.expect("write auth");
Arc::new(AuthManager::new(
tmp.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
))
Arc::new(
AuthManager::new(
tmp.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
)
}
fn chatgpt_auth_json(
@@ -975,7 +982,7 @@ mod tests {
manager: Arc<AuthManager>,
}
fn managed_auth_context(
async fn managed_auth_context(
plan_type: &str,
chatgpt_user_id: Option<&str>,
account_id: Option<&str>,
@@ -995,18 +1002,22 @@ mod tests {
)
.expect("write auth");
ManagedAuthContext {
manager: Arc::new(AuthManager::new(
home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)),
manager: Arc::new(
AuthManager::new(
home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
),
_home: home,
}
}
fn auth_manager_with_plan(plan_type: &str) -> Arc<AuthManager> {
async fn auth_manager_with_plan(plan_type: &str) -> Arc<AuthManager> {
auth_manager_with_plan_and_identity(plan_type, Some("user-12345"), Some("account-12345"))
.await
}
fn parse_for_fetch(contents: Option<&str>) -> Option<ConfigRequirementsToml> {
@@ -1118,7 +1129,7 @@ mod tests {
#[tokio::test]
async fn fetch_cloud_requirements_skips_non_chatgpt_auth() {
let auth_manager = auth_manager_with_api_key();
let auth_manager = auth_manager_with_api_key().await;
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager,
@@ -1134,7 +1145,7 @@ mod tests {
async fn fetch_cloud_requirements_skips_non_business_or_enterprise_plan() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("pro"),
auth_manager_with_plan("pro").await,
Arc::new(StaticFetcher { contents: None }),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1147,7 +1158,7 @@ mod tests {
async fn fetch_cloud_requirements_skips_team_like_usage_based_plan() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("self_serve_business_usage_based"),
auth_manager_with_plan("self_serve_business_usage_based").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1161,7 +1172,7 @@ mod tests {
async fn fetch_cloud_requirements_allows_business_plan() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1193,7 +1204,7 @@ mod tests {
async fn fetch_cloud_requirements_allows_business_like_usage_based_plan() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("enterprise_cbp_usage_based"),
auth_manager_with_plan("enterprise_cbp_usage_based").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1225,7 +1236,7 @@ mod tests {
async fn fetch_cloud_requirements_allows_hc_plan_as_enterprise() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("hc"),
auth_manager_with_plan("hc").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1329,7 +1340,7 @@ enabled = false
#[tokio::test(start_paused = true)]
async fn fetch_cloud_requirements_times_out() {
let auth_manager = auth_manager_with_plan("enterprise");
let auth_manager = auth_manager_with_plan("enterprise").await;
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager,
@@ -1356,7 +1367,7 @@ enabled = false
]));
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1405,12 +1416,15 @@ enabled = false
),
)
.expect("write initial auth");
let auth_manager = Arc::new(AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
));
let auth_manager = Arc::new(
AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
);
write_auth_json(
auth_home.path(),
@@ -1479,12 +1493,15 @@ enabled = false
),
)
.expect("write initial auth");
let auth_manager = Arc::new(AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
));
let auth_manager = Arc::new(
AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
);
write_auth_json(
auth_home.path(),
@@ -1559,7 +1576,8 @@ enabled = false
Some("account-12345"),
"stale-access-token",
"test-refresh-token",
);
)
.await;
write_auth_json(
auth._home.path(),
chatgpt_auth_json(
@@ -1611,12 +1629,15 @@ enabled = false
),
)
.expect("write auth");
let auth_manager = Arc::new(AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
));
let auth_manager = Arc::new(
AuthManager::new(
auth_home.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
)
.await,
);
let fetcher = Arc::new(UnauthorizedFetcher {
message:
@@ -1653,7 +1674,7 @@ enabled = false
]));
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1678,7 +1699,7 @@ enabled = false
))]));
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
fetcher,
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1700,7 +1721,7 @@ enabled = false
async fn fetch_cloud_requirements_uses_cache_when_valid() {
let codex_home = tempdir().expect("tempdir");
let prime_service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1711,7 +1732,7 @@ enabled = false
let fetcher = Arc::new(SequenceFetcher::new(vec![Err(request_error())]));
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1747,7 +1768,8 @@ enabled = false
"business",
/*chatgpt_user_id*/ None,
Some("account-12345"),
),
)
.await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1790,7 +1812,7 @@ enabled = false
async fn fetch_cloud_requirements_does_not_use_cache_when_auth_identity_is_incomplete() {
let codex_home = tempdir().expect("tempdir");
let prime_service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1807,7 +1829,8 @@ enabled = false
"business",
/*chatgpt_user_id*/ None,
Some("account-12345"),
),
)
.await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1843,7 +1866,8 @@ enabled = false
"business",
Some("user-12345"),
Some("account-12345"),
),
)
.await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1860,7 +1884,8 @@ enabled = false
"business",
Some("user-99999"),
Some("account-12345"),
),
)
.await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1892,7 +1917,7 @@ enabled = false
async fn fetch_cloud_requirements_ignores_tampered_cache() {
let codex_home = tempdir().expect("tempdir");
let prime_service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -1917,7 +1942,7 @@ enabled = false
"allowed_approval_policies = [\"never\"]".to_string(),
))]));
let service = CloudRequirementsService::new(
auth_manager_with_plan("enterprise"),
auth_manager_with_plan("enterprise").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -1975,7 +2000,7 @@ enabled = false
"allowed_approval_policies = [\"never\"]".to_string(),
))]));
let service = CloudRequirementsService::new(
auth_manager_with_plan("enterprise"),
auth_manager_with_plan("enterprise").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -2007,7 +2032,7 @@ enabled = false
async fn fetch_cloud_requirements_writes_signed_cache() {
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
Arc::new(StaticFetcher {
contents: Some("allowed_approval_policies = [\"never\"]".to_string()),
}),
@@ -2070,7 +2095,7 @@ enabled = false
let fetcher = Arc::new(SequenceFetcher::new(vec![Ok(None), Err(request_error())]));
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("enterprise"),
auth_manager_with_plan("enterprise").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -2088,7 +2113,7 @@ enabled = false
]));
let codex_home = tempdir().expect("tempdir");
let service = CloudRequirementsService::new(
auth_manager_with_plan("enterprise"),
auth_manager_with_plan("enterprise").await,
fetcher.clone(),
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
@@ -2121,7 +2146,7 @@ enabled = false
)),
]));
let service = CloudRequirementsService::new(
auth_manager_with_plan("business"),
auth_manager_with_plan("business").await,
fetcher,
codex_home.path().to_path_buf(),
CLOUD_REQUIREMENTS_TIMEOUT,
+9 -6
View File
@@ -44,12 +44,15 @@ pub fn normalize_base_url(input: &str) -> String {
pub async fn load_auth_manager(chatgpt_base_url: Option<String>) -> Option<AuthManager> {
// TODO: pass in cli overrides once cloud tasks properly support them.
let config = Config::load_with_cli_overrides(Vec::new()).await.ok()?;
Some(AuthManager::new(
config.codex_home.to_path_buf(),
/*enable_codex_api_key_env*/ false,
config.cli_auth_credentials_store_mode,
chatgpt_base_url.or(Some(config.chatgpt_base_url)),
))
Some(
AuthManager::new(
config.codex_home.to_path_buf(),
/*enable_codex_api_key_env*/ false,
config.cli_auth_credentials_store_mode,
chatgpt_base_url.or(Some(config.chatgpt_base_url)),
)
.await,
)
}
/// Build headers for ChatGPT-backed requests: `User-Agent`, optional `Authorization`,
+5 -12
View File
@@ -8,18 +8,11 @@ struct ConfigKeyAlias {
canonical_key: &'static str,
}
const CONFIG_KEY_ALIASES: &[ConfigKeyAlias] = &[
ConfigKeyAlias {
table_path: &["memories"],
legacy_key: "no_memories_if_mcp_or_web_search",
canonical_key: "disable_on_external_context",
},
ConfigKeyAlias {
table_path: &["agents"],
legacy_key: "max_concurrent_threads_per_session",
canonical_key: "max_threads",
},
];
const CONFIG_KEY_ALIASES: &[ConfigKeyAlias] = &[ConfigKeyAlias {
table_path: &["memories"],
legacy_key: "no_memories_if_mcp_or_web_search",
canonical_key: "disable_on_external_context",
}];
pub(crate) fn normalize_key_aliases(path: &[String], table: &mut TomlMap<String, TomlValue>) {
for alias in CONFIG_KEY_ALIASES {
+2
View File
@@ -39,6 +39,8 @@ codex-exec-server = { workspace = true }
codex-features = { workspace = true }
codex-feedback = { workspace = true }
codex-login = { workspace = true }
codex-memories-read = { workspace = true }
codex-memories-write = { workspace = true }
codex-mcp = { workspace = true }
codex-model-provider-info = { workspace = true }
codex-models-manager = { workspace = true }
+5
View File
@@ -1310,6 +1310,11 @@
"hide_spawn_agent_metadata": {
"type": "boolean"
},
"max_concurrent_threads_per_session": {
"format": "uint",
"minimum": 1.0,
"type": "integer"
},
"usage_hint_enabled": {
"type": "boolean"
},
+80
View File
@@ -7348,6 +7348,7 @@ async fn multi_agent_v2_config_from_feature_table() -> std::io::Result<()> {
codex_home.path().join(CONFIG_TOML_FILE),
r#"[features.multi_agent_v2]
enabled = true
max_concurrent_threads_per_session = 5
usage_hint_enabled = false
usage_hint_text = "Custom delegation guidance."
hide_spawn_agent_metadata = true
@@ -7361,6 +7362,8 @@ hide_spawn_agent_metadata = true
.await?;
assert!(config.features.enabled(Feature::MultiAgentV2));
assert_eq!(config.multi_agent_v2.max_concurrent_threads_per_session, 5);
assert_eq!(config.agent_max_threads, Some(4));
assert!(!config.multi_agent_v2.usage_hint_enabled);
assert_eq!(
config.multi_agent_v2.usage_hint_text.as_deref(),
@@ -7379,11 +7382,13 @@ async fn profile_multi_agent_v2_config_overrides_base() -> std::io::Result<()> {
r#"profile = "no_hint"
[features.multi_agent_v2]
max_concurrent_threads_per_session = 4
usage_hint_enabled = true
usage_hint_text = "base hint"
hide_spawn_agent_metadata = true
[profiles.no_hint.features.multi_agent_v2]
max_concurrent_threads_per_session = 6
usage_hint_enabled = false
usage_hint_text = "profile hint"
hide_spawn_agent_metadata = false
@@ -7396,6 +7401,7 @@ hide_spawn_agent_metadata = false
.build()
.await?;
assert_eq!(config.multi_agent_v2.max_concurrent_threads_per_session, 6);
assert!(!config.multi_agent_v2.usage_hint_enabled);
assert_eq!(
config.multi_agent_v2.usage_hint_text.as_deref(),
@@ -7406,6 +7412,80 @@ hide_spawn_agent_metadata = false
Ok(())
}
#[tokio::test]
async fn multi_agent_v2_default_session_thread_cap_counts_root() -> std::io::Result<()> {
let codex_home = TempDir::new()?;
std::fs::write(
codex_home.path().join(CONFIG_TOML_FILE),
r#"[features.multi_agent_v2]
enabled = true
"#,
)?;
let config = ConfigBuilder::without_managed_config_for_tests()
.codex_home(codex_home.path().to_path_buf())
.fallback_cwd(Some(codex_home.path().to_path_buf()))
.build()
.await?;
assert_eq!(config.multi_agent_v2.max_concurrent_threads_per_session, 4);
assert_eq!(config.agent_max_threads, Some(3));
Ok(())
}
#[tokio::test]
async fn multi_agent_v2_rejects_agents_max_threads() -> std::io::Result<()> {
let codex_home = TempDir::new()?;
std::fs::write(
codex_home.path().join(CONFIG_TOML_FILE),
r#"[features.multi_agent_v2]
enabled = true
[agents]
max_threads = 3
"#,
)?;
let err = ConfigBuilder::without_managed_config_for_tests()
.codex_home(codex_home.path().to_path_buf())
.fallback_cwd(Some(codex_home.path().to_path_buf()))
.build()
.await
.expect_err("agents.max_threads should conflict with multi_agent_v2");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
assert_eq!(
err.to_string(),
"agents.max_threads cannot be set when multi_agent_v2 is enabled"
);
Ok(())
}
#[tokio::test]
async fn multi_agent_v2_session_thread_cap_one_disallows_subagents() -> std::io::Result<()> {
let codex_home = TempDir::new()?;
std::fs::write(
codex_home.path().join(CONFIG_TOML_FILE),
r#"[features.multi_agent_v2]
enabled = true
max_concurrent_threads_per_session = 1
"#,
)?;
let config = ConfigBuilder::without_managed_config_for_tests()
.codex_home(codex_home.path().to_path_buf())
.fallback_cwd(Some(codex_home.path().to_path_buf()))
.build()
.await?;
assert_eq!(config.multi_agent_v2.max_concurrent_threads_per_session, 1);
assert_eq!(config.agent_max_threads, Some(0));
Ok(())
}
#[tokio::test]
async fn feature_requirements_normalize_runtime_feature_mutations() -> std::io::Result<()> {
let codex_home = TempDir::new()?;
+35 -8
View File
@@ -1,7 +1,6 @@
use crate::agents_md::AgentsMdManager;
use crate::config::edit::ConfigEdit;
use crate::config::edit::ConfigEditsBuilder;
use crate::memories::memory_root;
use crate::path_utils::normalize_for_native_workdir;
use crate::unified_exec::DEFAULT_MAX_BACKGROUND_TERMINAL_TIMEOUT_MS;
use crate::unified_exec::MIN_EMPTY_YIELD_TIME_MS;
@@ -63,6 +62,7 @@ use codex_features::MultiAgentV2ConfigToml;
use codex_git_utils::resolve_root_git_project_for_trust;
use codex_login::AuthManagerConfig;
use codex_mcp::McpConfig;
use codex_memories_write::memory_root;
use codex_model_provider_info::LEGACY_OLLAMA_CHAT_PROVIDER_ID;
use codex_model_provider_info::ModelProviderInfo;
use codex_model_provider_info::OLLAMA_CHAT_PROVIDER_REMOVED_ERROR;
@@ -131,6 +131,7 @@ pub use codex_git_utils::GhostSnapshotConfig;
/// the context window.
pub(crate) const AGENTS_MD_MAX_BYTES: usize = 32 * 1024; // 32 KiB
pub(crate) const DEFAULT_AGENT_MAX_THREADS: Option<usize> = Some(6);
pub(crate) const DEFAULT_MULTI_AGENT_V2_MAX_CONCURRENT_THREADS_PER_SESSION: usize = 4;
pub(crate) const DEFAULT_AGENT_MAX_DEPTH: i32 = 1;
pub(crate) const DEFAULT_AGENT_JOB_MAX_RUNTIME_SECONDS: Option<u64> = None;
const LOCAL_DEV_BUILD_VERSION: &str = "0.0.0";
@@ -704,6 +705,7 @@ pub struct Config {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MultiAgentV2Config {
pub max_concurrent_threads_per_session: usize,
pub usage_hint_enabled: bool,
pub usage_hint_text: Option<String>,
pub hide_spawn_agent_metadata: bool,
@@ -712,6 +714,8 @@ pub struct MultiAgentV2Config {
impl Default for MultiAgentV2Config {
fn default() -> Self {
Self {
max_concurrent_threads_per_session:
DEFAULT_MULTI_AGENT_V2_MAX_CONCURRENT_THREADS_PER_SESSION,
usage_hint_enabled: true,
usage_hint_text: None,
hide_spawn_agent_metadata: false,
@@ -1579,6 +1583,10 @@ fn resolve_multi_agent_v2_config(
let profile = multi_agent_v2_toml_config(config_profile.features.as_ref());
let default = MultiAgentV2Config::default();
let max_concurrent_threads_per_session = profile
.and_then(|config| config.max_concurrent_threads_per_session)
.or_else(|| base.and_then(|config| config.max_concurrent_threads_per_session))
.unwrap_or(default.max_concurrent_threads_per_session);
let usage_hint_enabled = profile
.and_then(|config| config.usage_hint_enabled)
.or_else(|| base.and_then(|config| config.usage_hint_enabled))
@@ -1594,6 +1602,7 @@ fn resolve_multi_agent_v2_config(
.unwrap_or(default.hide_spawn_agent_metadata);
MultiAgentV2Config {
max_concurrent_threads_per_session,
usage_hint_enabled,
usage_hint_text,
hide_spawn_agent_metadata,
@@ -2078,17 +2087,35 @@ impl Config {
let history = cfg.history.unwrap_or_default();
let agent_max_threads = cfg
.agents
.as_ref()
.and_then(|agents| agents.max_threads)
.or(DEFAULT_AGENT_MAX_THREADS);
if agent_max_threads == Some(0) {
if multi_agent_v2.max_concurrent_threads_per_session == 0 {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"agents.max_threads must be at least 1",
"features.multi_agent_v2.max_concurrent_threads_per_session must be at least 1",
));
}
let agent_max_threads_from_config = cfg.agents.as_ref().and_then(|agents| agents.max_threads);
let agent_max_threads = if features.enabled(Feature::MultiAgentV2) {
if agent_max_threads_from_config.is_some() {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"agents.max_threads cannot be set when multi_agent_v2 is enabled",
));
}
Some(
multi_agent_v2
.max_concurrent_threads_per_session
.saturating_sub(1),
)
} else {
let agent_max_threads = agent_max_threads_from_config.or(DEFAULT_AGENT_MAX_THREADS);
if agent_max_threads == Some(0) {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"agents.max_threads must be at least 1",
));
}
agent_max_threads
};
let agent_max_depth = cfg
.agents
.as_ref()
+3 -3
View File
@@ -144,7 +144,7 @@ pub async fn list_cached_accessible_connectors_from_mcp_tools(
config: &Config,
) -> Option<Vec<AppInfo>> {
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
let auth = auth_manager.auth().await;
if !config
.features
@@ -216,7 +216,7 @@ pub async fn list_accessible_connectors_from_mcp_tools_with_environment_manager(
environment_manager: &EnvironmentManager,
) -> anyhow::Result<AccessibleConnectorsStatus> {
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
let auth = auth_manager.auth().await;
if !config
.features
@@ -434,7 +434,7 @@ async fn list_directory_connectors_for_tool_suggest_with_auth(
Some(auth)
} else {
let auth_manager =
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(config, /*enable_codex_api_key_env*/ false).await;
loaded_auth = auth_manager.auth().await;
loaded_auth.as_ref()
};
+2 -1
View File
@@ -57,7 +57,7 @@ pub use codex_mcp::SandboxState;
mod mcp_openai_file;
mod mcp_tool_call;
mod memories;
pub use memories::clear_memory_roots_contents;
pub use codex_memories_write::clear_memory_roots_contents;
pub(crate) mod mention_syntax;
pub(crate) mod message_history;
pub(crate) mod utils;
@@ -200,4 +200,5 @@ pub mod compact;
pub(crate) mod memory_trace;
pub use memory_trace::BuiltMemory;
pub use memory_trace::build_memories_from_trace_files;
mod memory_usage;
pub mod otel_init;
-251
View File
@@ -1,251 +0,0 @@
use crate::memories::memory_extensions_root;
use chrono::DateTime;
use chrono::Duration;
use chrono::NaiveDateTime;
use chrono::Utc;
use std::path::Path;
use std::path::PathBuf;
use tracing::warn;
const FILENAME_TS_FORMAT: &str = "%Y-%m-%dT%H-%M-%S";
pub(super) const EXTENSION_RESOURCE_RETENTION_DAYS: i64 = 7;
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) struct RemovedExtensionResource {
pub(super) extension: String,
pub(super) resource_path: String,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) struct PendingExtensionResourceRemoval {
pub(super) removed: RemovedExtensionResource,
path: PathBuf,
}
pub(super) async fn find_old_extension_resources(
memory_root: &Path,
) -> Vec<PendingExtensionResourceRemoval> {
find_old_extension_resources_with_now(memory_root, Utc::now()).await
}
async fn find_old_extension_resources_with_now(
memory_root: &Path,
now: DateTime<Utc>,
) -> Vec<PendingExtensionResourceRemoval> {
let mut pending = Vec::new();
let cutoff = now - Duration::days(EXTENSION_RESOURCE_RETENTION_DAYS);
let extensions_root = memory_extensions_root(memory_root);
let mut extensions = match tokio::fs::read_dir(&extensions_root).await {
Ok(extensions) => extensions,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return pending,
Err(err) => {
warn!(
"failed reading memory extensions root {}: {err}",
extensions_root.display()
);
return pending;
}
};
while let Ok(Some(extension_entry)) = extensions.next_entry().await {
let extension_path = extension_entry.path();
let Ok(file_type) = extension_entry.file_type().await else {
continue;
};
if !file_type.is_dir() {
continue;
}
let Some(extension) = extension_path
.file_name()
.and_then(|name| name.to_str())
.map(ToOwned::to_owned)
else {
continue;
};
if !tokio::fs::try_exists(extension_path.join("instructions.md"))
.await
.unwrap_or(false)
{
continue;
}
let resources_path = extension_path.join("resources");
let mut resources = match tokio::fs::read_dir(&resources_path).await {
Ok(resources) => resources,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue,
Err(err) => {
warn!(
"failed reading memory extension resources {}: {err}",
resources_path.display()
);
continue;
}
};
while let Ok(Some(resource_entry)) = resources.next_entry().await {
let resource_file_path = resource_entry.path();
let Ok(file_type) = resource_entry.file_type().await else {
continue;
};
if !file_type.is_file() {
continue;
}
let Some(file_name) = resource_file_path
.file_name()
.and_then(|name| name.to_str())
else {
continue;
};
if !file_name.ends_with(".md") {
continue;
}
let Some(resource_timestamp) = resource_timestamp(file_name) else {
continue;
};
if resource_timestamp > cutoff {
continue;
}
pending.push(PendingExtensionResourceRemoval {
removed: RemovedExtensionResource {
extension: extension.clone(),
resource_path: format!("resources/{file_name}"),
},
path: resource_file_path,
});
}
}
pending.sort_by(|left, right| {
left.removed
.extension
.cmp(&right.removed.extension)
.then_with(|| left.removed.resource_path.cmp(&right.removed.resource_path))
});
pending
}
pub(super) async fn remove_extension_resources(resources: &[PendingExtensionResourceRemoval]) {
for resource in resources {
if let Err(err) = tokio::fs::remove_file(&resource.path).await
&& err.kind() != std::io::ErrorKind::NotFound
{
warn!(
"failed pruning old memory extension resource {}: {err}",
resource.path.display()
);
}
}
}
fn resource_timestamp(file_name: &str) -> Option<DateTime<Utc>> {
let timestamp = file_name.get(..19)?;
let naive = NaiveDateTime::parse_from_str(timestamp, FILENAME_TS_FORMAT).ok()?;
Some(DateTime::from_naive_utc_and_offset(naive, Utc))
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
use tempfile::TempDir;
#[tokio::test]
async fn finds_only_old_resources_from_extensions_with_instructions() {
let codex_home = TempDir::new().expect("create temp codex home");
let memory_root = codex_home.path().join("memories");
let extensions_root = memory_extensions_root(&memory_root);
let chronicle_resources = extensions_root.join("chronicle/resources");
tokio::fs::create_dir_all(&chronicle_resources)
.await
.expect("create chronicle resources");
tokio::fs::write(
extensions_root.join("chronicle/instructions.md"),
"instructions",
)
.await
.expect("write chronicle instructions");
let now = DateTime::from_naive_utc_and_offset(
NaiveDateTime::parse_from_str("2026-04-14T12-00-00", FILENAME_TS_FORMAT)
.expect("parse now"),
Utc,
);
let old_file = chronicle_resources.join("2026-04-06T11-59-59-abcd-10min-old.md");
let exact_cutoff_file =
chronicle_resources.join("2026-04-07T12-00-00-abcd-10min-cutoff.md");
let recent_file = chronicle_resources.join("2026-04-08T12-00-00-abcd-10min-recent.md");
let invalid_file = chronicle_resources.join("not-a-timestamp.md");
for file in [&old_file, &exact_cutoff_file, &recent_file, &invalid_file] {
tokio::fs::write(file, "resource")
.await
.expect("write chronicle resource");
}
let ignored_resources = extensions_root.join("ignored/resources");
tokio::fs::create_dir_all(&ignored_resources)
.await
.expect("create ignored resources");
let ignored_old_file = ignored_resources.join("2026-04-06T11-59-59-abcd-10min-old.md");
tokio::fs::write(&ignored_old_file, "ignored")
.await
.expect("write ignored resource");
let pending = find_old_extension_resources_with_now(&memory_root, now).await;
assert_eq!(
pending
.iter()
.map(|resource| resource.removed.clone())
.collect::<Vec<_>>(),
vec![
RemovedExtensionResource {
extension: "chronicle".to_string(),
resource_path: "resources/2026-04-06T11-59-59-abcd-10min-old.md".to_string(),
},
RemovedExtensionResource {
extension: "chronicle".to_string(),
resource_path: "resources/2026-04-07T12-00-00-abcd-10min-cutoff.md".to_string(),
},
]
);
assert!(
tokio::fs::try_exists(&old_file)
.await
.expect("check old file before remove")
);
assert!(
tokio::fs::try_exists(&exact_cutoff_file)
.await
.expect("check cutoff file before remove")
);
remove_extension_resources(&pending).await;
assert!(
!tokio::fs::try_exists(&old_file)
.await
.expect("check old file")
);
assert!(
!tokio::fs::try_exists(&exact_cutoff_file)
.await
.expect("check cutoff file")
);
assert!(
tokio::fs::try_exists(&recent_file)
.await
.expect("check recent file")
);
assert!(
tokio::fs::try_exists(&invalid_file)
.await
.expect("check invalid file")
);
assert!(
tokio::fs::try_exists(&ignored_old_file)
.await
.expect("check ignored old file")
);
}
}
+2 -52
View File
@@ -1,37 +1,23 @@
//! Memory subsystem for startup extraction and consolidation.
//! Memory startup extraction and consolidation orchestration.
//!
//! The startup memory pipeline is split into two phases:
//! - Phase 1: select rollouts, extract stage-1 raw memories, persist stage-1 outputs, and enqueue consolidation.
//! - Phase 2: claim a global consolidation lock, materialize consolidation inputs, and dispatch one consolidation agent.
pub(crate) mod citations;
mod control;
mod phase1;
mod phase2;
pub(crate) mod prompts;
mod start;
mod storage;
#[cfg(test)]
mod tests;
pub(crate) mod usage;
use codex_protocol::openai_models::ReasoningEffort;
pub use control::clear_memory_roots_contents;
/// Starts the memory startup pipeline for eligible root sessions.
/// This is the single entrypoint that `codex` uses to trigger memory startup.
///
/// This is the entry point to read and understand this module.
pub(crate) use start::start_memories_startup_task;
mod artifacts {
pub(super) const EXTENSIONS_SUBDIR: &str = "memories_extensions";
pub(super) const ROLLOUT_SUMMARIES_SUBDIR: &str = "rollout_summaries";
pub(super) const RAW_MEMORIES_FILENAME: &str = "raw_memories.md";
}
mod extensions;
/// Phase 1 (startup extraction).
mod phase_one {
/// Default model used for phase 1.
@@ -39,21 +25,9 @@ mod phase_one {
/// Default reasoning effort used for phase 1.
pub(super) const REASONING_EFFORT: super::ReasoningEffort = super::ReasoningEffort::Low;
/// Prompt used for phase 1.
pub(super) const PROMPT: &str = include_str!("../../templates/memories/stage_one_system.md");
pub(super) const PROMPT: &str = codex_memories_write::STAGE_ONE_PROMPT;
/// Concurrency cap for startup memory extraction and consolidation scheduling.
pub(super) const CONCURRENCY_LIMIT: usize = 8;
/// Fallback stage-1 rollout truncation limit (tokens) when model metadata
/// does not include a valid context window.
pub(super) const DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT: usize = 150_000;
/// Maximum number of tokens from `memory_summary.md` injected into memory
/// tool developer instructions.
pub(super) const MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT: usize = 5_000;
/// Portion of the model effective input window reserved for the stage-1
/// rollout input.
///
/// Keeping this below 100% leaves room for system instructions, prompt
/// framing, and model output.
pub(super) const CONTEXT_WINDOW_PERCENT: i64 = 70;
/// Lease duration (seconds) for phase-1 job ownership.
pub(super) const JOB_LEASE_SECONDS: i64 = 3_600;
/// Backoff delay (seconds) before retrying a failed stage-1 extraction job.
@@ -97,27 +71,3 @@ mod metrics {
/// Histogram for aggregate token usage across one phase-2 consolidation run.
pub(super) const MEMORY_PHASE_TWO_TOKEN_USAGE: &str = "codex.memory.phase2.token_usage";
}
use codex_utils_absolute_path::AbsolutePathBuf;
use std::path::Path;
use std::path::PathBuf;
pub fn memory_root(codex_home: &AbsolutePathBuf) -> AbsolutePathBuf {
codex_home.join("memories")
}
fn rollout_summaries_dir(root: &Path) -> PathBuf {
root.join(artifacts::ROLLOUT_SUMMARIES_SUBDIR)
}
fn memory_extensions_root(root: &Path) -> PathBuf {
root.with_file_name(artifacts::EXTENSIONS_SUBDIR)
}
fn raw_memories_file(root: &Path) -> PathBuf {
root.join(artifacts::RAW_MEMORIES_FILENAME)
}
async fn ensure_layout(root: &Path) -> std::io::Result<()> {
tokio::fs::create_dir_all(rollout_summaries_dir(root)).await
}
+1 -1
View File
@@ -5,13 +5,13 @@ use crate::context::is_memory_excluded_contextual_user_fragment;
use crate::memories::metrics;
use crate::memories::phase_one;
use crate::memories::phase_one::PRUNE_BATCH_SIZE;
use crate::memories::prompts::build_stage_one_input_message;
use crate::rollout::INTERACTIVE_SESSION_SOURCES;
use crate::rollout::policy::should_persist_response_item_for_memories;
use crate::session::session::Session;
use crate::session::turn_context::TurnContext;
use codex_api::ResponseEvent;
use codex_config::types::MemoriesConfig;
use codex_memories_write::build_stage_one_input_message;
use codex_otel::SessionTelemetry;
use codex_protocol::config_types::ReasoningSummary as ReasoningSummaryConfig;
use codex_protocol::config_types::ServiceTier;
+104 -87
View File
@@ -1,20 +1,21 @@
use crate::agent::AgentStatus;
use crate::agent::status::is_final as is_final_agent_status;
use crate::config::Config;
use crate::memories::extensions::PendingExtensionResourceRemoval;
use crate::memories::extensions::find_old_extension_resources;
use crate::memories::extensions::remove_extension_resources;
use crate::memories::memory_root;
use crate::memories::metrics;
use crate::memories::phase_two;
use crate::memories::prompts::build_consolidation_prompt;
use crate::memories::storage::rebuild_raw_memories_file_from_memories;
use crate::memories::storage::rollout_summary_file_stem;
use crate::memories::storage::sync_rollout_summaries_from_memories;
use crate::session::emit_subagent_session_started;
use crate::session::session::Session;
use codex_config::Constrained;
use codex_features::Feature;
use codex_memories_write::build_consolidation_prompt;
use codex_memories_write::memory_root;
use codex_memories_write::prune_old_extension_resources;
use codex_memories_write::rebuild_raw_memories_file_from_memories;
use codex_memories_write::sync_rollout_summaries_from_memories;
use codex_memories_write::workspace::memory_workspace_diff;
use codex_memories_write::workspace::prepare_memory_workspace;
use codex_memories_write::workspace::reset_memory_workspace_baseline;
use codex_memories_write::workspace::write_workspace_diff;
use codex_protocol::ThreadId;
use codex_protocol::protocol::AskForApproval;
use codex_protocol::protocol::SandboxPolicy;
@@ -25,7 +26,7 @@ use codex_protocol::user_input::UserInput;
use codex_state::Stage1Output;
use codex_state::StateRuntime;
use std::collections::HashMap;
use std::collections::HashSet;
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::watch;
@@ -59,7 +60,7 @@ pub(super) async fn run(session: &Arc<Session>, config: Arc<Config>) {
let max_raw_memories = config.memories.max_raw_memories_for_consolidation;
let max_unused_days = config.memories.max_unused_days;
// 1. Claim the job.
// 1. Claim the global Phase 2 lock before touching the memory workspace.
let claim = match job::claim(session, db).await {
Ok(claim) => claim,
Err(e) => {
@@ -72,71 +73,76 @@ pub(super) async fn run(session: &Arc<Session>, config: Arc<Config>) {
}
};
// 2. Get the config for the agent
let Some(agent_config) = agent::get_config(config.clone()) else {
// 2. Ensure the memories root has a git baseline repository.
if let Err(err) = prepare_memory_workspace(&root).await {
tracing::error!("failed preparing memory workspace: {err}");
job::failed(session, db, &claim, "failed_prepare_workspace").await;
return;
}
// 3. Build the locked-down config used by the consolidation agent.
let Some(agent_config) = agent::get_config(config.as_ref()) else {
// If we can't get the config, we can't consolidate.
tracing::error!("failed to get agent config");
job::failed(session, db, &claim, "failed_sandbox_policy").await;
return;
};
// 3. Query the memories
let selection = match db
// 4. Load current DB-backed Phase 2 inputs.
let raw_memories = match db
.get_phase2_input_selection(max_raw_memories, max_unused_days)
.await
{
Ok(selection) => selection,
Ok(raw_memories) => raw_memories,
Err(err) => {
tracing::error!("failed to list stage1 outputs from global: {}", err);
tracing::error!("failed to list stage1 outputs from global: {err}");
job::failed(session, db, &claim, "failed_load_stage1_outputs").await;
return;
}
};
let raw_memories = selection.selected.to_vec();
let artifact_memories = artifact_memories_for_phase2(&selection);
let raw_memory_count = raw_memories.len();
let new_watermark = get_watermark(claim.watermark, &raw_memories);
// 4. Update the file system by syncing the raw memories with the one extracted from DB at
// step 3
// [`rollout_summaries/`]
if let Err(err) =
sync_rollout_summaries_from_memories(&root, &artifact_memories, artifact_memories.len())
.await
{
tracing::error!("failed syncing local memory artifacts for global consolidation: {err}");
job::failed(session, db, &claim, "failed_sync_artifacts").await;
// 5. Sync the current inputs into the memory workspace.
if let Err(err) = sync_phase2_workspace_inputs(&root, &raw_memories).await {
tracing::error!("failed syncing phase2 workspace inputs: {err}");
job::failed(session, db, &claim, "failed_sync_workspace_inputs").await;
return;
}
// [`raw_memories.md`]
if let Err(err) =
rebuild_raw_memories_file_from_memories(&root, &artifact_memories, artifact_memories.len())
.await
{
tracing::error!("failed syncing local memory artifacts for global consolidation: {err}");
job::failed(session, db, &claim, "failed_rebuild_raw_memories").await;
return;
}
let pending_extension_resource_removals = find_old_extension_resources(&root).await;
let removed_extension_resources = pending_extension_resource_removals
.iter()
.map(|resource| resource.removed.clone())
.collect::<Vec<_>>();
if raw_memories.is_empty() && pending_extension_resource_removals.is_empty() {
// 6. Use git to decide whether the synced workspace actually changed.
let workspace_diff = match memory_workspace_diff(&root).await {
Ok(diff) => diff,
Err(err) => {
tracing::error!("failed checking memory workspace changes: {err}");
job::failed(session, db, &claim, "failed_workspace_status").await;
return;
}
};
if !workspace_diff.has_changes() {
tracing::error!("Phase 2 no changes");
// We check only after sync of the file system.
job::succeed(
session,
db,
&claim,
new_watermark,
&[],
"succeeded_no_input",
&raw_memories,
"succeeded_no_workspace_changes",
)
.await;
return;
}
// 5. Spawn the agent
let prompt = agent::get_prompt(config, &selection, &removed_extension_resources);
// 7. Persist the diff for the consolidation agent to inspect.
if let Err(err) = write_workspace_diff(&root, &workspace_diff).await {
tracing::error!("failed writing memory workspace diff file: {err}");
job::failed(session, db, &claim, "failed_workspace_diff_file").await;
return;
}
// 8. Spawn the consolidation agent.
let prompt = agent::get_prompt(&root);
let source = SessionSource::SubAgent(SubAgentSource::MemoryConsolidation);
let agent_control = session.services.agent_control.detached_registry();
let thread_id = match agent_control
@@ -172,39 +178,34 @@ pub(super) async fn run(session: &Arc<Session>, config: Arc<Config>) {
warn!("failed to load memory consolidation thread config for analytics: {thread_id}");
}
// 6. Spawn the agent handler.
// 9. Hand off completion handling, heartbeats, and baseline reset.
agent::handle(
session,
claim,
new_watermark,
raw_memories.clone(),
pending_extension_resource_removals,
root,
thread_id,
agent_control,
phase_two_e2e_timer,
);
// 7. Metrics and logs.
// 10. Emit dispatch metrics.
let counters = Counters {
input: raw_memories.len() as i64,
input: raw_memory_count as i64,
};
emit_metrics(session, counters);
}
fn artifact_memories_for_phase2(
selection: &codex_state::Phase2InputSelection,
) -> Vec<Stage1Output> {
let mut seen = HashSet::new();
let mut memories = selection.selected.clone();
for memory in &selection.selected {
seen.insert(rollout_summary_file_stem(memory));
}
for memory in &selection.previous_selected {
if seen.insert(rollout_summary_file_stem(memory)) {
memories.push(memory.clone());
}
}
memories
async fn sync_phase2_workspace_inputs(
root: &Path,
raw_memories: &[Stage1Output],
) -> std::io::Result<()> {
let raw_memory_count = raw_memories.len();
sync_rollout_summaries_from_memories(root, raw_memories, raw_memory_count).await?;
rebuild_raw_memories_file_from_memories(root, raw_memories, raw_memory_count).await?;
prune_old_extension_resources(root).await;
Ok(())
}
mod job {
@@ -234,7 +235,9 @@ mod job {
);
(ownership_token, input_watermark)
}
codex_state::Phase2JobClaimOutcome::SkippedNotDirty => return Err("skipped_not_dirty"),
codex_state::Phase2JobClaimOutcome::SkippedRetryUnavailable => {
return Err("skipped_retry_unavailable");
}
codex_state::Phase2JobClaimOutcome::SkippedRunning => return Err("skipped_running"),
};
@@ -293,9 +296,9 @@ mod job {
mod agent {
use super::*;
pub(super) fn get_config(config: Arc<Config>) -> Option<Config> {
pub(super) fn get_config(config: &Config) -> Option<Config> {
let root = memory_root(&config.codex_home);
let mut agent_config = config.as_ref().clone();
let mut agent_config = config.clone();
agent_config.cwd = root.clone();
// Consolidation threads must never feed back into phase-1 memory generation.
@@ -342,13 +345,8 @@ mod agent {
Some(agent_config)
}
pub(super) fn get_prompt(
config: Arc<Config>,
selection: &codex_state::Phase2InputSelection,
removed_extension_resources: &[crate::memories::extensions::RemovedExtensionResource],
) -> Vec<UserInput> {
let root = memory_root(&config.codex_home);
let prompt = build_consolidation_prompt(&root, selection, removed_extension_resources);
pub(super) fn get_prompt(root: &Path) -> Vec<UserInput> {
let prompt = build_consolidation_prompt(root);
vec![UserInput::Text {
text: prompt,
text_elements: vec![],
@@ -362,7 +360,7 @@ mod agent {
claim: Claim,
new_watermark: i64,
selected_outputs: Vec<codex_state::Stage1Output>,
pending_extension_resource_removals: Vec<PendingExtensionResourceRemoval>,
memory_root: codex_utils_absolute_path::AbsolutePathBuf,
thread_id: ThreadId,
agent_control: crate::agent::AgentControl,
phase_two_e2e_timer: Option<codex_otel::Timer>,
@@ -386,20 +384,38 @@ mod agent {
};
// Loop the agent until we have the final status.
let final_status = loop_agent(
db.clone(),
claim.token.clone(),
new_watermark,
thread_id,
rx,
)
.await;
let final_status = loop_agent(db.clone(), claim.token.clone(), thread_id, rx).await;
if matches!(final_status, AgentStatus::Completed(_)) {
if let Some(token_usage) = agent_control.get_total_token_usage(thread_id).await {
emit_token_usage_metrics(&session, &token_usage);
}
if job::succeed(
// Do not reset the workspace baseline if we lost the lock.
let Ok(still_owns_lock) = db
.heartbeat_global_phase2_job(&claim.token, phase_two::JOB_LEASE_SECONDS)
.await
.inspect_err(|err| {
tracing::error!(
"failed confirming global memory consolidation ownership before resetting workspace baseline: {err}"
);
})
else {
job::failed(&session, &db, &claim, "failed_confirm_ownership").await;
return;
};
if !still_owns_lock {
tracing::error!(
"lost global memory consolidation ownership before resetting workspace baseline"
);
return;
}
if let Err(err) = reset_memory_workspace_baseline(&memory_root).await {
tracing::error!("failed resetting memory workspace baseline: {err}");
job::failed(&session, &db, &claim, "failed_workspace_commit").await;
return;
}
if !job::succeed(
&session,
&db,
&claim,
@@ -409,7 +425,9 @@ mod agent {
)
.await
{
remove_extension_resources(&pending_extension_resource_removals).await;
tracing::error!(
"failed marking global memory consolidation job succeeded after resetting workspace baseline"
);
}
} else {
job::failed(&session, &db, &claim, "failed_agent").await;
@@ -433,7 +451,6 @@ mod agent {
async fn loop_agent(
db: Arc<StateRuntime>,
token: String,
_new_watermark: i64,
thread_id: ThreadId,
mut rx: watch::Receiver<AgentStatus>,
) -> AgentStatus {
@@ -491,7 +508,7 @@ pub(super) fn get_watermark(
.map(|memory| memory.source_updated_at.timestamp())
.max()
.unwrap_or(claimed_watermark)
.max(claimed_watermark) // todo double check the claimed here.
.max(claimed_watermark)
}
fn emit_metrics(session: &Arc<Session>, counters: Counters) {
-293
View File
@@ -1,293 +0,0 @@
use crate::memories::extensions::EXTENSION_RESOURCE_RETENTION_DAYS;
use crate::memories::extensions::RemovedExtensionResource;
use crate::memories::memory_extensions_root;
use crate::memories::memory_root;
use crate::memories::phase_one;
use crate::memories::storage::rollout_summary_file_stem_from_parts;
use codex_protocol::openai_models::ModelInfo;
use codex_state::Phase2InputSelection;
use codex_state::Stage1Output;
use codex_state::Stage1OutputRef;
use codex_utils_absolute_path::AbsolutePathBuf;
use codex_utils_output_truncation::TruncationPolicy;
use codex_utils_output_truncation::truncate_text;
use codex_utils_template::Template;
use std::fmt::Write as _;
use std::path::Path;
use std::sync::LazyLock;
use tokio::fs;
use tracing::warn;
static CONSOLIDATION_PROMPT_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../../templates/memories/consolidation.md"),
"memories/consolidation.md",
)
});
static STAGE_ONE_INPUT_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../../templates/memories/stage_one_input.md"),
"memories/stage_one_input.md",
)
});
static MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../../templates/memories/read_path.md"),
"memories/read_path.md",
)
});
static MEMORY_EXTENSIONS_FOLDER_STRUCTURE_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
MEMORY_EXTENSIONS_FOLDER_STRUCTURE,
"memories/extensions_folder_structure.md",
)
});
static MEMORY_EXTENSIONS_PRIMARY_INPUTS_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
MEMORY_EXTENSIONS_PRIMARY_INPUTS,
"memories/extensions_primary_inputs.md",
)
});
fn parse_embedded_template(source: &'static str, template_name: &str) -> Template {
match Template::parse(source) {
Ok(template) => template,
Err(err) => panic!("embedded template {template_name} is invalid: {err}"),
}
}
const MEMORY_EXTENSIONS_FOLDER_STRUCTURE: &str = r#"
Memory extensions (under {{ memory_extensions_root }}/):
- <extension_name>/instructions.md
- Source-specific guidance for interpreting additional memory signals. If an
extension folder exists, you must read its instructions.md to determine how to use this memory
source.
If the user has any memory extensions, you MUST read the instructions for each extension to
determine how to use the memory source. If the Phase 2 diff lists removed memory extension
resources, use that extension-specific deletion diff to remove stale memories derived only from
those resources. If it has no extension folders, continue with the standard memory inputs only.
"#;
const MEMORY_EXTENSIONS_PRIMARY_INPUTS: &str = r#"
Optional source-specific inputs:
Under `{{ memory_extensions_root }}/`:
- `<extension_name>/instructions.md`
- If extension folders exist, read each instructions.md first and follow it when interpreting
that extension's memory source.
If the Phase 2 diff lists removed memory extension resources, use that extension-specific deletion
diff to remove stale memories derived only from those resources.
"#;
/// Builds the consolidation subagent prompt for a specific memory root.
pub(super) fn build_consolidation_prompt(
memory_root: &Path,
selection: &Phase2InputSelection,
removed_extension_resources: &[RemovedExtensionResource],
) -> String {
let memory_extensions_root = memory_extensions_root(memory_root);
let memory_extensions_exist = memory_extensions_root.is_dir();
let memory_root = memory_root.display().to_string();
let memory_extensions_root = memory_extensions_root.display().to_string();
let memory_extensions_folder_structure = if memory_extensions_exist {
render_memory_extensions_block(
&MEMORY_EXTENSIONS_FOLDER_STRUCTURE_TEMPLATE,
&memory_extensions_root,
)
} else {
String::new()
};
let memory_extensions_primary_inputs = if memory_extensions_exist {
render_memory_extensions_block(
&MEMORY_EXTENSIONS_PRIMARY_INPUTS_TEMPLATE,
&memory_extensions_root,
)
} else {
String::new()
};
let phase2_input_selection =
render_phase2_input_selection(selection, removed_extension_resources);
CONSOLIDATION_PROMPT_TEMPLATE
.render([
("memory_root", memory_root.as_str()),
(
"memory_extensions_folder_structure",
memory_extensions_folder_structure.as_str(),
),
(
"memory_extensions_primary_inputs",
memory_extensions_primary_inputs.as_str(),
),
("phase2_input_selection", phase2_input_selection.as_str()),
])
.unwrap_or_else(|err| {
warn!("failed to render memories consolidation prompt template: {err}");
format!(
"## Memory Phase 2 (Consolidation)\nConsolidate Codex memories in: {memory_root}\n\n{phase2_input_selection}"
)
})
}
fn render_memory_extensions_block(template: &Template, memory_extensions_root: &str) -> String {
template
.render([("memory_extensions_root", memory_extensions_root)])
.unwrap_or_else(|err| {
warn!("failed to render memories extension prompt block: {err}");
String::new()
})
}
fn render_phase2_input_selection(
selection: &Phase2InputSelection,
removed_extension_resources: &[RemovedExtensionResource],
) -> String {
let retained = selection.retained_thread_ids.len();
let added = selection.selected.len().saturating_sub(retained);
let selected = if selection.selected.is_empty() {
"- none".to_string()
} else {
selection
.selected
.iter()
.map(|item| {
render_selected_input_line(
item,
selection.retained_thread_ids.contains(&item.thread_id),
)
})
.collect::<Vec<_>>()
.join("\n")
};
let removed = if selection.removed.is_empty() {
"- none".to_string()
} else {
selection
.removed
.iter()
.map(render_removed_input_line)
.collect::<Vec<_>>()
.join("\n")
};
let mut rendered = format!(
"- selected inputs this run: {}\n- newly added since the last successful Phase 2 run: {added}\n- retained from the last successful Phase 2 run: {retained}\n- removed from the last successful Phase 2 run: {}\n\nCurrent selected Phase 1 inputs:\n{selected}\n\nRemoved from the last successful Phase 2 selection:\n{removed}\n",
selection.selected.len(),
selection.removed.len(),
);
if !removed_extension_resources.is_empty() {
rendered.push_str("\nMemory extension resources removed by retention pruning:\n");
let _ = writeln!(
rendered,
"- retention window: {EXTENSION_RESOURCE_RETENTION_DAYS} days"
);
let mut current_extension = "";
for removed_resource in removed_extension_resources {
if removed_resource.extension != current_extension {
current_extension = &removed_resource.extension;
let _ = writeln!(rendered, "- extension: {current_extension}");
}
let _ = writeln!(rendered, " - {}", removed_resource.resource_path);
}
}
rendered
}
fn render_selected_input_line(item: &Stage1Output, retained: bool) -> String {
let status = if retained { "retained" } else { "added" };
let rollout_summary_file = format!(
"rollout_summaries/{}.md",
rollout_summary_file_stem_from_parts(
item.thread_id,
item.source_updated_at,
item.rollout_slug.as_deref(),
)
);
format!(
"- [{status}] thread_id={}, rollout_summary_file={rollout_summary_file}",
item.thread_id
)
}
fn render_removed_input_line(item: &Stage1OutputRef) -> String {
let rollout_summary_file = format!(
"rollout_summaries/{}.md",
rollout_summary_file_stem_from_parts(
item.thread_id,
item.source_updated_at,
item.rollout_slug.as_deref(),
)
);
format!(
"- thread_id={}, rollout_summary_file={rollout_summary_file}",
item.thread_id
)
}
/// Builds the stage-1 user message containing rollout metadata and content.
///
/// Large rollout payloads are truncated to 70% of the active model's effective
/// input window token budget while keeping both head and tail context.
pub(super) fn build_stage_one_input_message(
model_info: &ModelInfo,
rollout_path: &Path,
rollout_cwd: &Path,
rollout_contents: &str,
) -> anyhow::Result<String> {
let rollout_token_limit = model_info
.resolved_context_window()
.and_then(|limit| (limit > 0).then_some(limit))
.map(|limit| limit.saturating_mul(model_info.effective_context_window_percent) / 100)
.map(|limit| (limit.saturating_mul(phase_one::CONTEXT_WINDOW_PERCENT) / 100).max(1))
.and_then(|limit| usize::try_from(limit).ok())
.unwrap_or(phase_one::DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT);
let truncated_rollout_contents = truncate_text(
rollout_contents,
TruncationPolicy::Tokens(rollout_token_limit),
);
let rollout_path = rollout_path.display().to_string();
let rollout_cwd = rollout_cwd.display().to_string();
Ok(STAGE_ONE_INPUT_TEMPLATE.render([
("rollout_path", rollout_path.as_str()),
("rollout_cwd", rollout_cwd.as_str()),
("rollout_contents", truncated_rollout_contents.as_str()),
])?)
}
/// Build prompt used for read path. This prompt must be added to the developer instructions. In
/// case of large memory files, the `memory_summary.md` is truncated at
/// [phase_one::MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT].
pub(crate) async fn build_memory_tool_developer_instructions(
codex_home: &AbsolutePathBuf,
) -> Option<String> {
let base_path = memory_root(codex_home);
let memory_summary_path = base_path.join("memory_summary.md");
let memory_summary = fs::read_to_string(&memory_summary_path)
.await
.ok()?
.trim()
.to_string();
let memory_summary = truncate_text(
&memory_summary,
TruncationPolicy::Tokens(phase_one::MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT),
);
if memory_summary.is_empty() {
return None;
}
let base_path = base_path.display().to_string();
MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_TEMPLATE
.render([
("base_path", base_path.as_str()),
("memory_summary", memory_summary.as_str()),
])
.ok()
}
#[cfg(test)]
#[path = "prompts_tests.rs"]
mod tests;
-118
View File
@@ -1,118 +0,0 @@
use super::*;
use crate::memories::extensions::RemovedExtensionResource;
use codex_models_manager::model_info::model_info_from_slug;
use codex_state::Phase2InputSelection;
use core_test_support::PathExt;
use pretty_assertions::assert_eq;
use tempfile::tempdir;
use tokio::fs as tokio_fs;
#[test]
fn build_stage_one_input_message_truncates_rollout_using_model_context_window() {
let input = format!("{}{}{}", "a".repeat(700_000), "middle", "z".repeat(700_000));
let mut model_info = model_info_from_slug("gpt-5.3-codex");
model_info.context_window = Some(123_000);
let expected_rollout_token_limit = usize::try_from(
((123_000_i64 * model_info.effective_context_window_percent) / 100)
* phase_one::CONTEXT_WINDOW_PERCENT
/ 100,
)
.unwrap();
let expected_truncated = truncate_text(
&input,
TruncationPolicy::Tokens(expected_rollout_token_limit),
);
let message = build_stage_one_input_message(
&model_info,
Path::new("/tmp/rollout.jsonl"),
Path::new("/tmp"),
&input,
)
.unwrap();
assert!(expected_truncated.contains("tokens truncated"));
assert!(expected_truncated.starts_with('a'));
assert!(expected_truncated.ends_with('z'));
assert!(message.contains(&expected_truncated));
}
#[test]
fn build_stage_one_input_message_uses_default_limit_when_model_context_window_missing() {
let input = format!("{}{}{}", "a".repeat(700_000), "middle", "z".repeat(700_000));
let mut model_info = model_info_from_slug("gpt-5.3-codex");
model_info.context_window = None;
model_info.max_context_window = None;
let expected_truncated = truncate_text(
&input,
TruncationPolicy::Tokens(phase_one::DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT),
);
let message = build_stage_one_input_message(
&model_info,
Path::new("/tmp/rollout.jsonl"),
Path::new("/tmp"),
&input,
)
.unwrap();
assert!(message.contains(&expected_truncated));
}
#[test]
fn build_consolidation_prompt_includes_removed_extension_resources() {
let temp = tempdir().unwrap();
let memory_root = temp.path().join("memories");
std::fs::create_dir_all(temp.path().join("memories_extensions")).unwrap();
let removed_extension_resources = vec![
RemovedExtensionResource {
extension: "chronicle".to_string(),
resource_path: "resources/2026-04-06T11-59-59-abcd-10min-old.md".to_string(),
},
RemovedExtensionResource {
extension: "chronicle".to_string(),
resource_path: "resources/2026-04-07T12-00-00-abcd-10min-cutoff.md".to_string(),
},
];
let prompt = build_consolidation_prompt(
&memory_root,
&Phase2InputSelection::default(),
&removed_extension_resources,
);
assert!(prompt.contains("Memory extension resources removed by retention pruning:"));
assert!(prompt.contains("- retention window: 7 days"));
assert!(prompt.contains("- extension: chronicle"));
assert!(prompt.contains(" - resources/2026-04-06T11-59-59-abcd-10min-old.md"));
assert!(prompt.contains(" - resources/2026-04-07T12-00-00-abcd-10min-cutoff.md"));
assert!(prompt.contains("extension-specific deletion diff"));
}
#[tokio::test]
async fn build_memory_tool_developer_instructions_renders_embedded_template() {
let temp = tempdir().unwrap();
let codex_home = temp.path().abs();
let memories_dir = codex_home.join("memories");
tokio_fs::create_dir_all(&memories_dir).await.unwrap();
tokio_fs::write(
memories_dir.join("memory_summary.md"),
"Short memory summary for tests.",
)
.await
.unwrap();
let instructions = build_memory_tool_developer_instructions(&codex_home)
.await
.unwrap();
assert!(instructions.contains(&format!(
"- {}/memory_summary.md (already provided below; do NOT open again)",
memories_dir.display()
)));
assert!(instructions.contains("Short memory summary for tests."));
assert_eq!(
instructions
.matches("========= MEMORY_SUMMARY BEGINS =========")
.count(),
1
);
}
+323 -73
View File
@@ -1,18 +1,19 @@
use super::control::clear_memory_root_contents;
use super::storage::rebuild_raw_memories_file_from_memories;
use super::storage::sync_rollout_summaries_from_memories;
use crate::memories::ensure_layout;
use crate::memories::memory_root;
use crate::memories::raw_memories_file;
use crate::memories::rollout_summaries_dir;
use chrono::TimeZone;
use chrono::Utc;
use codex_config::types::DEFAULT_MEMORIES_MAX_RAW_MEMORIES_FOR_CONSOLIDATION;
use codex_memories_write::clear_memory_roots_contents;
use codex_memories_write::ensure_layout;
use codex_memories_write::memory_root;
use codex_memories_write::raw_memories_file;
use codex_memories_write::rebuild_raw_memories_file_from_memories;
use codex_memories_write::rollout_summaries_dir;
use codex_memories_write::sync_rollout_summaries_from_memories;
use codex_protocol::ThreadId;
use codex_state::Stage1Output;
use codex_utils_absolute_path::AbsolutePathBuf;
use pretty_assertions::assert_eq;
use serde_json::Value;
use std::path::Path;
use std::path::PathBuf;
use tempfile::tempdir;
@@ -67,7 +68,7 @@ fn stage_one_output_schema_requires_rollout_slug_and_keeps_it_nullable() {
#[tokio::test]
async fn clear_memory_root_contents_preserves_root_directory() {
let dir = tempdir().expect("tempdir");
let root = dir.path().join("memory");
let root = dir.path().join("memories");
let nested_dir = root.join("rollout_summaries");
tokio::fs::create_dir_all(&nested_dir)
.await
@@ -79,7 +80,7 @@ async fn clear_memory_root_contents_preserves_root_directory() {
.await
.expect("write rollout summary");
clear_memory_root_contents(&root)
clear_memory_roots_contents(dir.path())
.await
.expect("clear memory root contents");
@@ -115,10 +116,10 @@ async fn clear_memory_root_contents_rejects_symlinked_root() {
.await
.expect("write target file");
let root = dir.path().join("memory");
let root = dir.path().join("memories");
std::os::unix::fs::symlink(&target, &root).expect("create memory root symlink");
let err = clear_memory_root_contents(&root)
let err = clear_memory_roots_contents(dir.path())
.await
.expect_err("symlinked memory root should be rejected");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
@@ -130,6 +131,56 @@ async fn clear_memory_root_contents_rejects_symlinked_root() {
);
}
struct ConsolidatedOutputPaths {
memory_index: PathBuf,
memory_summary: PathBuf,
skill: PathBuf,
}
async fn write_consolidated_outputs(root: &Path) -> ConsolidatedOutputPaths {
let paths = ConsolidatedOutputPaths {
memory_index: root.join("MEMORY.md"),
memory_summary: root.join("memory_summary.md"),
skill: root.join("skills/demo/SKILL.md"),
};
tokio::fs::write(&paths.memory_index, "consolidated memory index\n")
.await
.expect("write memory index");
tokio::fs::write(&paths.memory_summary, "consolidated memory summary\n")
.await
.expect("write memory summary");
tokio::fs::create_dir_all(paths.skill.parent().expect("skill parent"))
.await
.expect("create skill dir");
tokio::fs::write(&paths.skill, "consolidated skill\n")
.await
.expect("write skill");
paths
}
async fn assert_consolidated_outputs_exist(paths: &ConsolidatedOutputPaths, context: &str) {
assert!(
tokio::fs::try_exists(&paths.memory_index)
.await
.expect("check memory index existence"),
"{context} should leave MEMORY.md untouched"
);
assert!(
tokio::fs::try_exists(&paths.memory_summary)
.await
.expect("check memory summary existence"),
"{context} should leave memory_summary.md untouched"
);
assert!(
tokio::fs::try_exists(&paths.skill)
.await
.expect("check skill existence"),
"{context} should leave skills untouched"
);
}
#[tokio::test]
async fn sync_rollout_summaries_and_raw_memories_file_keeps_latest_memories_only() {
let dir = tempdir().expect("tempdir");
@@ -236,6 +287,46 @@ async fn sync_rollout_summaries_and_raw_memories_file_keeps_latest_memories_only
assert!(rollout_path_pos < file_pos);
}
#[tokio::test]
async fn sync_empty_inputs_preserves_consolidated_outputs() {
let dir = tempdir().expect("tempdir");
let root = dir.path().join("memory");
ensure_layout(&root).await.expect("ensure layout");
let stale_rollout_summary_path = rollout_summaries_dir(&root).join("stale.md");
tokio::fs::write(&stale_rollout_summary_path, "stale summary\n")
.await
.expect("write stale rollout summary");
let outputs = write_consolidated_outputs(&root).await;
sync_rollout_summaries_from_memories(
&root,
&[],
DEFAULT_MEMORIES_MAX_RAW_MEMORIES_FOR_CONSOLIDATION,
)
.await
.expect("sync empty rollout summaries");
rebuild_raw_memories_file_from_memories(
&root,
&[],
DEFAULT_MEMORIES_MAX_RAW_MEMORIES_FOR_CONSOLIDATION,
)
.await
.expect("rebuild empty raw memories");
assert!(
!tokio::fs::try_exists(&stale_rollout_summary_path)
.await
.expect("check stale rollout summary existence"),
"empty sync should prune stale rollout summaries"
);
let raw_memories = tokio::fs::read_to_string(raw_memories_file(&root))
.await
.expect("read raw memories");
assert_eq!(raw_memories, "# Raw Memories\n\nNo raw memories yet.\n");
assert_consolidated_outputs_exist(&outputs, "empty sync").await;
}
#[tokio::test]
async fn sync_rollout_summaries_uses_timestamp_hash_and_sanitized_slug_filename() {
let dir = tempdir().expect("tempdir");
@@ -418,10 +509,7 @@ mod phase2 {
use crate::agent::AgentControl;
use crate::config::Config;
use crate::config::test_config;
use crate::memories::memory_root;
use crate::memories::phase2;
use crate::memories::raw_memories_file;
use crate::memories::rollout_summaries_dir;
use crate::session::session::Session;
use crate::session::tests::make_session_and_context;
use chrono::Duration as ChronoDuration;
@@ -430,6 +518,12 @@ mod phase2 {
use codex_config::types::McpServerConfig;
use codex_features::Feature;
use codex_login::CodexAuth;
use codex_memories_write::memory_root;
use codex_memories_write::raw_memories_file;
use codex_memories_write::rebuild_raw_memories_file_from_memories;
use codex_memories_write::rollout_summaries_dir;
use codex_memories_write::sync_rollout_summaries_from_memories;
use codex_memories_write::workspace::prepare_memory_workspace;
use codex_protocol::AgentPath;
use codex_protocol::ThreadId;
use codex_protocol::models::PermissionProfile;
@@ -520,7 +614,7 @@ mod phase2 {
}
}
async fn seed_stage1_output(&self, source_updated_at: i64) {
async fn seed_stage1_output(&self, source_updated_at: i64) -> ThreadId {
let thread_id = ThreadId::new();
let mut metadata_builder = ThreadMetadataBuilder::new(
thread_id,
@@ -569,6 +663,7 @@ mod phase2 {
.expect("mark stage-1 success"),
"stage-1 success should enqueue global consolidation"
);
thread_id
}
async fn shutdown_threads(&self) {
@@ -613,16 +708,85 @@ mod phase2 {
}
#[tokio::test]
async fn dispatch_skips_when_global_job_is_not_dirty() {
async fn dispatch_skips_when_memory_workspace_is_not_dirty() {
let harness = DispatchHarness::new().await;
let root = memory_root(&harness.config.codex_home);
rebuild_raw_memories_file_from_memories(
&root,
&[],
/*max_raw_memories_for_consolidation*/ 0,
)
.await
.expect("write empty raw memories baseline");
let outputs = super::write_consolidated_outputs(&root).await;
prepare_memory_workspace(&root)
.await
.expect("commit empty memory workspace as baseline");
phase2::run(&harness.session, Arc::clone(&harness.config)).await;
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
super::assert_consolidated_outputs_exist(&outputs, "clean no-input phase2").await;
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 0);
}
#[tokio::test]
async fn dispatch_uses_git_dirty_state_without_db_dirty_watermark() {
let harness = DispatchHarness::new().await;
let root = memory_root(&harness.config.codex_home);
rebuild_raw_memories_file_from_memories(
&root,
&[],
/*max_raw_memories_for_consolidation*/ 0,
)
.await
.expect("write empty raw memories baseline");
prepare_memory_workspace(&root)
.await
.expect("commit empty memory workspace as baseline");
let extension_resource = root
.join("extensions")
.join("chronicle")
.join("resources")
.join("2026-04-22T12-00-00-abcd-10min-memory.md");
tokio::fs::create_dir_all(
extension_resource
.parent()
.expect("extension resource parent"),
)
.await
.expect("create extension resource dir");
tokio::fs::write(
root.join("extensions/chronicle/instructions.md"),
"instructions\n",
)
.await
.expect("write extension instructions");
tokio::fs::write(&extension_resource, "extension memory\n")
.await
.expect("write extension resource");
phase2::run(&harness.session, Arc::clone(&harness.config)).await;
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 1);
let workspace_diff = tokio::fs::read_to_string(root.join("phase2_workspace_diff.md"))
.await
.expect("read workspace diff");
assert!(
workspace_diff.contains("- A extensions/chronicle/instructions.md"),
"git-only extension instructions should dirty phase2: {workspace_diff}"
);
assert!(
workspace_diff.contains("- A extensions/chronicle/resources/"),
"git-only extension resource should dirty phase2: {workspace_diff}"
);
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 1);
harness.shutdown_threads().await;
}
#[tokio::test]
async fn dispatch_skips_when_global_job_is_already_running() {
let harness = DispatchHarness::new().await;
@@ -696,7 +860,8 @@ mod phase2 {
assert!(
matches!(
post_dispatch_claim,
Phase2JobClaimOutcome::SkippedRunning | Phase2JobClaimOutcome::SkippedNotDirty
Phase2JobClaimOutcome::SkippedRunning
| Phase2JobClaimOutcome::SkippedRetryUnavailable
),
"stale-lock dispatch should either keep the reclaimed job running or finish it before re-claim"
);
@@ -835,7 +1000,7 @@ mod phase2 {
}
#[tokio::test]
async fn dispatch_with_empty_stage1_outputs_rebuilds_local_artifacts() {
async fn dispatch_with_empty_stage1_outputs_spawns_for_workspace_changes() {
let harness = DispatchHarness::new().await;
let root = memory_root(&harness.config.codex_home);
let summaries_dir = rollout_summaries_dir(&root);
@@ -851,25 +1016,7 @@ mod phase2 {
tokio::fs::write(&raw_memories_path, "stale raw memories\n")
.await
.expect("write stale raw memories");
let memory_index_path = root.join("MEMORY.md");
tokio::fs::write(&memory_index_path, "stale memory index\n")
.await
.expect("write stale memory index");
let memory_summary_path = root.join("memory_summary.md");
tokio::fs::write(&memory_summary_path, "stale memory summary\n")
.await
.expect("write stale memory summary");
let stale_skill_file = root.join("skills/demo/SKILL.md");
tokio::fs::create_dir_all(
stale_skill_file
.parent()
.expect("skills subdirectory parent should exist"),
)
.await
.expect("create stale skills dir");
tokio::fs::write(&stale_skill_file, "stale skill\n")
.await
.expect("write stale skill");
let outputs = super::write_consolidated_outputs(&root).await;
harness
.state_db
@@ -889,43 +1036,138 @@ mod phase2 {
.await
.expect("read rebuilt raw memories");
pretty_assertions::assert_eq!(raw_memories, "# Raw Memories\n\nNo raw memories yet.\n");
assert!(
!tokio::fs::try_exists(&memory_index_path)
.await
.expect("check memory index existence"),
"empty consolidation should remove stale MEMORY.md"
);
assert!(
!tokio::fs::try_exists(&memory_summary_path)
.await
.expect("check memory summary existence"),
"empty consolidation should remove stale memory_summary.md"
);
assert!(
!tokio::fs::try_exists(&stale_skill_file)
.await
.expect("check stale skill existence"),
"empty consolidation should remove stale skills artifacts"
);
assert!(
!tokio::fs::try_exists(root.join("skills"))
.await
.expect("check skills dir existence"),
"empty consolidation should remove stale skills directory"
);
super::assert_consolidated_outputs_exist(&outputs, "empty consolidation").await;
let next_claim = harness
.state_db
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim global job after empty consolidation success");
pretty_assertions::assert_eq!(next_claim, Phase2JobClaimOutcome::SkippedNotDirty);
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
.expect("claim global job after empty consolidation dispatch");
pretty_assertions::assert_eq!(next_claim, Phase2JobClaimOutcome::SkippedRunning);
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 1);
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 0);
pretty_assertions::assert_eq!(thread_ids.len(), 1);
harness.shutdown_threads().await;
}
#[tokio::test]
async fn dispatch_with_empty_selected_inputs_preserves_consolidated_outputs() {
let harness = DispatchHarness::new().await;
let source_updated_at = Utc::now().timestamp();
let thread_id = harness.seed_stage1_output(source_updated_at).await;
let root = memory_root(&harness.config.codex_home);
let selected = harness
.state_db
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 30)
.await
.expect("load phase2 input selection");
sync_rollout_summaries_from_memories(&root, &selected, selected.len())
.await
.expect("sync selected rollout summaries");
rebuild_raw_memories_file_from_memories(&root, &selected, selected.len())
.await
.expect("sync selected raw memories");
let outputs = super::write_consolidated_outputs(&root).await;
prepare_memory_workspace(&root)
.await
.expect("commit current memory workspace as baseline");
let claim = harness
.state_db
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim global phase2 job");
let Phase2JobClaimOutcome::Claimed {
ownership_token, ..
} = claim
else {
panic!("unexpected phase2 claim outcome: {claim:?}");
};
assert!(
harness
.state_db
.mark_global_phase2_job_succeeded(&ownership_token, source_updated_at, &selected)
.await
.expect("mark phase2 succeeded"),
"phase2 success should update selected baseline"
);
assert!(
harness
.state_db
.mark_thread_memory_mode_polluted(thread_id)
.await
.expect("mark thread polluted"),
"polluted selected thread should enqueue phase2 forgetting"
);
phase2::run(&harness.session, Arc::clone(&harness.config)).await;
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 1);
super::assert_consolidated_outputs_exist(&outputs, "empty selected phase2").await;
let workspace_diff = tokio::fs::read_to_string(root.join("phase2_workspace_diff.md"))
.await
.expect("read workspace diff");
assert!(
workspace_diff.contains("- D rollout_summaries/"),
"empty selected phase2 should surface deleted rollout summaries: {workspace_diff}"
);
assert!(
!workspace_diff.contains("- D MEMORY.md"),
"empty selected phase2 should not delete MEMORY.md directly: {workspace_diff}"
);
assert!(
!workspace_diff.contains("- D memory_summary.md"),
"empty selected phase2 should not delete memory_summary.md directly: {workspace_diff}"
);
assert!(
!workspace_diff.contains("- D skills/demo/SKILL.md"),
"empty selected phase2 should not delete skills directly: {workspace_diff}"
);
harness.shutdown_threads().await;
}
#[tokio::test]
async fn dispatch_with_clean_workspace_rebuilds_selected_phase2_baseline() {
let harness = DispatchHarness::new().await;
let source_updated_at = (Utc::now() - ChronoDuration::days(1)).timestamp();
let thread_id = harness.seed_stage1_output(source_updated_at).await;
let root = memory_root(&harness.config.codex_home);
let selected = harness
.state_db
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 30)
.await
.expect("load phase2 input selection");
sync_rollout_summaries_from_memories(&root, &selected, selected.len())
.await
.expect("sync selected rollout summaries");
rebuild_raw_memories_file_from_memories(&root, &selected, selected.len())
.await
.expect("sync selected raw memories");
prepare_memory_workspace(&root)
.await
.expect("commit current memory workspace as baseline");
phase2::run(&harness.session, Arc::clone(&harness.config)).await;
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
let pruned = harness
.state_db
.prune_stage1_outputs_for_retention(/*max_unused_days*/ 0, /*limit*/ 10)
.await
.expect("prune stage1 outputs after clean phase2");
pretty_assertions::assert_eq!(pruned, 0);
let selected = harness
.state_db
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 30)
.await
.expect("load phase2 input selection after clean workspace success");
pretty_assertions::assert_eq!(selected.len(), 1);
pretty_assertions::assert_eq!(selected[0].thread_id, thread_id);
}
#[tokio::test]
async fn dispatch_marks_job_for_retry_when_sandbox_policy_cannot_be_overridden() {
let harness = DispatchHarness::new().await;
@@ -946,7 +1188,7 @@ mod phase2 {
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim global job after sandbox policy failure");
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedNotDirty);
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedRetryUnavailable);
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 0);
@@ -968,7 +1210,7 @@ mod phase2 {
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim global job after sync failure");
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedNotDirty);
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedRetryUnavailable);
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 0);
@@ -990,7 +1232,7 @@ mod phase2 {
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim global job after rebuild failure");
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedNotDirty);
pretty_assertions::assert_eq!(retry_claim, Phase2JobClaimOutcome::SkippedRetryUnavailable);
pretty_assertions::assert_eq!(harness.user_input_ops_count(), 0);
let thread_ids = harness.manager.list_thread_ids().await;
pretty_assertions::assert_eq!(thread_ids.len(), 0);
@@ -1067,14 +1309,14 @@ mod phase2 {
let chronicle_resources = config
.codex_home
.join("memories_extensions/chronicle/resources");
.join("memories/extensions/chronicle/resources");
tokio::fs::create_dir_all(&chronicle_resources)
.await
.expect("create chronicle resources");
tokio::fs::write(
config
.codex_home
.join("memories_extensions/chronicle/instructions.md"),
.join("memories/extensions/chronicle/instructions.md"),
"instructions",
)
.await
@@ -1095,14 +1337,22 @@ mod phase2 {
.expect("claim global job after spawn failure");
pretty_assertions::assert_eq!(
retry_claim,
Phase2JobClaimOutcome::SkippedNotDirty,
Phase2JobClaimOutcome::SkippedRetryUnavailable,
"spawn failures should leave the job in retry backoff instead of running"
);
assert!(
tokio::fs::try_exists(&old_file)
!tokio::fs::try_exists(&old_file)
.await
.expect("check old extension resource"),
"spawn failures should not prune extension resources before retry"
"old extension resources should still be pruned on failed phase2 attempts"
);
let workspace_diff =
tokio::fs::read_to_string(config.codex_home.join("memories/phase2_workspace_diff.md"))
.await
.expect("read workspace diff");
assert!(
workspace_diff.contains("- D extensions/chronicle/resources/"),
"spawn failures should keep a retryable workspace diff: {workspace_diff}"
);
}
}
@@ -1,38 +1,17 @@
use crate::tools::context::ToolInvocation;
use crate::tools::context::ToolPayload;
use crate::tools::handlers::unified_exec::ExecCommandArgs;
use codex_memories_read::usage::MEMORIES_USAGE_METRIC;
use codex_memories_read::usage::memories_usage_kinds_from_command;
use codex_protocol::models::ShellCommandToolCallParams;
use codex_protocol::models::ShellToolCallParams;
use codex_protocol::parse_command::ParsedCommand;
use codex_shell_command::is_safe_command::is_known_safe_command;
use codex_shell_command::parse_command::parse_command;
use std::path::PathBuf;
const MEMORIES_USAGE_METRIC: &str = "codex.memories.usage";
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
enum MemoriesUsageKind {
MemoryMd,
MemorySummary,
RawMemories,
RolloutSummaries,
Skills,
}
impl MemoriesUsageKind {
fn as_tag(self) -> &'static str {
match self {
Self::MemoryMd => "memory_md",
Self::MemorySummary => "memory_summary",
Self::RawMemories => "raw_memories",
Self::RolloutSummaries => "rollout_summaries",
Self::Skills => "skills",
}
}
}
pub(crate) async fn emit_metric_for_tool_read(invocation: &ToolInvocation, success: bool) {
let kinds = memories_usage_kinds_from_invocation(invocation).await;
let Some((command, _)) = shell_command_for_invocation(invocation) else {
return;
};
let kinds = memories_usage_kinds_from_command(&command);
if kinds.is_empty() {
return;
}
@@ -52,27 +31,6 @@ pub(crate) async fn emit_metric_for_tool_read(invocation: &ToolInvocation, succe
}
}
async fn memories_usage_kinds_from_invocation(
invocation: &ToolInvocation,
) -> Vec<MemoriesUsageKind> {
let Some((command, _)) = shell_command_for_invocation(invocation) else {
return Vec::new();
};
if !is_known_safe_command(&command) {
return Vec::new();
}
let parsed_commands = parse_command(&command);
parsed_commands
.into_iter()
.filter_map(|command| match command {
ParsedCommand::Read { path, .. } => get_memory_kind(path.display().to_string()),
ParsedCommand::Search { path, .. } => path.and_then(get_memory_kind),
ParsedCommand::ListFiles { .. } | ParsedCommand::Unknown { .. } => None,
})
.collect()
}
fn shell_command_for_invocation(invocation: &ToolInvocation) -> Option<(Vec<String>, PathBuf)> {
let ToolPayload::Function { arguments } = &invocation.payload else {
return None;
@@ -129,19 +87,3 @@ fn shell_command_for_invocation(invocation: &ToolInvocation) -> Option<(Vec<Stri
(Some(_), _) | (None, _) => None,
}
}
fn get_memory_kind(path: String) -> Option<MemoriesUsageKind> {
if path.contains("memories/MEMORY.md") {
Some(MemoriesUsageKind::MemoryMd)
} else if path.contains("memories/memory_summary.md") {
Some(MemoriesUsageKind::MemorySummary)
} else if path.contains("memories/raw_memories.md") {
Some(MemoriesUsageKind::RawMemories)
} else if path.contains("memories/rollout_summaries/") {
Some(MemoriesUsageKind::RolloutSummaries)
} else if path.contains("memories/skills/") {
Some(MemoriesUsageKind::Skills)
} else {
None
}
}
+1 -1
View File
@@ -29,7 +29,7 @@ pub async fn build_prompt_input(
config.ephemeral = true;
let auth_manager =
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false);
AuthManager::shared_from_config(&config, /*enable_codex_api_key_env*/ false).await;
let local_runtime_paths = ExecServerRuntimePaths::from_optional_paths(
config.codex_self_exe.clone(),
+2 -2
View File
@@ -681,7 +681,7 @@ pub async fn drop_memories(sess: &Arc<Session>, config: &Arc<Config>, sub_id: St
errors.push("state db unavailable; memory rows were not cleared".to_string());
}
if let Err(err) = crate::memories::clear_memory_roots_contents(&config.codex_home).await {
if let Err(err) = codex_memories_write::clear_memory_roots_contents(&config.codex_home).await {
errors.push(format!(
"failed clearing memory directories under {}: {err}",
config.codex_home.display()
@@ -689,7 +689,7 @@ pub async fn drop_memories(sess: &Arc<Session>, config: &Arc<Config>, sub_id: St
}
if errors.is_empty() {
let memory_root = crate::memories::memory_root(&config.codex_home);
let memory_root = codex_memories_write::memory_root(&config.codex_home);
sess.send_event_raw(Event {
id: sub_id,
msg: EventMsg::Warning(WarningEvent {
+1 -1
View File
@@ -3323,7 +3323,7 @@ fn errors_to_info(errors: &[SkillError]) -> Vec<SkillErrorInfo> {
.collect()
}
use crate::memories::prompts::build_memory_tool_developer_instructions;
use codex_memories_read::build_memory_tool_developer_instructions;
#[cfg(test)]
pub(crate) mod tests;
+16 -2
View File
@@ -194,7 +194,12 @@ impl TurnContext {
.with_spawn_agent_usage_hint_text(config.multi_agent_v2.usage_hint_text.clone())
.with_hide_spawn_agent_metadata(config.multi_agent_v2.hide_spawn_agent_metadata)
.with_goal_tools_allowed(self.tools_config.goal_tools)
.with_max_concurrent_threads_per_session(config.agent_max_threads)
.with_max_concurrent_threads_per_session(
config
.features
.enabled(Feature::MultiAgentV2)
.then_some(config.multi_agent_v2.max_concurrent_threads_per_session),
)
.with_agent_type_description(crate::agent::role::spawn_tool_spec::build(
&config.agent_roles,
));
@@ -459,7 +464,16 @@ impl Session {
.with_spawn_agent_usage_hint_text(per_turn_config.multi_agent_v2.usage_hint_text.clone())
.with_hide_spawn_agent_metadata(per_turn_config.multi_agent_v2.hide_spawn_agent_metadata)
.with_goal_tools_allowed(goal_tools_supported)
.with_max_concurrent_threads_per_session(per_turn_config.agent_max_threads)
.with_max_concurrent_threads_per_session(
per_turn_config
.features
.enabled(Feature::MultiAgentV2)
.then_some(
per_turn_config
.multi_agent_v2
.max_concurrent_threads_per_session,
),
)
.with_agent_type_description(crate::agent::role::spawn_tool_spec::build(
&per_turn_config.agent_roles,
));
+2 -2
View File
@@ -11,13 +11,13 @@ use tokio_util::sync::CancellationToken;
use crate::context::ContextualUserFragment;
use crate::context::ImageGenerationInstructions;
use crate::function_tool::FunctionCallError;
use crate::memories::citations::parse_memory_citation;
use crate::memories::citations::thread_ids_from_memory_citation;
use crate::parse_turn_item;
use crate::session::session::Session;
use crate::session::turn_context::TurnContext;
use crate::tools::parallel::ToolCallRuntime;
use crate::tools::router::ToolRouter;
use codex_memories_read::citations::parse_memory_citation;
use codex_memories_read::citations::thread_ids_from_memory_citation;
use codex_protocol::error::CodexErr;
use codex_protocol::error::Result;
use codex_protocol::models::FunctionCallOutputBody;
@@ -534,6 +534,11 @@ async fn build_runner_options(
"agent depth limit reached; this session cannot spawn more subagents".to_string(),
));
}
if turn.config.agent_max_threads == Some(0) {
return Err(FunctionCallError::RespondToModel(
"agent thread limit reached; this session cannot spawn more subagents".to_string(),
));
}
let max_concurrency =
normalize_concurrency(requested_concurrency, turn.config.agent_max_threads);
let base_instructions = session.get_base_instructions().await;
+1 -1
View File
@@ -8,7 +8,7 @@ use crate::goals::GoalRuntimeEvent;
use crate::hook_runtime::record_additional_contexts;
use crate::hook_runtime::run_post_tool_use_hooks;
use crate::hook_runtime::run_pre_tool_use_hooks;
use crate::memories::usage::emit_metric_for_tool_read;
use crate::memory_usage::emit_metric_for_tool_read;
use crate::sandbox_tags::permission_profile_policy_tag;
use crate::sandbox_tags::permission_profile_sandbox_tag;
use crate::session::turn_context::TurnContext;
+2 -1
View File
@@ -1091,7 +1091,8 @@ async fn prefers_apikey_when_config_prefers_apikey_even_with_chatgpt_tokens() {
config.model_provider = model_provider;
let auth_manager =
match CodexAuth::from_auth_storage(codex_home.path(), AuthCredentialsStoreMode::File) {
match CodexAuth::from_auth_storage(codex_home.path(), AuthCredentialsStoreMode::File).await
{
Ok(Some(auth)) => codex_core::test_support::auth_manager_from_auth(auth),
Ok(None) => panic!("No CodexAuth found in codex_home"),
Err(e) => panic!("Failed to load CodexAuth: {e}"),
+52 -292
View File
@@ -2,6 +2,8 @@ use anyhow::Result;
use chrono::Duration as ChronoDuration;
use chrono::Utc;
use codex_features::Feature;
use codex_git_utils::diff_since_latest_init;
use codex_git_utils::reset_git_repository;
use codex_protocol::ThreadId;
use codex_protocol::protocol::EventMsg;
use codex_protocol::protocol::Op;
@@ -11,9 +13,7 @@ use core_test_support::responses::ResponsesRequest;
use core_test_support::responses::ev_assistant_message;
use core_test_support::responses::ev_completed;
use core_test_support::responses::ev_response_created;
use core_test_support::responses::ev_web_search_call_done;
use core_test_support::responses::mount_sse_once;
use core_test_support::responses::mount_sse_sequence;
use core_test_support::responses::sse;
use core_test_support::responses::start_mock_server;
use core_test_support::test_codex::TestCodex;
@@ -27,13 +27,14 @@ use tokio::time::Duration;
use tokio::time::Instant;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn memories_startup_phase2_tracks_added_and_removed_inputs_across_runs() -> Result<()> {
async fn memories_startup_phase2_tracks_workspace_diff_across_runs() -> Result<()> {
let server = start_mock_server().await;
let home = Arc::new(TempDir::new()?);
let db = init_state_db(&home).await?;
let memory_root = home.path().join("memories");
let now = Utc::now();
let thread_a = seed_stage1_output(
let _thread_a = seed_stage1_output(
db.as_ref(),
home.path(),
now - ChronoDuration::hours(2),
@@ -43,53 +44,21 @@ async fn memories_startup_phase2_tracks_added_and_removed_inputs_across_runs() -
)
.await?;
let first_phase2 = mount_sse_once(
&server,
sse(vec![
ev_response_created("resp-phase2-1"),
ev_assistant_message("msg-phase2-1", "phase2 complete"),
ev_completed("resp-phase2-1"),
]),
let rollout_summaries_root = memory_root.join("rollout_summaries");
tokio::fs::create_dir_all(&rollout_summaries_root).await?;
tokio::fs::write(
memory_root.join("raw_memories.md"),
"# Raw Memories\n\nraw memory A\n",
)
.await;
.await?;
tokio::fs::write(
rollout_summaries_root.join("rollout-a.md"),
"git_branch: branch-rollout-a\n\nrollout summary A\n",
)
.await?;
reset_git_repository(&memory_root).await?;
let first = build_test_codex(&server, home.clone()).await?;
let first_request = wait_for_single_request(&first_phase2).await;
let first_prompt = phase2_prompt_text(&first_request);
assert!(
first_prompt.contains("- selected inputs this run: 1"),
"expected selected count in first prompt: {first_prompt}"
);
assert!(
first_prompt.contains("- newly added since the last successful Phase 2 run: 1"),
"expected added count in first prompt: {first_prompt}"
);
assert!(
first_prompt.contains("- removed from the last successful Phase 2 run: 0"),
"expected removed count in first prompt: {first_prompt}"
);
assert!(
first_prompt.contains(&format!("- [added] thread_id={thread_a},")),
"expected thread A to be marked added: {first_prompt}"
);
assert!(
first_prompt.contains("Removed from the last successful Phase 2 selection:\n- none"),
"expected no removed items in first prompt: {first_prompt}"
);
wait_for_phase2_success(db.as_ref(), thread_a).await?;
let memory_root = home.path().join("memories");
let raw_memories = tokio::fs::read_to_string(memory_root.join("raw_memories.md")).await?;
assert!(raw_memories.contains("raw memory A"));
assert!(!raw_memories.contains("raw memory B"));
let rollout_summaries = read_rollout_summary_bodies(&memory_root).await?;
assert_eq!(rollout_summaries.len(), 1);
assert!(rollout_summaries[0].contains("rollout summary A"));
assert!(rollout_summaries[0].contains("git_branch: branch-rollout-a"));
shutdown_test_codex(&first).await?;
let thread_b = seed_stage1_output(
let _thread_b = seed_stage1_output(
db.as_ref(),
home.path(),
now - ChronoDuration::hours(1),
@@ -99,46 +68,30 @@ async fn memories_startup_phase2_tracks_added_and_removed_inputs_across_runs() -
)
.await?;
let second_phase2 = mount_sse_once(
let phase2 = mount_sse_once(
&server,
sse(vec![
ev_response_created("resp-phase2-2"),
ev_assistant_message("msg-phase2-2", "phase2 complete"),
ev_completed("resp-phase2-2"),
ev_response_created("resp-phase2"),
ev_assistant_message("msg-phase2", "phase2 complete"),
ev_completed("resp-phase2"),
]),
)
.await;
let second = build_test_codex(&server, home.clone()).await?;
let second_request = wait_for_single_request(&second_phase2).await;
let second_prompt = phase2_prompt_text(&second_request);
let codex = build_test_codex(&server, home.clone()).await?;
let request = wait_for_single_request(&phase2).await;
let prompt = phase2_prompt_text(&request);
assert!(
second_prompt.contains("- selected inputs this run: 1"),
"expected selected count in second prompt: {second_prompt}"
);
assert!(
second_prompt.contains("- newly added since the last successful Phase 2 run: 1"),
"expected added count in second prompt: {second_prompt}"
);
assert!(
second_prompt.contains("- removed from the last successful Phase 2 run: 1"),
"expected removed count in second prompt: {second_prompt}"
);
assert!(
second_prompt.contains(&format!("- [added] thread_id={thread_b},")),
"expected thread B to be marked added: {second_prompt}"
);
assert!(
second_prompt.contains(&format!("- thread_id={thread_a},")),
"expected thread A to be marked removed: {second_prompt}"
prompt.contains("phase2_workspace_diff.md"),
"expected workspace diff file in prompt: {prompt}"
);
wait_for_phase2_success(db.as_ref(), thread_b).await?;
wait_for_phase2_workspace_reset(&memory_root).await?;
let raw_memories = tokio::fs::read_to_string(memory_root.join("raw_memories.md")).await?;
assert!(raw_memories.contains("raw memory B"));
assert!(raw_memories.contains("raw memory A"));
assert!(!raw_memories.contains("raw memory A"));
let rollout_summaries = read_rollout_summary_bodies(&memory_root).await?;
assert_eq!(rollout_summaries.len(), 2);
assert_eq!(rollout_summaries.len(), 1);
assert!(
rollout_summaries
.iter()
@@ -152,20 +105,20 @@ async fn memories_startup_phase2_tracks_added_and_removed_inputs_across_runs() -
assert!(
rollout_summaries
.iter()
.any(|summary| summary.contains("rollout summary A"))
.all(|summary| !summary.contains("rollout summary A"))
);
shutdown_test_codex(&second).await?;
shutdown_test_codex(&codex).await?;
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn memories_startup_phase2_prunes_old_extension_resources_and_reports_them() -> Result<()> {
async fn memories_startup_phase2_prunes_old_extension_resources() -> Result<()> {
let server = start_mock_server().await;
let home = Arc::new(TempDir::new()?);
let db = init_state_db(&home).await?;
let now = Utc::now();
let thread_id = seed_stage1_output(
let _thread_id = seed_stage1_output(
db.as_ref(),
home.path(),
now - ChronoDuration::hours(1),
@@ -175,11 +128,11 @@ async fn memories_startup_phase2_prunes_old_extension_resources_and_reports_them
)
.await?;
let chronicle_resources = home.path().join("memories_extensions/chronicle/resources");
let chronicle_resources = home.path().join("memories/extensions/chronicle/resources");
tokio::fs::create_dir_all(&chronicle_resources).await?;
tokio::fs::write(
home.path()
.join("memories_extensions/chronicle/instructions.md"),
.join("memories/extensions/chronicle/instructions.md"),
"instructions",
)
.await?;
@@ -210,23 +163,11 @@ async fn memories_startup_phase2_prunes_old_extension_resources_and_reports_them
let prompt = phase2_prompt_text(&request);
assert!(
prompt.contains("Memory extension resources removed by retention pruning:"),
"expected extension resource prune report in prompt: {prompt}"
);
assert!(
prompt.contains("- retention window: 7 days"),
"expected retention window in prompt: {prompt}"
);
assert!(
prompt.contains("- extension: chronicle"),
"expected extension name in prompt: {prompt}"
);
assert!(
prompt.contains(&format!(" - resources/{old_file_name}")),
"expected old resource in prompt: {prompt}"
prompt.contains("phase2_workspace_diff.md"),
"expected workspace diff file in prompt: {prompt}"
);
wait_for_phase2_success(db.as_ref(), thread_id).await?;
wait_for_phase2_workspace_reset(&home.path().join("memories")).await?;
wait_for_file_removed(&old_file).await?;
assert!(
!tokio::fs::try_exists(&old_file).await?,
@@ -242,8 +183,8 @@ async fn memories_startup_phase2_prunes_old_extension_resources_and_reports_them
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn memories_startup_phase2_processes_old_extension_resources_without_stage1_input()
-> Result<()> {
async fn memories_startup_phase2_prunes_old_extension_resources_without_stage1_input() -> Result<()>
{
let server = start_mock_server().await;
let home = Arc::new(TempDir::new()?);
let db = init_state_db(&home).await?;
@@ -251,11 +192,11 @@ async fn memories_startup_phase2_processes_old_extension_resources_without_stage
.await?;
let now = Utc::now();
let chronicle_resources = home.path().join("memories_extensions/chronicle/resources");
let chronicle_resources = home.path().join("memories/extensions/chronicle/resources");
tokio::fs::create_dir_all(&chronicle_resources).await?;
tokio::fs::write(
home.path()
.join("memories_extensions/chronicle/instructions.md"),
.join("memories/extensions/chronicle/instructions.md"),
"instructions",
)
.await?;
@@ -281,189 +222,16 @@ async fn memories_startup_phase2_processes_old_extension_resources_without_stage
let prompt = phase2_prompt_text(&request);
assert!(
prompt.contains("- selected inputs this run: 0"),
"expected no selected raw inputs in prompt: {prompt}"
);
assert!(
prompt.contains("Memory extension resources removed by retention pruning:"),
"expected extension resource prune report in prompt: {prompt}"
);
assert!(
prompt.contains(&format!(" - resources/{old_file_name}")),
"expected old resource in prompt: {prompt}"
prompt.contains("phase2_workspace_diff.md"),
"expected workspace diff file in prompt: {prompt}"
);
wait_for_file_removed(&old_file).await?;
wait_for_phase2_workspace_reset(&home.path().join("memories")).await?;
shutdown_test_codex(&codex).await?;
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn web_search_pollution_moves_selected_thread_into_removed_phase2_inputs() -> Result<()> {
let server = start_mock_server().await;
let home = Arc::new(TempDir::new()?);
let db = init_state_db(&home).await?;
let mut initial_builder = test_codex().with_home(home.clone()).with_config(|config| {
config
.features
.enable(Feature::Sqlite)
.expect("test config should allow feature update");
config
.features
.enable(Feature::MemoryTool)
.expect("test config should allow feature update");
config.memories.max_raw_memories_for_consolidation = 1;
config.memories.disable_on_external_context = true;
});
let initial = initial_builder.build(&server).await?;
mount_sse_once(
&server,
sse(vec![
ev_response_created("resp-initial-1"),
ev_assistant_message("msg-initial-1", "initial turn complete"),
ev_completed("resp-initial-1"),
]),
)
.await;
initial.submit_turn("hello before memories").await?;
let rollout_path = initial
.session_configured
.rollout_path
.clone()
.expect("rollout path");
let thread_id = initial.session_configured.session_id;
let updated_at = {
let deadline = Instant::now() + Duration::from_secs(10);
loop {
if let Some(metadata) = db.get_thread(thread_id).await? {
break metadata.updated_at;
}
assert!(
Instant::now() < deadline,
"timed out waiting for thread metadata for {thread_id}"
);
tokio::time::sleep(Duration::from_millis(50)).await;
}
};
seed_stage1_output_for_existing_thread(
db.as_ref(),
thread_id,
updated_at.timestamp(),
"raw memory seeded for web search pollution",
"rollout summary seeded for web search pollution",
Some("pollution-rollout"),
)
.await?;
shutdown_test_codex(&initial).await?;
let responses = mount_sse_sequence(
&server,
vec![
sse(vec![
ev_response_created("resp-phase2-1"),
ev_assistant_message("msg-phase2-1", "phase2 complete"),
ev_completed("resp-phase2-1"),
]),
sse(vec![
ev_response_created("resp-web-1"),
ev_web_search_call_done("ws-1", "completed", "weather seattle"),
ev_completed("resp-web-1"),
]),
],
)
.await;
let mut resumed_builder = test_codex().with_home(home.clone()).with_config(|config| {
config
.features
.enable(Feature::Sqlite)
.expect("test config should allow feature update");
config
.features
.enable(Feature::MemoryTool)
.expect("test config should allow feature update");
config.memories.max_raw_memories_for_consolidation = 1;
config.memories.disable_on_external_context = true;
});
let resumed = resumed_builder
.resume(&server, home.clone(), rollout_path.clone())
.await?;
let first_phase2_request = wait_for_request(&responses, /*expected_count*/ 1)
.await
.remove(0);
let first_phase2_prompt = phase2_prompt_text(&first_phase2_request);
assert!(
first_phase2_prompt.contains("- selected inputs this run: 1"),
"expected seeded thread to be selected before pollution: {first_phase2_prompt}"
);
assert!(
first_phase2_prompt.contains("- newly added since the last successful Phase 2 run: 1"),
"expected seeded thread to be added before pollution: {first_phase2_prompt}"
);
assert!(
first_phase2_prompt.contains(&format!("- [added] thread_id={thread_id},")),
"expected selected thread in first phase2 prompt: {first_phase2_prompt}"
);
wait_for_phase2_success(db.as_ref(), thread_id).await?;
resumed
.submit_turn("search the web for weather seattle")
.await?;
assert_eq!(
{
let deadline = Instant::now() + Duration::from_secs(10);
loop {
let memory_mode = db.get_thread_memory_mode(thread_id).await?;
if memory_mode.as_deref() == Some("polluted") {
break memory_mode;
}
assert!(
Instant::now() < deadline,
"timed out waiting for polluted memory mode for {thread_id}"
);
tokio::time::sleep(Duration::from_millis(50)).await;
}
}
.as_deref(),
Some("polluted")
);
let selection = {
let deadline = Instant::now() + Duration::from_secs(10);
loop {
let selection = db
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 30)
.await?;
if selection.selected.is_empty()
&& selection.retained_thread_ids.is_empty()
&& selection.removed.len() == 1
&& selection.removed[0].thread_id == thread_id
{
break selection;
}
assert!(
Instant::now() < deadline,
"timed out waiting for polluted thread to move into removed phase2 inputs: \
{selection:?}"
);
tokio::time::sleep(Duration::from_millis(50)).await;
}
};
assert_eq!(responses.requests().len(), 2);
assert!(selection.selected.is_empty());
assert_eq!(selection.retained_thread_ids, Vec::<ThreadId>::new());
assert_eq!(selection.removed.len(), 1);
assert_eq!(selection.removed[0].thread_id, thread_id);
shutdown_test_codex(&resumed).await?;
Ok(())
}
async fn build_test_codex(server: &wiremock::MockServer, home: Arc<TempDir>) -> Result<TestCodex> {
#[allow(clippy::expect_used)]
let mut builder = test_codex().with_home(home).with_config(|config| {
@@ -560,30 +328,22 @@ fn phase2_prompt_text(request: &ResponsesRequest) -> String {
request
.message_input_texts("user")
.into_iter()
.find(|text| text.contains("Current selected Phase 1 inputs:"))
.find(|text| text.contains("Memory workspace diff:"))
.expect("phase2 prompt text")
}
async fn wait_for_phase2_success(
db: &codex_state::StateRuntime,
expected_thread_id: ThreadId,
) -> Result<()> {
async fn wait_for_phase2_workspace_reset(memory_root: &Path) -> Result<()> {
wait_for_file_removed(&memory_root.join("phase2_workspace_diff.md")).await?;
let deadline = Instant::now() + Duration::from_secs(10);
loop {
let selection = db
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 30)
.await?;
if selection.selected.len() == 1
&& selection.selected[0].thread_id == expected_thread_id
&& selection.retained_thread_ids == vec![expected_thread_id]
&& selection.removed.is_empty()
if let Ok(diff) = diff_since_latest_init(memory_root).await
&& !diff.has_changes()
{
return Ok(());
}
assert!(
Instant::now() < deadline,
"timed out waiting for phase2 success for {expected_thread_id}"
"timed out waiting for clean memory workspace baseline"
);
tokio::time::sleep(Duration::from_millis(50)).await;
}
+5 -2
View File
@@ -348,7 +348,8 @@ pub async fn run_main(cli: Cli, arg0_paths: Arg0DispatchPaths) -> anyhow::Result
/*enable_codex_api_key_env*/ false,
config_toml.cli_auth_credentials_store.unwrap_or_default(),
chatgpt_base_url,
);
)
.await;
let run_cli_overrides = cli_kv_overrides.clone();
let run_loader_overrides = loader_overrides.clone();
let run_cloud_requirements = cloud_requirements.clone();
@@ -438,7 +439,9 @@ pub async fn run_main(cli: Cli, arg0_paths: Arg0DispatchPaths) -> anyhow::Result
auth_credentials_store_mode: config.cli_auth_credentials_store_mode,
forced_login_method: config.forced_login_method,
forced_chatgpt_workspace_id: config.forced_chatgpt_workspace_id.clone(),
}) {
})
.await
{
eprintln!("{err}");
std::process::exit(1);
}
+3
View File
@@ -9,6 +9,9 @@ pub struct MultiAgentV2ConfigToml {
#[serde(skip_serializing_if = "Option::is_none")]
pub enabled: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
#[schemars(range(min = 1))]
pub max_concurrent_threads_per_session: Option<usize>,
#[serde(skip_serializing_if = "Option::is_none")]
pub usage_hint_enabled: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub usage_hint_text: Option<String>,
+3
View File
@@ -396,6 +396,7 @@ fn multi_agent_v2_feature_config_deserializes_table() {
r#"
[multi_agent_v2]
enabled = true
max_concurrent_threads_per_session = 4
usage_hint_enabled = false
usage_hint_text = "Custom delegation guidance."
hide_spawn_agent_metadata = true
@@ -411,6 +412,7 @@ hide_spawn_agent_metadata = true
features.multi_agent_v2,
Some(crate::FeatureToml::Config(crate::MultiAgentV2ConfigToml {
enabled: Some(true),
max_concurrent_threads_per_session: Some(4),
usage_hint_enabled: Some(false),
usage_hint_text: Some("Custom delegation guidance.".to_string()),
hide_spawn_agent_metadata: Some(true),
@@ -442,6 +444,7 @@ usage_hint_enabled = false
features_toml.multi_agent_v2,
Some(crate::FeatureToml::Config(crate::MultiAgentV2ConfigToml {
enabled: None,
max_concurrent_threads_per_session: None,
usage_hint_enabled: Some(false),
usage_hint_text: None,
hide_spawn_agent_metadata: None,
+5 -3
View File
@@ -3,9 +3,11 @@
Helpers for interacting with git, including patch application and worktree
snapshot utilities. The crate also exposes a lightweight baseline API for
internal directories that use git only as a resettable diff mechanism:
`reset_git_repository` replaces `root/.git` with a fresh one-commit baseline,
and `diff_since_latest_init` returns structured file changes plus a unified
diff from that baseline to the current directory contents.
`ensure_git_baseline_repository` preserves a usable `root/.git` baseline or
creates one when it is missing or unusable, `reset_git_repository` replaces
`root/.git` with a fresh one-commit baseline, and `diff_since_latest_init`
returns structured file changes plus a unified diff from that baseline to the
current directory contents.
```rust,no_run
use std::path::Path;
+70 -4
View File
@@ -12,6 +12,8 @@ use std::path::Path;
use std::path::PathBuf;
use tokio::task;
use crate::operations::run_git_for_status;
const BASELINE_COMMIT_MESSAGE: &str =
"Initialize Codex git baseline\n\nCo-authored-by: Codex <noreply@openai.com>";
@@ -65,18 +67,40 @@ struct GitBaselineFileEntry {
/// This is intentionally destructive for `root/.git`. It is meant for internal directories where
/// git is used only as a baseline/diff implementation detail, not for user repositories.
pub async fn reset_git_repository(root: &Path) -> anyhow::Result<()> {
let root = root.to_path_buf();
task::spawn_blocking(move || reset_git_repository_sync(&root)).await?
}
/// Ensures `root` has a usable git baseline repository.
///
/// Existing usable `.git/` metadata is preserved. Missing or unusable metadata is replaced with a
/// fresh one-commit baseline.
pub async fn ensure_git_baseline_repository(root: &Path) -> anyhow::Result<()> {
let root = root.to_path_buf();
task::spawn_blocking(move || {
fs::create_dir_all(&root)
.with_context(|| format!("create git baseline root {}", root.display()))?;
remove_git_metadata(&root)?;
let repo = gix::init(&root).with_context(|| format!("init git repo {}", root.display()))?;
commit_current_tree(&repo, BASELINE_COMMIT_MESSAGE)?;
anyhow::Ok(())
if root.join(".git").is_dir()
&& let Ok(repo) = gix::open(&root)
&& head_file_entries(&repo).is_ok()
{
return Ok(());
}
reset_git_repository_sync(&root)
})
.await?
}
fn reset_git_repository_sync(root: &Path) -> anyhow::Result<()> {
fs::create_dir_all(root)
.with_context(|| format!("create git baseline root {}", root.display()))?;
remove_git_metadata(root)?;
let repo = gix::init(root).with_context(|| format!("init git repo {}", root.display()))?;
commit_current_tree(&repo, BASELINE_COMMIT_MESSAGE)?;
write_index_from_head(root)?;
Ok(())
}
/// Returns the diff between the latest baseline reset and the current directory contents.
pub async fn diff_since_latest_init(root: &Path) -> anyhow::Result<GitBaselineDiff> {
let root = root.to_path_buf();
@@ -130,6 +154,11 @@ fn commit_current_tree(repo: &gix::Repository, message: &str) -> anyhow::Result<
Ok(())
}
fn write_index_from_head(root: &Path) -> anyhow::Result<()> {
run_git_for_status(root, ["read-tree", "--reset", "HEAD"], /*env*/ None)
.context("write git baseline index from HEAD")
}
fn codex_signature() -> gix::actor::Signature {
gix::actor::Signature {
name: "Codex".into(),
@@ -501,8 +530,24 @@ mod tests {
use super::*;
use pretty_assertions::assert_eq;
use std::fs;
use std::process::Command;
use tempfile::TempDir;
fn git_stdout(root: &Path, args: &[&str]) -> String {
let output = Command::new("git")
.current_dir(root)
.args(args)
.output()
.expect("run git command");
assert!(
output.status.success(),
"git command failed: {args:?}\nstdout:\n{}\nstderr:\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8_lossy(&output.stdout).to_string()
}
#[tokio::test]
async fn reset_creates_fresh_baseline() {
let home = TempDir::new().expect("tempdir");
@@ -513,9 +558,30 @@ mod tests {
reset_git_repository(&root).await.expect("reset repo");
assert!(root.join(".git").is_dir());
assert!(root.join(".git/index").is_file());
let diff = diff_since_latest_init(&root).await.expect("diff");
assert!(!diff.has_changes());
assert_eq!(diff.unified_diff, "");
assert_eq!(git_stdout(&root, &["status", "--porcelain"]), "");
assert_eq!(git_stdout(&root, &["ls-files"]), "MEMORY.md\n");
}
#[tokio::test]
async fn ensure_recovers_from_unborn_repository() {
let home = TempDir::new().expect("tempdir");
let root = home.path().join("repo");
fs::create_dir_all(&root).expect("create root");
fs::write(root.join("MEMORY.md"), "memory").expect("write memory");
gix::init(&root).expect("init git repo without baseline commit");
ensure_git_baseline_repository(&root)
.await
.expect("ensure repo");
let diff = diff_since_latest_init(&root).await.expect("diff");
assert!(!diff.has_changes());
assert_eq!(git_stdout(&root, &["status", "--porcelain"]), "");
assert_eq!(git_stdout(&root, &["ls-files"]), "MEMORY.md\n");
}
#[tokio::test]
+1
View File
@@ -17,6 +17,7 @@ pub use baseline::GitBaselineChange;
pub use baseline::GitBaselineChangeStatus;
pub use baseline::GitBaselineDiff;
pub use baseline::diff_since_latest_init;
pub use baseline::ensure_git_baseline_repository;
pub use baseline::reset_git_repository;
pub use branch::merge_base_with_head;
pub use codex_protocol::models::GhostCommit;
+45 -25
View File
@@ -116,10 +116,11 @@ fn login_with_agent_identity_rejects_invalid_jwt() {
);
}
#[test]
fn missing_auth_json_returns_none() {
#[tokio::test]
async fn missing_auth_json_returns_none() {
let dir = tempdir().unwrap();
let auth = CodexAuth::from_auth_storage(dir.path(), AuthCredentialsStoreMode::File)
.await
.expect("call should succeed");
assert_eq!(auth, None);
}
@@ -143,6 +144,7 @@ async fn pro_account_with_no_api_key_uses_chatgpt_auth() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.unwrap()
.unwrap();
assert_eq!(None, auth.api_key());
@@ -196,6 +198,7 @@ async fn loads_api_key_from_auth_json() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.unwrap()
.unwrap();
assert_eq!(auth.auth_mode(), AuthMode::ApiKey);
@@ -222,15 +225,16 @@ fn logout_removes_auth_file() -> Result<(), std::io::Error> {
Ok(())
}
#[test]
fn unauthorized_recovery_reports_mode_and_step_names() {
#[tokio::test]
async fn unauthorized_recovery_reports_mode_and_step_names() {
let dir = tempdir().unwrap();
let manager = AuthManager::shared(
dir.path().to_path_buf(),
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
let managed = UnauthorizedRecovery {
manager: Arc::clone(&manager),
step: UnauthorizedRecoveryStep::Reload,
@@ -250,8 +254,8 @@ fn unauthorized_recovery_reports_mode_and_step_names() {
assert_eq!(external.step_name(), "external_refresh");
}
#[test]
fn refresh_failure_is_scoped_to_the_matching_auth_snapshot() {
#[tokio::test]
async fn refresh_failure_is_scoped_to_the_matching_auth_snapshot() {
let codex_home = tempdir().unwrap();
write_auth_file(
AuthFileParams {
@@ -268,6 +272,7 @@ fn refresh_failure_is_scoped_to_the_matching_auth_snapshot() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
let mut updated_auth_dot_json = auth
@@ -284,6 +289,7 @@ fn refresh_failure_is_scoped_to_the_matching_auth_snapshot() {
updated_auth_dot_json,
AuthCredentialsStoreMode::File,
)
.await
.expect("updated auth should parse");
let manager = AuthManager::from_auth_for_testing(auth.clone());
@@ -619,9 +625,9 @@ impl Drop for EnvVarGuard {
}
}
#[test]
#[tokio::test]
#[serial(codex_auth_env)]
fn load_auth_reads_agent_identity_from_env() {
async fn load_auth_reads_agent_identity_from_env() {
let codex_home = tempdir().unwrap();
let expected_record = agent_identity_record("account-123");
let agent_identity = fake_agent_identity_jwt(&expected_record).expect("fake agent identity");
@@ -632,6 +638,7 @@ fn load_auth_reads_agent_identity_from_env() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("env auth should load")
.expect("env auth should be present");
@@ -645,9 +652,9 @@ fn load_auth_reads_agent_identity_from_env() {
);
}
#[test]
#[tokio::test]
#[serial(codex_auth_env)]
fn load_auth_keeps_codex_api_key_env_precedence() {
async fn load_auth_keeps_codex_api_key_env_precedence() {
let codex_home = tempdir().unwrap();
let record = agent_identity_record("account-123");
let agent_identity = fake_agent_identity_jwt(&record).expect("fake agent identity");
@@ -659,6 +666,7 @@ fn load_auth_keeps_codex_api_key_env_precedence() {
/*enable_codex_api_key_env*/ true,
AuthCredentialsStoreMode::File,
)
.await
.expect("env auth should load")
.expect("env auth should be present");
@@ -679,8 +687,9 @@ async fn enforce_login_restrictions_logs_out_for_method_mismatch() {
)
.await;
let err =
super::enforce_login_restrictions(&config).expect_err("expected method mismatch to error");
let err = super::enforce_login_restrictions(&config)
.await
.expect_err("expected method mismatch to error");
assert!(err.to_string().contains("ChatGPT login is required"));
assert!(
!codex_home.path().join("auth.json").exists(),
@@ -710,6 +719,7 @@ async fn enforce_login_restrictions_logs_out_for_workspace_mismatch() {
.await;
let err = super::enforce_login_restrictions(&config)
.await
.expect_err("expected workspace mismatch to error");
assert!(err.to_string().contains("workspace org_mine"));
assert!(
@@ -739,7 +749,9 @@ async fn enforce_login_restrictions_allows_matching_workspace() {
)
.await;
super::enforce_login_restrictions(&config).expect("matching workspace should succeed");
super::enforce_login_restrictions(&config)
.await
.expect("matching workspace should succeed");
assert!(
codex_home.path().join("auth.json").exists(),
"auth.json should remain when restrictions pass"
@@ -761,7 +773,9 @@ async fn enforce_login_restrictions_allows_api_key_if_login_method_not_set_but_f
)
.await;
super::enforce_login_restrictions(&config).expect("matching workspace should succeed");
super::enforce_login_restrictions(&config)
.await
.expect("matching workspace should succeed");
assert!(
codex_home.path().join("auth.json").exists(),
"auth.json should remain when restrictions pass"
@@ -782,6 +796,7 @@ async fn enforce_login_restrictions_blocks_env_api_key_when_chatgpt_required() {
.await;
let err = super::enforce_login_restrictions(&config)
.await
.expect_err("environment API key should not satisfy forced ChatGPT login");
assert!(
err.to_string()
@@ -818,8 +833,8 @@ fn fake_agent_identity_jwt(record: &AgentIdentityAuthRecord) -> std::io::Result<
Ok(format!("{header_b64}.{payload_b64}.{signature_b64}"))
}
#[test]
fn plan_type_maps_known_plan() {
#[tokio::test]
async fn plan_type_maps_known_plan() {
let codex_home = tempdir().unwrap();
let _jwt = write_auth_file(
AuthFileParams {
@@ -836,14 +851,15 @@ fn plan_type_maps_known_plan() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
pretty_assertions::assert_eq!(auth.account_plan_type(), Some(AccountPlanType::Pro));
}
#[test]
fn plan_type_maps_self_serve_business_usage_based_plan() {
#[tokio::test]
async fn plan_type_maps_self_serve_business_usage_based_plan() {
let codex_home = tempdir().unwrap();
let _jwt = write_auth_file(
AuthFileParams {
@@ -860,6 +876,7 @@ fn plan_type_maps_self_serve_business_usage_based_plan() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
@@ -869,8 +886,8 @@ fn plan_type_maps_self_serve_business_usage_based_plan() {
);
}
#[test]
fn plan_type_maps_enterprise_cbp_usage_based_plan() {
#[tokio::test]
async fn plan_type_maps_enterprise_cbp_usage_based_plan() {
let codex_home = tempdir().unwrap();
let _jwt = write_auth_file(
AuthFileParams {
@@ -887,6 +904,7 @@ fn plan_type_maps_enterprise_cbp_usage_based_plan() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
@@ -896,8 +914,8 @@ fn plan_type_maps_enterprise_cbp_usage_based_plan() {
);
}
#[test]
fn plan_type_maps_unknown_to_unknown() {
#[tokio::test]
async fn plan_type_maps_unknown_to_unknown() {
let codex_home = tempdir().unwrap();
let _jwt = write_auth_file(
AuthFileParams {
@@ -914,14 +932,15 @@ fn plan_type_maps_unknown_to_unknown() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
pretty_assertions::assert_eq!(auth.account_plan_type(), Some(AccountPlanType::Unknown));
}
#[test]
fn missing_plan_type_maps_to_unknown() {
#[tokio::test]
async fn missing_plan_type_maps_to_unknown() {
let codex_home = tempdir().unwrap();
let _jwt = write_auth_file(
AuthFileParams {
@@ -938,6 +957,7 @@ fn missing_plan_type_maps_to_unknown() {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
)
.await
.expect("load auth")
.expect("auth available");
+51 -32
View File
@@ -193,7 +193,7 @@ impl From<RefreshTokenError> for std::io::Error {
}
impl CodexAuth {
fn from_auth_dot_json(
async fn from_auth_dot_json(
codex_home: &Path,
auth_dot_json: AuthDotJson,
auth_credentials_store_mode: AuthCredentialsStoreMode,
@@ -234,7 +234,7 @@ impl CodexAuth {
}
}
pub fn from_auth_storage(
pub async fn from_auth_storage(
codex_home: &Path,
auth_credentials_store_mode: AuthCredentialsStoreMode,
) -> std::io::Result<Option<Self>> {
@@ -243,6 +243,7 @@ impl CodexAuth {
/*enable_codex_api_key_env*/ false,
auth_credentials_store_mode,
)
.await
}
pub fn from_agent_identity_jwt(jwt: &str) -> std::io::Result<Self> {
@@ -522,6 +523,7 @@ pub async fn logout_with_revoke(
auth_credentials_store_mode,
/*chatgpt_base_url*/ None,
)
.await
.logout_with_revoke()
.await
}
@@ -609,12 +611,13 @@ pub struct AuthConfig {
pub forced_chatgpt_workspace_id: Option<String>,
}
pub fn enforce_login_restrictions(config: &AuthConfig) -> std::io::Result<()> {
pub async fn enforce_login_restrictions(config: &AuthConfig) -> std::io::Result<()> {
let Some(auth) = load_auth(
&config.codex_home,
/*enable_codex_api_key_env*/ true,
config.auth_credentials_store_mode,
)?
)
.await?
else {
return Ok(());
};
@@ -714,15 +717,11 @@ fn logout_all_stores(
Ok(removed_ephemeral || removed_managed)
}
fn load_auth(
async fn load_auth(
codex_home: &Path,
enable_codex_api_key_env: bool,
auth_credentials_store_mode: AuthCredentialsStoreMode,
) -> std::io::Result<Option<CodexAuth>> {
let build_auth = |auth_dot_json: AuthDotJson, storage_mode| {
CodexAuth::from_auth_dot_json(codex_home, auth_dot_json, storage_mode)
};
// API key via env var takes precedence over any other auth method.
if enable_codex_api_key_env && let Some(api_key) = read_codex_api_key_from_env() {
return Ok(Some(CodexAuth::from_api_key(api_key.as_str())));
@@ -735,7 +734,12 @@ fn load_auth(
AuthCredentialsStoreMode::Ephemeral,
);
if let Some(auth_dot_json) = ephemeral_storage.load()? {
let auth = build_auth(auth_dot_json, AuthCredentialsStoreMode::Ephemeral)?;
let auth = CodexAuth::from_auth_dot_json(
codex_home,
auth_dot_json,
AuthCredentialsStoreMode::Ephemeral,
)
.await?;
return Ok(Some(auth));
}
@@ -755,7 +759,9 @@ fn load_auth(
None => return Ok(None),
};
let auth = build_auth(auth_dot_json, auth_credentials_store_mode)?;
let auth =
CodexAuth::from_auth_dot_json(codex_home, auth_dot_json, auth_credentials_store_mode)
.await?;
Ok(Some(auth))
}
@@ -1169,6 +1175,7 @@ impl UnauthorizedRecovery {
match self
.manager
.reload_if_account_id_matches(self.expected_account_id.as_deref())
.await
{
ReloadOutcome::ReloadedChanged => {
self.step = UnauthorizedRecoveryStep::RefreshToken;
@@ -1279,7 +1286,7 @@ impl AuthManager {
/// preferred auth method. Errors loading auth are swallowed; `auth()` will
/// simply return `None` in that case so callers can treat it as an
/// unauthenticated state.
pub fn new(
pub async fn new(
codex_home: PathBuf,
enable_codex_api_key_env: bool,
auth_credentials_store_mode: AuthCredentialsStoreMode,
@@ -1290,6 +1297,7 @@ impl AuthManager {
enable_codex_api_key_env,
auth_credentials_store_mode,
)
.await
.ok()
.flatten();
Self {
@@ -1402,13 +1410,16 @@ impl AuthManager {
/// Force a reload of the auth information from auth.json. Returns
/// whether the auth value changed.
pub fn reload(&self) -> bool {
pub async fn reload(&self) -> bool {
tracing::info!("Reloading auth");
let new_auth = self.load_auth_from_storage();
let new_auth = self.load_auth_from_storage().await;
self.set_cached_auth(new_auth)
}
fn reload_if_account_id_matches(&self, expected_account_id: Option<&str>) -> ReloadOutcome {
async fn reload_if_account_id_matches(
&self,
expected_account_id: Option<&str>,
) -> ReloadOutcome {
let expected_account_id = match expected_account_id {
Some(account_id) => account_id,
None => {
@@ -1417,7 +1428,7 @@ impl AuthManager {
}
};
let new_auth = self.load_auth_from_storage();
let new_auth = self.load_auth_from_storage().await;
let new_account_id = new_auth.as_ref().and_then(CodexAuth::get_account_id);
if new_account_id.as_deref() != Some(expected_account_id) {
@@ -1488,12 +1499,13 @@ impl AuthManager {
}
}
fn load_auth_from_storage(&self) -> Option<CodexAuth> {
async fn load_auth_from_storage(&self) -> Option<CodexAuth> {
load_auth(
&self.codex_home,
self.enable_codex_api_key_env,
self.auth_credentials_store_mode,
)
.await
.ok()
.flatten()
}
@@ -1557,22 +1569,25 @@ impl AuthManager {
}
/// Convenience constructor returning an `Arc` wrapper.
pub fn shared(
pub async fn shared(
codex_home: PathBuf,
enable_codex_api_key_env: bool,
auth_credentials_store_mode: AuthCredentialsStoreMode,
chatgpt_base_url: Option<String>,
) -> Arc<Self> {
Arc::new(Self::new(
codex_home,
enable_codex_api_key_env,
auth_credentials_store_mode,
chatgpt_base_url,
))
Arc::new(
Self::new(
codex_home,
enable_codex_api_key_env,
auth_credentials_store_mode,
chatgpt_base_url,
)
.await,
)
}
/// Convenience constructor returning an `Arc` wrapper from resolved config.
pub fn shared_from_config(
pub async fn shared_from_config(
config: &impl AuthManagerConfig,
enable_codex_api_key_env: bool,
) -> Arc<Self> {
@@ -1581,7 +1596,8 @@ impl AuthManager {
enable_codex_api_key_env,
config.cli_auth_credentials_store_mode(),
Some(config.chatgpt_base_url()),
);
)
.await;
auth_manager.set_forced_chatgpt_workspace_id(config.forced_chatgpt_workspace_id());
auth_manager
}
@@ -1647,7 +1663,10 @@ impl AuthManager {
.as_ref()
.and_then(CodexAuth::get_account_id);
match self.reload_if_account_id_matches(expected_account_id.as_deref()) {
match self
.reload_if_account_id_matches(expected_account_id.as_deref())
.await
{
ReloadOutcome::ReloadedChanged => {
tracing::info!("Skipping token refresh because auth changed after guarded reload.");
Ok(())
@@ -1714,10 +1733,10 @@ impl AuthManager {
/// if a file was removed, Ok(false) if no auth file existed. On success,
/// reloads the in‑memory auth cache so callers immediately observe the
/// unauthenticated state.
pub fn logout(&self) -> std::io::Result<bool> {
pub async fn logout(&self) -> std::io::Result<bool> {
let removed = logout_all_stores(&self.codex_home, self.auth_credentials_store_mode)?;
// Always reload to clear any cached auth (even if file absent).
self.reload();
self.reload().await;
Ok(removed)
}
@@ -1730,7 +1749,7 @@ impl AuthManager {
}
let result = logout_all_stores(&self.codex_home, self.auth_credentials_store_mode)?;
// Always reload to clear any cached auth (even if file absent).
self.reload();
self.reload().await;
Ok(result)
}
@@ -1826,7 +1845,7 @@ impl AuthManager {
AuthCredentialsStoreMode::Ephemeral,
)
.map_err(RefreshTokenError::Transient)?;
self.reload();
self.reload().await;
Ok(())
}
@@ -1846,7 +1865,7 @@ impl AuthManager {
refresh_response.refresh_token,
)
.map_err(RefreshTokenError::from)?;
self.reload();
self.reload().await;
Ok(())
}
+35 -34
View File
@@ -46,7 +46,7 @@ async fn refresh_token_succeeds_updates_storage() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -56,7 +56,7 @@ async fn refresh_token_succeeds_updates_storage() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
ctx.auth_manager
.refresh_token_from_authority()
@@ -110,7 +110,7 @@ async fn refresh_token_refreshes_when_auth_is_unchanged() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -120,7 +120,7 @@ async fn refresh_token_refreshes_when_auth_is_unchanged() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
ctx.auth_manager
.refresh_token()
@@ -164,7 +164,7 @@ async fn refresh_token_skips_refresh_when_auth_changed() -> Result<()> {
skip_if_no_network!(Ok(()));
let server = MockServer::start().await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
@@ -175,7 +175,7 @@ async fn refresh_token_skips_refresh_when_auth_changed() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
let disk_auth = AuthDotJson {
@@ -230,7 +230,7 @@ async fn refresh_token_errors_on_account_mismatch() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -240,7 +240,7 @@ async fn refresh_token_errors_on_account_mismatch() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let mut disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
disk_tokens.account_id = Some("other-account".to_string());
@@ -299,7 +299,7 @@ async fn returns_fresh_tokens_as_is() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let stale_refresh = Utc::now() - Duration::days(9);
let fresh_access_token = access_token_with_expiration(Utc::now() + Duration::hours(1));
let initial_tokens = build_tokens(&fresh_access_token, INITIAL_REFRESH_TOKEN);
@@ -310,7 +310,7 @@ async fn returns_fresh_tokens_as_is() -> Result<()> {
last_refresh: Some(stale_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let cached_auth = ctx
.auth_manager
@@ -347,7 +347,7 @@ async fn refreshes_token_when_access_token_is_expired() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let fresh_refresh = Utc::now() - Duration::days(1);
let expired_access_token = access_token_with_expiration(Utc::now() - Duration::hours(1));
let initial_tokens = build_tokens(&expired_access_token, INITIAL_REFRESH_TOKEN);
@@ -358,7 +358,7 @@ async fn refreshes_token_when_access_token_is_expired() -> Result<()> {
last_refresh: Some(fresh_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let cached_auth = ctx
.auth_manager
@@ -398,7 +398,7 @@ async fn auth_reloads_disk_auth_when_cached_auth_is_stale() -> Result<()> {
let server = MockServer::start().await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let stale_refresh = Utc::now() - Duration::days(9);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -408,7 +408,7 @@ async fn auth_reloads_disk_auth_when_cached_auth_is_stale() -> Result<()> {
last_refresh: Some(stale_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let fresh_refresh = Utc::now() - Duration::days(1);
let disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
@@ -461,7 +461,7 @@ async fn auth_reloads_disk_auth_without_calling_expired_refresh_token() -> Resul
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let stale_refresh = Utc::now() - Duration::days(9);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -471,7 +471,7 @@ async fn auth_reloads_disk_auth_without_calling_expired_refresh_token() -> Resul
last_refresh: Some(stale_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let fresh_refresh = Utc::now() - Duration::days(1);
let disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
@@ -522,7 +522,7 @@ async fn refresh_token_returns_permanent_error_for_expired_refresh_token() -> Re
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -532,7 +532,7 @@ async fn refresh_token_returns_permanent_error_for_expired_refresh_token() -> Re
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let err = ctx
.auth_manager
@@ -575,7 +575,7 @@ async fn refresh_token_does_not_retry_after_permanent_failure() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -585,7 +585,7 @@ async fn refresh_token_does_not_retry_after_permanent_failure() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let first_err = ctx
.auth_manager
@@ -642,7 +642,7 @@ async fn refresh_token_reloads_changed_auth_after_permanent_failure() -> Result<
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -652,7 +652,7 @@ async fn refresh_token_reloads_changed_auth_after_permanent_failure() -> Result<
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let first_err = ctx
.auth_manager
@@ -723,7 +723,7 @@ async fn refresh_token_returns_transient_error_on_server_failure() -> Result<()>
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -733,7 +733,7 @@ async fn refresh_token_returns_transient_error_on_server_failure() -> Result<()>
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let err = ctx
.auth_manager
@@ -776,7 +776,7 @@ async fn unauthorized_recovery_reloads_then_refreshes_tokens() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -786,7 +786,7 @@ async fn unauthorized_recovery_reloads_then_refreshes_tokens() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
let disk_auth = AuthDotJson {
@@ -870,7 +870,7 @@ async fn unauthorized_recovery_errors_on_account_mismatch() -> Result<()> {
.mount(&server)
.await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let initial_last_refresh = Utc::now() - Duration::days(1);
let initial_tokens = build_tokens(INITIAL_ACCESS_TOKEN, INITIAL_REFRESH_TOKEN);
let initial_auth = AuthDotJson {
@@ -880,7 +880,7 @@ async fn unauthorized_recovery_errors_on_account_mismatch() -> Result<()> {
last_refresh: Some(initial_last_refresh),
agent_identity: None,
};
ctx.write_auth(&initial_auth)?;
ctx.write_auth(&initial_auth).await?;
let mut disk_tokens = build_tokens("disk-access-token", "disk-refresh-token");
disk_tokens.account_id = Some("other-account".to_string());
@@ -941,7 +941,7 @@ async fn unauthorized_recovery_requires_chatgpt_auth() -> Result<()> {
skip_if_no_network!(Ok(()));
let server = MockServer::start().await;
let ctx = RefreshTokenTestContext::new(&server)?;
let ctx = RefreshTokenTestContext::new(&server).await?;
let auth = AuthDotJson {
auth_mode: Some(AuthMode::ApiKey),
openai_api_key: Some("sk-test".to_string()),
@@ -949,7 +949,7 @@ async fn unauthorized_recovery_requires_chatgpt_auth() -> Result<()> {
last_refresh: None,
agent_identity: None,
};
ctx.write_auth(&auth)?;
ctx.write_auth(&auth).await?;
let mut recovery = ctx.auth_manager.unauthorized_recovery();
assert!(!recovery.has_next());
@@ -974,7 +974,7 @@ struct RefreshTokenTestContext {
}
impl RefreshTokenTestContext {
fn new(server: &MockServer) -> Result<Self> {
async fn new(server: &MockServer) -> Result<Self> {
let codex_home = TempDir::new()?;
let endpoint = format!("{}/oauth/token", server.uri());
@@ -985,7 +985,8 @@ impl RefreshTokenTestContext {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
Ok(Self {
codex_home,
@@ -1000,13 +1001,13 @@ impl RefreshTokenTestContext {
.context("auth.json should exist")
}
fn write_auth(&self, auth_dot_json: &AuthDotJson) -> Result<()> {
async fn write_auth(&self, auth_dot_json: &AuthDotJson) -> Result<()> {
save_auth(
self.codex_home.path(),
auth_dot_json,
AuthCredentialsStoreMode::File,
)?;
self.auth_manager.reload();
self.auth_manager.reload().await;
Ok(())
}
}
+2 -1
View File
@@ -143,7 +143,8 @@ async fn auth_manager_logout_with_revoke_uses_cached_auth() -> Result<()> {
/*enable_codex_api_key_env*/ false,
AuthCredentialsStoreMode::File,
/*chatgpt_base_url*/ None,
);
)
.await;
save_auth(
codex_home.path(),
&chatgpt_auth_with_refresh_token("newer-disk-refresh-token"),
+2 -1
View File
@@ -141,7 +141,8 @@ pub async fn run_main(
arg0_paths,
Arc::new(config),
environment_manager,
);
)
.await;
async move {
while let Some(msg) = incoming_rx.recv().await {
match msg {
+3 -2
View File
@@ -49,7 +49,7 @@ pub(crate) struct MessageProcessor {
impl MessageProcessor {
/// Create a new `MessageProcessor`, retaining a handle to the outgoing
/// `Sender` so handlers can enqueue messages to be written to stdout.
pub(crate) fn new(
pub(crate) async fn new(
outgoing: OutgoingMessageSender,
arg0_paths: Arg0DispatchPaths,
config: Arc<Config>,
@@ -59,7 +59,8 @@ impl MessageProcessor {
let auth_manager = AuthManager::shared_from_config(
config.as_ref(),
/*enable_codex_api_key_env*/ false,
);
)
.await;
let thread_manager = Arc::new(ThreadManager::new(
config.as_ref(),
auth_manager,
@@ -1,16 +1,28 @@
# Memories Pipeline (Core)
# Memories
This module runs a startup memory pipeline for eligible sessions.
This directory owns reusable memory crates and the memory pipeline documentation.
Runtime orchestration for Phase 1 and Phase 2 still lives in `codex-core` under
`codex-rs/core/src/memories/`.
## Crates
- `codex-rs/memories/read` (`codex-memories-read`) owns the read path:
memory developer-instruction injection, memory citation parsing, and
read-usage telemetry classification.
- `codex-rs/memories/write` (`codex-memories-write`) owns the write path:
Phase 1 and Phase 2 prompt rendering, filesystem artifact helpers,
workspace diff helpers, and extension resource pruning.
## Prompt Templates
Memory prompt templates live under `codex-rs/core/templates/memories/`.
Memory prompt templates live with the crate that uses them:
- The undated template files are the canonical latest versions used at runtime:
- `stage_one_system.md`
- `stage_one_input.md`
- `consolidation.md`
- `read_path.md`
- `read/templates/memories/read_path.md`
- `write/templates/memories/stage_one_system.md`
- `write/templates/memories/stage_one_input.md`
- `write/templates/memories/consolidation.md`
- In `codex`, edit those undated template files in place.
- The dated snapshot-copy workflow is used in the separate `openai/project/agent_memory/write` harness repo, not here.
@@ -70,7 +82,8 @@ Phase 2 consolidates the latest stage-1 outputs into the filesystem memory artif
What it does:
- claims a single global phase-2 job (so only one consolidation runs at a time)
- claims a single global phase-2 lock before touching the memories root (so only one consolidation
inspects or mutates the workspace at a time)
- loads a bounded set of stage-1 outputs from the state DB using phase-2
selection rules:
- ignores memories whose `last_usage` falls outside the configured
@@ -82,53 +95,58 @@ What it does:
- computes a completion watermark from the claimed watermark + newest input timestamps
- syncs local memory artifacts under the memories root:
- `raw_memories.md` (merged raw memories, latest first)
- `rollout_summaries/` (one summary file per retained rollout)
- prunes stale rollout summaries that are no longer retained
- finds old resource files from memory extensions under
`memories_extensions/<extension>/resources/` for extension directories that
have an `instructions.md`, using the memory module retention window
- if there are no Phase 1 inputs or old extension resources, marks the job
successful and exits
- `rollout_summaries/` (one summary file per selected rollout)
- keeps the memories root itself as a git-baseline directory, initialized under
`~/.codex/memories/.git` by `codex-git-utils`
- prunes stale rollout summaries that are no longer selected
- prunes memory extension resource files older than the extension retention
window, so cleanup appears in the workspace diff
- writes `phase2_workspace_diff.md` in the memories root with the git-style diff
from the previous successful Phase 2 baseline to the current worktree
- if the memory workspace has no changes after artifact sync/pruning, marks the
job successful and exits
If there is input, it then:
If the memory workspace has changes, it then:
- spawns an internal consolidation sub-agent
- builds the Phase 2 prompt with a diff of the current Phase 1 input
selection versus the last successful Phase 2 selection (`added`,
`retained`, `removed`)
- includes old extension resource paths in the prompt diff
- builds the Phase 2 prompt with the path to the generated workspace diff
- points the agent at `phase2_workspace_diff.md` for the detailed diff context
- runs it with no approvals, no network, and local write access only
- disables collab for that agent (to prevent recursive delegation)
- watches the agent status and heartbeats the global job lease while it runs
- resets the memory git baseline after the agent completes successfully; the
generated diff file is removed before this reset so deleted content is not
kept in the prompt artifact or unreachable git objects
- marks the phase-2 job success/failure in the state DB when the agent finishes
- prunes old extension resource files after the consolidation agent completes
and the successful Phase 2 job is recorded
Selection diff behavior:
Selection and workspace-diff behavior:
- successful Phase 2 runs mark the exact stage-1 snapshots they consumed with
`selected_for_phase2 = 1` and persist the matching
`selected_for_phase2_source_updated_at`
- Phase 1 upserts preserve the previous `selected_for_phase2` baseline until
the next successful Phase 2 run rewrites it
- the next Phase 2 run compares the current top-N stage-1 inputs against that
prior snapshot selection to label inputs as `added` or `retained`; a
refreshed thread stays `added` until Phase 2 successfully selects its newer
snapshot
- rows that were previously selected but still exist outside the current top-N
selection are surfaced as `removed`
- before the agent starts, local `rollout_summaries/` and `raw_memories.md`
keep the union of the current selection and the previous successful
selection, so removed-thread evidence stays available during forgetting
- Phase 2 loads only the current top-N selected stage-1 inputs, syncs
`rollout_summaries/` and `raw_memories.md` directly to that selection, then
lets the git-style workspace diff surface additions, modifications, and
deletions against the previous successful memory baseline
- when the selected input set is empty, stale `rollout_summaries/` files are
removed and `raw_memories.md` is rewritten to the empty-input placeholder;
consolidated outputs such as `MEMORY.md`, `memory_summary.md`, and `skills/`
are left for the agent to update
Watermark behavior:
- The global phase-2 job claim includes an input watermark representing the latest input timestamp known when the job was claimed.
- The global phase-2 lock does not use DB watermarks as a dirty check; git
workspace dirtiness decides whether an agent needs to run.
- The global phase-2 job row still tracks an input watermark as bookkeeping
for the latest DB input timestamp known when the job was claimed.
- Phase 2 recomputes a `new_watermark` using the max of:
- the claimed watermark
- the newest `source_updated_at` timestamp in the stage-1 inputs it actually loaded
- On success, Phase 2 stores that completion watermark in the DB.
- This lets later phase-2 runs know whether new stage-1 data arrived since the last successful consolidation (dirty vs not dirty), while also avoiding moving the watermark backwards.
- This avoids moving the recorded completion watermark backwards, but does not
decide whether Phase 2 has work.
In practice, this phase is responsible for refreshing the on-disk memory workspace and producing/updating the higher-level consolidated memory outputs.
+9
View File
@@ -0,0 +1,9 @@
load("//:defs.bzl", "codex_rust_crate")
codex_rust_crate(
name = "read",
crate_name = "codex_memories_read",
compile_data = glob([
"templates/**",
]),
)
+25
View File
@@ -0,0 +1,25 @@
[package]
edition.workspace = true
license.workspace = true
name = "codex-memories-read"
version.workspace = true
[lib]
name = "codex_memories_read"
path = "src/lib.rs"
[lints]
workspace = true
[dependencies]
codex-protocol = { workspace = true }
codex-shell-command = { workspace = true }
codex-utils-absolute-path = { workspace = true }
codex-utils-output-truncation = { workspace = true }
codex-utils-template = { workspace = true }
tokio = { workspace = true, features = ["fs"] }
[dev-dependencies]
pretty_assertions = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["fs", "macros"] }
+19
View File
@@ -0,0 +1,19 @@
//! Read-path helpers for Codex memories.
//!
//! This crate owns memory injection, memory citation parsing, and telemetry
//! classification for read access to the memory folder. It intentionally does
//! not depend on the memory write pipeline.
pub mod citations;
mod prompts;
pub mod usage;
use codex_utils_absolute_path::AbsolutePathBuf;
pub use prompts::build_memory_tool_developer_instructions;
const MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT: usize = 5_000;
pub fn memory_root(codex_home: &AbsolutePathBuf) -> AbsolutePathBuf {
codex_home.join("memories")
}
+56
View File
@@ -0,0 +1,56 @@
use crate::MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT;
use crate::memory_root;
use codex_utils_absolute_path::AbsolutePathBuf;
use codex_utils_output_truncation::TruncationPolicy;
use codex_utils_output_truncation::truncate_text;
use codex_utils_template::Template;
use std::sync::LazyLock;
use tokio::fs;
static MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../templates/memories/read_path.md"),
"memories/read_path.md",
)
});
fn parse_embedded_template(source: &'static str, template_name: &str) -> Template {
match Template::parse(source) {
Ok(template) => template,
Err(err) => panic!("embedded template {template_name} is invalid: {err}"),
}
}
/// Build the read-path prompt that is added to developer instructions.
///
/// Large `memory_summary.md` files are truncated at
/// [MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT].
pub async fn build_memory_tool_developer_instructions(
codex_home: &AbsolutePathBuf,
) -> Option<String> {
let base_path = memory_root(codex_home);
let memory_summary_path = base_path.join("memory_summary.md");
let memory_summary = fs::read_to_string(&memory_summary_path)
.await
.ok()?
.trim()
.to_string();
let memory_summary = truncate_text(
&memory_summary,
TruncationPolicy::Tokens(MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_SUMMARY_TOKEN_LIMIT),
);
if memory_summary.is_empty() {
return None;
}
let base_path = base_path.display().to_string();
MEMORY_TOOL_DEVELOPER_INSTRUCTIONS_TEMPLATE
.render([
("base_path", base_path.as_str()),
("memory_summary", memory_summary.as_str()),
])
.ok()
}
#[cfg(test)]
#[path = "prompts_tests.rs"]
mod tests;
@@ -0,0 +1,35 @@
use super::*;
use codex_utils_absolute_path::AbsolutePathBuf;
use pretty_assertions::assert_eq;
use tempfile::tempdir;
use tokio::fs as tokio_fs;
#[tokio::test]
async fn build_memory_tool_developer_instructions_renders_embedded_template() {
let temp = tempdir().unwrap();
let codex_home = AbsolutePathBuf::from_absolute_path(temp.path()).unwrap();
let memories_dir = codex_home.join("memories");
tokio_fs::create_dir_all(&memories_dir).await.unwrap();
tokio_fs::write(
memories_dir.join("memory_summary.md"),
"Short memory summary for tests.",
)
.await
.unwrap();
let instructions = build_memory_tool_developer_instructions(&codex_home)
.await
.unwrap();
assert!(instructions.contains(&format!(
"- {}/memory_summary.md (already provided below; do NOT open again)",
memories_dir.display()
)));
assert!(instructions.contains("Short memory summary for tests."));
assert_eq!(
instructions
.matches("========= MEMORY_SUMMARY BEGINS =========")
.count(),
1
);
}
+57
View File
@@ -0,0 +1,57 @@
use codex_protocol::parse_command::ParsedCommand;
use codex_shell_command::is_safe_command::is_known_safe_command;
use codex_shell_command::parse_command::parse_command;
pub const MEMORIES_USAGE_METRIC: &str = "codex.memories.usage";
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum MemoriesUsageKind {
MemoryMd,
MemorySummary,
RawMemories,
RolloutSummaries,
Skills,
}
impl MemoriesUsageKind {
pub fn as_tag(self) -> &'static str {
match self {
Self::MemoryMd => "memory_md",
Self::MemorySummary => "memory_summary",
Self::RawMemories => "raw_memories",
Self::RolloutSummaries => "rollout_summaries",
Self::Skills => "skills",
}
}
}
pub fn memories_usage_kinds_from_command(command: &[String]) -> Vec<MemoriesUsageKind> {
if !is_known_safe_command(command) {
return Vec::new();
}
parse_command(command)
.into_iter()
.filter_map(|command| match command {
ParsedCommand::Read { path, .. } => get_memory_kind(path.display().to_string()),
ParsedCommand::Search { path, .. } => path.and_then(get_memory_kind),
ParsedCommand::ListFiles { .. } | ParsedCommand::Unknown { .. } => None,
})
.collect()
}
fn get_memory_kind(path: String) -> Option<MemoriesUsageKind> {
if path.contains("memories/MEMORY.md") {
Some(MemoriesUsageKind::MemoryMd)
} else if path.contains("memories/memory_summary.md") {
Some(MemoriesUsageKind::MemorySummary)
} else if path.contains("memories/raw_memories.md") {
Some(MemoriesUsageKind::RawMemories)
} else if path.contains("memories/rollout_summaries/") {
Some(MemoriesUsageKind::RolloutSummaries)
} else if path.contains("memories/skills/") {
Some(MemoriesUsageKind::Skills)
} else {
None
}
}
+9
View File
@@ -0,0 +1,9 @@
load("//:defs.bzl", "codex_rust_crate")
codex_rust_crate(
name = "write",
crate_name = "codex_memories_write",
compile_data = glob([
"templates/**",
]),
)
+31
View File
@@ -0,0 +1,31 @@
[package]
edition.workspace = true
license.workspace = true
name = "codex-memories-write"
version.workspace = true
[lib]
name = "codex_memories_write"
path = "src/lib.rs"
[lints]
workspace = true
[dependencies]
anyhow = { workspace = true }
chrono = { workspace = true }
codex-git-utils = { workspace = true }
codex-protocol = { workspace = true }
codex-state = { workspace = true }
codex-utils-absolute-path = { workspace = true }
codex-utils-output-truncation = { workspace = true }
codex-utils-template = { workspace = true }
tokio = { workspace = true, features = ["fs"] }
tracing = { workspace = true, features = ["log"] }
uuid = { workspace = true, features = ["v4", "v5"] }
[dev-dependencies]
codex-models-manager = { workspace = true }
pretty_assertions = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["fs", "macros"] }
+101
View File
@@ -0,0 +1,101 @@
use crate::memory_extensions_root;
use chrono::DateTime;
use chrono::Duration;
use chrono::NaiveDateTime;
use chrono::Utc;
use std::path::Path;
use tracing::warn;
const FILENAME_TS_FORMAT: &str = "%Y-%m-%dT%H-%M-%S";
const EXTENSION_RESOURCE_RETENTION_DAYS: i64 = 7;
pub async fn prune_old_extension_resources(memory_root: &Path) {
prune_old_extension_resources_with_now(memory_root, Utc::now()).await
}
async fn prune_old_extension_resources_with_now(memory_root: &Path, now: DateTime<Utc>) {
let cutoff = now - Duration::days(EXTENSION_RESOURCE_RETENTION_DAYS);
let extensions_root = memory_extensions_root(memory_root);
let mut extensions = match tokio::fs::read_dir(&extensions_root).await {
Ok(extensions) => extensions,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return,
Err(err) => {
warn!(
"failed reading memory extensions root {}: {err}",
extensions_root.display()
);
return;
}
};
while let Ok(Some(extension_entry)) = extensions.next_entry().await {
let extension_path = extension_entry.path();
let Ok(file_type) = extension_entry.file_type().await else {
continue;
};
if !file_type.is_dir()
|| !tokio::fs::try_exists(extension_path.join("instructions.md"))
.await
.unwrap_or(false)
{
continue;
}
let resources_path = extension_path.join("resources");
let mut resources = match tokio::fs::read_dir(&resources_path).await {
Ok(resources) => resources,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue,
Err(err) => {
warn!(
"failed reading memory extension resources {}: {err}",
resources_path.display()
);
continue;
}
};
while let Ok(Some(resource_entry)) = resources.next_entry().await {
let resource_file_path = resource_entry.path();
let Ok(file_type) = resource_entry.file_type().await else {
continue;
};
if !file_type.is_file() {
continue;
}
let Some(file_name) = resource_file_path
.file_name()
.and_then(|name| name.to_str())
else {
continue;
};
if !file_name.ends_with(".md") {
continue;
}
let Some(resource_timestamp) = resource_timestamp(file_name) else {
continue;
};
if resource_timestamp > cutoff {
continue;
}
if let Err(err) = tokio::fs::remove_file(&resource_file_path).await
&& err.kind() != std::io::ErrorKind::NotFound
{
warn!(
"failed pruning old memory extension resource {}: {err}",
resource_file_path.display()
);
}
}
}
}
fn resource_timestamp(file_name: &str) -> Option<DateTime<Utc>> {
let timestamp = file_name.get(..19)?;
let naive = NaiveDateTime::parse_from_str(timestamp, FILENAME_TS_FORMAT).ok()?;
Some(DateTime::from_naive_utc_and_offset(naive, Utc))
}
#[cfg(test)]
#[path = "extensions_tests.rs"]
mod tests;
@@ -0,0 +1,81 @@
use super::*;
use pretty_assertions::assert_eq;
use tempfile::TempDir;
#[tokio::test]
async fn prunes_only_old_resources_from_extensions_with_instructions() {
let codex_home = TempDir::new().expect("create temp codex home");
let memory_root = codex_home.path().join("memories");
let extensions_root = memory_extensions_root(&memory_root);
let chronicle_resources = extensions_root.join("chronicle/resources");
tokio::fs::create_dir_all(&chronicle_resources)
.await
.expect("create chronicle resources");
tokio::fs::write(
extensions_root.join("chronicle/instructions.md"),
"instructions",
)
.await
.expect("write chronicle instructions");
let now = DateTime::from_naive_utc_and_offset(
NaiveDateTime::parse_from_str("2026-04-14T12-00-00", FILENAME_TS_FORMAT)
.expect("parse now"),
Utc,
);
let old_file = chronicle_resources.join("2026-04-06T11-59-59-abcd-10min-old.md");
let exact_cutoff_file = chronicle_resources.join("2026-04-07T12-00-00-abcd-10min-cutoff.md");
let recent_file = chronicle_resources.join("2026-04-08T12-00-00-abcd-10min-recent.md");
let invalid_file = chronicle_resources.join("not-a-timestamp.md");
for file in [&old_file, &exact_cutoff_file, &recent_file, &invalid_file] {
tokio::fs::write(file, "resource")
.await
.expect("write chronicle resource");
}
let ignored_resources = extensions_root.join("ignored/resources");
tokio::fs::create_dir_all(&ignored_resources)
.await
.expect("create ignored resources");
let ignored_old_file = ignored_resources.join("2026-04-06T11-59-59-abcd-10min-old.md");
tokio::fs::write(&ignored_old_file, "ignored")
.await
.expect("write ignored resource");
prune_old_extension_resources_with_now(&memory_root, now).await;
assert!(
!tokio::fs::try_exists(&old_file)
.await
.expect("check old file")
);
assert!(
!tokio::fs::try_exists(&exact_cutoff_file)
.await
.expect("check cutoff file")
);
assert!(
tokio::fs::try_exists(&recent_file)
.await
.expect("check recent file")
);
assert!(
tokio::fs::try_exists(&invalid_file)
.await
.expect("check invalid file")
);
assert!(
tokio::fs::try_exists(&ignored_old_file)
.await
.expect("check ignored file")
);
}
#[test]
fn parses_timestamp_prefix_from_resource_file_name() {
let parsed = resource_timestamp("2026-04-06T11-59-59-abcd-10min-old.md")
.expect("timestamp should parse");
assert_eq!(parsed.timestamp(), 1_775_476_799);
assert!(resource_timestamp("not-a-timestamp.md").is_none());
}
+63
View File
@@ -0,0 +1,63 @@
//! Write-path helpers for Codex memories.
//!
//! This crate owns the file-backed memory artifact helpers, Phase 1 and Phase
//! 2 prompt rendering, extension pruning, and workspace diffing. Runtime
//! orchestration for Phase 1 and Phase 2 remains in `codex-core`.
mod control;
mod extensions;
mod prompts;
mod storage;
pub mod workspace;
use codex_utils_absolute_path::AbsolutePathBuf;
use std::path::Path;
use std::path::PathBuf;
pub use control::clear_memory_roots_contents;
pub use extensions::prune_old_extension_resources;
pub use prompts::build_consolidation_prompt;
pub use prompts::build_stage_one_input_message;
pub use storage::rebuild_raw_memories_file_from_memories;
pub use storage::rollout_summary_file_stem;
pub use storage::sync_rollout_summaries_from_memories;
/// Prompt used for phase 1 extraction.
pub const STAGE_ONE_PROMPT: &str = include_str!("../templates/memories/stage_one_system.md");
/// Fallback stage-1 rollout truncation limit (tokens) when model metadata
/// does not include a valid context window.
pub const DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT: usize = 150_000;
/// Portion of the model effective input window reserved for the stage-1
/// rollout input.
///
/// Keeping this below 100% leaves room for system instructions, prompt framing,
/// and model output.
pub const STAGE_ONE_CONTEXT_WINDOW_PERCENT: i64 = 70;
mod artifacts {
pub(super) const EXTENSIONS_SUBDIR: &str = "extensions";
pub(super) const ROLLOUT_SUMMARIES_SUBDIR: &str = "rollout_summaries";
pub(super) const RAW_MEMORIES_FILENAME: &str = "raw_memories.md";
}
pub fn memory_root(codex_home: &AbsolutePathBuf) -> AbsolutePathBuf {
codex_home.join("memories")
}
pub fn rollout_summaries_dir(root: &Path) -> PathBuf {
root.join(artifacts::ROLLOUT_SUMMARIES_SUBDIR)
}
pub fn memory_extensions_root(root: &Path) -> PathBuf {
root.join(artifacts::EXTENSIONS_SUBDIR)
}
pub fn raw_memories_file(root: &Path) -> PathBuf {
root.join(artifacts::RAW_MEMORIES_FILENAME)
}
pub async fn ensure_layout(root: &Path) -> std::io::Result<()> {
tokio::fs::create_dir_all(rollout_summaries_dir(root)).await
}
+160
View File
@@ -0,0 +1,160 @@
use crate::DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT;
use crate::STAGE_ONE_CONTEXT_WINDOW_PERCENT;
use crate::memory_extensions_root;
use crate::workspace::WORKSPACE_DIFF_FILENAME;
use codex_protocol::openai_models::ModelInfo;
use codex_utils_output_truncation::TruncationPolicy;
use codex_utils_output_truncation::truncate_text;
use codex_utils_template::Template;
use std::path::Path;
use std::sync::LazyLock;
use tracing::warn;
static CONSOLIDATION_PROMPT_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../templates/memories/consolidation.md"),
"memories/consolidation.md",
)
});
static STAGE_ONE_INPUT_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
include_str!("../templates/memories/stage_one_input.md"),
"memories/stage_one_input.md",
)
});
static MEMORY_EXTENSIONS_FOLDER_STRUCTURE_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
MEMORY_EXTENSIONS_FOLDER_STRUCTURE,
"memories/extensions_folder_structure.md",
)
});
static MEMORY_EXTENSIONS_PRIMARY_INPUTS_TEMPLATE: LazyLock<Template> = LazyLock::new(|| {
parse_embedded_template(
MEMORY_EXTENSIONS_PRIMARY_INPUTS,
"memories/extensions_primary_inputs.md",
)
});
fn parse_embedded_template(source: &'static str, template_name: &str) -> Template {
match Template::parse(source) {
Ok(template) => template,
Err(err) => panic!("embedded template {template_name} is invalid: {err}"),
}
}
const MEMORY_EXTENSIONS_FOLDER_STRUCTURE: &str = r#"
Memory extensions (under {{ memory_extensions_root }}/):
- <extension_name>/instructions.md
- Source-specific guidance for interpreting additional memory signals. If an
extension folder exists, you must read its instructions.md to determine how to use this memory
source.
If the user has any memory extensions, you MUST read the instructions for each extension to
determine how to use the memory source. If the workspace diff shows deleted extension resource files,
remove stale memories derived only from those resources. If it has no extension folders, continue
with the standard memory inputs only.
"#;
const MEMORY_EXTENSIONS_PRIMARY_INPUTS: &str = r#"
Optional source-specific inputs:
Under `{{ memory_extensions_root }}/`:
- `<extension_name>/instructions.md`
- If extension folders exist, read each instructions.md first and follow it when interpreting
that extension's memory source.
If the workspace diff shows deleted memory extension resources, use that extension-specific deletion
signal to remove stale memories derived only from those resources.
"#;
/// Builds the consolidation subagent prompt for a specific memory root.
pub fn build_consolidation_prompt(memory_root: &Path) -> String {
let memory_extensions_root = memory_extensions_root(memory_root);
let memory_extensions_exist = memory_extensions_root.is_dir();
let memory_root = memory_root.display().to_string();
let memory_extensions_root = memory_extensions_root.display().to_string();
let phase2_workspace_diff_file = WORKSPACE_DIFF_FILENAME.to_string();
let memory_extensions_folder_structure = if memory_extensions_exist {
render_memory_extensions_block(
&MEMORY_EXTENSIONS_FOLDER_STRUCTURE_TEMPLATE,
&memory_extensions_root,
)
} else {
String::new()
};
let memory_extensions_primary_inputs = if memory_extensions_exist {
render_memory_extensions_block(
&MEMORY_EXTENSIONS_PRIMARY_INPUTS_TEMPLATE,
&memory_extensions_root,
)
} else {
String::new()
};
CONSOLIDATION_PROMPT_TEMPLATE
.render([
("memory_root", memory_root.as_str()),
(
"memory_extensions_folder_structure",
memory_extensions_folder_structure.as_str(),
),
(
"memory_extensions_primary_inputs",
memory_extensions_primary_inputs.as_str(),
),
(
"phase2_workspace_diff_file",
phase2_workspace_diff_file.as_str(),
),
])
.unwrap_or_else(|err| {
warn!("failed to render memories consolidation prompt template: {err}");
format!(
"## Memory Phase 2 (Consolidation)\nConsolidate Codex memories in: {memory_root}\n\nRead {phase2_workspace_diff_file} first."
)
})
}
fn render_memory_extensions_block(template: &Template, memory_extensions_root: &str) -> String {
template
.render([("memory_extensions_root", memory_extensions_root)])
.unwrap_or_else(|err| {
warn!("failed to render memories extension prompt block: {err}");
String::new()
})
}
/// Builds the stage-1 user message containing rollout metadata and content.
///
/// Large rollout payloads are truncated to 70% of the active model's effective
/// input window token budget while keeping both head and tail context.
pub fn build_stage_one_input_message(
model_info: &ModelInfo,
rollout_path: &Path,
rollout_cwd: &Path,
rollout_contents: &str,
) -> anyhow::Result<String> {
let rollout_token_limit = model_info
.resolved_context_window()
.and_then(|limit| (limit > 0).then_some(limit))
.map(|limit| limit.saturating_mul(model_info.effective_context_window_percent) / 100)
.map(|limit| (limit.saturating_mul(STAGE_ONE_CONTEXT_WINDOW_PERCENT) / 100).max(1))
.and_then(|limit| usize::try_from(limit).ok())
.unwrap_or(DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT);
let truncated_rollout_contents = truncate_text(
rollout_contents,
TruncationPolicy::Tokens(rollout_token_limit),
);
let rollout_path = rollout_path.display().to_string();
let rollout_cwd = rollout_cwd.display().to_string();
Ok(STAGE_ONE_INPUT_TEMPLATE.render([
("rollout_path", rollout_path.as_str()),
("rollout_cwd", rollout_cwd.as_str()),
("rollout_contents", truncated_rollout_contents.as_str()),
])?)
}
#[cfg(test)]
#[path = "prompts_tests.rs"]
mod tests;
@@ -0,0 +1,71 @@
use super::*;
use codex_models_manager::model_info::model_info_from_slug;
use tempfile::tempdir;
#[test]
fn build_stage_one_input_message_truncates_rollout_using_model_context_window() {
let input = format!("{}{}{}", "a".repeat(700_000), "middle", "z".repeat(700_000));
let mut model_info = model_info_from_slug("gpt-5.3-codex");
model_info.context_window = Some(123_000);
let expected_rollout_token_limit = usize::try_from(
((123_000_i64 * model_info.effective_context_window_percent) / 100)
* STAGE_ONE_CONTEXT_WINDOW_PERCENT
/ 100,
)
.unwrap();
let expected_truncated = truncate_text(
&input,
TruncationPolicy::Tokens(expected_rollout_token_limit),
);
let message = build_stage_one_input_message(
&model_info,
Path::new("/tmp/rollout.jsonl"),
Path::new("/tmp"),
&input,
)
.unwrap();
assert!(expected_truncated.contains("tokens truncated"));
assert!(expected_truncated.starts_with('a'));
assert!(expected_truncated.ends_with('z'));
assert!(message.contains(&expected_truncated));
}
#[test]
fn build_stage_one_input_message_uses_default_limit_when_model_context_window_missing() {
let input = format!("{}{}{}", "a".repeat(700_000), "middle", "z".repeat(700_000));
let mut model_info = model_info_from_slug("gpt-5.3-codex");
model_info.context_window = None;
model_info.max_context_window = None;
let expected_truncated = truncate_text(
&input,
TruncationPolicy::Tokens(DEFAULT_STAGE_ONE_ROLLOUT_TOKEN_LIMIT),
);
let message = build_stage_one_input_message(
&model_info,
Path::new("/tmp/rollout.jsonl"),
Path::new("/tmp"),
&input,
)
.unwrap();
assert!(message.contains(&expected_truncated));
}
#[test]
fn build_consolidation_prompt_points_to_workspace_diff_and_extension_tree() {
let temp = tempdir().unwrap();
let memory_root = temp.path().join("memories");
let memory_extensions_root = memory_root.join("extensions");
std::fs::create_dir_all(&memory_extensions_root).unwrap();
let prompt = build_consolidation_prompt(&memory_root);
assert!(prompt.contains("Memory workspace diff:"));
assert!(prompt.contains("phase2_workspace_diff.md"));
assert!(prompt.contains(&format!(
"Memory extensions (under {}/):",
memory_extensions_root.display()
)));
assert!(prompt.contains("workspace diff shows deleted extension resource files"));
}
@@ -5,12 +5,12 @@ use std::path::Path;
use tracing::warn;
use uuid::Uuid;
use crate::memories::ensure_layout;
use crate::memories::raw_memories_file;
use crate::memories::rollout_summaries_dir;
use crate::ensure_layout;
use crate::raw_memories_file;
use crate::rollout_summaries_dir;
/// Rebuild `raw_memories.md` from DB-backed stage-1 outputs.
pub(super) async fn rebuild_raw_memories_file_from_memories(
pub async fn rebuild_raw_memories_file_from_memories(
root: &Path,
memories: &[Stage1Output],
max_raw_memories_for_consolidation: usize,
@@ -20,7 +20,7 @@ pub(super) async fn rebuild_raw_memories_file_from_memories(
}
/// Syncs canonical rollout summary files from DB-backed stage-1 output rows.
pub(super) async fn sync_rollout_summaries_from_memories(
pub async fn sync_rollout_summaries_from_memories(
root: &Path,
memories: &[Stage1Output],
max_raw_memories_for_consolidation: usize,
@@ -38,24 +38,6 @@ pub(super) async fn sync_rollout_summaries_from_memories(
write_rollout_summary_for_thread(root, memory).await?;
}
if retained.is_empty() {
for file_name in ["MEMORY.md", "memory_summary.md"] {
let path = root.join(file_name);
if let Err(err) = tokio::fs::remove_file(path).await
&& err.kind() != std::io::ErrorKind::NotFound
{
return Err(err);
}
}
let skills_dir = root.join("skills");
if let Err(err) = tokio::fs::remove_dir_all(skills_dir).await
&& err.kind() != std::io::ErrorKind::NotFound
{
return Err(err);
}
}
Ok(())
}
@@ -168,7 +150,7 @@ fn rollout_summary_format_error(err: std::fmt::Error) -> std::io::Error {
std::io::Error::other(format!("format rollout summary: {err}"))
}
pub(crate) fn rollout_summary_file_stem(memory: &Stage1Output) -> String {
pub fn rollout_summary_file_stem(memory: &Stage1Output) -> String {
rollout_summary_file_stem_from_parts(
memory.thread_id,
memory.source_updated_at,
@@ -176,7 +158,7 @@ pub(crate) fn rollout_summary_file_stem(memory: &Stage1Output) -> String {
)
}
pub(super) fn rollout_summary_file_stem_from_parts(
fn rollout_summary_file_stem_from_parts(
thread_id: codex_protocol::ThreadId,
source_updated_at: chrono::DateTime<chrono::Utc>,
rollout_slug: Option<&str>,
@@ -1,5 +1,4 @@
use super::rollout_summary_file_stem;
use super::rollout_summary_file_stem_from_parts;
use chrono::TimeZone;
use chrono::Utc;
use codex_protocol::ThreadId;
@@ -32,14 +31,6 @@ fn rollout_summary_file_stem_uses_uuid_timestamp_and_hash_when_slug_missing() {
let memory = stage1_output_with_slug(thread_id, /*rollout_slug*/ None);
assert_eq!(rollout_summary_file_stem(&memory), FIXED_PREFIX);
assert_eq!(
rollout_summary_file_stem_from_parts(
memory.thread_id,
memory.source_updated_at,
memory.rollout_slug.as_deref(),
),
FIXED_PREFIX
);
}
#[test]
+121
View File
@@ -0,0 +1,121 @@
use anyhow::Context;
use codex_git_utils::GitBaselineDiff;
use codex_git_utils::diff_since_latest_init;
use codex_git_utils::ensure_git_baseline_repository;
use codex_git_utils::reset_git_repository;
use std::path::Path;
/// Generated diff file the Phase 2 consolidation agent reads before editing memories.
pub const WORKSPACE_DIFF_FILENAME: &str = "phase2_workspace_diff.md";
const WORKSPACE_DIFF_MAX_BYTES: usize = 4 * 1024 * 1024;
/// Prepares the memory directory for git-baseline diffing.
///
/// This keeps an existing usable `.git/` baseline intact. It initializes a new git baseline when the
/// metadata is missing or unusable, and removes any stale generated `phase2_workspace_diff.md` file
/// so that the next diff does not include a previous prompt artifact.
pub async fn prepare_memory_workspace(root: &Path) -> anyhow::Result<()> {
tokio::fs::create_dir_all(root)
.await
.with_context(|| format!("create memory workspace {}", root.display()))?;
remove_workspace_diff(root).await?;
ensure_git_baseline_repository(root).await?;
Ok(())
}
/// Returns the current workspace diff after removing any stale generated diff artifact.
///
/// The removed file is only `phase2_workspace_diff.md`; memory artifacts and `.git/` metadata are
/// left intact.
pub async fn memory_workspace_diff(root: &Path) -> anyhow::Result<GitBaselineDiff> {
remove_workspace_diff(root).await?;
diff_since_latest_init(root).await
}
/// Writes `phase2_workspace_diff.md` with a bounded git-style diff from the current baseline.
pub async fn write_workspace_diff(root: &Path, diff: &GitBaselineDiff) -> anyhow::Result<()> {
let path = root.join(WORKSPACE_DIFF_FILENAME);
tokio::fs::write(&path, render_workspace_diff_file(diff))
.await
.with_context(|| format!("write memory workspace diff file {}", path.display()))
}
/// Marks the current memory root as the new baseline.
///
/// The generated diff file is removed before resetting the baseline so deleted memory content is
/// not retained in the prompt artifact or in unreachable git objects.
pub async fn reset_memory_workspace_baseline(root: &Path) -> anyhow::Result<()> {
remove_workspace_diff(root).await?;
reset_git_repository(root).await
}
/// Removes the generated `phase2_workspace_diff.md` prompt artifact.
///
/// This does not remove `.git/`, reset the baseline, or delete memory content. It is used before
/// diffing and before baseline reset so the generated diff file itself is not treated as memory
/// workspace input.
pub(super) async fn remove_workspace_diff(root: &Path) -> anyhow::Result<()> {
let path = root.join(WORKSPACE_DIFF_FILENAME);
match tokio::fs::remove_file(&path).await {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err)
.with_context(|| format!("remove memory workspace diff file {}", path.display())),
}
}
fn render_workspace_diff_file(diff: &GitBaselineDiff) -> String {
let mut rendered = String::from(
"# Memory Workspace Diff\n\n\
Generated by Codex before Phase 2 memory consolidation. Read this file first and do not edit it.\n\n\
## Status\n",
);
if !diff.has_changes() {
rendered.push_str("- none\n");
return rendered;
}
for change in &diff.changes {
rendered.push_str(&format!("- {} {}\n", change.status.label(), change.path));
}
rendered.push_str("\n## Diff\n\n```diff\n");
append_bounded_diff(&mut rendered, &diff.unified_diff);
rendered.push_str("```\n");
rendered
}
fn append_bounded_diff(rendered: &mut String, diff: &str) {
if diff.len() <= WORKSPACE_DIFF_MAX_BYTES {
rendered.push_str(diff);
if !diff.ends_with('\n') {
rendered.push('\n');
}
return;
}
let boundary = previous_char_boundary(diff, WORKSPACE_DIFF_MAX_BYTES);
rendered.push_str(&diff[..boundary]);
if !rendered.ends_with('\n') {
rendered.push('\n');
}
rendered.push_str(&format!(
"\n[workspace diff truncated at {WORKSPACE_DIFF_MAX_BYTES} bytes]\n"
));
}
fn previous_char_boundary(value: &str, max_bytes: usize) -> usize {
if max_bytes >= value.len() {
return value.len();
}
let mut index = max_bytes;
while !value.is_char_boundary(index) {
index -= 1;
}
index
}
#[cfg(test)]
#[path = "workspace_tests.rs"]
mod tests;
@@ -0,0 +1,78 @@
use super::*;
use codex_git_utils::GitBaselineChange;
use codex_git_utils::GitBaselineChangeStatus;
use pretty_assertions::assert_eq;
use std::fs;
use tempfile::TempDir;
#[test]
fn render_workspace_diff_file_bounds_large_diff() {
let diff = GitBaselineDiff {
changes: vec![GitBaselineChange {
status: GitBaselineChangeStatus::Modified,
path: "MEMORY.md".to_string(),
}],
unified_diff: "a".repeat(WORKSPACE_DIFF_MAX_BYTES + 128),
};
let rendered = render_workspace_diff_file(&diff);
assert!(rendered.contains("- M MEMORY.md"));
assert!(rendered.contains("[workspace diff truncated at 4194304 bytes]"));
assert!(rendered.ends_with("```\n"));
}
#[tokio::test]
async fn reset_memory_workspace_baseline_removes_generated_diff() {
let home = TempDir::new().expect("tempdir");
let root = home.path().join("memories");
prepare_memory_workspace(&root)
.await
.expect("prepare memory workspace");
fs::write(root.join("MEMORY.md"), "memory").expect("write memory");
write_workspace_diff(
&root,
&GitBaselineDiff {
changes: vec![GitBaselineChange {
status: GitBaselineChangeStatus::Added,
path: "MEMORY.md".to_string(),
}],
unified_diff: "+memory\n".to_string(),
},
)
.await
.expect("write workspace diff");
reset_memory_workspace_baseline(&root)
.await
.expect("reset baseline");
assert!(!root.join(WORKSPACE_DIFF_FILENAME).exists());
let diff = memory_workspace_diff(&root)
.await
.expect("load workspace diff");
assert_eq!(diff.changes, Vec::new());
}
#[tokio::test]
async fn prepare_memory_workspace_recovers_unusable_git_dir() {
let home = TempDir::new().expect("tempdir");
let root = home.path().join("memories");
fs::create_dir_all(root.join(".git")).expect("create unusable git dir");
fs::write(root.join("MEMORY.md"), "memory").expect("write memory");
prepare_memory_workspace(&root)
.await
.expect("prepare memory workspace");
let diff = memory_workspace_diff(&root)
.await
.expect("load workspace diff");
assert_eq!(diff.changes, Vec::new());
}
#[test]
fn previous_char_boundary_handles_multibyte_text() {
let text = "aé";
assert_eq!(previous_char_boundary(text, /*max_bytes*/ 2), 1);
}
@@ -143,29 +143,34 @@ Mode selection:
- INCREMENTAL UPDATE: existing artifacts already exist and `raw_memories.md`
mostly contains new additions.
Incremental thread diff snapshot (computed before the current artifact sync rewrites local files):
Memory workspace diff:
**Diff since last consolidation:**
{{ phase2_input_selection }}
The folder `{{ memory_root }}/` is a git repository managed by Codex. Read
`{{ phase2_workspace_diff_file }}` in this same folder first. It contains the git-style diff from
the previous successful Phase 2 baseline to the current worktree. It is generated by Codex for
this run and is not part of the committed memory artifacts.
Incremental update and forgetting mechanism:
- Use the diff provided
- Use the git-style diff in `{{ phase2_workspace_diff_file }}` to identify relevant changed
sections and deleted inputs.
- Every changes in `{{ phase2_workspace_diff_file }}` are authoritative and must propagated and consolidated. If a
changes appears to be randomly placed in the files, it is probably a user change and you shouldn't just drop it.
Make sure to add it to the overall memories consolidation
- Do not open raw sessions / original rollout transcripts.
- For each added thread id, search it in `raw_memories.md`, read that raw-memory section, and
read the corresponding `rollout_summaries/*.md` file only when needed for stronger evidence,
task placement, or conflict resolution.
- For added or modified `raw_memories.md` and `rollout_summaries/*.md` files, read the changed
raw-memory sections and the corresponding rollout summaries only when needed for stronger
evidence, task placement, or conflict resolution.
- When scanning a raw-memory section, read the task-level `Preference signals:` subsections
first, then the rest of the task blocks.
- For each removed thread id, search it in `MEMORY.md` and delete only the memory supported by
that thread. Use `thread_id=<thread_id>` in `### rollout_summary_files` when available; if not,
fall back to rollout summary filenames plus the corresponding `rollout_summaries/*.md` files.
- If a `MEMORY.md` block contains both removed and undeleted threads, do not delete the whole
block. Remove only the removed thread's references and thread-local guidance, preserve shared
or still-supported content, and split or rewrite the block only if needed to keep the undeleted
threads intact.
- For deleted `rollout_summaries/*.md` or `extensions/*/resources/*.md` files, search their
filenames, paths, and thread ids (when present) in `MEMORY.md`. Delete only memory supported
by deleted inputs.
- If a `MEMORY.md` block contains both deleted and still-present evidence, do not delete the whole
block. Remove only stale references and stale local guidance, preserve shared or still-supported
content, and split or rewrite the block only if needed.
- After `MEMORY.md` cleanup is done, revisit `memory_summary.md` and remove or rewrite stale
summary/index content that was only supported by removed thread ids.
summary/index content that was only supported by deleted files.
Outputs:
Under `{{ memory_root }}/`:
@@ -743,26 +748,28 @@ WORKFLOW
3. INCREMENTAL UPDATE behavior:
- Read existing `MEMORY.md` and `memory_summary.md` first for continuity and to locate
existing references that may need surgical cleanup.
- Use the injected thread-diff snapshot as the first routing pass:
- added thread ids = ingestion queue
- removed thread ids = forgetting / stale-cleanup queue
- Use the injected git-style workspace changes as the first routing pass:
- added/modified `raw_memories.md` and `rollout_summaries/*.md` = ingestion queue
- deleted `rollout_summaries/*.md` and `extensions/*/resources/*.md` = forgetting /
stale-cleanup queue
- Build an index of rollout references already present in existing `MEMORY.md` before
scanning raw memories so you can route net-new evidence into the right blocks.
- Work in this order:
1. For newly added thread ids, search them in `raw_memories.md`, read those sections, and
open the corresponding `rollout_summaries/*.md` files when necessary.
1. For added or modified rollout inputs, search their paths/thread ids in `raw_memories.md`,
read those sections, and open the corresponding `rollout_summaries/*.md` files when
necessary.
2. Route the new signal into existing `MEMORY.md` blocks or create new ones when needed.
3. For removed thread ids, search `MEMORY.md` and surgically delete or rewrite only the
unsupported thread-local memory.
4. If a block mixes removed and undeleted threads, preserve the undeleted-thread content;
split or rewrite the block if that is the cleanest way to delete only the removed part.
3. For deleted inputs, search `MEMORY.md` and surgically delete or rewrite only the
unsupported memory.
4. If a block mixes deleted and still-present evidence, preserve the still-supported content;
split or rewrite the block if that is the cleanest way to delete only the stale part.
5. After `MEMORY.md` is correct, revisit `memory_summary.md` and remove or rewrite stale
summary/index content that no longer has undeleted support.
summary/index content that no longer has current support.
- Integrate new signal into existing artifacts by:
- scanning the newly added raw-memory entries in recency order and identifying which existing blocks they should update
- scanning added or modified raw-memory entries in recency order and identifying which existing blocks they should update
- updating existing knowledge with better/newer evidence
- updating stale or contradicting guidance
- pruning or downgrading memory whose only provenance comes from removed thread ids
- pruning or downgrading memory whose only provenance comes from deleted inputs
- expanding terse old blocks when new summaries/raw memories make the task family clearer
- doing light clustering and merging if needed
- refreshing `MEMORY.md` top-of-file ordering so recent high-utility task families stay easy to find
@@ -774,8 +781,8 @@ WORKFLOW
target, keep its wording, label, and relative order mostly stable. Rewrite/reorder/rename/
split/merge only when fixing a real problem (staleness, ambiguity, schema drift, wrong
boundaries) or when meaningful new evidence materially improves retrieval clarity/searchability.
- Spend most of your deep-dive budget on newly added thread ids and on mixed blocks touched by
removed thread ids. Do not re-read unchanged older threads unless you need them for
- Spend most of your deep-dive budget on added/modified inputs and on mixed blocks touched by
deleted inputs. Do not re-read unchanged older threads unless you need them for
conflict resolution, clustering, or provenance repair.
4. Evidence deep-dive rule (both modes):
@@ -793,8 +800,7 @@ WORKFLOW
evidence, procedural detail, validation signals, and user feedback before finalizing
`MEMORY.md`.
- When deleting stale memory from a mixed block, use the relevant rollout summaries to decide
which details are uniquely supported by removed threads versus still supported by undeleted
threads.
which details are uniquely supported by deleted inputs versus still-supported evidence.
- Use `updated_at` and validation strength together to resolve stale/conflicting notes.
- For user-profile or preference claims, recurrence matters: repeated evidence across
rollouts should generally outrank a single polished but isolated summary.
@@ -811,7 +817,7 @@ WORKFLOW
- remove duplication in memory_summary, skills/, and MEMORY.md
- remove stale or low-signal blocks that are less likely to be useful in the future
- remove or rewrite blocks/task sections whose supporting rollout references point only to
removed thread ids or missing rollout summary files
deleted inputs or missing rollout summary files
- run a global rollout-reference audit on final `MEMORY.md` and fix accidental duplicate
entries / redundant repetition, while preserving intentional multi-task or multi-block
reuse when it adds distinct task-local value
+3 -2
View File
@@ -206,7 +206,7 @@ fn static_manager_for_tests(model_catalog: ModelsResponse) -> StaticModelsManage
)
}
fn chatgpt_auth_tokens_for_tests(codex_home: &Path) -> CodexAuth {
async fn chatgpt_auth_tokens_for_tests(codex_home: &Path) -> CodexAuth {
let auth_dot_json = codex_login::AuthDotJson {
auth_mode: Some(AuthMode::ChatgptAuthTokens),
openai_api_key: None,
@@ -232,6 +232,7 @@ c2ln",
.expect("auth.json should be written");
CodexAuth::from_auth_storage(codex_home, AuthCredentialsStoreMode::File)
.await
.expect("auth should load")
.expect("auth should be present")
}
@@ -685,7 +686,7 @@ async fn refresh_available_models_fetches_with_chatgpt_auth_tokens() {
"ChatGPT Auth Tokens",
/*priority*/ 1,
)]]);
let auth = chatgpt_auth_tokens_for_tests(codex_home.path());
let auth = chatgpt_auth_tokens_for_tests(codex_home.path()).await;
let manager = openai_manager_for_tests_with_auth(
codex_home.path().to_path_buf(),
endpoint.clone(),
+1 -1
View File
@@ -82,7 +82,7 @@ pub enum CodexErr {
ContextWindowExceeded,
#[error("no thread with id: {0}")]
ThreadNotFound(ThreadId),
#[error("agent thread limit reached (max {max_threads})")]
#[error("agent thread limit reached")]
AgentLimitReached { max_threads: usize },
#[error("session configured event was not the first event in the stream")]
SessionConfiguredNotFirstEvent,
-2
View File
@@ -14,7 +14,6 @@ mod runtime;
pub use model::LogEntry;
pub use model::LogQuery;
pub use model::LogRow;
pub use model::Phase2InputSelection;
pub use model::Phase2JobClaimOutcome;
/// Preferred entrypoint: owns configuration and metrics.
pub use runtime::StateRuntime;
@@ -42,7 +41,6 @@ pub use model::SortKey;
pub use model::Stage1JobClaim;
pub use model::Stage1JobClaimOutcome;
pub use model::Stage1Output;
pub use model::Stage1OutputRef;
pub use model::Stage1StartupClaimParams;
pub use model::ThreadGoal;
pub use model::ThreadGoalStatus;
+3 -30
View File
@@ -22,21 +22,6 @@ pub struct Stage1Output {
pub generated_at: DateTime<Utc>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Stage1OutputRef {
pub thread_id: ThreadId,
pub source_updated_at: DateTime<Utc>,
pub rollout_slug: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct Phase2InputSelection {
pub selected: Vec<Stage1Output>,
pub previous_selected: Vec<Stage1Output>,
pub retained_thread_ids: Vec<ThreadId>,
pub removed: Vec<Stage1OutputRef>,
}
#[derive(Debug)]
pub(crate) struct Stage1OutputRow {
thread_id: String,
@@ -89,18 +74,6 @@ fn epoch_seconds_to_datetime(secs: i64) -> Result<DateTime<Utc>> {
.ok_or_else(|| anyhow::anyhow!("invalid unix timestamp: {secs}"))
}
pub(crate) fn stage1_output_ref_from_parts(
thread_id: String,
source_updated_at: i64,
rollout_slug: Option<String>,
) -> Result<Stage1OutputRef> {
Ok(Stage1OutputRef {
thread_id: ThreadId::try_from(thread_id)?,
source_updated_at: epoch_seconds_to_datetime(source_updated_at)?,
rollout_slug,
})
}
/// Result of trying to claim a stage-1 memory extraction job.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Stage1JobClaimOutcome {
@@ -136,14 +109,14 @@ pub struct Stage1StartupClaimParams<'a> {
/// Result of trying to claim a phase-2 consolidation job.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Phase2JobClaimOutcome {
/// The caller owns the global lock and should spawn consolidation.
/// The caller owns the global lock and may inspect the memory workspace.
Claimed {
ownership_token: String,
/// Snapshot of `input_watermark` at claim time.
input_watermark: i64,
},
/// The global job is not pending consolidation (or is already up to date).
SkippedNotDirty,
/// The global job is in retry backoff.
SkippedRetryUnavailable,
/// Another worker currently owns a fresh global consolidation lease.
SkippedRunning,
}
-3
View File
@@ -19,12 +19,10 @@ pub use graph::DirectionalThreadSpawnEdgeStatus;
pub use log::LogEntry;
pub use log::LogQuery;
pub use log::LogRow;
pub use memories::Phase2InputSelection;
pub use memories::Phase2JobClaimOutcome;
pub use memories::Stage1JobClaim;
pub use memories::Stage1JobClaimOutcome;
pub use memories::Stage1Output;
pub use memories::Stage1OutputRef;
pub use memories::Stage1StartupClaimParams;
pub use thread_goal::ThreadGoal;
pub use thread_goal::ThreadGoalStatus;
@@ -40,7 +38,6 @@ pub use thread_metadata::ThreadsPage;
pub(crate) use agent_job::AgentJobItemRow;
pub(crate) use agent_job::AgentJobRow;
pub(crate) use memories::Stage1OutputRow;
pub(crate) use memories::stage1_output_ref_from_parts;
pub(crate) use thread_goal::ThreadGoalRow;
pub(crate) use thread_metadata::ThreadRow;
pub(crate) use thread_metadata::anchor_from_item;
+249 -222
View File
@@ -3,7 +3,6 @@ use super::threads::push_thread_filters;
use super::threads::push_thread_order_and_limit;
use super::*;
use crate::SortDirection;
use crate::model::Phase2InputSelection;
use crate::model::Phase2JobClaimOutcome;
use crate::model::Stage1JobClaim;
use crate::model::Stage1JobClaimOutcome;
@@ -11,12 +10,10 @@ use crate::model::Stage1Output;
use crate::model::Stage1OutputRow;
use crate::model::Stage1StartupClaimParams;
use crate::model::ThreadRow;
use crate::model::stage1_output_ref_from_parts;
use chrono::Duration;
use sqlx::Executor;
use sqlx::QueryBuilder;
use sqlx::Sqlite;
use std::collections::HashSet;
use uuid::Uuid;
const JOB_KIND_MEMORY_STAGE1: &str = "memory_stage1";
@@ -331,8 +328,7 @@ WHERE thread_id IN (
Ok(rows_affected as usize)
}
/// Returns the current phase-2 input set along with its diff against the
/// last successful phase-2 selection.
/// Returns the current phase-2 input set.
///
/// Query behavior:
/// - current selection keeps only non-empty stage-1 outputs whose
@@ -342,22 +338,17 @@ WHERE thread_id IN (
/// - eligible rows are ordered by `usage_count DESC`,
/// `COALESCE(last_usage, source_updated_at) DESC`, `source_updated_at DESC`,
/// `thread_id DESC`
/// - previously selected rows are identified by `selected_for_phase2 = 1`
/// - `previous_selected` contains the current persisted rows that belonged
/// to the last successful phase-2 baseline, even if those threads are no
/// longer memory-eligible
/// - `retained_thread_ids` records which current rows still match the exact
/// snapshot selected in the last successful phase-2 run
/// - removed rows are previously selected rows that are still present in
/// `stage1_outputs` but are no longer in the current selection, including
/// threads that are no longer memory-eligible
///
/// The returned rows are the complete Phase 2 filesystem input. Phase 2
/// syncs these rows directly; deletions are represented by the workspace
/// diff against the previous successful memory baseline.
pub async fn get_phase2_input_selection(
&self,
n: usize,
max_unused_days: i64,
) -> anyhow::Result<Phase2InputSelection> {
) -> anyhow::Result<Vec<Stage1Output>> {
if n == 0 {
return Ok(Phase2InputSelection::default());
return Ok(Vec::new());
}
let cutoff = (Utc::now() - Duration::days(max_unused_days.max(0))).timestamp();
@@ -372,9 +363,7 @@ SELECT
so.rollout_slug,
so.generated_at,
COALESCE(t.cwd, '') AS cwd,
t.git_branch AS git_branch,
so.selected_for_phase2,
so.selected_for_phase2_source_updated_at
t.git_branch AS git_branch
FROM stage1_outputs AS so
LEFT JOIN threads AS t
ON t.id = so.thread_id
@@ -398,70 +387,14 @@ LIMIT ?
.fetch_all(self.pool.as_ref())
.await?;
let mut current_thread_ids = HashSet::with_capacity(current_rows.len());
let mut selected = Vec::with_capacity(current_rows.len());
let mut retained_thread_ids = Vec::new();
for row in current_rows {
let thread_id = row.try_get::<String, _>("thread_id")?;
current_thread_ids.insert(thread_id.clone());
let source_updated_at = row.try_get::<i64, _>("source_updated_at")?;
if row.try_get::<i64, _>("selected_for_phase2")? != 0
&& row.try_get::<Option<i64>, _>("selected_for_phase2_source_updated_at")?
== Some(source_updated_at)
{
retained_thread_ids.push(ThreadId::try_from(thread_id.clone())?);
}
selected.push(Stage1Output::try_from(Stage1OutputRow::try_from_row(
&row,
)?)?);
}
let previous_rows = sqlx::query(
r#"
SELECT
so.thread_id,
COALESCE(t.rollout_path, '') AS rollout_path,
so.source_updated_at,
so.raw_memory,
so.rollout_summary,
so.rollout_slug,
so.generated_at,
COALESCE(t.cwd, '') AS cwd,
t.git_branch AS git_branch
FROM stage1_outputs AS so
LEFT JOIN threads AS t
ON t.id = so.thread_id
WHERE so.selected_for_phase2 = 1
ORDER BY so.source_updated_at DESC, so.thread_id DESC
"#,
)
.fetch_all(self.pool.as_ref())
.await?;
let previous_selected = previous_rows
.iter()
.map(Stage1OutputRow::try_from_row)
.map(|row| row.and_then(Stage1Output::try_from))
.collect::<Result<Vec<_>, _>>()?;
let mut removed = Vec::new();
for row in previous_rows {
let thread_id = row.try_get::<String, _>("thread_id")?;
if current_thread_ids.contains(thread_id.as_str()) {
continue;
}
removed.push(stage1_output_ref_from_parts(
thread_id,
row.try_get("source_updated_at")?,
row.try_get("rollout_slug")?,
)?);
}
Ok(Phase2InputSelection {
selected,
previous_selected,
retained_thread_ids,
removed,
})
Ok(selected)
}
/// Marks a thread as polluted and enqueues phase-2 forgetting when the
@@ -909,19 +842,22 @@ WHERE kind = ? AND job_key = ?
/// Enqueues or advances the global phase-2 consolidation job watermark.
///
/// The underlying upsert keeps the job `running` when already running, resets
/// `pending/error` jobs to `pending`, and advances `input_watermark` so each
/// enqueue marks new consolidation work even when `source_updated_at` is
/// older than prior maxima.
/// `pending/error` jobs to `pending`, and advances `input_watermark` as
/// bookkeeping even when `source_updated_at` is older than prior maxima.
/// Phase 2 does not use this watermark as a dirty check; git workspace diffing
/// decides whether consolidation work exists after the lock is claimed.
pub async fn enqueue_global_consolidation(&self, input_watermark: i64) -> anyhow::Result<()> {
enqueue_global_consolidation_with_executor(self.pool.as_ref(), input_watermark).await
}
/// Attempts to claim the global phase-2 consolidation job.
/// Attempts to claim the global phase-2 consolidation lock.
///
/// Claim semantics:
/// - reads the singleton global job row (`kind='memory_consolidate_global'`)
/// - returns `SkippedNotDirty` when `input_watermark <= last_success_watermark`
/// - returns `SkippedNotDirty` when retries are exhausted or retry backoff is active
/// - creates and claims the singleton row when it does not exist yet
/// - does not use DB watermarks to decide whether Phase 2 has work; git workspace
/// dirtiness is the source of truth after the caller materializes inputs
/// - returns `SkippedRetryUnavailable` when retry backoff is active
/// - returns `SkippedRunning` when an active running lease exists
/// - otherwise updates the row to `running`, sets ownership + lease, and
/// returns `Claimed`
@@ -939,7 +875,7 @@ WHERE kind = ? AND job_key = ?
let existing_job = sqlx::query(
r#"
SELECT status, lease_until, retry_at, retry_remaining, input_watermark, last_success_watermark
SELECT status, lease_until, retry_at, input_watermark
FROM jobs
WHERE kind = ? AND job_key = ?
"#,
@@ -950,30 +886,55 @@ WHERE kind = ? AND job_key = ?
.await?;
let Some(existing_job) = existing_job else {
let rows_affected = sqlx::query(
r#"
INSERT INTO jobs (
kind,
job_key,
status,
worker_id,
ownership_token,
started_at,
finished_at,
lease_until,
retry_at,
retry_remaining,
last_error,
input_watermark,
last_success_watermark
) VALUES (?, ?, 'running', ?, ?, ?, NULL, ?, NULL, ?, NULL, 0, 0)
"#,
)
.bind(JOB_KIND_MEMORY_CONSOLIDATE_GLOBAL)
.bind(MEMORY_CONSOLIDATION_JOB_KEY)
.bind(worker_id.as_str())
.bind(ownership_token.as_str())
.bind(now)
.bind(lease_until)
.bind(DEFAULT_RETRY_REMAINING)
.execute(&mut *tx)
.await?
.rows_affected();
tx.commit().await?;
return Ok(Phase2JobClaimOutcome::SkippedNotDirty);
return if rows_affected == 0 {
Ok(Phase2JobClaimOutcome::SkippedRunning)
} else {
Ok(Phase2JobClaimOutcome::Claimed {
ownership_token,
input_watermark: 0,
})
};
};
let input_watermark: Option<i64> = existing_job.try_get("input_watermark")?;
let input_watermark_value = input_watermark.unwrap_or(0);
let last_success_watermark: Option<i64> = existing_job.try_get("last_success_watermark")?;
if input_watermark_value <= last_success_watermark.unwrap_or(0) {
tx.commit().await?;
return Ok(Phase2JobClaimOutcome::SkippedNotDirty);
}
let status: String = existing_job.try_get("status")?;
let existing_lease_until: Option<i64> = existing_job.try_get("lease_until")?;
let retry_at: Option<i64> = existing_job.try_get("retry_at")?;
let retry_remaining: i64 = existing_job.try_get("retry_remaining")?;
if retry_remaining <= 0 {
tx.commit().await?;
return Ok(Phase2JobClaimOutcome::SkippedNotDirty);
}
if retry_at.is_some_and(|retry_at| retry_at > now) {
tx.commit().await?;
return Ok(Phase2JobClaimOutcome::SkippedNotDirty);
return Ok(Phase2JobClaimOutcome::SkippedRetryUnavailable);
}
if status == "running" && existing_lease_until.is_some_and(|lease_until| lease_until > now)
{
@@ -994,10 +955,8 @@ SET
retry_at = NULL,
last_error = NULL
WHERE kind = ? AND job_key = ?
AND input_watermark > COALESCE(last_success_watermark, 0)
AND (status != 'running' OR lease_until IS NULL OR lease_until <= ?)
AND (retry_at IS NULL OR retry_at <= ?)
AND retry_remaining > 0
"#,
)
.bind(worker_id.as_str())
@@ -1063,37 +1022,17 @@ WHERE kind = ? AND job_key = ?
/// `max(existing_last_success_watermark, completed_watermark)`
/// - rewrites `selected_for_phase2` so only the exact selected stage-1
/// snapshots remain marked as part of the latest successful phase-2
/// selection, and persists each selected snapshot's
/// `source_updated_at` for future retained-vs-added diffing
/// selection, and persists each selected snapshot's `source_updated_at`
pub async fn mark_global_phase2_job_succeeded(
&self,
ownership_token: &str,
completed_watermark: i64,
selected_outputs: &[Stage1Output],
) -> anyhow::Result<bool> {
let now = Utc::now().timestamp();
let mut tx = self.pool.begin().await?;
let rows_affected = sqlx::query(
r#"
UPDATE jobs
SET
status = 'done',
finished_at = ?,
lease_until = NULL,
last_error = NULL,
last_success_watermark = max(COALESCE(last_success_watermark, 0), ?)
WHERE kind = ? AND job_key = ?
AND status = 'running' AND ownership_token = ?
"#,
)
.bind(now)
.bind(completed_watermark)
.bind(JOB_KIND_MEMORY_CONSOLIDATE_GLOBAL)
.bind(MEMORY_CONSOLIDATION_JOB_KEY)
.bind(ownership_token)
.execute(&mut *tx)
.await?
.rows_affected();
let rows_affected =
mark_global_phase2_job_succeeded_row(&mut *tx, ownership_token, completed_watermark)
.await?;
if rows_affected == 0 {
tx.commit().await?;
@@ -1139,7 +1078,7 @@ WHERE thread_id = ? AND source_updated_at = ?
/// - updates only the owned running singleton global row
/// - sets `status='error'`, clears lease
/// - writes failure reason and retry time
/// - decrements `retry_remaining`
/// - decrements `retry_remaining` without going below zero
pub async fn mark_global_phase2_job_failed(
&self,
ownership_token: &str,
@@ -1156,7 +1095,7 @@ SET
finished_at = ?,
lease_until = NULL,
retry_at = ?,
retry_remaining = retry_remaining - 1,
retry_remaining = max(retry_remaining - 1, 0),
last_error = ?
WHERE kind = ? AND job_key = ?
AND status = 'running' AND ownership_token = ?
@@ -1197,7 +1136,7 @@ SET
finished_at = ?,
lease_until = NULL,
retry_at = ?,
retry_remaining = retry_remaining - 1,
retry_remaining = max(retry_remaining - 1, 0),
last_error = ?
WHERE kind = ? AND job_key = ?
AND status = 'running'
@@ -1218,6 +1157,40 @@ WHERE kind = ? AND job_key = ?
}
}
async fn mark_global_phase2_job_succeeded_row<'e, E>(
executor: E,
ownership_token: &str,
completed_watermark: i64,
) -> anyhow::Result<u64>
where
E: Executor<'e, Database = Sqlite>,
{
let now = Utc::now().timestamp();
let rows_affected = sqlx::query(
r#"
UPDATE jobs
SET
status = 'done',
finished_at = ?,
lease_until = NULL,
last_error = NULL,
last_success_watermark = max(COALESCE(last_success_watermark, 0), ?)
WHERE kind = ? AND job_key = ?
AND status = 'running' AND ownership_token = ?
"#,
)
.bind(now)
.bind(completed_watermark)
.bind(JOB_KIND_MEMORY_CONSOLIDATE_GLOBAL)
.bind(MEMORY_CONSOLIDATION_JOB_KEY)
.bind(ownership_token)
.execute(executor)
.await?
.rows_affected();
Ok(rows_affected)
}
async fn enqueue_global_consolidation_with_executor<'e, E>(
executor: E,
input_watermark: i64,
@@ -2271,16 +2244,6 @@ WHERE kind = 'memory_stage1'
"no-output without an existing stage1 output should not enqueue phase2"
);
let claim_phase2 = runtime
.try_claim_global_phase2_job(owner, /*lease_seconds*/ 3600)
.await
.expect("claim phase2");
assert_eq!(
claim_phase2,
Phase2JobClaimOutcome::SkippedNotDirty,
"phase2 should remain clean when no-output deleted nothing"
);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
@@ -2350,7 +2313,7 @@ WHERE kind = 'memory_stage1'
)
.await
.expect("mark initial phase2 succeeded"),
"initial phase2 success should clear global dirty state"
"initial phase2 success should finalize the global job"
);
let no_output_claim = runtime
@@ -2505,7 +2468,7 @@ WHERE kind = 'memory_stage1'
}
#[tokio::test]
async fn phase2_global_consolidation_reruns_when_watermark_advances() {
async fn phase2_global_lock_can_be_reclaimed_after_success_without_new_watermark() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -2537,24 +2500,83 @@ WHERE kind = 'memory_stage1'
"phase2 success should finalize for current token"
);
let claim_up_to_date = runtime
let claim_after_success = runtime
.try_claim_global_phase2_job(owner, /*lease_seconds*/ 3600)
.await
.expect("claim phase2 up-to-date");
assert_eq!(claim_up_to_date, Phase2JobClaimOutcome::SkippedNotDirty);
.expect("claim phase2 after success");
assert!(
matches!(claim_after_success, Phase2JobClaimOutcome::Claimed { .. }),
"the DB claim is only a lock; git workspace diff decides whether there is work"
);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
#[tokio::test]
async fn phase2_global_lock_can_be_claimed_after_retry_budget_is_exhausted() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
.expect("initialize runtime");
runtime
.enqueue_global_consolidation(/*input_watermark*/ 101)
.enqueue_global_consolidation(/*input_watermark*/ 100)
.await
.expect("enqueue global consolidation again");
.expect("enqueue global consolidation");
let claim_rerun = runtime
.try_claim_global_phase2_job(owner, /*lease_seconds*/ 3600)
let owner = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("owner id");
for attempt in 0..3 {
let claim = runtime
.try_claim_global_phase2_job(owner, /*lease_seconds*/ 3_600)
.await
.expect("claim phase2 before retry exhaustion");
let ownership_token = match claim {
Phase2JobClaimOutcome::Claimed {
ownership_token, ..
} => ownership_token,
other => panic!(
"attempt {} should claim phase2 before retries are exhausted: {other:?}",
attempt + 1
),
};
assert!(
runtime
.mark_global_phase2_job_failed(
ownership_token.as_str(),
"boom",
/*retry_delay_seconds*/ 0,
)
.await
.expect("mark phase2 failed"),
"attempt {} should decrement retry budget",
attempt + 1
);
}
let job_row =
sqlx::query("SELECT retry_remaining FROM jobs WHERE kind = ? AND job_key = ?")
.bind("memory_consolidate_global")
.bind("global")
.fetch_one(runtime.pool.as_ref())
.await
.expect("load phase2 job row after retry exhaustion");
assert_eq!(
job_row
.try_get::<i64, _>("retry_remaining")
.expect("retry_remaining"),
0
);
let claim_after_exhaustion = runtime
.try_claim_global_phase2_job(owner, /*lease_seconds*/ 3_600)
.await
.expect("claim phase2 rerun");
.expect("claim phase2 after retry exhaustion");
assert!(
matches!(claim_rerun, Phase2JobClaimOutcome::Claimed { .. }),
"advanced watermark should be claimable"
matches!(
claim_after_exhaustion,
Phase2JobClaimOutcome::Claimed { .. }
),
"phase2 claim should only lock; workspace diffing decides whether there is work"
);
let _ = tokio::fs::remove_dir_all(codex_home).await;
@@ -2801,7 +2823,7 @@ VALUES (?, ?, ?, ?, ?)
}
#[tokio::test]
async fn get_phase2_input_selection_reports_added_retained_and_removed_rows() {
async fn get_phase2_input_selection_returns_current_selected_rows() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -2895,28 +2917,19 @@ VALUES (?, ?, ?, ?, ?)
.await
.expect("load phase2 input selection");
assert_eq!(selection.selected.len(), 2);
assert_eq!(selection.previous_selected.len(), 2);
assert_eq!(selection.selected[0].thread_id, thread_id_c);
assert_eq!(selection.len(), 2);
assert_eq!(selection[0].thread_id, thread_id_c);
assert_eq!(
selection.selected[0].rollout_path,
selection[0].rollout_path,
codex_home.join(format!("rollout-{thread_id_c}.jsonl"))
);
assert_eq!(selection.selected[1].thread_id, thread_id_b);
assert_eq!(selection.retained_thread_ids, vec![thread_id_c]);
assert_eq!(selection.removed.len(), 1);
assert_eq!(selection.removed[0].thread_id, thread_id_a);
assert_eq!(
selection.removed[0].rollout_slug.as_deref(),
Some("rollout-a")
);
assert_eq!(selection[1].thread_id, thread_id_b);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
#[tokio::test]
async fn get_phase2_input_selection_marks_polluted_previous_selection_as_removed() {
async fn get_phase2_input_selection_excludes_polluted_previous_selection() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -3002,24 +3015,8 @@ VALUES (?, ?, ?, ?, ?)
.await
.expect("load phase2 input selection");
assert_eq!(selection.selected.len(), 1);
assert_eq!(selection.selected[0].thread_id, thread_id_enabled);
assert_eq!(selection.previous_selected.len(), 2);
assert!(
selection
.previous_selected
.iter()
.any(|item| item.thread_id == thread_id_enabled)
);
assert!(
selection
.previous_selected
.iter()
.any(|item| item.thread_id == thread_id_polluted)
);
assert_eq!(selection.retained_thread_ids, vec![thread_id_enabled]);
assert_eq!(selection.removed.len(), 1);
assert_eq!(selection.removed[0].thread_id, thread_id_polluted);
assert_eq!(selection.len(), 1);
assert_eq!(selection[0].thread_id, thread_id_enabled);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
@@ -3113,7 +3110,7 @@ VALUES (?, ?, ?, ?, ?)
}
#[tokio::test]
async fn get_phase2_input_selection_treats_regenerated_selected_rows_as_added() {
async fn get_phase2_input_selection_returns_regenerated_selected_rows() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -3213,12 +3210,9 @@ VALUES (?, ?, ?, ?, ?)
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 36_500)
.await
.expect("load phase2 input selection");
assert_eq!(selection.selected.len(), 1);
assert_eq!(selection.previous_selected.len(), 1);
assert_eq!(selection.selected[0].thread_id, thread_id);
assert_eq!(selection.selected[0].source_updated_at.timestamp(), 101);
assert!(selection.retained_thread_ids.is_empty());
assert!(selection.removed.is_empty());
assert_eq!(selection.len(), 1);
assert_eq!(selection[0].thread_id, thread_id);
assert_eq!(selection[0].source_updated_at.timestamp(), 101);
let (selected_for_phase2, selected_for_phase2_source_updated_at) =
sqlx::query_as::<_, (i64, Option<i64>)>(
@@ -3235,7 +3229,7 @@ VALUES (?, ?, ?, ?, ?)
}
#[tokio::test]
async fn get_phase2_input_selection_reports_regenerated_previous_selection_as_removed() {
async fn get_phase2_input_selection_uses_current_ranking_after_refreshes() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -3368,29 +3362,11 @@ VALUES (?, ?, ?, ?, ?)
.expect("load phase2 input selection");
assert_eq!(
selection
.selected
.iter()
.map(|output| output.thread_id)
.collect::<Vec<_>>(),
vec![thread_id_d, thread_id_c]
);
assert_eq!(
selection
.previous_selected
.iter()
.map(|output| output.thread_id)
.collect::<Vec<_>>(),
vec![thread_id_a, thread_id_b]
);
assert!(selection.retained_thread_ids.is_empty());
assert_eq!(
selection
.removed
.iter()
.map(|output| (output.thread_id, output.source_updated_at.timestamp()))
.collect::<Vec<_>>(),
vec![(thread_id_a, 102), (thread_id_b, 101)]
);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
@@ -3527,7 +3503,8 @@ VALUES (?, ?, ?, ?, ?)
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 36_500)
.await
.expect("load phase2 input selection after refresh");
assert_eq!(selection.retained_thread_ids, vec![thread_id]);
assert_eq!(selection.len(), 1);
assert_eq!(selection[0].thread_id, thread_id);
let (selected_for_phase2, selected_for_phase2_source_updated_at) =
sqlx::query_as::<_, (i64, Option<i64>)>(
@@ -3657,9 +3634,8 @@ VALUES (?, ?, ?, ?, ?)
.get_phase2_input_selection(/*n*/ 1, /*max_unused_days*/ 36_500)
.await
.expect("load phase2 input selection");
assert_eq!(selection.selected.len(), 1);
assert_eq!(selection.selected[0].source_updated_at.timestamp(), 101);
assert!(selection.retained_thread_ids.is_empty());
assert_eq!(selection.len(), 1);
assert_eq!(selection[0].source_updated_at.timestamp(), 101);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
@@ -3870,7 +3846,6 @@ VALUES (?, ?, ?, ?, ?)
assert_eq!(
selection
.selected
.iter()
.map(|output| output.thread_id)
.collect::<Vec<_>>(),
@@ -3967,7 +3942,6 @@ VALUES (?, ?, ?, ?, ?)
assert_eq!(
selection
.selected
.iter()
.map(|output| output.thread_id)
.collect::<Vec<_>>(),
@@ -4056,9 +4030,9 @@ VALUES (?, ?, ?, ?, ?)
.await
.expect("load phase2 input selection");
assert_eq!(selection.selected.len(), 1);
assert_eq!(selection.selected[0].thread_id, newer_thread);
assert_eq!(selection.selected[0].source_updated_at.timestamp(), 200);
assert_eq!(selection.len(), 1);
assert_eq!(selection[0].thread_id, newer_thread);
assert_eq!(selection[0].source_updated_at.timestamp(), 200);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
@@ -4411,6 +4385,59 @@ VALUES (?, ?, ?, ?, ?)
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
#[tokio::test]
async fn phase2_global_lock_creates_missing_job_row() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
.expect("initialize runtime");
let owner_a = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("owner a");
let owner_b = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("owner b");
let claim = runtime
.try_claim_global_phase2_job(owner_a, /*lease_seconds*/ 3_600)
.await
.expect("claim global phase2 lock");
let ownership_token = match claim {
Phase2JobClaimOutcome::Claimed {
ownership_token,
input_watermark,
} => {
assert_eq!(input_watermark, 0);
ownership_token
}
other => panic!("unexpected phase2 lock claim outcome: {other:?}"),
};
let second_claim = runtime
.try_claim_global_phase2_job(owner_b, /*lease_seconds*/ 3_600)
.await
.expect("claim global phase2 lock from second owner");
assert_eq!(second_claim, Phase2JobClaimOutcome::SkippedRunning);
assert!(
runtime
.mark_global_phase2_job_succeeded(
ownership_token.as_str(),
/*completed_watermark*/ 0,
&[]
)
.await
.expect("mark phase2 lock success")
);
let claim_after_success = runtime
.try_claim_global_phase2_job(owner_b, /*lease_seconds*/ 3_600)
.await
.expect("claim global phase2 lock after success");
assert!(
matches!(claim_after_success, Phase2JobClaimOutcome::Claimed { .. }),
"git workspace diff, not the DB watermark, decides whether the claimed lock has work"
);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
#[tokio::test]
async fn phase2_global_lock_stale_lease_allows_takeover() {
let codex_home = unique_temp_dir();
@@ -4487,7 +4514,7 @@ VALUES (?, ?, ?, ?, ?)
}
#[tokio::test]
async fn phase2_backfilled_inputs_below_last_success_still_become_dirty() {
async fn enqueue_global_consolidation_keeps_phase2_input_watermark_monotonic() {
let codex_home = unique_temp_dir();
let runtime = StateRuntime::init(codex_home.clone(), "test-provider".to_string())
.await
@@ -4527,23 +4554,23 @@ VALUES (?, ?, ?, ?, ?)
runtime
.enqueue_global_consolidation(/*input_watermark*/ 400)
.await
.expect("enqueue backfilled consolidation");
.expect("enqueue lower-watermark consolidation");
let owner_b = ThreadId::from_string(&Uuid::new_v4().to_string()).expect("owner b");
let claim_b = runtime
.try_claim_global_phase2_job(owner_b, /*lease_seconds*/ 3_600)
.await
.expect("claim backfilled consolidation");
.expect("claim lower-watermark consolidation");
match claim_b {
Phase2JobClaimOutcome::Claimed {
input_watermark, ..
} => {
assert!(
input_watermark > 500,
"backfilled enqueue should advance dirty watermark beyond last success"
"lower-watermark enqueue should still advance the bookkeeping watermark"
);
}
other => panic!("unexpected backfilled phase2 claim outcome: {other:?}"),
other => panic!("unexpected lower-watermark phase2 claim outcome: {other:?}"),
}
let _ = tokio::fs::remove_dir_all(codex_home).await;
@@ -4608,7 +4635,7 @@ VALUES (?, ?, ?, ?, ?)
.try_claim_global_phase2_job(ThreadId::new(), /*lease_seconds*/ 3_600)
.await
.expect("claim after fallback failure");
assert_eq!(claim, Phase2JobClaimOutcome::SkippedNotDirty);
assert_eq!(claim, Phase2JobClaimOutcome::SkippedRetryUnavailable);
let _ = tokio::fs::remove_dir_all(codex_home).await;
}
+75 -2
View File
@@ -30,6 +30,9 @@ use codex_app_server_protocol::AdditionalPermissionProfile;
use codex_app_server_protocol::AgentMessageDeltaNotification;
use codex_app_server_protocol::CommandExecutionRequestApprovalParams;
use codex_app_server_protocol::ConfigWarningNotification;
use codex_app_server_protocol::FileChangeRequestApprovalParams;
use codex_app_server_protocol::FileUpdateChange;
use codex_app_server_protocol::ItemStartedNotification;
use codex_app_server_protocol::JSONRPCErrorError;
use codex_app_server_protocol::McpServerStartupState;
use codex_app_server_protocol::McpServerStatusUpdatedNotification;
@@ -38,6 +41,7 @@ use codex_app_server_protocol::NetworkApprovalProtocol as AppServerNetworkApprov
use codex_app_server_protocol::NetworkPolicyAmendment as AppServerNetworkPolicyAmendment;
use codex_app_server_protocol::NetworkPolicyRuleAction as AppServerNetworkPolicyRuleAction;
use codex_app_server_protocol::NonSteerableTurnKind as AppServerNonSteerableTurnKind;
use codex_app_server_protocol::PatchChangeKind;
use codex_app_server_protocol::PermissionsRequestApprovalParams;
use codex_app_server_protocol::RequestId as AppServerRequestId;
use codex_app_server_protocol::ServerNotification;
@@ -70,6 +74,7 @@ use codex_protocol::models::PermissionProfile;
use codex_protocol::protocol::AskForApproval;
use codex_protocol::protocol::Event;
use codex_protocol::protocol::EventMsg;
use codex_protocol::protocol::FileChange;
use codex_protocol::protocol::NetworkApprovalContext;
use codex_protocol::protocol::NetworkApprovalProtocol;
use codex_protocol::protocol::RolloutItem;
@@ -1579,7 +1584,7 @@ async fn reset_memories_clears_local_memory_directories() -> Result<()> {
app.config.sqlite_home = codex_home.path().to_path_buf();
let memory_root = codex_home.path().join("memories");
let extensions_root = codex_home.path().join("memories_extensions");
let extensions_root = memory_root.join("extensions");
std::fs::create_dir_all(memory_root.join("rollout_summaries"))?;
std::fs::create_dir_all(&extensions_root)?;
std::fs::write(memory_root.join("MEMORY.md"), "stale memory\n")?;
@@ -1594,7 +1599,6 @@ async fn reset_memories_clears_local_memory_directories() -> Result<()> {
app.reset_memories_with_app_server(&mut app_server).await;
assert_eq!(std::fs::read_dir(&memory_root)?.count(), 0);
assert_eq!(std::fs::read_dir(&extensions_root)?.count(), 0);
app_server.shutdown().await?;
Ok(())
@@ -2523,6 +2527,75 @@ async fn inactive_thread_exec_approval_splits_shell_wrapped_command() {
);
}
#[tokio::test]
async fn inactive_thread_file_change_approval_recovers_buffered_changes() {
let (mut app, mut app_event_rx, _op_rx) = make_test_app_with_channels().await;
let thread_id = ThreadId::new();
app.enqueue_thread_notification(
thread_id,
ServerNotification::ItemStarted(ItemStartedNotification {
thread_id: thread_id.to_string(),
turn_id: "turn-approval".to_string(),
item: ThreadItem::FileChange {
id: "patch-approval".to_string(),
changes: vec![FileUpdateChange {
path: "README.md".to_string(),
kind: PatchChangeKind::Add,
diff: "hello\n".to_string(),
}],
status: codex_app_server_protocol::PatchApplyStatus::InProgress,
},
}),
)
.await
.expect("enqueue file change item");
let request = ServerRequest::FileChangeRequestApproval {
request_id: AppServerRequestId::Integer(9),
params: FileChangeRequestApprovalParams {
thread_id: thread_id.to_string(),
turn_id: "turn-approval".to_string(),
item_id: "patch-approval".to_string(),
reason: Some("command failed; retry without sandbox?".to_string()),
grant_root: None,
},
};
let request = app
.interactive_request_for_thread_request(thread_id, &request)
.await
.expect("expected file change approval request");
let ThreadInteractiveRequest::Approval(ApprovalRequest::ApplyPatch {
changes, reason, ..
}) = &request
else {
panic!("expected apply-patch approval request");
};
assert_eq!(
changes,
&HashMap::from([(
PathBuf::from("README.md"),
FileChange::Add {
content: "hello\n".to_string(),
},
)])
);
assert_eq!(
reason,
&Some("command failed; retry without sandbox?".to_string())
);
app.push_thread_interactive_request(request);
let cell = match app_event_rx.try_recv() {
Ok(AppEvent::InsertHistoryCell(cell)) => cell,
other => panic!("expected patch preview history cell, saw {other:?}"),
};
let rendered = lines_to_single_string(&cell.display_lines(/*width*/ 80));
assert!(rendered.contains("• Added README.md (+1 -0)"));
assert!(rendered.contains("1 +hello"));
}
#[tokio::test]
async fn inactive_thread_permissions_approval_preserves_file_system_permissions() {
let app = make_test_app().await;
+48
View File
@@ -157,6 +157,40 @@ impl ThreadEventStore {
.collect()
}
pub(super) fn file_change_changes(
&self,
turn_id: &str,
item_id: &str,
) -> Option<Vec<codex_app_server_protocol::FileUpdateChange>> {
self.buffer
.iter()
.rev()
.find_map(|event| match event {
ThreadBufferedEvent::Notification(ServerNotification::ItemStarted(
notification,
)) if turn_id_matches(turn_id, &notification.turn_id) => {
file_change_item_changes(&notification.item, item_id)
}
ThreadBufferedEvent::Notification(ServerNotification::ItemCompleted(
notification,
)) if turn_id_matches(turn_id, &notification.turn_id) => {
file_change_item_changes(&notification.item, item_id)
}
ThreadBufferedEvent::Request(_)
| ThreadBufferedEvent::Notification(_)
| ThreadBufferedEvent::HistoryEntryResponse(_)
| ThreadBufferedEvent::FeedbackSubmission(_) => None,
})
.or_else(|| {
self.turns
.iter()
.rev()
.filter(|turn| turn_id_matches(turn_id, &turn.id))
.flat_map(|turn| turn.items.iter().rev())
.find_map(|item| file_change_item_changes(item, item_id))
})
}
pub(super) fn apply_thread_rollback(&mut self, response: &ThreadRollbackResponse) {
self.turns = response.thread.turns.clone();
self.buffer.clear();
@@ -231,6 +265,20 @@ impl ThreadEventStore {
}
}
fn turn_id_matches(request_turn_id: &str, candidate_turn_id: &str) -> bool {
request_turn_id.is_empty() || request_turn_id == candidate_turn_id
}
fn file_change_item_changes(
item: &ThreadItem,
item_id: &str,
) -> Option<Vec<codex_app_server_protocol::FileUpdateChange>> {
match item {
ThreadItem::FileChange { id, changes, .. } if id == item_id => Some(changes.clone()),
_ => None,
}
}
#[derive(Debug)]
pub(super) struct ThreadEventChannel {
pub(super) sender: mpsc::Sender<ThreadBufferedEvent>,
+34 -1
View File
@@ -194,6 +194,17 @@ impl App {
store.session.as_ref().map(|session| session.cwd.clone())
}
async fn thread_file_change_changes(
&self,
thread_id: ThreadId,
turn_id: &str,
item_id: &str,
) -> Option<Vec<codex_app_server_protocol::FileUpdateChange>> {
let channel = self.thread_event_channels.get(&thread_id)?;
let store = channel.store.lock().await;
store.file_change_changes(turn_id, item_id)
}
pub(super) async fn interactive_request_for_thread_request(
&self,
thread_id: ThreadId,
@@ -264,7 +275,11 @@ impl App {
.thread_cwd(thread_id)
.await
.unwrap_or_else(|| self.config.cwd.clone()),
changes: HashMap::new(),
changes: self
.thread_file_change_changes(thread_id, &params.turn_id, &params.item_id)
.await
.map(crate::app_server_approval_conversions::file_update_changes_to_core)
.unwrap_or_default(),
}),
),
ServerRequest::McpServerElicitationRequest { request_id, params } => {
@@ -311,6 +326,7 @@ impl App {
pub(super) fn push_thread_interactive_request(&mut self, request: ThreadInteractiveRequest) {
match request {
ThreadInteractiveRequest::Approval(request) => {
self.render_inactive_patch_preview(&request);
self.chat_widget.push_approval_request(request);
}
ThreadInteractiveRequest::McpServerElicitation(request) => {
@@ -320,6 +336,23 @@ impl App {
}
}
fn render_inactive_patch_preview(&mut self, request: &ApprovalRequest) {
let ApprovalRequest::ApplyPatch {
thread_label,
cwd,
changes,
..
} = request
else {
return;
};
if thread_label.is_none() || changes.is_empty() {
return;
}
self.chat_widget
.add_to_history(history_cell::new_patch_event(changes.clone(), cwd));
}
pub(super) async fn pending_inactive_thread_requests(&self) -> Vec<(ThreadId, ServerRequest)> {
let channels: Vec<(ThreadId, Arc<Mutex<ThreadEventStore>>)> = self
.thread_event_channels
+76 -2
View File
@@ -10,7 +10,8 @@
//!
//! Backtrack operates as a small state machine:
//! - The first `Esc` in the main view "primes" the feature and captures a base thread id.
//! - A subsequent `Esc` opens the transcript overlay (`Ctrl+T`) and highlights a user message.
//! - A subsequent `Esc` opens the transcript overlay (`Ctrl+T`) and highlights a user message when
//! there is a rewind target.
//! - `Enter` requests a rollback from core and records a `pending_rollback` guard.
//! - On `EventMsg::ThreadRolledBack`, we either finish an in-flight backtrack request or queue a
//! rollback trim so it runs in event order with transcript inserts.
@@ -44,6 +45,8 @@ use crossterm::event::KeyCode;
use crossterm::event::KeyEvent;
use crossterm::event::KeyEventKind;
const NO_PREVIOUS_MESSAGE_TO_EDIT: &str = "No previous message to edit.";
/// Aggregates all backtrack-related state used by the App.
#[derive(Default)]
pub(crate) struct BacktrackState {
@@ -266,11 +269,21 @@ impl App {
self.backtrack.primed = true;
self.backtrack.nth_user_message = usize::MAX;
self.backtrack.base_id = self.chat_widget.thread_id();
self.chat_widget.show_esc_backtrack_hint();
if has_backtrack_target(&self.transcript_cells) {
self.chat_widget.show_esc_backtrack_hint();
}
}
/// Open overlay and begin backtrack preview flow (first step + highlight).
fn open_backtrack_preview(&mut self, tui: &mut tui::Tui) {
if !has_backtrack_target(&self.transcript_cells) {
self.reset_backtrack_state();
self.chat_widget
.add_info_message(NO_PREVIOUS_MESSAGE_TO_EDIT.to_string(), /*hint*/ None);
tui.frame_requester().schedule_frame();
return;
}
self.open_transcript_overlay(tui);
self.backtrack.overlay_preview_active = true;
// Composer is hidden by overlay; clear its hint.
@@ -280,6 +293,14 @@ impl App {
/// When overlay is already open, begin preview mode and select latest user message.
fn begin_overlay_backtrack_preview(&mut self, tui: &mut tui::Tui) {
if !has_backtrack_target(&self.transcript_cells) {
self.close_transcript_overlay(tui);
self.chat_widget
.add_info_message(NO_PREVIOUS_MESSAGE_TO_EDIT.to_string(), /*hint*/ None);
tui.frame_requester().schedule_frame();
return;
}
self.backtrack.primed = true;
self.backtrack.base_id = self.chat_widget.thread_id();
self.backtrack.overlay_preview_active = true;
@@ -613,6 +634,10 @@ pub(crate) fn user_count(cells: &[Arc<dyn crate::history_cell::HistoryCell>]) ->
user_positions_iter(cells).count()
}
fn has_backtrack_target(cells: &[Arc<dyn crate::history_cell::HistoryCell>]) -> bool {
user_count(cells) > 0
}
fn nth_user_position(
cells: &[Arc<dyn crate::history_cell::HistoryCell>],
nth: usize,
@@ -674,9 +699,22 @@ mod tests {
use super::*;
use crate::history_cell::AgentMessageCell;
use crate::history_cell::HistoryCell;
use pretty_assertions::assert_eq;
use ratatui::prelude::Line;
use std::sync::Arc;
fn render_lines(lines: &[Line<'static>]) -> Vec<String> {
lines
.iter()
.map(|line| {
line.spans
.iter()
.map(|span| span.content.as_ref())
.collect::<String>()
})
.collect()
}
#[test]
fn trim_transcript_for_first_user_drops_user_and_newer_cells() {
let mut cells: Vec<Arc<dyn HistoryCell>> = vec![
@@ -885,4 +923,40 @@ mod tests {
assert_eq!(agent_group_count(&cells), 2);
}
#[test]
fn backtrack_target_requires_user_message() {
let mut cells: Vec<Arc<dyn HistoryCell>> = vec![
Arc::new(AgentMessageCell::new(
vec![Line::from("assistant")],
/*is_first_line*/ true,
)) as Arc<dyn HistoryCell>,
Arc::new(crate::history_cell::new_info_event(
"Context compacted".to_string(),
/*hint*/ None,
)) as Arc<dyn HistoryCell>,
];
assert!(!has_backtrack_target(&cells));
cells.push(Arc::new(UserHistoryCell {
message: "hello".to_string(),
text_elements: Vec::new(),
local_image_paths: Vec::new(),
remote_image_urls: Vec::new(),
}) as Arc<dyn HistoryCell>);
assert!(has_backtrack_target(&cells));
}
#[test]
fn backtrack_unavailable_info_message_snapshot() {
let cell = crate::history_cell::new_info_event(
NO_PREVIOUS_MESSAGE_TO_EDIT.to_string(),
/*hint*/ None,
);
let rendered = render_lines(&cell.display_lines(/*width*/ 80)).join("\n");
insta::assert_snapshot!(rendered);
}
}
@@ -1,9 +1,14 @@
use codex_app_server_protocol::AdditionalNetworkPermissions;
use codex_app_server_protocol::FileUpdateChange;
use codex_app_server_protocol::GrantedPermissionProfile;
use codex_app_server_protocol::NetworkApprovalContext as AppServerNetworkApprovalContext;
use codex_app_server_protocol::PatchChangeKind;
use codex_protocol::protocol::FileChange;
use codex_protocol::protocol::NetworkApprovalContext;
use codex_protocol::protocol::NetworkApprovalProtocol;
use codex_protocol::request_permissions::RequestPermissionProfile as CoreRequestPermissionProfile;
use std::collections::HashMap;
use std::path::PathBuf;
pub(crate) fn network_approval_context_to_core(
value: AppServerNetworkApprovalContext,
@@ -38,21 +43,50 @@ pub(crate) fn granted_permission_profile_from_request(
}
}
pub(crate) fn file_update_changes_to_core(
changes: Vec<FileUpdateChange>,
) -> HashMap<PathBuf, FileChange> {
changes
.into_iter()
.map(|change| {
let path = PathBuf::from(change.path);
let file_change = match change.kind {
PatchChangeKind::Add => FileChange::Add {
content: change.diff,
},
PatchChangeKind::Delete => FileChange::Delete {
content: change.diff,
},
PatchChangeKind::Update { move_path } => FileChange::Update {
unified_diff: change.diff,
move_path,
},
};
(path, file_change)
})
.collect()
}
#[cfg(test)]
mod tests {
use super::file_update_changes_to_core;
use super::granted_permission_profile_from_request;
use super::network_approval_context_to_core;
use codex_app_server_protocol::FileUpdateChange;
use codex_app_server_protocol::PatchChangeKind;
use codex_protocol::models::FileSystemPermissions;
use codex_protocol::models::NetworkPermissions;
use codex_protocol::permissions::FileSystemAccessMode;
use codex_protocol::permissions::FileSystemPath;
use codex_protocol::permissions::FileSystemSandboxEntry;
use codex_protocol::permissions::FileSystemSpecialPath;
use codex_protocol::protocol::FileChange;
use codex_protocol::protocol::NetworkApprovalContext;
use codex_protocol::protocol::NetworkApprovalProtocol;
use codex_protocol::request_permissions::RequestPermissionProfile as CoreRequestPermissionProfile;
use codex_utils_absolute_path::AbsolutePathBuf;
use pretty_assertions::assert_eq;
use std::collections::HashMap;
use std::path::PathBuf;
fn absolute_path(path: &str) -> AbsolutePathBuf {
@@ -73,6 +107,23 @@ mod tests {
);
}
#[test]
fn converts_file_update_changes_to_core() {
assert_eq!(
file_update_changes_to_core(vec![FileUpdateChange {
path: "foo.txt".to_string(),
kind: PatchChangeKind::Add,
diff: "hello\n".to_string(),
}]),
HashMap::from([(
PathBuf::from("foo.txt"),
FileChange::Add {
content: "hello\n".to_string(),
},
)])
);
}
#[test]
fn converts_request_permissions_into_granted_permissions() {
assert_eq!(
@@ -9,7 +9,6 @@ use crate::bottom_pane::CancellationEvent;
use crate::bottom_pane::list_selection_view::ListSelectionView;
use crate::bottom_pane::list_selection_view::SelectionItem;
use crate::bottom_pane::list_selection_view::SelectionViewParams;
use crate::diff_render::DiffSummary;
use crate::exec_command::strip_bash_lc_and_escape;
use crate::history_cell;
use crate::key_hint;
@@ -633,8 +632,6 @@ fn build_header(request: &ApprovalRequest) -> Box<dyn Renderable> {
ApprovalRequest::ApplyPatch {
thread_label,
reason,
cwd,
changes,
..
} => {
let mut header: Vec<Box<dyn Renderable>> = Vec::new();
@@ -643,11 +640,13 @@ fn build_header(request: &ApprovalRequest) -> Box<dyn Renderable> {
"Thread: ".into(),
thread_label.clone().bold(),
])));
header.push(Box::new(Line::from("")));
}
if let Some(reason) = reason
&& !reason.is_empty()
{
if !header.is_empty() {
header.push(Box::new(Line::from("")));
}
header.push(Box::new(
Paragraph::new(Line::from_iter([
"Reason: ".into(),
@@ -655,9 +654,7 @@ fn build_header(request: &ApprovalRequest) -> Box<dyn Renderable> {
]))
.wrap(Wrap { trim: false }),
));
header.push(Box::new(Line::from("")));
}
header.push(DiffSummary::new(changes.clone(), cwd.clone()).into());
Box::new(ColumnRenderable::with(header))
}
ApprovalRequest::McpElicitation {
@@ -1555,6 +1552,32 @@ mod tests {
);
}
#[test]
fn apply_patch_prompt_with_thread_label_omits_command_line() {
let (tx, _rx) = unbounded_channel::<AppEvent>();
let tx = AppEventSender::new(tx);
let mut changes = HashMap::new();
changes.insert(
PathBuf::from("bug1.txt"),
FileChange::Add {
content: "one\ntwo\nthree\n".to_string(),
},
);
let request = ApprovalRequest::ApplyPatch {
thread_id: ThreadId::new(),
thread_label: Some("Banach [worker]".to_string()),
id: "test".to_string(),
reason: None,
cwd: absolute_path("/tmp"),
changes,
};
let view = ApprovalOverlay::new(request, tx, Features::with_defaults());
let rendered = render_overlay_lines(&view, /*width*/ 120);
assert!(rendered.contains("Thread: Banach [worker]"));
assert!(rendered.contains("o to open thread"));
assert!(!rendered.contains("$ apply_patch"));
}
#[test]
fn network_exec_prompt_title_includes_host() {
let (tx, _rx) = unbounded_channel::<AppEvent>();
@@ -1160,7 +1160,7 @@ impl ChatComposer {
/// Convert canonical composer text into the textarea's internal representation.
///
/// Bash mode stores the leading `!` as prompt state instead of editable text,
/// Shell mode stores the leading `!` as prompt state instead of editable text,
/// so full-buffer imports must absorb that prefix before rebuilding the textarea.
fn imported_text_for_textarea(
&mut self,
@@ -2934,7 +2934,7 @@ impl ChatComposer {
fn shell_mode_footer_line(&self) -> Option<Line<'static>> {
self.is_bang_shell_command()
.then_some(())
.map(|_| Line::from(vec![Span::from("Bash mode").light_red()]))
.map(|_| Line::from(vec![Span::from("Shell mode").light_red()]))
}
/// Applies any due `PasteBurst` flush at time `now`.
@@ -4660,7 +4660,7 @@ mod tests {
}
#[test]
fn shell_command_uses_bash_accent_style() {
fn shell_command_uses_shell_accent_style() {
let (tx, _rx) = unbounded_channel::<AppEvent>();
let sender = AppEventSender::new(tx);
let mut composer = ChatComposer::new(
@@ -4688,11 +4688,11 @@ mod tests {
let footer_text = (0..area.width)
.map(|x| buf[(x, footer_y)].symbol().chars().next().unwrap_or(' '))
.collect::<String>();
let bash_label_x = footer_text
.find("Bash mode")
.expect("expected bash mode footer label");
let shell_label_x = footer_text
.find("Shell mode")
.expect("expected shell mode footer label");
assert_eq!(
buf[(bash_label_x as u16, footer_y)].style().fg,
buf[(shell_label_x as u16, footer_y)].style().fg,
Some(Color::LightRed)
);
}
@@ -10,4 +10,4 @@ expression: terminal.backend()
" "
" "
" "
" gpt-5.4 high fast · ~/code/codex-1 · Context 0% used Bash mode "
" gpt-5.4 high fast · ~/code/codex-1 · Context 0% used Shell mode "
+22 -34
View File
@@ -47,6 +47,7 @@ use self::realtime::PendingSteerCompareKey;
use crate::app::app_server_requests::ResolvedAppServerRequest;
use crate::app_command::AppCommand;
use crate::app_event::RealtimeAudioDeviceKind;
use crate::app_server_approval_conversions::file_update_changes_to_core;
use crate::app_server_approval_conversions::network_approval_context_to_core;
use crate::app_server_session::ThreadSessionState;
#[cfg(not(target_os = "linux"))]
@@ -1774,36 +1775,6 @@ fn patch_approval_request_from_params(
}
}
fn app_server_patch_changes_to_core(
changes: Vec<codex_app_server_protocol::FileUpdateChange>,
) -> HashMap<PathBuf, codex_protocol::protocol::FileChange> {
changes
.into_iter()
.map(|change| {
let path = PathBuf::from(change.path);
let file_change = match change.kind {
codex_app_server_protocol::PatchChangeKind::Add => {
codex_protocol::protocol::FileChange::Add {
content: change.diff,
}
}
codex_app_server_protocol::PatchChangeKind::Delete => {
codex_protocol::protocol::FileChange::Delete {
content: change.diff,
}
}
codex_app_server_protocol::PatchChangeKind::Update { move_path } => {
codex_protocol::protocol::FileChange::Update {
unified_diff: change.diff,
move_path,
}
}
};
(path, file_change)
})
.collect()
}
fn app_server_collab_thread_id_to_core(thread_id: &str) -> Option<ThreadId> {
match ThreadId::from_string(thread_id) {
Ok(thread_id) => Some(thread_id),
@@ -6151,9 +6122,26 @@ impl ChatWidget {
}
}
fn submit_shell_command_with_history(
&mut self,
command: &str,
history_text: &str,
) -> QueueDrain {
let drain = self.submit_shell_command(command);
if drain == QueueDrain::Stop {
self.submit_op(Op::AddToHistory {
text: history_text.to_string(),
});
}
drain
}
fn submit_queued_shell_prompt(&mut self, user_message: UserMessage) -> QueueDrain {
match user_message.text.strip_prefix('!') {
Some(command) => self.submit_shell_command(command),
Some(command) => {
let history_text = user_message.text.clone();
self.submit_shell_command_with_history(command, &history_text)
}
None => {
self.submit_user_message(user_message);
QueueDrain::Stop
@@ -6249,7 +6237,7 @@ impl ChatWidget {
if shell_escape_policy == ShellEscapePolicy::Allow
&& let Some(stripped) = text.strip_prefix('!')
{
let app_command = match self.submit_shell_command(stripped) {
let app_command = match self.submit_shell_command_with_history(stripped, &text) {
QueueDrain::Continue => None,
QueueDrain::Stop => Some(AppCommand::run_user_shell_command(
stripped.trim().to_string(),
@@ -6780,7 +6768,7 @@ impl ChatWidget {
status,
codex_app_server_protocol::PatchApplyStatus::Failed
),
changes: app_server_patch_changes_to_core(changes),
changes: file_update_changes_to_core(changes),
status: match status {
codex_app_server_protocol::PatchApplyStatus::Completed => {
codex_protocol::protocol::PatchApplyStatus::Completed
@@ -7339,7 +7327,7 @@ impl ChatWidget {
call_id: id,
turn_id: notification.turn_id,
auto_approved: false,
changes: app_server_patch_changes_to_core(changes),
changes: file_update_changes_to_core(changes),
});
}
ThreadItem::McpToolCall {
@@ -1,18 +1,11 @@
---
source: tui/src/chatwidget/tests.rs
expression: terminal.backend().vt100().screen().contents()
source: tui/src/chatwidget/tests/exec_flow.rs
expression: contents
---
Would you like to make the following edits?
Reason: The model wants to apply changes
README.md (+2 -0)
1 +hello
2 +world
› 1. Yes, proceed (y)
2. Yes, and don't ask again for these files (a)
3. No, and tell Codex what to do differently (esc)
@@ -349,25 +349,6 @@ async fn live_app_server_command_execution_strips_shell_wrapper() {
);
}
#[test]
fn app_server_patch_changes_to_core_preserves_diffs() {
let changes = app_server_patch_changes_to_core(vec![FileUpdateChange {
path: "foo.txt".to_string(),
kind: PatchChangeKind::Add,
diff: "hello\n".to_string(),
}]);
assert_eq!(
changes,
HashMap::from([(
PathBuf::from("foo.txt"),
FileChange::Add {
content: "hello\n".to_string(),
},
)])
);
}
#[tokio::test]
async fn live_app_server_collab_wait_items_render_history() {
let (mut chat, mut rx, _op_rx) = make_chatwidget_manual(/*model_override*/ None).await;

Some files were not shown because too many files have changed in this diff Show More