## Why
Agent messages that arrive after a model request starts must survive compaction until a normal inference request includes them. Compactor output alone does not establish that those messages were delivered.
## What changed
- Persist history sequences and sampling checkpoints in harness metadata, capturing the input prefix for each normal inference request.
- Retain pending agent messages within existing size budgets during local and remote compaction, placing the summary before pending messages and subsequent caller input.
- Preserve sampling progress across history replacement and resume, and drain queued mail after interrupted sampling even when compaction leaves context above the token limit.
## Testing
Add regression coverage for request-boundary sampling checkpoints, pending messages across local compaction and repeated remote compaction with resume, and injected messages that interrupt sampling before mid-turn compaction.
GitOrigin-RevId: 26c04ab96b5a1c3cc1d6625ccc74a0f00201d14c
## Why
Uploaded audio always reserved 12,000 tokens per file, even for short clips. This could discard recordings during compaction that would fit within the retained-message budget.
## What changed
- Add optional audio duration metadata to attachments and cache measured durations per session, resolving metadata for new input and replayed history.
- Use measured durations for history estimates and compaction budgets at 10 tokens per second, rounded up and capped at 12,000 tokens. Keep the full budget for missing or unusable metadata.
- Bound metadata lookups to 250 milliseconds per batch with up to eight concurrent requests, and record accepted input before lookup so interruption preserves the batch.
## Testing
Add coverage for duration rounding and fallback estimates, mixed measured and unknown audio, retention through compaction and cold resume, and interruption during metadata lookup.
GitOrigin-RevId: 26fe320c7024944d9d4c371d5fdcda84442b5cc5
## Why
Remote compaction did not charge retained file audio against its token budget, allowing audio-heavy history to exceed the intended limit.
## What changed
- Charge file audio using `FILE_AUDIO_TOKENS_ESTIMATE`, independently of image budgeting.
- Retain the newest media content that fits, keeping images and audio together with their adjacent labels and preserving content annotations. Stop at oversized media instead of backfilling older content.
- Share media-label detection with user-input event mapping.
- Add download timeouts and a Debian mirror fallback for the musl build's `libcap` source archive.
## Testing
Add unit coverage for audio budgeting in both image-budget modes, atomic media labels, and truncation boundaries. Add integration coverage showing that compaction retains the newest five of seven file-audio items within the retained-message budget, including a mixed-media case.
GitOrigin-RevId: 4b7cd69f53966461fdeec14096712671d480187b
## Why
Uploaded audio was rejected even for audio-capable models. Invalid or expired recordings also need to be excluded from later requests so users can continue the conversation.
## What changed
- Accept uploaded audio when the selected model supports audio; otherwise, preserve valid sibling input and report the unsupported attachment before inference.
- Track successful audio decoding by model and provider, and attribute provider rejections against the exact submitted request. Preserve attachments when attribution is ambiguous.
- Persist confirmed rejected file IDs across compaction and resume, omitting them from future requests without rewriting retained user history.
- Check audio availability before remote compaction and retry once without an unambiguously rejected recording. Include guidance to remove or replace affected attachments in audio errors.
## Testing
Add unit and scenario coverage for uploaded audio across follow-up turns, compaction, cold and paginated resume, model switching, ambiguous rejections, expired files, and queued or interrupted input.
GitOrigin-RevId: e701901e7b1c0865c9f2a5918197215f549b06c0
## Why
Image files can become unavailable between turns. Model requests should reflect their current availability while preserving the original references in saved history.
## What changed
- Add opt-in availability checks through `AttachmentStore::is_resolvable`, retaining references by default and on backend or access errors.
- Check distinct file IDs concurrently for each prepared request and replace confirmed-missing images with `[image unavailable]` in messages and tool outputs.
- Apply the same handling to compaction requests, with background image checks running outside the session lock without delaying foreground turn completion.
## Testing
Add unit tests and multi-turn scenarios covering placeholders, preserved history and annotations, concurrent deduplicated checks, backend errors, and nonblocking background compaction.
GitOrigin-RevId: ec98c7014c6e6b5063d87b943d2cd66a9ce7c1e3
## Why
File images contributed zero to history and compaction token estimates and were dropped from Guardian context. This undercounted context usage and omitted image evidence from reviews.
## What changed
- Estimate file images using the fixed resized-image cost, or 10,000 patch tokens for `original` detail because file references do not expose dimensions.
- Charge file images against remote compaction and Guardian request budgets.
- Preserve file images from user messages and tool outputs through Guardian selection and delivery, and deduplicate review images by file ID.
- Apply transcript image count and byte limits to both reference types, counting file ID bytes rather than referenced file contents.
## Testing
Add coverage for file-image estimates, selection limits, delivery, and compaction retention. Extend Guardian tests to verify file-image evidence reaches reviews and uploaded original-detail images trigger reviewer-history compaction.
GitOrigin-RevId: ee024204dcbe16e277fea9347c6a4d7fc120e18d
## Why
Compaction and host-injected context can rewrite the model-visible conversation
without changing what the user authorized. Treating those updates as authorization
changes prevents Guardian from reusing an otherwise valid review.
## What changed
- Track a host-owned user-message revision separately from the conversation history
generation.
- Advance the revision for genuine user messages and history resets, while preserving
it across compaction and internal context injection.
- Use message content-kind metadata to distinguish host context from user input,
conservatively treating unknown or incomplete metadata as user authorization.
## Testing
Added coverage that cached Guardian authorization survives compaction and internal
context, but is invalidated by user input and rollback.
GitOrigin-RevId: c7960fa182d1686042c23764d2abc58acbc4b882
## What changed
- Add a positive `output_token_limit` setting to each entry under an MCP server's `tools` configuration.
- Apply the most restrictive limit when plugin and user policies overlap, while keeping approval policy independent.
- Carry the effective MCP output budget in conversation history so tool output, post-tool hook responses, and resumed sessions use the same truncation limit.
## Testing
- Cover configuration parsing, serialization, schema validation, and plugin policy merging.
- Cover MCP output below and above the configured limit, post-tool hook responses, and session resume.
GitOrigin-RevId: d0beb4fca9ba6055d9e1d31c137373b465d50d61
## Why
Remote compaction's retained-message budget counted text but not images, so
image-heavy history could retain more context than the budget represented.
## What changed
- Add the opt-in `compaction_image_budget` feature to charge retained images
using the existing image size estimate.
- Keep images and their adjacent labels atomic when truncating a boundary
message, while preserving existing text, audio, metadata, and
client-authored developer-message behavior.
- Stop backfilling older messages when an image at the truncation boundary
does not fit.
## Testing
- Add unit coverage for image boundaries, annotations, audio, text-only
messages, and client-authored developer messages.
- Add an integration test covering repeated remote compaction with the feature
enabled, disabled, and left at its default.
GitOrigin-RevId: 88868bbd09d2858939dbae3f8402e1f497b4e3c6
## Why
Remote compaction can process large histories. Repeatedly estimating and replacing the full history, and cloning it when tracing is disabled, adds avoidable CPU and memory overhead.
## What changed
- Estimate each history item's token count once, update the total as trailing tool outputs are rewritten, and replace history only after all rewrites are selected.
- Preserve unclamped token totals while calculating removed tokens so saturated estimates do not hide overflow.
- Snapshot compaction input history only when rollout tracing is enabled.
- Reuse the v2 request input instead of cloning it before adding the compaction trigger.
## Testing
- Cover enabled and disabled compaction trace contexts through `is_enabled()` assertions.
GitOrigin-RevId: 8de9c9704ba29532fba05430eb204bbae2e8bf83
## Why
`log_remote_compact_failure` was the only consumer of the
compact-request logging payload and most of the token-usage breakdown
fields. Once that failure log is removed, keeping the surrounding
carrier types leaves dead plumbing in the compaction path and context
manager.
## What changed
- Remove `log_remote_compact_failure`, `CompactRequestLogData`, and the
v2 wrapper that only fed that log.
- Let both remote compaction implementations return the original
compaction error directly.
- Replace `TotalTokenUsageBreakdown` with a narrow helper that returns
only the remaining value needed by compaction analytics.
- Keep `estimate_response_item_model_visible_bytes` private to the
context manager implementation.
## Validation
- `cargo check -p codex-core`
## Why
When trying to fit history under compaction limit rewrite output items
instead of removing them entirely. Otherwise we're breaking
incrementality in relation to the previous response.
## Why
Large MCP tool call outputs can make rollout JSONL files enormous. In
the session that motivated this change, the biggest JSONL records were:
- `event_msg/mcp_tool_call_end`
- `response_item/function_call_output`
both containing the same unbounded MCP payloads - just 3 MCP tool calls
that each were multi-hundred MBs 😱
This PR truncates both of those JSONL records.
## How
#### For `response_item/function_call_output`
Unified exec already bounds tool output before it is injected into
model-facing history, which also keeps the corresponding rollout
`response_item/function_call_output` records small.
MCP should follow the same pattern: truncate the model-facing tool
output at the tool-output boundary, while leaving code-mode/raw hook
consumers alone.
#### For `event_msg/mcp_tool_call_end`
`McpToolCallEnd` also needs its own bounded event copy because it is the
app-server/replay/UI event shape that backs `ThreadItem::McpToolCall`.
Unfortunately this is _not_ downstream of the `ToolOutput` trait.
## Model behavior
Model behavior is actually unchanged as a result of this PR.
Before this PR, MCP output was:
1. Converted to `FunctionCallOutput`.
2. Recorded into in-memory history.
3. Truncated by `ContextManager::record_items()` before later model
turns saw it.
After this branch, MCP output is truncated earlier, in
`McpToolOutput::response_payload()`, using the same helper. Then
`ContextManager::record_items()` sees an already-truncated output and
effectively has little/no additional work to do.
So the model should still see the same kind of truncated function-call
output. The practical difference is where truncation happens: earlier,
before rollout persistence/app-server emission can see the giant
payload.
## Verification
- `cargo test -p codex-core mcp_tool_output`
- `cargo test -p codex-core
mcp_tool_call::tests::truncate_mcp_tool_result_for_event`
- `cargo test -p codex-core
mcp_post_tool_use_payload_uses_model_tool_name_args_and_result`
- `just fmt`
- `just fix -p codex-core`
- `git diff --check`
## Summary
This PR centralizes model-visible state diffing for turn context updates
into one module, while keeping existing behavior and call sites stable.
### What changed
- Added `core/src/context_updates.rs` with the consolidated diffing
logic for:
- environment context updates
- permissions/policy updates
- collaboration mode updates
- model-instruction switch updates
- personality updates
- Added `BuildSettingsUpdateItemsParams` so required dependencies are
passed explicitly.
- Updated `Session::build_settings_update_items` in `core/src/codex.rs`
to delegate to the centralized module.
- Reused the same centralized `personality_message_for` helper from
initial-context assembly to avoid duplicated logic.
- Registered the new module in `core/src/lib.rs`.
## Why
This is a minimal, shippable step toward the model-visible-state design:
all state diff decisions for turn-context update items now live in one
place, improving reviewability and reducing drift risk without expanding
scope.
## Behavior
- Intended to be behavior-preserving.
- No protocol/schema changes.
- No call-site behavior changes beyond routing through the new
centralized logic.
## Testing
Ran targeted tests in this worktree:
- `cargo test -p codex-core
build_settings_update_items_emits_environment_item_for_network_changes`
- `cargo test -p codex-core collaboration_instructions --test all`
Both passed.
## Codex author
`codex resume 019c540f-3951-7352-a3fa-6f07b834d4ce`
## Summary
- add targeted remote-compaction failure diagnostics in compact_remote
logging
- log the specific values needed to explain overflow timing:
- last_api_response_total_tokens
- estimated_tokens_of_items_added_since_last_successful_api_response
- estimated_bytes_of_items_added_since_last_successful_api_response
- failing_compaction_request_body_bytes
- simplify breakdown naming and remove
last_api_response_total_bytes_estimate (it was an approximation and not
useful for debugging)
## Why
When compaction fails with context_length_exceeded, we need concrete,
low-ambiguity numbers that map directly to:
1) what the API most recently reported, and
2) what local history added since then.
This keeps the failure logs actionable without adding broad, noisy
metrics.
## Testing
- just fmt
- cargo test -p codex-core
Two fixes:
1. Include trailing tool output in the total context size calculation.
Otherwise when checking whether compaction should run we ignore newly
added outputs.
2. Trim trailing tool output/tool calls until we can fit the request
into the model context size. Otherwise the compaction endpoint will fail
to compact. We only trim items that can be reproduced again by the model
(tool calls, tool call outputs).
- This PR is to make it on path for truncating by tokens. This path will
be initially used by unified exec and context manager (responsible for
MCP calls mainly).
- We are exposing new config `calls_output_max_tokens`
- Use `tokens` as the main budget unit but truncate based on the model
family by Introducing `TruncationPolicy`.
- Introduce `truncate_text` as a router for truncation based on the
mode.
In next PRs:
- remove truncate_with_line_bytes_budget
- Add the ability to the model to override the token budget.
This is just a refactor of `conversation_history` file by breaking it up
into multiple smaller ones with helper. This refactor will help us move
more functionality related to context management here. in a clean way.