Support typed content in contextual prompt fragments (#54776)

## Why

Contextual fragments and extension-owned world-state sections currently return strings, limiting them to text content.

## What changed

- Return `ContentItem` from fragment bodies and rendering, preserving classification and attribution. Apply text markers only to `InputText`; pass other content through unchanged.
- Allow extension prompt and world-state contributions to carry typed content, and preserve media in context-window hints.
- Hash non-text world-state fragments by their serialized content and role while retaining the existing text hashing behavior.
- Add shared content-to-tool-output conversion that preserves image references, image detail, and audio.

## Testing

Add coverage for typed world-state hashing and preservation of image content and attribution. Expand code-mode output tests to cover input text, output text, file-backed and inline images, and audio.

GitOrigin-RevId: 2ed00d466a3f169b151202f3eef71e667ea64be1
This commit is contained in:
pakrym-oai
2026-10-17 04:02:21 +00:00
committed by copyberry
parent 3b8f43a06b
commit ce0323f2f2
162 changed files with 1109 additions and 570 deletions
@@ -2,6 +2,7 @@
use super::*;
use crate::bottom_pane::RestrictedInputMode;
use codex_protocol::items::AsyncUserInputQuestion;
use codex_protocol_test_support::ContentItemTestExt;
use pretty_assertions::assert_eq;
fn questions() -> Vec<AsyncUserInputQuestion> {
@@ -94,7 +95,9 @@ async fn async_question_answers_preserve_ambiguous_skill_selection_and_dismiss_r
items,
vec![
UserInput::Text {
text: AnsweredQuestion::new(&id, "Which way?", "Use $route").render(),
text: AnsweredQuestion::new(&id, "Which way?", "Use $route")
.render()
.expect_text(),
text_elements: Vec::new(),
},
UserInput::Skill {
@@ -1037,7 +1040,9 @@ async fn distinct_async_question_replies_with_identical_text_both_render() {
drain_insert_history(&mut rx);
for id in ["first", "first", "second"] {
let items = [UserInput::Text {
text: AnsweredQuestion::new(id, "Continue?", "Yes").render(),
text: AnsweredQuestion::new(id, "Continue?", "Yes")
.render()
.expect_text(),
text_elements: Vec::new(),
}];
chat.on_committed_user_message(