Add audio file references with explicit runtime rejection (#58430)

## What changed

- Allow audio inputs to carry uploaded file references via `fileId` in the app-server API and `file_id` in model content, preserving existing inline audio shapes and Python class names.
- Preserve mixed inline and file-backed audio ordering through history reconstruction and migration, and budget file-backed audio conservatively when duration is unknown.
- Keep uploaded audio disabled at runtime: reject it before inference or raw history injection, preserve accompanying text, and prevent rejected attachments from reaching follow-up or resumed requests.

## Testing

Add coverage for wire-format round trips, SDK compatibility, mixed audio history ordering, and rejection during turn start, steering, task completion, and raw injection.

GitOrigin-RevId: 70895c47a2494c0c9936348ad1250eb1b7b892a6
This commit is contained in:
Hung
2026-10-30 20:53:25 +00:00
committed by copyberry
parent b9858217d8
commit 2546265e3e
83 changed files with 2122 additions and 300 deletions
@@ -1101,8 +1101,16 @@ class InputAudioContentItem(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
)
audio_url: str
type: Annotated[Literal["input_audio"], Field(title="InputAudioContentItemType")]
audio_url: str
class FileIdContentItem1(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
)
type: Annotated[Literal["input_audio"], Field(title="InputAudioContentItemType")]
file_id: str
class OutputTextContentItem(BaseModel):
@@ -6762,6 +6770,14 @@ class AudioUserInput(BaseModel):
url: str
class FileIdUserInput1(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
)
type: Annotated[Literal["audio"], Field(title="AudioUserInputType")]
file_id: Annotated[str, Field(alias="fileId")]
class LocalAudioUserInput(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
@@ -6795,6 +6811,7 @@ class UserInput(
| FileIdUserInput
| LocalImageUserInput
| AudioUserInput
| FileIdUserInput1
| LocalAudioUserInput
| SkillUserInput
| MentionUserInput
@@ -6806,6 +6823,7 @@ class UserInput(
| FileIdUserInput
| LocalImageUserInput
| AudioUserInput
| FileIdUserInput1
| LocalAudioUserInput
| SkillUserInput
| MentionUserInput
@@ -8375,6 +8393,7 @@ class ContentItem(
| InputImageContentItem
| FileIdContentItem
| InputAudioContentItem
| FileIdContentItem1
| OutputTextContentItem
]
):
@@ -8383,6 +8402,7 @@ class ContentItem(
| InputImageContentItem
| FileIdContentItem
| InputAudioContentItem
| FileIdContentItem1
| OutputTextContentItem
)