Preserve file-backed audio in tool outputs (#60735)

## Why

Converting message content to tool output previously reduced file-backed audio to a text file ID, losing its audio type.

## What changed

- Allow `input_audio` tool output items to carry either `audio_url` or `file_id`, preserving both forms during content conversion.
- Account for file-backed tool audio in history estimates and output truncation, and prepare only inline audio data.
- Reject uploaded audio in raw function and custom tool output injection, extending the existing requirement to submit uploaded audio through typed turn input.
- Regenerate app-server schemas and TypeScript/Python types while preserving the Python inline audio class name.

## Testing

Add coverage for tool audio wire-format round trips, repeated file audio sharing text and audio output budgets, and rejection of raw uploaded audio injection across messages and tool outputs. Extend Python SDK coverage for the inline tool audio shape.

GitOrigin-RevId: f4704466e6151662d2c8b8dc3f5ca2ab2f5710a2
This commit is contained in:
Hung
2026-11-11 06:41:15 +00:00
committed by copyberry
parent ce6f7d9503
commit 9018f3a3b0
49 changed files with 882 additions and 235 deletions
@@ -1781,10 +1781,20 @@ class InputAudioFunctionCallOutputContentItem(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
)
audio_url: str
type: Annotated[
Literal["input_audio"], Field(title="InputAudioFunctionCallOutputContentItemType")
]
audio_url: str
class FileIdFunctionCallOutputContentItem1(BaseModel):
model_config = ConfigDict(
populate_by_name=True,
)
type: Annotated[
Literal["input_audio"], Field(title="InputAudioFunctionCallOutputContentItemType")
]
file_id: str
class EncryptedContentFunctionCallOutputContentItem(BaseModel):
@@ -8930,6 +8940,7 @@ class FunctionCallOutputContentItem(
| InputImageFunctionCallOutputContentItem
| FileIdFunctionCallOutputContentItem
| InputAudioFunctionCallOutputContentItem
| FileIdFunctionCallOutputContentItem1
| EncryptedContentFunctionCallOutputContentItem
]
):
@@ -8938,6 +8949,7 @@ class FunctionCallOutputContentItem(
| InputImageFunctionCallOutputContentItem
| FileIdFunctionCallOutputContentItem
| InputAudioFunctionCallOutputContentItem
| FileIdFunctionCallOutputContentItem1
| EncryptedContentFunctionCallOutputContentItem,
Field(
description="Responses API compatible content items that can be returned by a tool call. This is a subset of ContentItem with the types we support as function call outputs."