Add audio variants to user input protocols (#33923)

## What changed

- Add remote and local audio variants to core and app-server user inputs, including generated JSON and TypeScript schemas.
- Preserve audio fields when converting app-server inputs and response content into thread items.
- Keep unsupported audio out of model requests, TUI rendering, text extraction, and truncated tool output, using explicit text placeholders where audio is submitted as a model input.

## Testing

- Cover app-server media conversion, unsupported-input placeholders, TUI omission, and tool-output truncation.

GitOrigin-RevId: 16152bf3af674575ce569764a8b472e032e7569d
This commit is contained in:
nhamidi-oai
2026-07-17 23:14:37 +00:00
committed by copyberry
parent e98765c212
commit 97dbd01a3d
49 changed files with 1330 additions and 33 deletions
@@ -2050,6 +2050,46 @@
"title": "LocalImageUserInput",
"type": "object"
},
{
"properties": {
"type": {
"enum": [
"audio"
],
"title": "AudioUserInputType",
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"type",
"url"
],
"title": "AudioUserInput",
"type": "object"
},
{
"properties": {
"path": {
"type": "string"
},
"type": {
"enum": [
"localAudio"
],
"title": "LocalAudioUserInputType",
"type": "string"
}
},
"required": [
"path",
"type"
],
"title": "LocalAudioUserInput",
"type": "object"
},
{
"properties": {
"name": {