Add audio variants to user input protocols (#33923)

## What changed

- Add remote and local audio variants to core and app-server user inputs, including generated JSON and TypeScript schemas.
- Preserve audio fields when converting app-server inputs and response content into thread items.
- Keep unsupported audio out of model requests, TUI rendering, text extraction, and truncated tool output, using explicit text placeholders where audio is submitted as a model input.

## Testing

- Cover app-server media conversion, unsupported-input placeholders, TUI omission, and tool-output truncation.

GitOrigin-RevId: 16152bf3af674575ce569764a8b472e032e7569d
This commit is contained in:
nhamidi-oai
2026-07-17 23:14:37 +00:00
committed by copyberry
parent e98765c212
commit 97dbd01a3d
49 changed files with 1330 additions and 33 deletions
@@ -559,10 +559,14 @@ fn create_collab_input_items_schema() -> JsonSchema {
"image_url".to_string(),
JsonSchema::string(Some("Image URL when type is image.".to_string())),
),
(
"audio_url".to_string(),
JsonSchema::string(Some("Audio URL when type is audio.".to_string())),
),
(
"path".to_string(),
JsonSchema::string(Some(
"Path when type is local_image/skill, or structured mention target such as app://<connector-id> or plugin://<plugin-name>@<marketplace-name> when type is mention."
"Path when type is local_image/local_audio/skill, or structured mention target such as app://<connector-id> or plugin://<plugin-name>@<marketplace-name> when type is mention."
.to_string(),
)),
),