Gate audio history by model input modalities (#33982)

## What changed

- Add `audio` to the model input modalities exposed by the protocol and generated app-server schemas.
- Preserve audio in prompts for models that advertise audio input, and replace historical audio with an omission marker for models that do not.

## Testing

- Cover prompt normalization with and without audio support.
- Extend the model-switching integration test to verify that switching from a multimodal model to a text-only model strips prior image and audio content.

GitOrigin-RevId: c18e47ad602720216303ff9abb5bca4d508391ee
This commit is contained in:
nhamidi-oai
2026-07-18 09:13:38 +00:00
committed by copyberry
parent 5c0e582c59
commit 56395bddaf
11 changed files with 146 additions and 20 deletions
@@ -11550,6 +11550,13 @@
"image"
],
"type": "string"
},
{
"description": "Audio attachments included in user turns.",
"enum": [
"audio"
],
"type": "string"
}
]
},
@@ -7907,6 +7907,13 @@
"image"
],
"type": "string"
},
{
"description": "Audio attachments included in user turns.",
"enum": [
"audio"
],
"type": "string"
}
]
},
@@ -17,6 +17,13 @@
"image"
],
"type": "string"
},
{
"description": "Audio attachments included in user turns.",
"enum": [
"audio"
],
"type": "string"
}
]
},
@@ -5,4 +5,4 @@
/**
* Canonical user-input modality tags advertised by a model.
*/
export type InputModality = "text" | "image";
export type InputModality = "text" | "image" | "audio";