jif 5e96aabd68
Avoid full-string scans in token-budget truncation ( #49712 ) ...
## Why
Token-budget truncation uses byte counts to estimate tokens, but previously scanned the entire string to count characters, including the discarded middle.
## What changed
Use UTF-8 boundary lookups to select the retained prefix and suffix for token-budget truncation. Skip character counting for a zero-token budget and defer it to paths that need character counts.
## Testing
Add regression coverage for multibyte characters crossing both retained boundaries, checking the truncated output and original token estimate.
GitOrigin-RevId: 37fee0c8d4ede8e74b491d3d641b0ea0e9003dd4
2026-09-30 19:14:56 +00:00
2026-09-29 22:50:07 +00:00
2026-09-30 17:33:59 +00:00
2026-09-30 05:29:56 +00:00
2026-09-30 00:51:39 +00:00
2026-09-30 18:29:46 +00:00
2026-09-26 23:17:41 +00:00
2026-09-30 01:35:21 +00:00
2026-09-30 11:58:03 +00:00
2026-09-22 06:36:09 +00:00
2026-09-29 18:26:40 +00:00
2026-09-23 18:25:07 +00:00
2026-09-18 00:25:37 +00:00
2026-09-21 22:58:49 +00:00
2026-09-23 00:47:34 +00:00
2026-09-25 00:20:02 +00:00
2026-09-30 18:49:29 +00:00
2026-09-30 19:00:00 +00:00
2026-09-29 13:58:09 +00:00
2026-09-29 03:32:19 +00:00
2026-09-25 16:57:42 +00:00
2026-09-25 16:28:36 +00:00
2026-09-25 15:42:35 +00:00
2026-09-25 20:34:52 +00:00
2026-09-30 16:57:01 +00:00
2026-09-25 00:20:02 +00:00
2026-09-24 01:50:36 +00:00
2026-09-29 23:01:46 +00:00
2026-09-30 14:37:56 +00:00
2026-09-18 22:40:37 +00:00
2026-09-30 17:50:58 +00:00
2026-09-26 23:17:41 +00:00
2026-09-23 22:20:34 +00:00
2026-09-29 13:08:37 +00:00
2026-09-26 23:17:41 +00:00
2026-09-30 18:37:18 +00:00
2026-09-29 20:40:21 +00:00
2026-09-30 18:29:46 +00:00
2026-09-30 11:59:13 +00:00
2026-09-30 17:20:35 +00:00
2026-09-24 17:04:24 +00:00
2026-09-19 01:21:32 +00:00
2026-09-29 04:38:37 +00:00
2026-09-28 17:40:25 +00:00
2026-09-30 11:59:13 +00:00
2026-09-29 22:04:56 +00:00
2026-09-24 14:50:27 +00:00
2026-09-29 22:24:09 +00:00
2026-09-29 11:51:21 +00:00
2026-09-30 05:29:56 +00:00
2026-09-28 14:46:32 +00:00
2026-09-28 06:30:53 +00:00
2026-09-29 21:25:26 +00:00
2026-09-29 18:50:36 +00:00
2026-09-29 17:22:13 +00:00
2026-09-25 19:14:31 +00:00
2026-09-30 04:30:01 +00:00
2026-09-30 17:45:25 +00:00
2026-09-23 22:20:34 +00:00
2026-09-29 01:14:41 +00:00
2026-09-30 04:30:01 +00:00
2026-09-30 01:58:03 +00:00
2026-09-29 01:14:41 +00:00
2026-09-30 04:30:01 +00:00
2026-09-30 18:51:29 +00:00
2026-09-26 23:08:50 +00:00
2026-09-24 18:08:31 +00:00
2026-09-28 15:58:17 +00:00
2026-09-27 03:26:57 +00:00
2026-09-30 19:00:00 +00:00
2026-09-29 13:58:09 +00:00
2026-09-30 18:03:22 +00:00
2026-09-24 20:39:37 +00:00
2026-09-30 19:00:00 +00:00
2026-09-27 19:14:24 +00:00
2026-09-30 19:14:56 +00:00
2026-09-30 01:58:03 +00:00
2026-09-23 04:25:28 +00:00
2026-09-22 23:37:39 +00:00
2026-09-30 17:50:58 +00:00
2026-09-28 19:56:33 +00:00
2026-09-23 00:47:07 +00:00
2026-09-30 18:29:46 +00:00
2026-09-30 18:29:46 +00:00
2026-09-30 03:33:43 +00:00
jif
5e96aabd68