jif 5e96aabd68 Avoid full-string scans in token-budget truncation (#49712) ...

## Why

Token-budget truncation uses byte counts to estimate tokens, but previously scanned the entire string to count characters, including the discarded middle.

## What changed

Use UTF-8 boundary lookups to select the retained prefix and suffix for token-budget truncation. Skip character counting for a zero-token budget and defer it to paths that need character counts.

## Testing

Add regression coverage for multibyte characters crossing both retained boundaries, checking the truncated output and original token estimate.

GitOrigin-RevId: 37fee0c8d4ede8e74b491d3d641b0ea0e9003dd4

2026-09-30 19:14:56 +00:00

..

src

Avoid full-string scans in token-budget truncation (#49712)

2026-09-30 19:14:56 +00:00

BUILD.bazel

feat: add support for building with Bazel (#8875)

2026-01-09 11:09:43 -08:00

Cargo.toml

Disable empty Cargo test targets (#21584)

2026-05-07 15:44:17 -07:00