mirror of
https://github.com/openai/codex.git
synced 2026-10-18 17:18:52 +00:00
## Why Token-budget truncation uses byte counts to estimate tokens, but previously scanned the entire string to count characters, including the discarded middle. ## What changed Use UTF-8 boundary lookups to select the retained prefix and suffix for token-budget truncation. Skip character counting for a zero-token budget and defer it to paths that need character counts. ## Testing Add regression coverage for multibyte characters crossing both retained boundaries, checking the truncated output and original token estimate. GitOrigin-RevId: 37fee0c8d4ede8e74b491d3d641b0ea0e9003dd4