Block a user
Prefix cache reuse plateaus mid-session — deep agentic turns re-prefill ~23k tokens every turn
Retracting the prefill figures in this issue — both of them
#295 shows the prefill EMA tracks recent traffic shape rather than capability, so every number I quoted here was a snapshot of what…
A restart collapses the advertised context window to a sixth, until real traffic warms the prefill EMA
Responses API terminal frames were unreadable: truncation and crash both reported as success
Raised #294 to fix the class rather than the instance: pin OpenAI's published OpenAPI spec, validate our emitted shapes against it in CI, and require every deviation to be declared with a…
Validate our OpenAI-compatible wire shapes against the published spec in CI, with declared deviations
Responses API terminal frames were unreadable: truncation and crash both reported as success
Prefix cache reuse plateaus mid-session — deep agentic turns re-prefill ~23k tokens every turn
Check (2) answered: there is an explicit cap, and it is small
crates/neuron/src/config.rs:
fn default_prefix_cache_enabled() -> bool { true }
fn default_prefix_cache_budget_mb()…
Prefix cache reuse plateaus mid-session — deep agentic turns re-prefill ~23k tokens every turn
The plateau is not only a latency cost — it shrinks the advertised context window
This turned out to be the important finding, and it was not what I expected when filing.
beast derives its…
grenade
created branch fix/client-capability-misattributions in helexa/helexa
2026-08-26 09:45:52 +00:00
Prefix cache reuse plateaus mid-session — deep agentic turns re-prefill ~23k tokens every turn
The #291 change is now live on beast, which sets up check (1) with a clean before/after.
Before (through both sessions in the issue body, and unchanged since 2026-08-25 16:33):
batch…
Prefix cache reuse plateaus mid-session — deep agentic turns re-prefill ~23k tokens every turn
grenade
pushed to fix/291-max-in-flight-and-config-only-ci at helexa/helexa
2026-08-26 08:40:08 +00:00