rob thijssen
6e0f15c888
Some checks failed
CI / CUDA type-check (push) Failing after 1m40s
CI / Format (push) Successful in 42s
CI / Clippy (push) Successful in 2m51s
CI / Test (push) Successful in 6m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The harness emitted only token counts on InferenceEvent::Finish; all
timing was client-side SSE arrival, so bench "TTFT" conflated tokenize
+ prefill and decode tok/s was a window estimate.
Add FinishTiming { prefill_ms, decode_ms, prefill_tokens } to the
Finish event, populated by all three streaming producers (TP, worker,
and local CPU paths), and surface it on the OpenAI chat
`usage.helexa_timing` extension so helexa-bench can compute true
prefill vs decode tok/s. cortex forwards usage verbatim, so the field
survives proxying. Non-streaming and Responses paths carry None for
now (bench reads the streaming chat path).
Keystone for the Performance observability epic (#83).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq