feat(helexa-bench): percentiles + prefill/decode split in store & report (#86) #101

Merged

grenade merged 1 commits from feat/86-bench-percentiles into main 2026-06-27 09:08:53 +00:00

Conversation 0 Commits 1 Files Changed 5 +234 -23

grenade commented 2026-06-27 09:08:44 +00:00

Owner

Copy Link

Part of the Performance observability epic (#83); builds on #85 (now merged).

Bench reported only a single median per cell, hiding tail latency and unable to record the server-measured prefill/decode split now emitted on usage.helexa_timing (#85).

• scenario: parse usage.helexa_timing into ScenarioMetrics (prefill_ms, decode_ms, prefill_tokens).
• store: persist the three columns via an additive, PRAGMA-guarded ensure_columns migration (pre-#85 DBs backfill as NULL); aggregation now emits p50/p95/p99 for TTFT and total (nearest-rank) plus a prefill-tok/s median derived from the split.
• report: markdown gains prefill tok/s, TTFT p95, total p95 columns; JSON gains p95/p99 + prefill_ms/decode_ms/prefill_tps medians.

Tests: nearest-rank percentile, idempotent backfill migration, and a report cell asserting percentiles + prefill split. All four validation jobs green.

Closes #86.

Part of the Performance observability epic (#83); builds on #85 (now merged). Bench reported only a single median per cell, hiding tail latency and unable to record the server-measured prefill/decode split now emitted on `usage.helexa_timing` (#85). - **scenario**: parse `usage.helexa_timing` into `ScenarioMetrics` (prefill_ms, decode_ms, prefill_tokens). - **store**: persist the three columns via an additive, PRAGMA-guarded `ensure_columns` migration (pre-#85 DBs backfill as NULL); aggregation now emits p50/p95/p99 for TTFT and total (nearest-rank) plus a prefill-tok/s median derived from the split. - **report**: markdown gains prefill tok/s, TTFT p95, total p95 columns; JSON gains p95/p99 + prefill_ms/decode_ms/prefill_tps medians. Tests: nearest-rank percentile, idempotent backfill migration, and a report cell asserting percentiles + prefill split. All four validation jobs green. Closes #86.

grenade added 1 commit 2026-06-27 09:08:44 +00:00

feat(helexa-bench): percentiles + prefill/decode split in store & report (#86) ... afc1f7a706

Bench reported only a single median per cell, hiding tail latency and
unable to record the server-measured prefill/decode split now emitted
on `usage.helexa_timing` (#85).

- scenario: parse `usage.helexa_timing` into ScenarioMetrics
  (prefill_ms, decode_ms, prefill_tokens).
- store: persist the three columns (additive PRAGMA-guarded migration
  via ensure_columns, so pre-#85 DBs backfill as NULL); aggregate now
  emits p50/p95/p99 for TTFT and total (nearest-rank) plus a
  prefill-tok/s median derived from the split.
- report: markdown gains prefill tok/s, TTFT p95, total p95 columns;
  JSON gains p95/p99 + prefill_ms/decode_ms/prefill_tps medians.

Tests: nearest-rank percentile, idempotent backfill migration, and a
report cell asserting percentiles + prefill split.

Part of the Performance observability epic (#83). Stacked on #85.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq

grenade merged commit 06a36566d1 into main 2026-06-27 09:08:53 +00:00

grenade deleted branch feat/86-bench-percentiles 2026-06-27 09:08:54 +00:00

grenade referenced this issue from a commit 2026-06-27 09:08:54 +00:00

Merge pull request 'feat(helexa-bench): percentiles + prefill/decode split in store & report (#86)' (#101) from feat/86-bench-percentiles into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

out-of-scope

Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably

p1-now

Active pain or unblocking work — pick up first

p2-next

Engine levers — start once P1 baselines exist

p3-later

Dependent on earlier work, or demand-driven

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

grenade

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: helexa/helexa#101