feat(helexa-bench): context-length scaling view (#88) #104

Merged

grenade merged 1 commits from feat/88-bench-context-scaling into main 2026-06-27 11:03:36 +00:00

Conversation 0 Commits 1 Files Changed 5 +244 -6

grenade commented 2026-06-27 11:03:26 +00:00

Owner

Copy Link

Part of the Performance observability epic (#83), O4. Validates the long-context property that makes the 80B-A3B frontier model (#84) viable.

The chat:<n> cells already capture prefill & decode tok/s per context (via #85/#86); this pivots them into a per-(target,model) scaling curve and computes decode-flatness — decode tok/s at the largest context ÷ the smallest. ~1.0 confirms the Gated-DeltaNet O(1)-in-sequence-length decode; a sharp drop locates where the model stops scaling for free.

• store: Store::scaling() pivots the latest-build chat:<n> report cells into ScalingCurve/ScalingPoint, ordered by context, with the flatness ratio (concurrency: and other scenarios excluded).
• report: render_scaling_markdown (one block per model: prefill/decode tok/s vs ctx + flatness verdict) and render_scaling_json.
• cli: helexa-bench report --scaling.
• api: GET /api/scaling for the bench UI.
• example config documents widening prompt_sizes into a scaling ladder.

No new request shape — reuses the chat-latency measurement points, so a denser curve is just more prompt_sizes entries (operators widen deliberately; large contexts cost more per sample).

Tests: scaling pivot + flatness + scenario exclusion, markdown render. All four validation jobs green. Closes #88.

Part of the Performance observability epic (#83), O4. Validates the long-context property that makes the 80B-A3B frontier model (#84) viable. The `chat:<n>` cells already capture prefill & decode tok/s per context (via #85/#86); this pivots them into a per-(target,model) scaling curve and computes **decode-flatness** — decode tok/s at the largest context ÷ the smallest. ~1.0 confirms the Gated-DeltaNet O(1)-in-sequence-length decode; a sharp drop locates where the model stops scaling for free. - **store**: `Store::scaling()` pivots the latest-build `chat:<n>` report cells into `ScalingCurve`/`ScalingPoint`, ordered by context, with the flatness ratio (concurrency:<n> and other scenarios excluded). - **report**: `render_scaling_markdown` (one block per model: prefill/decode tok/s vs ctx + flatness verdict) and `render_scaling_json`. - **cli**: `helexa-bench report --scaling`. - **api**: `GET /api/scaling` for the bench UI. - example config documents widening `prompt_sizes` into a scaling ladder. No new request shape — reuses the chat-latency measurement points, so a denser curve is just more `prompt_sizes` entries (operators widen deliberately; large contexts cost more per sample). Tests: scaling pivot + flatness + scenario exclusion, markdown render. All four validation jobs green. Closes #88.

grenade added 1 commit 2026-06-27 11:03:27 +00:00

feat(helexa-bench): context-length scaling view (#88) ... d472b6428a

The chat:<n> cells already capture prefill & decode tok/s per context
(via #85/#86); this pivots them into a per-(target,model) scaling curve
and computes decode-flatness — decode tok/s at the largest context ÷ the
smallest. ~1.0 confirms the Gated-DeltaNet O(1)-in-sequence-length
decode; a sharp drop locates where the model stops scaling for free.

- store: Store::scaling() pivots the latest-build chat:<n> report cells
  into ScalingCurve/ScalingPoint, ordered by context, with the flatness
  ratio (concurrency:<n> and other scenarios are excluded).
- report: render_scaling_markdown (one block per model: prefill/decode
  tok/s vs ctx + a flatness verdict) and render_scaling_json.
- cli: `helexa-bench report --scaling` selects the view.
- api: GET /api/scaling for the bench UI.
- example config documents widening prompt_sizes into a scaling ladder.

No new request shape — reuses the chat-latency measurement points, so a
denser curve is just more prompt_sizes entries (operators widen the
ladder deliberately; large contexts cost more per sample).

Tests: scaling pivot + flatness + scenario exclusion, markdown render.

Part of the Performance observability epic (#83), O4. Validates the
long-context property that makes the 80B-A3B frontier model (#84) viable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq

grenade merged commit a488ade675 into main 2026-06-27 11:03:36 +00:00

grenade deleted branch feat/88-bench-context-scaling 2026-06-27 11:03:36 +00:00

grenade referenced this issue from a commit 2026-06-27 11:03:37 +00:00

Merge pull request 'feat(helexa-bench): context-length scaling view (#88)' (#104) from feat/88-bench-context-scaling into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

out-of-scope

Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably

p1-now

Active pain or unblocking work — pick up first

p2-next

Engine levers — start once P1 baselines exist

p3-later

Dependent on earlier work, or demand-driven

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

grenade

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: helexa/helexa#104