helexa/helexa
1
0
Fork 0
Code Issues 134 Pull Requests Actions Packages Projects Releases Wiki Activity

helexa-bench should measure only quiet neurons — and stamp every sample with the contention it saw #298

New Issue
Open
opened 2026-08-26 14:12:46 +00:00 by grenade · 0 comments
grenade commented 2026-08-26 14:12:46 +00:00
Owner
Copy Link
Copy Source

Today a bench sweep and a live agentic session ran against beast at the
same time. Both were damaged: the session's decode was slowed, and bench
recorded 9.8 tok/s for concurrency:8 against 168.8 measured on a
quiet host — then wrote that into the trend series as if it were fleet
performance.

Nothing in the pipeline noticed, and nothing in the stored row says the
number is unusable.

Why it happened, and why it will keep happening

The sweep is version-aware: a new build SHA unsatisfies every cell and
triggers a full pass. That means bench starts measuring the instant a
deploy lands — precisely when the fleet is cold-loading and whoever
did the deploy is testing it. The trigger is correlated with the busiest
moment, by construction.

What is already available

/health publishes everything a gate needs, per model:

pub in_flight: usize,
pub queue_depth: usize,
pub max_in_flight: usize,      // the denominator

and bench already polls /version + /models cheaply each tick, so
adding /health costs nothing.

Three traps a naive "skip when busy" would walk into

1. beast may never be idle. Its 27B co-serves a continuous CCTV
motion-triage workload at ~6 requests/min. A strict idle gate could mean
the host we care most about silently accumulates no data. That is
worse than contaminated data: a wrong number eventually contradicts
something, while an empty series just looks like nobody got round to it.
Any gate needs a starvation guard.

2. Instantaneous idle is not idle. At 6 req/min the gaps between
CCTV requests are seconds long, so a single poll reading in_flight: 0
lands in a gap and proves nothing. Either sample across a window, or —
better — have neuron publish how long it has been quiet.

3. Gating alone would not have caught today. Bench's own traffic
makes in_flight > 0 the moment it starts, and concurrency:8
deliberately creates load. The thing that matters is foreign arrivals
during a measurement, which a pre-flight check cannot see.

Proposal

1. idle_secs on neuron /health. Seconds since the last request
finished for this model (0 while any are in flight). One poll then
answers "has this been quiet for 60 s?" reliably, and the bench side
stops being a heuristic over sampled polls.

2. Defer, don't skip. A cell whose target is not quiet is
rescheduled with backoff, not dropped. The sweep revisits.

3. Starvation guard — the part not to omit. After N deferrals,
measure anyway and mark the row, and log it loudly. A target that can
never be measured cleanly is itself a finding (it is true of beast, and
we should be able to see that stated rather than infer it from a gap in
a chart). Deferring forever in silence is the failure mode this
introduces, so it needs designing in from the start.

4. Stamp every sample with the contention it observed. Record
in_flight / queue_depth at start and end, and the peak during the
run. Bench knows its own concurrency, so anything above that is foreign
traffic.

This is the highest-value part and worth doing even if the rest waits:
it is what makes a clean row provably clean and a dirty row
filterable, instead of both looking identical in the database. Stamping
alone would have caught today's contamination with no gating at all.
Gating only reduces how much work gets thrown away.

5. Settle delay after a SHA change. Wait for the fleet to be
serving normally before starting the post-deploy sweep, rather than
racing the cold-load.

UI

bench/src/pages/Trends.tsx should flag or filter contaminated points
rather than plotting them silently. Same instinct as regime.rs — a
number whose meaning is compromised should say so on the chart, not in
someone's memory.

Retrospective note

Today's beast rows for d3341cb are contaminated in both directions and
should not be trusted. Worth deciding whether to mark or drop them once
there is a mechanism to mark with.

Related

#288 (bench identity — the last time bench measured something other
than what it thought), the regime.rs measurement-boundary precedent,
#137 (max_in_flight / load surface), #291.

Today a bench sweep and a live agentic session ran against beast at the same time. Both were damaged: the session's decode was slowed, and bench recorded **9.8 tok/s** for `concurrency:8` against 168.8 measured on a quiet host — then wrote that into the trend series as if it were fleet performance. Nothing in the pipeline noticed, and nothing in the stored row says the number is unusable. ## Why it happened, and why it will keep happening The sweep is version-aware: a new build SHA unsatisfies every cell and triggers a full pass. That means bench starts measuring **the instant a deploy lands** — precisely when the fleet is cold-loading and whoever did the deploy is testing it. The trigger is correlated with the busiest moment, by construction. ## What is already available `/health` publishes everything a gate needs, per model: ```rust pub in_flight: usize, pub queue_depth: usize, pub max_in_flight: usize, // the denominator ``` and bench already polls `/version` + `/models` cheaply each tick, so adding `/health` costs nothing. ## Three traps a naive "skip when busy" would walk into **1. beast may never be idle.** Its 27B co-serves a continuous CCTV motion-triage workload at ~6 requests/min. A strict idle gate could mean the host we care most about silently accumulates *no* data. That is worse than contaminated data: a wrong number eventually contradicts something, while an empty series just looks like nobody got round to it. Any gate needs a starvation guard. **2. Instantaneous idle is not idle.** At 6 req/min the gaps between CCTV requests are seconds long, so a single poll reading `in_flight: 0` lands in a gap and proves nothing. Either sample across a window, or — better — have neuron publish how long it has been quiet. **3. Gating alone would not have caught today.** Bench's own traffic makes `in_flight > 0` the moment it starts, and `concurrency:8` deliberately creates load. The thing that matters is *foreign* arrivals during a measurement, which a pre-flight check cannot see. ## Proposal **1. `idle_secs` on neuron `/health`.** Seconds since the last request finished for this model (0 while any are in flight). One poll then answers "has this been quiet for 60 s?" reliably, and the bench side stops being a heuristic over sampled polls. **2. Defer, don't skip.** A cell whose target is not quiet is rescheduled with backoff, not dropped. The sweep revisits. **3. Starvation guard — the part not to omit.** After N deferrals, measure anyway and mark the row, and log it loudly. A target that can never be measured cleanly is itself a finding (it is true of beast, and we should be able to see that stated rather than infer it from a gap in a chart). Deferring forever in silence is the failure mode this introduces, so it needs designing in from the start. **4. Stamp every sample with the contention it observed.** Record `in_flight` / `queue_depth` at start and end, and the peak during the run. Bench knows its own concurrency, so anything above that is foreign traffic. This is the highest-value part and worth doing even if the rest waits: it is what makes a clean row *provably* clean and a dirty row filterable, instead of both looking identical in the database. **Stamping alone would have caught today's contamination with no gating at all.** Gating only reduces how much work gets thrown away. **5. Settle delay after a SHA change.** Wait for the fleet to be serving normally before starting the post-deploy sweep, rather than racing the cold-load. ## UI `bench/src/pages/Trends.tsx` should flag or filter contaminated points rather than plotting them silently. Same instinct as `regime.rs` — a number whose meaning is compromised should say so on the chart, not in someone's memory. ## Retrospective note Today's beast rows for `d3341cb` are contaminated in both directions and should not be trusted. Worth deciding whether to mark or drop them once there is a mechanism to mark with. ## Related #288 (bench identity — the last time bench measured something other than what it thought), the `regime.rs` measurement-boundary precedent, #137 (`max_in_flight` / load surface), #291.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#298