helexa/helexa
1
0
Fork 0
Code Issues 134 Pull Requests Actions Packages Projects Releases Wiki Activity

monitoring: scrape neuron /metrics, and give it a request counter #328

New Issue
Open
opened 2026-09-22 20:07:08 +00:00 by grenade · 1 comment
grenade commented 2026-09-22 20:07:08 +00:00
Owner
Copy Link
Copy Source

Two gaps, found while trying to answer "how many requests has beast served in the last 15 minutes"

1. The neuron scrape job exists in the repo but is not applied.

asset/monitoring/prometheus-neuron.scrape.yml landed with 5458b285 ("feat(neuron): serve /metrics, with the decode phases as histograms") and carries its own rationale for why cortex's 10s-sampled gauges cannot answer where a request's time goes. But Prometheus on golgafrinchans has only ever scraped cortex:

$ curl -s "$P/api/v1/targets?state=active" | jq -r '.data.activeTargets[]|"\(.job) \(.scrapeUrl) \(.health)"'
cortex http://hanzalova.internal:31314/metrics up

So every neuron_* series — the decode-phase histograms the commit was written for — exists on each host and is collected by nobody. The file says it needs no new firewalld rule (13131 is already open to the control plane), so applying it is: append the job to /etc/prometheus/prometheus.yml, curl -X POST http://localhost:26559/-/reload. Worth doing through whatever manages that host's config rather than by hand, so it survives the next rebuild.

2. neuron's /metrics has no request counter.

Today it publishes only:

  • neuron_prefill_seconds, neuron_decode_seconds (histograms)
  • neuron_decode_phase_seconds{phase=forward|sample|emit|residual}
  • neuron_completion_tokens_total

There is no "requests received" or "responses sent". The histograms' _count is a usable proxy for completed requests, but only for requests that reached decode, and it says nothing about requests that were rejected, refused for context length, or are still in flight.

cortex_requests_total{node,model} counts only what cortex proxied — it misses helexa-bench, direct probes, and anything else that talks to 13131. Today the only complete answer is counting lines in the journal:

ssh beast 'J=$(journalctl -u neuron --since -15min --no-pager);
  grep -c "(stream): starting" <<<"$J";
  grep "done prompt_tokens" <<<"$J" | grep -vc "TP chat_completion"'

which is also a trap: a TP request logs its completion twice, so the naive grep double-counts.

Proposal

Add to neuron's exporter, labelled {model, wire} (openai-chat / openai-responses / images) so the split that the journal shows is queryable:

  • neuron_requests_total — admitted requests, incremented at the start of serving;
  • neuron_responses_total{outcome} — stop / length / tool_calls / error / client_gone; the finish reason is already in the completion log line;
  • neuron_rejections_total{reason} — matching the /health counters (queue_full, wait_timeout, per_principal, anon_yield, kv_timeout, kv_unservable) plus context_length_exceeded, which currently exists only as a WARN;
  • neuron_in_flight / neuron_queue_depth gauges, mirroring /health so the same numbers are queryable over time without cortex in the path.

Then sum(increase(neuron_requests_total{instance=~"beast.*"}[15m])) is the answer, and it covers every caller rather than only cortex-proxied ones.

Acceptance

  • /metrics on a serving neuron exposes the counters above, and they agree with the journal over the same window (the count that motivated this: 10 started / 10 completed / 0 rejected in 15 min, 5 openai-chat + 5 openai-responses).
  • A TP request increments neuron_requests_total exactly once (the double-logging trap above).
  • A rejected request increments neuron_rejections_total and not neuron_responses_total.
  • The scrape job is applied on the Prometheus host and its target is up for every reachable neuron.
  • Grafana's fleet dashboard gets a panel for requests and rejections by reason sourced from neuron, or the existing panels move to it where the neuron series is the better source.

Related

  • #327 (detect and report GPU memory-clock locks) adds GPU clock/power fields to the same health surface; both are the same "the signals exist in memory but nothing exports them" gap the 2026-07-09 addendum describes.
  • tok_s_decode on /health can report nonsense (18,191 tok/s observed 2026-09-22); anything built on the live decode EMA inherits that until it is fixed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA

## Two gaps, found while trying to answer "how many requests has beast served in the last 15 minutes" **1. The neuron scrape job exists in the repo but is not applied.** `asset/monitoring/prometheus-neuron.scrape.yml` landed with `5458b285` ("feat(neuron): serve /metrics, with the decode phases as histograms") and carries its own rationale for why cortex's 10s-sampled gauges cannot answer where a request's time goes. But Prometheus on golgafrinchans has only ever scraped cortex: ``` $ curl -s "$P/api/v1/targets?state=active" | jq -r '.data.activeTargets[]|"\(.job) \(.scrapeUrl) \(.health)"' cortex http://hanzalova.internal:31314/metrics up ``` So every `neuron_*` series — the decode-phase histograms the commit was written for — exists on each host and is collected by nobody. The file says it needs no new firewalld rule (13131 is already open to the control plane), so applying it is: append the job to `/etc/prometheus/prometheus.yml`, `curl -X POST http://localhost:26559/-/reload`. Worth doing through whatever manages that host's config rather than by hand, so it survives the next rebuild. **2. neuron's `/metrics` has no request counter.** Today it publishes only: - `neuron_prefill_seconds`, `neuron_decode_seconds` (histograms) - `neuron_decode_phase_seconds{phase=forward|sample|emit|residual}` - `neuron_completion_tokens_total` There is no "requests received" or "responses sent". The histograms' `_count` is a usable proxy for *completed* requests, but only for requests that reached decode, and it says nothing about requests that were rejected, refused for context length, or are still in flight. `cortex_requests_total{node,model}` counts only what cortex proxied — it misses `helexa-bench`, direct probes, and anything else that talks to 13131. Today the only complete answer is counting lines in the journal: ```sh ssh beast 'J=$(journalctl -u neuron --since -15min --no-pager); grep -c "(stream): starting" <<<"$J"; grep "done prompt_tokens" <<<"$J" | grep -vc "TP chat_completion"' ``` which is also a trap: a TP request logs its completion twice, so the naive grep double-counts. ## Proposal Add to neuron's exporter, labelled `{model, wire}` (`openai-chat` / `openai-responses` / `images`) so the split that the journal shows is queryable: - `neuron_requests_total` — admitted requests, incremented at the start of serving; - `neuron_responses_total{outcome}` — `stop` / `length` / `tool_calls` / `error` / `client_gone`; the finish reason is already in the completion log line; - `neuron_rejections_total{reason}` — matching the `/health` counters (`queue_full`, `wait_timeout`, `per_principal`, `anon_yield`, `kv_timeout`, `kv_unservable`) plus `context_length_exceeded`, which currently exists only as a WARN; - `neuron_in_flight` / `neuron_queue_depth` gauges, mirroring `/health` so the same numbers are queryable over time without cortex in the path. Then `sum(increase(neuron_requests_total{instance=~"beast.*"}[15m]))` is the answer, and it covers every caller rather than only cortex-proxied ones. ## Acceptance - `/metrics` on a serving neuron exposes the counters above, and they agree with the journal over the same window (the count that motivated this: 10 started / 10 completed / 0 rejected in 15 min, 5 `openai-chat` + 5 `openai-responses`). - A TP request increments `neuron_requests_total` exactly **once** (the double-logging trap above). - A rejected request increments `neuron_rejections_total` and **not** `neuron_responses_total`. - The scrape job is applied on the Prometheus host and its target is `up` for every reachable neuron. - Grafana's fleet dashboard gets a panel for requests and rejections by reason sourced from neuron, or the existing panels move to it where the neuron series is the better source. ## Related - #327 (detect and report GPU memory-clock locks) adds GPU clock/power fields to the same health surface; both are the same "the signals exist in memory but nothing exports them" gap the 2026-07-09 addendum describes. - `tok_s_decode` on `/health` can report nonsense (18,191 tok/s observed 2026-09-22); anything built on the live decode EMA inherits that until it is fixed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade commented 2026-09-22 20:27:10 +00:00
Author
Owner
Copy Link
Copy Source

Two more observability gaps from the same session, both "the neuron knows this and never says it". Filing here rather than separately because they are the same surface as the counters above — what a request was, not just how many there were.

1. The resolved reasoning effort is never logged

apply_effort_kwarg (crates/neuron/src/harness/candle.rs:6348) reads extra.reasoning.effort, snaps it to a rung the model's template accepts, and puts reasoning_effort in the template kwargs. It logs only the interesting-but-rare cases: an INFO when the requested effort had to be mapped to a neighbouring rung, and a DEBUG when there is no comparable rung. The ordinary case — request asked medium, model got medium — logs nothing, and neither does the request-start line, which carries temperature, top_p, top_k, seed and max_new but not effort.

So "what effort did this request actually run at?" cannot be answered from the journal. Answering it took reading four files across pi, pi-ai and neuron to prove the value survives each hop, then an A/B against the live host to prove the knob reaches the template:

effort reasoning tokens (same trivial prompt)
low 56
medium 129
xhigh 54

(One sample each on a question with nothing to think about, so this shows the knob is wired, nothing more.)

It also matters because the template's own default is the strongest rung (xhigh for Qwen/Qwen3.8-27B), so a caller that sends nothing silently gets the most expensive setting. Whether that happened is currently unanswerable after the fact.

Proposal: add the resolved effort to the existing TP chat_completion (stream): starting INFO line — effort=medium (or effort=default:xhigh when the caller sent none and the template's default applies). One field, no new line, and it makes the reasoning-token totals interpretable.

2. A request's principal is not in the neuron journal

Identifying which client was sending a recurring workload meant grepping cortex's TRACE-level inbound bodies on the gateway host, because neuron's log lines carry req_id, model and wire but nothing about who the caller is. cortex already stamps principal headers for fair-share admission (#54) and neuron already parses them — principal never reaches the log.

The body grep worked, but TRACE bodies are a blunt instrument: they log entire request payloads, including base64 image data, to the journal of the gateway.

Proposal: add principal (account/key, already available at the admission boundary) to the request-start and request-done lines, and to the neuron_requests_total labels proposed above if cardinality allows — a per-principal request count is exactly what "who is my second tenant" needs. None for direct/unauthenticated requests, as today.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA

Two more observability gaps from the same session, both "the neuron knows this and never says it". Filing here rather than separately because they are the same surface as the counters above — what a request *was*, not just how many there were. ## 1. The resolved reasoning effort is never logged `apply_effort_kwarg` (`crates/neuron/src/harness/candle.rs:6348`) reads `extra.reasoning.effort`, snaps it to a rung the model's template accepts, and puts `reasoning_effort` in the template kwargs. It logs **only** the interesting-but-rare cases: an INFO when the requested effort had to be mapped to a neighbouring rung, and a DEBUG when there is no comparable rung. The ordinary case — request asked `medium`, model got `medium` — logs nothing, and neither does the request-start line, which carries `temperature`, `top_p`, `top_k`, `seed` and `max_new` but not effort. So "what effort did this request actually run at?" cannot be answered from the journal. Answering it took reading four files across pi, pi-ai and neuron to prove the value survives each hop, then an A/B against the live host to prove the knob reaches the template: | effort | reasoning tokens (same trivial prompt) | |---|---| | low | 56 | | medium | 129 | | xhigh | 54 | (One sample each on a question with nothing to think about, so this shows the knob is wired, nothing more.) It also matters because the template's own default is the *strongest* rung (`xhigh` for `Qwen/Qwen3.8-27B`), so a caller that sends nothing silently gets the most expensive setting. Whether that happened is currently unanswerable after the fact. **Proposal:** add the resolved effort to the existing `TP chat_completion (stream): starting` INFO line — `effort=medium` (or `effort=default:xhigh` when the caller sent none and the template's default applies). One field, no new line, and it makes the reasoning-token totals interpretable. ## 2. A request's principal is not in the neuron journal Identifying *which client* was sending a recurring workload meant grepping cortex's TRACE-level inbound bodies on the gateway host, because neuron's log lines carry `req_id`, `model` and `wire` but nothing about who the caller is. cortex already stamps principal headers for fair-share admission (#54) and neuron already parses them — `principal` never reaches the log. The body grep worked, but TRACE bodies are a blunt instrument: they log entire request payloads, including base64 image data, to the journal of the gateway. **Proposal:** add `principal` (account/key, already available at the admission boundary) to the request-start and request-done lines, and to the `neuron_requests_total` labels proposed above if cardinality allows — a per-principal request count is exactly what "who is my second tenant" needs. `None` for direct/unauthenticated requests, as today. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#328