feat(observability): export live per-neuron:model capacity metrics (#137 T1+T2) #153
Reference in New Issue
Block a user
Delete Branch "feat/137-capacity-metrics-t1"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes the #137 observability gap (T1 + T2). Every capacity signal cortex already had in hand but discarded is now on the Prometheus surface, plus the throughput and load-shedding numbers.
What lands
T1 — live load + GPU headroom (previously polled for routing, then dropped)
cortex_model_{in_flight,queue_depth,max_in_flight,max_queue_depth}{node,model}— saturation = in_flight / max_in_flightcortex_device_{vram_used_mb,vram_free_mb,utilization_pct,temp_c}{node,device}T2 — throughput + backpressure
cortex_model_tok_s_{prefill,decode}{node,model}— live tokens/sec EMA, folded in at every serving finish site (single-GPU worker, TP task, all four batch-engine finish_slot sites), so decode tok/s is honest under real concurrency, not just batch-1cortex_model_rejections_total{node,model,reason}— queue_full / wait_timeout / per_principal, the load-shedding signalNotes
ModelLoadadditions are#[serde(default)]for rolling-deploy back-compat (a pre-#137 neuron simply reports 0 / "unknown").🤖 Generated with Claude Code
Adds the load-shedding signal. AdmissionController tallies rejections by reason (queue_full / wait_timeout / per_principal) with lock-free atomics, counted since the model loaded. Surfaced on /health via ModelLoad and exported by cortex as cortex_model_rejections_total{node,model,reason} (counter via .absolute — a model reload resets to 0, read as a normal Prometheus counter reset). This is the definitive "at capacity / shedding" metric: a rising wait_timeout/queue_full rate means the neuron:model is turning work away. Tests: admission unit test asserts a refused request tallies under queue_full only; the cortex export test asserts the counter renders with the reason label and value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012GaW6F2j3zsQ6yABKPsXXeThe headline capacity number: live decode throughput per neuron:model. Evolves context_limit::PrefillRateEma into ThroughputEma — a dual prefill+decode tokens/sec EMA (type alias keeps the ~15 prefill call sites, incl. cuda-gated, unchanged). Decode is folded in at every finish site the model already routes through: the single-GPU device-worker path, the TP orchestration task, and all four batch-engine finish_slot sites (engine derives the model's rate object from the active backend/session, so Single and TP both record into the same tracker /health reads). The pure-CPU arch.forward fallback has no rate object and is left untracked (not a served-throughput number). Surfaced on /health via ModelLoad.{tok_s_prefill,tok_s_decode} and exported by cortex as gauges cortex_model_tok_s_{prefill,decode}{node,model}. Tests: ThroughputEma decode EMA tracks independently of prefill and ignores degenerate samples; cortex export test asserts both tok/s gauges render with the polled values. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012GaW6F2j3zsQ6yABKPsXXe