chore(monitoring): prometheus scrape + grafana dashboard for #137 metrics

Fleet visibility for the #137 capacity metrics now that cortex exports
them. asset/monitoring/:
- prometheus-cortex.scrape.yml — the single scrape job (cortex on
  hanzalova:31314 is the only target; neuron has no /metrics)
- firewalld-cortex-metrics.sh — scoped rule opening :31314 to the
  monitoring host (10.3.101.4) only; firewalld had no rule so cross-host
  scrapes timed out
- grafana-helexa-fleet.json — dashboard (saturation, tok/s, rejection
  rate, TTFT, per-device VRAM/util/temp), datasource + node/model vars
- README.md — apply runbook (firewall → scrape reload → dashboard import)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GaW6F2j3zsQ6yABKPsXXe

asset/monitoring/README.md

@@ -0,0 +1,79 @@
+# Fleet monitoring — Prometheus scrape + Grafana dashboard
+
+Visibility into the cortex `#137` capacity metrics (per-neuron:model load,
+saturation, tok/s, load-shedding, and per-device GPU health) beyond what the
+bench UI shows.
+
+## Topology
+
+- **cortex** runs on `hanzalova.internal` (10.6.0.46) and is the **only**
+  Prometheus target — it exposes every `cortex_*` metric on `:31314`, already
+  labelled by `{node,model}` / `{node,device}` from its neuron poller. neuron
+  has no `/metrics` endpoint.
+- **Prometheus + Grafana** run on `golgafrinchans.kosherinata.internal`
+  (Prometheus is a podman quadlet, config bind-mounted at
+  `/etc/prometheus/prometheus.yml`, `--web.enable-lifecycle` on). Its mesh IP
+  toward hanzalova is `10.3.101.4`.
+
+## Apply (three steps, in order)
+
+### 1. Open the metrics port on the cortex host
+
+cortex binds `0.0.0.0:31314` but firewalld has no rule for it, so a
+cross-host scrape times out. Open it to the monitoring host **only**:
+
+```sh
+# on hanzalova.internal, as root
+./firewalld-cortex-metrics.sh
+```
+
+### 2. Add the scrape job on the monitoring host
+
+Append `prometheus-cortex.scrape.yml` into the `scrape_configs:` list of
+`/etc/prometheus/prometheus.yml` on `golgafrinchans.kosherinata.internal`,
+then hot-reload (no restart, lifecycle API is enabled):
+
+```sh
+curl -X POST http://localhost:9090/-/reload
+# verify the target is UP:
+curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[]|select(.labels.job=="cortex")|{health,lastError}'
+```
+
+### 3. Import the dashboard
+
+`grafana-helexa-fleet.json` is a raw dashboard model with a `datasource`
+template variable, so it binds to whichever Prometheus data source you pick
+at import. Import via the API (creds in `/etc/grafana/grafana.env`):
+
+```sh
+# on golgafrinchans, GF_SECURITY_ADMIN_USER/PASSWORD live in the env file
+set -a; . /etc/grafana/grafana.env; set +a
+jq -n --slurpfile d grafana-helexa-fleet.json \
+  '{dashboard: $d[0], overwrite: true, folderUid: null}' \
+| curl -s -u "$GF_SECURITY_ADMIN_USER:$GF_SECURITY_ADMIN_PASSWORD" \
+    -H 'Content-Type: application/json' \
+    -d @- http://localhost:3000/api/dashboards/db | jq '{status,uid,version}'
+```
+
+The dashboard lands at uid `helexa-fleet`. Re-importing with `overwrite:true`
+updates it in place.
+
+## What the dashboard shows
+
+| Row | Panels |
+|---|---|
+| Capacity & saturation | saturation % (in_flight ÷ max_in_flight), in-flight vs ceiling, queue depth |
+| Throughput | decode tok/s (revenue capacity), prefill tok/s |
+| Backpressure & traffic | rejection rate by reason, request & error rate, TTFT p95/p50 |
+| GPU health | VRAM used per device, GPU utilization %, temperature |
+
+Templating: pick the Prometheus data source, then filter by `neuron` (node)
+and `model`. Values are live at the cortex poll cadence (~10 s) scraped every
+15 s.
+
+## Note on the empirical knee
+
+The live gauges above answer "how loaded is it right now". The
+**sustainable-concurrency knee** (max N before latency/shedding breaks) is a
+load-test result, not a live gauge — see `helexa-bench report --concurrency`
+and `GET /api/concurrency` (#137 T3).

asset/monitoring/firewalld-cortex-metrics.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Open cortex's Prometheus metrics port (:31314) on the cortex gateway host
+# (hanzalova.internal, 10.6.0.46) to the monitoring host ONLY
+# (golgafrinchans.kosherinata.internal, which reaches hanzalova as 10.3.101.4).
+#
+# cortex binds 0.0.0.0:31314 but firewalld has no rule for it, so a
+# cross-host scrape times out. This adds a scoped rich rule — the metrics
+# stay closed to everything except the Prometheus host.
+#
+# Run as root on hanzalova.internal. Idempotent.
+set -euo pipefail
+
+MONITOR_IP="10.3.101.4"   # golgafrinchans.kosherinata.internal (mesh source)
+PORT="31314"
+
+rule="rule family=\"ipv4\" source address=\"${MONITOR_IP}/32\" port port=\"${PORT}\" protocol=\"tcp\" accept"
+
+if firewall-cmd --permanent --query-rich-rule="${rule}" >/dev/null 2>&1; then
+  echo "rich rule already present; nothing to do"
+else
+  firewall-cmd --permanent --add-rich-rule="${rule}"
+  firewall-cmd --reload
+  echo "opened tcp/${PORT} to ${MONITOR_IP} and reloaded firewalld"
+fi
+
+# Verify from this host; the real check is a scrape from the monitoring host:
+#   curl -s -o /dev/null -w '%{http_code}\n' http://hanzalova.internal:31314/metrics
+firewall-cmd --list-rich-rules | grep "${PORT}" || true

asset/monitoring/grafana-helexa-fleet.json

@@ -0,0 +1,149 @@
+{
+  "uid": "helexa-fleet",
+  "title": "helexa fleet — capacity & throughput",
+  "tags": ["helexa", "cortex", "capacity"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "15s",
+  "time": { "from": "now-3h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "datasource",
+        "label": "Prometheus",
+        "type": "datasource",
+        "query": "prometheus",
+        "current": {},
+        "hide": 0
+      },
+      {
+        "name": "node",
+        "label": "neuron",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "${datasource}" },
+        "query": "label_values(cortex_model_max_in_flight, node)",
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".*",
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "model",
+        "label": "model",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "${datasource}" },
+        "query": "label_values(cortex_model_max_in_flight{node=~\"$node\"}, model)",
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "allValue": ".*",
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    { "id": 1, "type": "row", "title": "Capacity & saturation", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 } },
+    {
+      "id": 2, "type": "timeseries", "title": "Saturation %  (in_flight ÷ max_in_flight)",
+      "description": "How full each neuron:model is against its admission ceiling. 100% = every slot busy; sustained >100% headroom pressure means it's about to queue/shed.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 1 },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 70 }, { "color": "red", "value": 90 } ] } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "100 * cortex_model_in_flight{node=~\"$node\", model=~\"$model\"} / clamp_min(cortex_model_max_in_flight{node=~\"$node\", model=~\"$model\"}, 1)", "legendFormat": "{{node}} · {{model}}" } ]
+    },
+    {
+      "id": 3, "type": "timeseries", "title": "In-flight vs ceiling",
+      "description": "Live concurrent requests (in_flight) against the configured max_in_flight ceiling.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 1 },
+      "fieldConfig": { "defaults": { "unit": "none", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never" } }, "overrides": [ { "matcher": { "id": "byRegexp", "options": ".*ceiling.*" }, "properties": [ { "id": "custom.lineStyle", "value": { "dash": [8, 6], "fill": "dash" } }, { "id": "custom.fillOpacity", "value": 0 } ] } ] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_model_in_flight{node=~\"$node\", model=~\"$model\"}", "legendFormat": "{{node}} · {{model}}" }, { "refId": "B", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_model_max_in_flight{node=~\"$node\", model=~\"$model\"}", "legendFormat": "{{node}} · {{model}} ceiling" } ]
+    },
+    {
+      "id": 4, "type": "timeseries", "title": "Queue depth",
+      "description": "Requests waiting in admission beyond the in-flight slots — the backpressure building before rejection.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 1 },
+      "fieldConfig": { "defaults": { "unit": "none", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_model_queue_depth{node=~\"$node\", model=~\"$model\"}", "legendFormat": "{{node}} · {{model}}" } ]
+    },
+    { "id": 10, "type": "row", "title": "Throughput", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 } },
+    {
+      "id": 11, "type": "timeseries", "title": "Decode tok/s  (generation throughput)",
+      "description": "Live decode tokens/sec EMA per neuron:model — the revenue-capacity number. Honest under batch concurrency (folded in at every serving finish site).",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 10 },
+      "fieldConfig": { "defaults": { "unit": "none", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never", "axisLabel": "tok/s" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "mean", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_model_tok_s_decode{node=~\"$node\", model=~\"$model\"}", "legendFormat": "{{node}} · {{model}}" } ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Prefill tok/s  (prompt-processing throughput)",
+      "description": "Live prefill tokens/sec EMA per neuron:model.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 10 },
+      "fieldConfig": { "defaults": { "unit": "none", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never", "axisLabel": "tok/s" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "mean", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_model_tok_s_prefill{node=~\"$node\", model=~\"$model\"}", "legendFormat": "{{node}} · {{model}}" } ]
+    },
+    { "id": 20, "type": "row", "title": "Backpressure & traffic", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 } },
+    {
+      "id": 21, "type": "timeseries", "title": "Rejection rate by reason  (load shedding)",
+      "description": "rate() of admission rejections. Any sustained non-zero here means the neuron:model is turning work away — queue_full/wait_timeout = at capacity, per_principal = a client hit its fair-share cap.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 19 },
+      "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "custom": { "fillOpacity": 15, "showPoints": "never", "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "sum"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "sum by (node, model, reason) (rate(cortex_model_rejections_total{node=~\"$node\", model=~\"$model\"}[5m]))", "legendFormat": "{{node}} · {{model}} · {{reason}}" } ]
+    },
+    {
+      "id": 22, "type": "timeseries", "title": "Request & error rate",
+      "description": "Proxied request rate and error rate per neuron:model.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 19 },
+      "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never" } }, "overrides": [ { "matcher": { "id": "byRegexp", "options": ".*error.*" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } ] } ] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "sum by (node, model) (rate(cortex_requests_total{node=~\"$node\", model=~\"$model\"}[5m]))", "legendFormat": "{{node}} · {{model}}" }, { "refId": "B", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "sum by (node, model) (rate(cortex_request_errors_total{node=~\"$node\", model=~\"$model\"}[5m]))", "legendFormat": "{{node}} · {{model}} error" } ]
+    },
+    {
+      "id": 23, "type": "timeseries", "title": "TTFT p95 / p50",
+      "description": "Time-to-first-token latency percentiles from the request histogram.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 19 },
+      "fieldConfig": { "defaults": { "unit": "s", "min": 0, "custom": { "fillOpacity": 0, "showPoints": "never" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "histogram_quantile(0.95, sum by (le, node, model) (rate(cortex_time_to_first_token_seconds_bucket{node=~\"$node\", model=~\"$model\"}[5m])))", "legendFormat": "p95 {{node}} · {{model}}" }, { "refId": "B", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "histogram_quantile(0.50, sum by (le, node, model) (rate(cortex_time_to_first_token_seconds_bucket{node=~\"$node\", model=~\"$model\"}[5m])))", "legendFormat": "p50 {{node}} · {{model}}" } ]
+    },
+    { "id": 30, "type": "row", "title": "GPU health", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 27 } },
+    {
+      "id": 31, "type": "timeseries", "title": "VRAM used per device",
+      "description": "Per-device VRAM used (MB). Pair with free to see headroom for loading bigger models.",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 28 },
+      "fieldConfig": { "defaults": { "unit": "decmbytes", "min": 0, "custom": { "fillOpacity": 10, "showPoints": "never" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_device_vram_used_mb{node=~\"$node\"}", "legendFormat": "{{node}} · gpu{{device}}" } ]
+    },
+    {
+      "id": 32, "type": "timeseries", "title": "GPU utilization %",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 28 },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "custom": { "fillOpacity": 10, "showPoints": "never" } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "mean"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_device_utilization_pct{node=~\"$node\"}", "legendFormat": "{{node}} · gpu{{device}}" } ]
+    },
+    {
+      "id": 33, "type": "timeseries", "title": "GPU temperature °C",
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 28 },
+      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0, "custom": { "fillOpacity": 0, "showPoints": "never" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 75 }, { "color": "red", "value": 87 } ] } }, "overrides": [] },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] } },
+      "targets": [ { "refId": "A", "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "cortex_device_temp_c{node=~\"$node\"}", "legendFormat": "{{node}} · gpu{{device}}" } ]
+    }
+  ]
+}

asset/monitoring/prometheus-cortex.scrape.yml

@@ -0,0 +1,26 @@
+# helexa cortex gateway scrape job — append into the `scrape_configs:` list
+# of /etc/prometheus/prometheus.yml on golgafrinchans.kosherinata.internal,
+# then reload:  curl -X POST http://localhost:9090/-/reload
+#
+# cortex runs on hanzalova.internal (10.6.0.46) and exposes ALL fleet metrics
+# on :31314 — it is the only Prometheus target in the stack (neuron has no
+# /metrics; cortex's poller already relabels everything by {node,model} /
+# {node,device}). Reaching it cross-DC (kosherinata -> hanzalova) requires the
+# firewalld rule in asset/monitoring/firewalld-cortex-metrics.sh, which opens
+# :31314 to this host's mesh IP (10.3.101.4) only.
+#
+# Metrics exposed here include (see #137):
+#   cortex_model_in_flight / _queue_depth / _max_in_flight / _max_queue_depth
+#   cortex_model_tok_s_decode / _tok_s_prefill
+#   cortex_model_rejections_total{reason=queue_full|wait_timeout|per_principal}
+#   cortex_device_vram_used_mb / _vram_free_mb / _utilization_pct / _temp_c
+#   cortex_requests_total / _request_errors_total / _cold_starts_total
+#   cortex_request_duration_seconds / _time_to_first_token_seconds (histograms)
+#   cortex_prompt_tokens_total / _completion_tokens_total / cortex_spend_*
+- job_name: cortex
+  # cortex refreshes the load/device gauges from each neuron's /health on a
+  # ~10s poll; 15s scrape (the global default) tracks it without aliasing.
+  static_configs:
+    - targets: ['hanzalova.internal:31314']
+      labels:
+        fleet: hanzalova