rob thijssen
1231276cc6
Some checks failed
CI / Test (push) Waiting to run
CI / CUDA type-check (push) Waiting to run
CI / Format (push) Successful in 8s
CI / Format (pull_request) Successful in 8s
CI / Clippy (push) Successful in 2m21s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
CI / Clippy (pull_request) Successful in 2m28s
CI / Test (pull_request) Successful in 5m56s
CI / CUDA type-check (pull_request) Successful in 12m47s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
Fleet visibility for the #137 capacity metrics now that cortex exports them. asset/monitoring/: - prometheus-cortex.scrape.yml — the single scrape job (cortex on hanzalova:31314 is the only target; neuron has no /metrics) - firewalld-cortex-metrics.sh — scoped rule opening :31314 to the monitoring host (10.3.101.4) only; firewalld had no rule so cross-host scrapes timed out - grafana-helexa-fleet.json — dashboard (saturation, tok/s, rejection rate, TTFT, per-device VRAM/util/temp), datasource + node/model vars - README.md — apply runbook (firewall → scrape reload → dashboard import) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012GaW6F2j3zsQ6yABKPsXXe
29 lines
1.2 KiB
Bash
Executable File
29 lines
1.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Open cortex's Prometheus metrics port (:31314) on the cortex gateway host
|
|
# (hanzalova.internal, 10.6.0.46) to the monitoring host ONLY
|
|
# (golgafrinchans.kosherinata.internal, which reaches hanzalova as 10.3.101.4).
|
|
#
|
|
# cortex binds 0.0.0.0:31314 but firewalld has no rule for it, so a
|
|
# cross-host scrape times out. This adds a scoped rich rule — the metrics
|
|
# stay closed to everything except the Prometheus host.
|
|
#
|
|
# Run as root on hanzalova.internal. Idempotent.
|
|
set -euo pipefail
|
|
|
|
MONITOR_IP="10.3.101.4" # golgafrinchans.kosherinata.internal (mesh source)
|
|
PORT="31314"
|
|
|
|
rule="rule family=\"ipv4\" source address=\"${MONITOR_IP}/32\" port port=\"${PORT}\" protocol=\"tcp\" accept"
|
|
|
|
if firewall-cmd --permanent --query-rich-rule="${rule}" >/dev/null 2>&1; then
|
|
echo "rich rule already present; nothing to do"
|
|
else
|
|
firewall-cmd --permanent --add-rich-rule="${rule}"
|
|
firewall-cmd --reload
|
|
echo "opened tcp/${PORT} to ${MONITOR_IP} and reloaded firewalld"
|
|
fi
|
|
|
|
# Verify from this host; the real check is a scrape from the monitoring host:
|
|
# curl -s -o /dev/null -w '%{http_code}\n' http://hanzalova.internal:31314/metrics
|
|
firewall-cmd --list-rich-rules | grep "${PORT}" || true
|