lair/quantus
2
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fix: validate metrics without shell-escaped python or grafana auth

Browse Source 
Two bugs, both in validation only -- the deploy half worked and both
scrape targets are up.

The targets check used python3 -c with \" escapes inside an f-string,
which Python reads as a line continuation:

  SyntaxError: unexpected character after line continuation character

Rewritten as a quoted heredoc reading a file, so nothing is escaped
through the shell. It now also reports per-target health rather than
grepping for ' up '.

The Grafana check called /api/search, which returns 401 -- the API
requires auth and this job holds no Grafana credentials by design.
Replaced with the unauthenticated /api/health, a file presence check for
both dashboard and provider, and a scan of the grafana journal for
dashboard provisioning errors, which is what actually distinguishes a
dashboard that loaded from one that failed to parse.
This commit is contained in:
Rob Thijssen
2026-08-31 18:55:58 +03:00
parent 76fd8073c0
commit 2cd478646f
2 changed files with 73 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/deploy.yaml
+57 -28
View File
@@ -724,41 +724,70 @@ jobs:
fail=0
echo "--- prometheus targets ---"
# The point of this check: a scrape target that drifted out of the
# matrices, or a firewalld rule that never landed, shows up here as a
# missing or DOWN target instead of an empty graph nobody looks at.
# The point of this check: a target that drifted out of the matrices,
# or a firewalld rule that never landed, shows up here as missing or
# DOWN instead of an empty graph nobody looks at.
#
# Written as a quoted heredoc reading a file, NOT `python3 -c '...'`:
# escaping quotes through the shell into an f-string produced
# "SyntaxError: unexpected character after line continuation character",
# because Python reads \" inside the expression as a continuation.
for i in $(seq 1 20); do
body=$(run "curl -fsS 'http://127.0.0.1:${{ env.PROM_PORT }}/api/v1/targets?state=active'")
got=$(python3 -c '
import json,sys
d=json.load(sys.stdin)
for t in d["data"]["activeTargets"]:
j=t["labels"].get("job","")
if j.startswith("quantus-"):
print(f"{j} {t[\"labels\"].get(\"instance\")} {t[\"health\"]} {t.get(\"lastError\",\"\")}")
' <<<"$body" | sort)
[ -n "$got" ] && break
run "curl -fsS 'http://127.0.0.1:${{ env.PROM_PORT }}/api/v1/targets?state=active'" > targets.json
if grep -q 'quantus-' targets.json; then break; fi
sleep 3
done
if [ -z "${got:-}" ]; then
echo " FAIL prometheus has no quantus-* targets — is scrape_config_files wired in?" >&2
echo " (one-time: script/infra-setup.sh --metrics-hosts)" >&2
exit 1
if python3 - <<'PY'
import json, sys
d = json.load(open("targets.json"))
rows = [t for t in d["data"]["activeTargets"]
if t["labels"].get("job", "").startswith("quantus-")]
if not rows:
print(" FAIL prometheus has no quantus-* targets", file=sys.stderr)
print(" is scrape_config_files wired into prometheus.yml?", file=sys.stderr)
print(" one-time: script/infra-setup.sh --metrics-hosts", file=sys.stderr)
sys.exit(1)
bad = 0
for t in sorted(rows, key=lambda x: (x["labels"].get("job", ""),
x["labels"].get("instance", ""))):
j = t["labels"].get("job")
i = t["labels"].get("instance")
h = t["health"]
e = t.get("lastError", "")
print((" ok " if h == "up" else " FAIL ") + " ".join(filter(None, [j, i, h, e])))
if h != "up":
bad += 1
sys.exit(1 if bad else 0)
PY
then
echo " all quantus targets up"
else
echo " FAIL not every quantus target is up" >&2; fail=1
fi
printf '%s\n' "$got" | sed 's/^/ /'
printf '%s\n' "$got" | grep -qv " up " && { echo " FAIL not every quantus target is up" >&2; fail=1; } || true
echo "--- grafana dashboard ---"
# Assert Grafana actually LOADED it, not merely that a file is on disk.
# A provisioned dashboard that failed to parse leaves the JSON sitting
# there looking fine while the folder stays empty.
if run "curl -fsS 'http://127.0.0.1:${{ env.GRAFANA_PORT }}/api/search?query=Quantus'" \
| grep -q 'quantus-node-miner'; then
echo " ok grafana has loaded the dashboard (uid quantus-node-miner)"
# Grafana's /api/search requires auth (401 unauthenticated), and this
# job holds no Grafana credentials by design — so assert provisioning
# from the journal instead, which gitea_ci can read via systemd-journal.
run "curl -fsS http://127.0.0.1:${{ env.GRAFANA_PORT }}/api/health" > /dev/null \
&& echo " ok grafana is healthy" \
|| { echo " FAIL grafana health endpoint not responding" >&2; fail=1; }
if run "test -s /etc/grafana/provisioning/dashboards/quantus/quantus.json" \
&& run "test -s /etc/grafana/provisioning/dashboards/quantus-dashboards.yaml"; then
echo " ok dashboard and provider present on the host"
else
echo " FAIL grafana has not loaded the dashboard" >&2
echo " provider present? check /etc/grafana/provisioning/dashboards/" >&2
fail=1
echo " FAIL dashboard JSON or provider missing" >&2; fail=1
fi
# A dashboard that fails to parse is logged by the provisioner and then
# silently absent from the UI, so treat any provisioning error as fatal.
if run "journalctl -u grafana.service --since '-10min' --no-pager" \
| grep -E 'logger=provisioning\.dashboard.*level=error'; then
echo " FAIL grafana reported a dashboard provisioning error" >&2; fail=1
else
echo " ok no dashboard provisioning errors in the last 10 minutes"
fi
exit $fail
CLAUDE.md
+16
View File
@@ -100,6 +100,22 @@ immediate assertion reads a starting node as a broken one. Validate waits for
readiness on a bounded poll. Do not replace that with a fixed `sleep`, and do not
remove it because "it passed locally".
## Inline Python in workflow steps
Use a quoted heredoc (`python3 - <<'PY'`) reading a file or an env var. Do NOT
use `python3 -c '...'` with escaped quotes: `\"` inside an f-string expression is
read by Python as a line continuation and fails with "unexpected character after
line continuation character". A quoted heredoc needs no shell escaping at all.
## Grafana checks cannot use its API
`/api/search` and most of Grafana's API require auth (401 unauthenticated), and
this workflow deliberately holds no Grafana credentials. Assert provisioning from
`journalctl -u grafana.service` instead — `gitea_ci` is in `systemd-journal`, and
a dashboard that fails to parse logs `logger=provisioning.dashboard level=error`
and is then silently absent from the UI. `/api/health` is unauthenticated and is
fine for liveness.
## Pipes in validate steps
The workflow shell is `bash -e -o pipefail`. Any pipe whose *reader* exits early