fix: wait for first scrape before judging target health

Run 8's validate reported FAIL quantus-arena bob:25033 unknown FAIL quantus-miner quadbrat:9900 unknown ok quantus-miner benjy:9900 up ok quantus-node bob:9615 up which is not a failure: a target Prometheus has only just learned about reports 'unknown' until its first scrape completes, and this step runs seconds after the config reload with a 15s scrape interval. Both were up moments later. The old retry loop broke as soon as ANY quantus target existed, which was immediately true because the node target predated this run. It now waits for the unknowns to resolve, bounded at 120s. A genuinely dead target reports 'down' rather than 'unknown', so this does not paper over real failures -- the same distinction the readiness wait on the node makes. Third race of this shape: deploy finishes before the thing it deployed is observable.
2026-09-01 03:43:03 +03:00
parent c748da08cb
commit adc04f5bc5
2 changed files with 29 additions and 3 deletions
@@ -779,10 +779,29 @@ jobs:
          # escaping quotes through the shell into an f-string produced
          # "SyntaxError: unexpected character after line continuation character",
          # because Python reads \" inside the expression as a continuation.
-          for i in $(seq 1 20); do
+          # A target Prometheus has only just learned about reports health
+          # "unknown" until its FIRST scrape completes. With a 15s scrape
+          # interval and this step running seconds after the config reload, any
+          # run that adds a target would otherwise fail on a target that is
+          # perfectly healthy. Wait for the unknowns to resolve, bounded; a
+          # genuinely dead target goes to "down", not "unknown", so this does
+          # not paper over real failures.
+          cat > count_unknown.py <<'PY'
+          import json
+          d = json.load(open("targets.json"))
+          rows = [t for t in d["data"]["activeTargets"]
+                  if t["labels"].get("job", "").startswith("quantus-")]
+          print(sum(1 for t in rows if t["health"] == "unknown") if rows else 1)
+          PY
+          for i in $(seq 1 24); do
            run "curl -fsS 'http://127.0.0.1:${{ env.PROM_PORT }}/api/v1/targets?state=active'" > targets.json
-            if grep -q 'quantus-' targets.json; then break; fi
-            sleep 3
+            pending=$(python3 count_unknown.py)
+            if [ "$pending" = "0" ]; then
+              [ "$i" -gt 1 ] && echo "  settled after ~$((i*5))s"
+              break
+            fi
+            [ "$i" = "24" ] && echo "  ${pending} target(s) still unscraped after 120s" >&2
+            sleep 5
          done

          if python3 - <<'PY'
@@ -91,6 +91,13 @@ deploy resolves the node host on the MINER host at deploy time and renders the
 literal into `miner.env` — which also keeps `10.x` out of the repo. Do not
 "simplify" it back to `${{ matrix.node }}:$PORT`.

+## Prometheus targets report "unknown" before their first scrape
+
+Any run that ADDS a scrape target will see it as health `unknown`, not `up` —
+Prometheus has not scraped it yet, and validate runs seconds after the config
+reload. `validate metrics` waits for unknowns to resolve before judging. A
+genuinely dead target reports `down`, so the wait does not hide real failures.
+
 ## Validate runs seconds after a restart

 `systemctl restart` returns as soon as the process is exec'd, not when it is