fix: emit a heartbeat while waiting so the step is not killed as idle

v1.0.2 moved the verdict off watch-build's exit code, but left the step
silent for the whole build: COPR emits nothing between state transitions,
and the poller only spoke at the end.

That silence is itself the failure. Two monsoon releases died after ~24-27
minutes of no output at all — run 48 at 08:35:50 with COPR having succeeded
at 08:30:30, and run 50 at 09:35:50, 66 seconds *before* COPR reported
success at 09:36:56. A build finishing either side of the kill rules out a
build-duration timeout and points at an inactivity timeout on the runner.

Print a line on every poll with elapsed time and current state. It keeps the
step producing output for as long as the build runs, and makes the log show
progress instead of a 25-minute gap. Heartbeats go to stderr because stdout
carries wait_for_terminal_state's return value.

Test covers a build that stays running across several polls before
succeeding, asserting the heartbeat appears.
This commit is contained in:
2026-08-07 12:39:51 +03:00
parent 3701a483af
commit 5f27687438
2 changed files with 18 additions and 3 deletions

View File

@@ -39,14 +39,24 @@ build_state() {
# terminal state, or "unknown" if we ran out of patience.
wait_for_terminal_state() {
local build_id="$1"
local deadline=$(($(date +%s) + BUILD_TIMEOUT))
local state
local started deadline state now
started=$(date +%s)
deadline=$((started + BUILD_TIMEOUT))
state="$(build_state "$build_id")"
while ! is_terminal_state "$state"; do
if [ "$(date +%s)" -ge "$deadline" ]; then
now=$(date +%s)
if [ "$now" -ge "$deadline" ]; then
echo "unknown"
return
fi
# Heartbeat on stderr (stdout carries the return value). A COPR build emits
# nothing between state transitions, so a long build leaves the step silent
# for its whole duration — and a silent step gets killed by the runner's
# inactivity timeout before it can finish. Two builds died this way after
# ~24-27 minutes of no output, one of them 66 seconds before COPR reported
# success. Printing every poll keeps the step alive and shows progress.
printf ' [%3dm %3ds] build %s: %s\n' \
$(((now - started) / 60)) $(((now - started) % 60)) "$build_id" "${state:-unknown}" >&2
sleep "$POLL_INTERVAL"
state="$(build_state "$build_id")"
done

View File

@@ -92,6 +92,11 @@ run_case "canceled build exits 1" \
run_case "skipped build exits 0" \
0 "was skipped" STUB_WATCH_HANGS=0 STUB_STATES="skipped"
# A build that takes a while must keep producing output, or the runner's
# inactivity timeout kills the step before COPR finishes.
CASE_TIMEOUT=6 run_case "slow build emits a heartbeat while waiting" \
0 "build 12345: running" STUB_WATCH_HANGS=1 STUB_STATES="running running running succeeded"
# Never settles: bounded, and reported as a timeout rather than a build failure.
CASE_TIMEOUT=3 run_case "build that never settles times out" \
1 "gave up after" STUB_WATCH_HANGS=1 STUB_STATES="running"