v1.0.3's heartbeat did its job: it showed exactly where the next failure was.
In monsoon run 56 the heartbeats stopped at 14m29s while the build was still
running, and the job was killed at 10:20:50 — five minutes after COPR had
reported success at 10:15:20.
The poll loop was blocked inside `$(build_state ...)`. copr-cli makes network
requests with no internal timeout, so `copr-cli status` hung for over twelve
minutes and took the loop with it. Same failure mode as watch-build in run 48,
just relocated.
Bound all of them: status (60s), submit (600s), download-build (600s), each
overridable. A timed-out status yields empty output, which reads as
non-terminal, so the loop simply heartbeats and retries on the next tick.
Test reproduces it with a stub whose first status call hangs; it deadlocks
against v1.0.3 and passes here.
v1.0.2 moved the verdict off watch-build's exit code, but left the step
silent for the whole build: COPR emits nothing between state transitions,
and the poller only spoke at the end.
That silence is itself the failure. Two monsoon releases died after ~24-27
minutes of no output at all — run 48 at 08:35:50 with COPR having succeeded
at 08:30:30, and run 50 at 09:35:50, 66 seconds *before* COPR reported
success at 09:36:56. A build finishing either side of the kill rules out a
build-duration timeout and points at an inactivity timeout on the runner.
Print a line on every poll with elapsed time and current state. It keeps the
step producing output for as long as the build runs, and makes the log show
progress instead of a 25-minute gap. Heartbeats go to stderr because stdout
carries wait_for_terminal_state's return value.
Test covers a build that stays running across several polls before
succeeding, asserting the heartbeat appears.
`copr-cli watch-build` holds a long-lived connection to COPR and can stop
responding while the build carries on and finishes normally. The script
treated its exit status as the build's outcome, so a watcher that lost its
connection was reported as a failed build.
Seen in monsoon run 48: COPR build 10835047 succeeded at 08:30:30, the
watcher went silent after 08:09:11, and the runner killed the step at
08:35:50. The job was marked failed even though the RPM built and published
fine, and that false failure also blocked the dependent version-bump job
from running at all.
Keep watch-build for its progress stream, but move it to the background,
bound it with a timeout, and ignore its exit code. Poll `copr-cli status`
for the authoritative state and derive the exit code from that. Polling also
means the script returns as soon as the build settles instead of waiting on
the watcher to notice.
succeeded and skipped (already built) pass; failed and canceled fail; and
never reaching a terminal state within COPR_BUILD_TIMEOUT is reported as a
timeout rather than silently as a build failure.
Adds tests/test-copr-build.sh, which drives the script against a stub
copr-cli. The regression case — hung watcher, successful build — fails
against the previous script by hanging until killed, which is the production
symptom.
Capture the non-obvious details of working in this repo: Gitea hosting,
the floating-v1-tag release workflow, the builder-live.log.gz gotcha,
and how to fetch consumer-job logs for verification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
copr-cli download-build fetches builder-live.log.gz (the on-mirror form
for completed builds), but the dump loop looked for plain builder-live.log
and silently emitted nothing. Try .gz first with zcat, fall back to plain,
and note the absence explicitly if neither is present.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wraps copr-cli submit/watch/download-build with per-chroot log
dumping as collapsible ::group:: blocks. Moves the logic previously
living at .gitea/scripts/copr-build.sh in helexa/cortex into a
reusable action so every consumer gets live log visibility and
consistent failure diagnostics.
Inputs: project, srpm (glob OK), copr-config (secret).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
