Two things.
A build-level "succeeded" says the build finished, not that every chroot
produced an RPM. The job could go green while fedora-43, fedora-44 or rawhide
had failed or was still building. Poll build-chroot/list alongside the build
state, keep waiting while any chroot is unfinished, and fail if any finished in
anything other than succeeded or skipped. Per-chroot results are printed at the
end and included in the heartbeat, so a slow chroot is visible while it runs.
copr-cli has no per-chroot status subcommand, so this reads COPR's public API
directly with python3 — already a hard dependency of copr-cli, so nothing new
is required. COPR_API_BASE overrides the instance.
Separately, renaming BUILD_TIMEOUT to WAIT_BUDGET in v1.0.5 missed the
watch-build invocation, leaving `timeout "" copr-cli watch-build`. That failed
immediately, so the progress stream has not actually run since v1.0.5 —
harmless only because the verdict comes from polling, and caught here because
the new tests surfaced the error text.
The unfinished-versus-failed distinction is deliberate and load-bearing:
reporting a still-running chroot as a failure would be the same mistake this
action has now made in three different guises.
v1.0.5 cut the wait budget to 720s because gongfoo's output-stall reaper
sampled the runner container's log, which never carries step output — any step
over ~15 minutes was killed however much it printed, and successful builds were
reported as failures.
gongfoo#4 is fixed and deployed: the reaper now derives progress from the Gitea
job log, which does see our heartbeat. So the short budget is now the wrong
trade — it returns success while the build is still running, which is how
monsoon run 66 reported green on a build that had another ten minutes to go.
Raise the budget to 2700s. That is comfortably above a ~25 minute Rust release
build and under gongfoo's default 3600s absolute wall-clock cap, so we always
return a verdict of our own rather than being reaped mid-wait.
The heartbeat is now load-bearing rather than diagnostic: it is the signal
keeping a long job alive. Noted in CLAUDE.md so it does not get tidied away.
The verdict logic was right; the waiting was impossible. Gitea's runner kills
a step at ~15 minutes. monsoon runs 56 and 60 stopped producing output at
14m29s and 14m30s after step start — within one second of each other, with
perfectly regular 31s heartbeats right up to the cut — while their COPR builds
carried on and succeeded at 10:15:20 and 11:34:55. The job was then marked
failed by a reaper ticking at :05:50/:20:50/:35:50/:50:50.
So the step was never going to survive a ~25 minute build, and every fix so
far addressed a real defect that was not this one.
Default the wait budget to 720s, comfortably inside the limit, and change what
expiry means: a build still running is reported with its URL and exits 0,
because it has not failed. A build that reaches failed or canceled inside the
budget still fails the job, which catches the early failures that make up most
build breakage. Skip the artifact download entirely when the build has not
finished, since there is nothing to fetch and no time to spend.
The trade is explicit: a build that fails after the budget will not be caught
by CI. That is strictly better than the previous behaviour of failing every
successful build and blocking dependent jobs.
v1.0.3's heartbeat did its job: it showed exactly where the next failure was.
In monsoon run 56 the heartbeats stopped at 14m29s while the build was still
running, and the job was killed at 10:20:50 — five minutes after COPR had
reported success at 10:15:20.
The poll loop was blocked inside `$(build_state ...)`. copr-cli makes network
requests with no internal timeout, so `copr-cli status` hung for over twelve
minutes and took the loop with it. Same failure mode as watch-build in run 48,
just relocated.
Bound all of them: status (60s), submit (600s), download-build (600s), each
overridable. A timed-out status yields empty output, which reads as
non-terminal, so the loop simply heartbeats and retries on the next tick.
Test reproduces it with a stub whose first status call hangs; it deadlocks
against v1.0.3 and passes here.
v1.0.2 moved the verdict off watch-build's exit code, but left the step
silent for the whole build: COPR emits nothing between state transitions,
and the poller only spoke at the end.
That silence is itself the failure. Two monsoon releases died after ~24-27
minutes of no output at all — run 48 at 08:35:50 with COPR having succeeded
at 08:30:30, and run 50 at 09:35:50, 66 seconds *before* COPR reported
success at 09:36:56. A build finishing either side of the kill rules out a
build-duration timeout and points at an inactivity timeout on the runner.
Print a line on every poll with elapsed time and current state. It keeps the
step producing output for as long as the build runs, and makes the log show
progress instead of a 25-minute gap. Heartbeats go to stderr because stdout
carries wait_for_terminal_state's return value.
Test covers a build that stays running across several polls before
succeeding, asserting the heartbeat appears.
`copr-cli watch-build` holds a long-lived connection to COPR and can stop
responding while the build carries on and finishes normally. The script
treated its exit status as the build's outcome, so a watcher that lost its
connection was reported as a failed build.
Seen in monsoon run 48: COPR build 10835047 succeeded at 08:30:30, the
watcher went silent after 08:09:11, and the runner killed the step at
08:35:50. The job was marked failed even though the RPM built and published
fine, and that false failure also blocked the dependent version-bump job
from running at all.
Keep watch-build for its progress stream, but move it to the background,
bound it with a timeout, and ignore its exit code. Poll `copr-cli status`
for the authoritative state and derive the exit code from that. Polling also
means the script returns as soon as the build settles instead of waiting on
the watcher to notice.
succeeded and skipped (already built) pass; failed and canceled fail; and
never reaching a terminal state within COPR_BUILD_TIMEOUT is reported as a
timeout rather than silently as a build failure.
Adds tests/test-copr-build.sh, which drives the script against a stub
copr-cli. The regression case — hung watcher, successful build — fails
against the previous script by hanging until killed, which is the production
symptom.
Capture the non-obvious details of working in this repo: Gitea hosting,
the floating-v1-tag release workflow, the builder-live.log.gz gotcha,
and how to fetch consumer-job logs for verification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
copr-cli download-build fetches builder-live.log.gz (the on-mirror form
for completed builds), but the dump loop looked for plain builder-live.log
and silently emitted nothing. Try .gz first with zcat, fall back to plain,
and note the absence explicitly if neither is present.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wraps copr-cli submit/watch/download-build with per-chroot log
dumping as collapsible ::group:: blocks. Moves the logic previously
living at .gitea/scripts/copr-build.sh in helexa/cortex into a
reusable action so every consumer gets live log visibility and
consistent failure diagnostics.
Inputs: project, srpm (glob OK), copr-config (secret).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
