fix: stop waiting past the runner's step limit and calling it a failure

The verdict logic was right; the waiting was impossible. Gitea's runner kills
a step at ~15 minutes. monsoon runs 56 and 60 stopped producing output at
14m29s and 14m30s after step start — within one second of each other, with
perfectly regular 31s heartbeats right up to the cut — while their COPR builds
carried on and succeeded at 10:15:20 and 11:34:55. The job was then marked
failed by a reaper ticking at :05:50/:20:50/:35:50/:50:50.

So the step was never going to survive a ~25 minute build, and every fix so
far addressed a real defect that was not this one.

Default the wait budget to 720s, comfortably inside the limit, and change what
expiry means: a build still running is reported with its URL and exits 0,
because it has not failed. A build that reaches failed or canceled inside the
budget still fails the job, which catches the early failures that make up most
build breakage. Skip the artifact download entirely when the build has not
finished, since there is nothing to fetch and no time to spend.

The trade is explicit: a build that fails after the budget will not be caught
by CI. That is strictly better than the previous behaviour of failing every
successful build and blocking dependent jobs.
This commit is contained in:
2026-08-07 14:48:50 +03:00
parent d690d5aaa8
commit 95973e731b
4 changed files with 49 additions and 15 deletions

View File

@@ -42,6 +42,15 @@ inside a command substitution for twelve minutes and the job was killed five
minutes after COPR had reported success. Any new `copr-cli` invocation gets a
`timeout`.
**The runner kills a step at ~15 minutes.** Measured, not guessed: monsoon runs
56 and 60 stopped producing output at 14m29s and 14m30s after step start, with
their COPR builds still running and succeeding minutes later. The job is then
marked failed by a reaper that ticks at :05:50/:20:50/:35:50/:50:50. A monsoon
build takes ~25 minutes, so **waiting for it to finish is not possible here**.
The wait budget defaults to 720s and a build still running at expiry exits 0
with a note rather than a failure. Do not raise the budget past ~13 minutes
without first confirming the runner limit has changed.
**Never leave the step silent.** A long build produces no COPR output between
state transitions, and a silent step is killed by the runner's inactivity
timeout. The poll loop prints a heartbeat every interval; that is load-bearing,

View File

@@ -18,15 +18,16 @@ This action:
than from the health of a long-lived connection.
- On completion, fetches each chroot's `builder-live.log` via
`copr-cli download-build` and emits them as `::group::` blocks.
- Fails CI if the build fails, but always dumps logs first.
- Gives up after `COPR_BUILD_TIMEOUT` (default 7200s) and says so, rather than
hanging until the runner kills the job.
- Fails CI if the build **fails**. A build merely still running when the wait
budget expires is reported, not failed.
- Returns within `COPR_WAIT_BUDGET` whatever happens, so the step finishes
before a CI runner's step limit kills it.
### Environment overrides
| Variable | Default | Purpose |
|---|---|---|
| `COPR_BUILD_TIMEOUT` | `7200` | Seconds to wait for a build to reach a terminal state. |
| `COPR_WAIT_BUDGET` | `720` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
| `COPR_POLL_INTERVAL` | `30` | Seconds between build-state polls. |
## Requirements

View File

@@ -9,9 +9,16 @@
set -o pipefail
# How long to wait for a build to reach a terminal state before giving up, and
# how often to ask COPR where it has got to.
BUILD_TIMEOUT="${COPR_BUILD_TIMEOUT:-7200}"
# How long we are willing to wait for the build before handing control back to
# CI, and how often to ask COPR where it has got to.
#
# Deliberately well under the runner's step limit rather than the length of a
# build. Gitea's runner kills a step at ~15 minutes: two monsoon releases had
# their step killed at 14m29s and 14m30s with the build still running, and the
# job then reported a successful build as a failure. Waiting longer is simply
# not available to us, so past this budget the build is reported as still
# running and COPR is left to finish it.
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-720}}"
POLL_INTERVAL="${COPR_POLL_INTERVAL:-30}"
# Every copr-cli invocation is bounded. They perform network calls with no
# internal timeout, and a hang in any of them stalls the whole job.
@@ -49,16 +56,19 @@ build_state() {
# Poll COPR until the build settles, or until BUILD_TIMEOUT elapses. Echoes the
# terminal state, or "unknown" if we ran out of patience.
# Poll until the build settles or the budget runs out. Echoes the last state
# seen, which the caller checks for terminality — a non-terminal state means we
# ran out of patience, not that anything went wrong.
wait_for_terminal_state() {
local build_id="$1"
local started deadline state now
started=$(date +%s)
deadline=$((started + BUILD_TIMEOUT))
deadline=$((started + WAIT_BUDGET))
state="$(build_state "$build_id")"
while ! is_terminal_state "$state"; do
now=$(date +%s)
if [ "$now" -ge "$deadline" ]; then
echo "unknown"
echo "${state:-unknown}"
return
fi
# Heartbeat on stderr (stdout carries the return value). A COPR build emits
@@ -128,13 +138,22 @@ case "$BUILD_STATE" in
STATUS=1
;;
*)
echo "error: gave up after ${BUILD_TIMEOUT}s waiting for COPR build $BUILD_ID to finish" >&2
echo " check https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID" >&2
STATUS=1
# Still going, or we could not tell. Either way the build has not failed,
# and reporting it as a failure is what made every green release red.
echo "note: build $BUILD_ID has not finished within our ${WAIT_BUDGET}s wait budget"
echo " (last state: ${BUILD_STATE:-unknown}). COPR will finish it on its own."
echo " Follow: https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID"
STATUS=0
;;
esac
# Fetch per-chroot results (logs + rpms). Anonymous download — no auth needed.
# Only meaningful once the build has finished; there is nothing to fetch for one
# that is still running, and trying would burn the little time we have left.
if ! is_terminal_state "$BUILD_STATE"; then
exit "$STATUS"
fi
LOG_DIR="$(mktemp -d -t copr-logs.XXXXXX)"
timeout "$DOWNLOAD_TIMEOUT" copr-cli download-build --dest "$LOG_DIR" "$BUILD_ID" || {
echo "warning: failed to download build artifacts" >&2

View File

@@ -111,9 +111,14 @@ CASE_TIMEOUT=30 run_case "a hung status call does not stall the poll loop" \
0 "final state: succeeded" STUB_WATCH_HANGS=1 STUB_STATES="running succeeded" \
STUB_STATUS_HANGS_UNTIL=1 COPR_STATUS_TIMEOUT=2
# Never settles: bounded, and reported as a timeout rather than a build failure.
CASE_TIMEOUT=3 run_case "build that never settles times out" \
1 "gave up after" STUB_WATCH_HANGS=1 STUB_STATES="running"
# The step limit case: the build is still running when our wait budget expires.
# This must NOT be reported as a failure — that is what turned every green
# release red — and it must not try to fetch logs that do not exist yet.
CASE_TIMEOUT=3 run_case "still-running build at budget expiry exits 0" \
0 "has not finished within" STUB_WATCH_HANGS=1 STUB_STATES="running"
CASE_TIMEOUT=3 run_case "still-running build says so rather than claiming failure" \
0 "COPR will finish it" STUB_WATCH_HANGS=1 STUB_STATES="running"
echo
echo "passed: $PASS failed: $FAIL"