fix: stop waiting past the runner's step limit and calling it a failure
The verdict logic was right; the waiting was impossible. Gitea's runner kills a step at ~15 minutes. monsoon runs 56 and 60 stopped producing output at 14m29s and 14m30s after step start — within one second of each other, with perfectly regular 31s heartbeats right up to the cut — while their COPR builds carried on and succeeded at 10:15:20 and 11:34:55. The job was then marked failed by a reaper ticking at :05:50/:20:50/:35:50/:50:50. So the step was never going to survive a ~25 minute build, and every fix so far addressed a real defect that was not this one. Default the wait budget to 720s, comfortably inside the limit, and change what expiry means: a build still running is reported with its URL and exits 0, because it has not failed. A build that reaches failed or canceled inside the budget still fails the job, which catches the early failures that make up most build breakage. Skip the artifact download entirely when the build has not finished, since there is nothing to fetch and no time to spend. The trade is explicit: a build that fails after the budget will not be caught by CI. That is strictly better than the previous behaviour of failing every successful build and blocking dependent jobs.
This commit is contained in:
@@ -42,6 +42,15 @@ inside a command substitution for twelve minutes and the job was killed five
|
||||
minutes after COPR had reported success. Any new `copr-cli` invocation gets a
|
||||
`timeout`.
|
||||
|
||||
**The runner kills a step at ~15 minutes.** Measured, not guessed: monsoon runs
|
||||
56 and 60 stopped producing output at 14m29s and 14m30s after step start, with
|
||||
their COPR builds still running and succeeding minutes later. The job is then
|
||||
marked failed by a reaper that ticks at :05:50/:20:50/:35:50/:50:50. A monsoon
|
||||
build takes ~25 minutes, so **waiting for it to finish is not possible here**.
|
||||
The wait budget defaults to 720s and a build still running at expiry exits 0
|
||||
with a note rather than a failure. Do not raise the budget past ~13 minutes
|
||||
without first confirming the runner limit has changed.
|
||||
|
||||
**Never leave the step silent.** A long build produces no COPR output between
|
||||
state transitions, and a silent step is killed by the runner's inactivity
|
||||
timeout. The poll loop prints a heartbeat every interval; that is load-bearing,
|
||||
|
||||
@@ -18,15 +18,16 @@ This action:
|
||||
than from the health of a long-lived connection.
|
||||
- On completion, fetches each chroot's `builder-live.log` via
|
||||
`copr-cli download-build` and emits them as `::group::` blocks.
|
||||
- Fails CI if the build fails, but always dumps logs first.
|
||||
- Gives up after `COPR_BUILD_TIMEOUT` (default 7200s) and says so, rather than
|
||||
hanging until the runner kills the job.
|
||||
- Fails CI if the build **fails**. A build merely still running when the wait
|
||||
budget expires is reported, not failed.
|
||||
- Returns within `COPR_WAIT_BUDGET` whatever happens, so the step finishes
|
||||
before a CI runner's step limit kills it.
|
||||
|
||||
### Environment overrides
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `COPR_BUILD_TIMEOUT` | `7200` | Seconds to wait for a build to reach a terminal state. |
|
||||
| `COPR_WAIT_BUDGET` | `720` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
|
||||
| `COPR_POLL_INTERVAL` | `30` | Seconds between build-state polls. |
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -9,9 +9,16 @@
|
||||
|
||||
set -o pipefail
|
||||
|
||||
# How long to wait for a build to reach a terminal state before giving up, and
|
||||
# how often to ask COPR where it has got to.
|
||||
BUILD_TIMEOUT="${COPR_BUILD_TIMEOUT:-7200}"
|
||||
# How long we are willing to wait for the build before handing control back to
|
||||
# CI, and how often to ask COPR where it has got to.
|
||||
#
|
||||
# Deliberately well under the runner's step limit rather than the length of a
|
||||
# build. Gitea's runner kills a step at ~15 minutes: two monsoon releases had
|
||||
# their step killed at 14m29s and 14m30s with the build still running, and the
|
||||
# job then reported a successful build as a failure. Waiting longer is simply
|
||||
# not available to us, so past this budget the build is reported as still
|
||||
# running and COPR is left to finish it.
|
||||
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-720}}"
|
||||
POLL_INTERVAL="${COPR_POLL_INTERVAL:-30}"
|
||||
# Every copr-cli invocation is bounded. They perform network calls with no
|
||||
# internal timeout, and a hang in any of them stalls the whole job.
|
||||
@@ -49,16 +56,19 @@ build_state() {
|
||||
|
||||
# Poll COPR until the build settles, or until BUILD_TIMEOUT elapses. Echoes the
|
||||
# terminal state, or "unknown" if we ran out of patience.
|
||||
# Poll until the build settles or the budget runs out. Echoes the last state
|
||||
# seen, which the caller checks for terminality — a non-terminal state means we
|
||||
# ran out of patience, not that anything went wrong.
|
||||
wait_for_terminal_state() {
|
||||
local build_id="$1"
|
||||
local started deadline state now
|
||||
started=$(date +%s)
|
||||
deadline=$((started + BUILD_TIMEOUT))
|
||||
deadline=$((started + WAIT_BUDGET))
|
||||
state="$(build_state "$build_id")"
|
||||
while ! is_terminal_state "$state"; do
|
||||
now=$(date +%s)
|
||||
if [ "$now" -ge "$deadline" ]; then
|
||||
echo "unknown"
|
||||
echo "${state:-unknown}"
|
||||
return
|
||||
fi
|
||||
# Heartbeat on stderr (stdout carries the return value). A COPR build emits
|
||||
@@ -128,13 +138,22 @@ case "$BUILD_STATE" in
|
||||
STATUS=1
|
||||
;;
|
||||
*)
|
||||
echo "error: gave up after ${BUILD_TIMEOUT}s waiting for COPR build $BUILD_ID to finish" >&2
|
||||
echo " check https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID" >&2
|
||||
STATUS=1
|
||||
# Still going, or we could not tell. Either way the build has not failed,
|
||||
# and reporting it as a failure is what made every green release red.
|
||||
echo "note: build $BUILD_ID has not finished within our ${WAIT_BUDGET}s wait budget"
|
||||
echo " (last state: ${BUILD_STATE:-unknown}). COPR will finish it on its own."
|
||||
echo " Follow: https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID"
|
||||
STATUS=0
|
||||
;;
|
||||
esac
|
||||
|
||||
# Fetch per-chroot results (logs + rpms). Anonymous download — no auth needed.
|
||||
# Only meaningful once the build has finished; there is nothing to fetch for one
|
||||
# that is still running, and trying would burn the little time we have left.
|
||||
if ! is_terminal_state "$BUILD_STATE"; then
|
||||
exit "$STATUS"
|
||||
fi
|
||||
|
||||
LOG_DIR="$(mktemp -d -t copr-logs.XXXXXX)"
|
||||
timeout "$DOWNLOAD_TIMEOUT" copr-cli download-build --dest "$LOG_DIR" "$BUILD_ID" || {
|
||||
echo "warning: failed to download build artifacts" >&2
|
||||
|
||||
@@ -111,9 +111,14 @@ CASE_TIMEOUT=30 run_case "a hung status call does not stall the poll loop" \
|
||||
0 "final state: succeeded" STUB_WATCH_HANGS=1 STUB_STATES="running succeeded" \
|
||||
STUB_STATUS_HANGS_UNTIL=1 COPR_STATUS_TIMEOUT=2
|
||||
|
||||
# Never settles: bounded, and reported as a timeout rather than a build failure.
|
||||
CASE_TIMEOUT=3 run_case "build that never settles times out" \
|
||||
1 "gave up after" STUB_WATCH_HANGS=1 STUB_STATES="running"
|
||||
# The step limit case: the build is still running when our wait budget expires.
|
||||
# This must NOT be reported as a failure — that is what turned every green
|
||||
# release red — and it must not try to fetch logs that do not exist yet.
|
||||
CASE_TIMEOUT=3 run_case "still-running build at budget expiry exits 0" \
|
||||
0 "has not finished within" STUB_WATCH_HANGS=1 STUB_STATES="running"
|
||||
|
||||
CASE_TIMEOUT=3 run_case "still-running build says so rather than claiming failure" \
|
||||
0 "COPR will finish it" STUB_WATCH_HANGS=1 STUB_STATES="running"
|
||||
|
||||
echo
|
||||
echo "passed: $PASS failed: $FAIL"
|
||||
|
||||
Reference in New Issue
Block a user