fix: wait for the build again now the runner reaper reads the job log

v1.0.5 cut the wait budget to 720s because gongfoo's output-stall reaper
sampled the runner container's log, which never carries step output — any step
over ~15 minutes was killed however much it printed, and successful builds were
reported as failures.

gongfoo#4 is fixed and deployed: the reaper now derives progress from the Gitea
job log, which does see our heartbeat. So the short budget is now the wrong
trade — it returns success while the build is still running, which is how
monsoon run 66 reported green on a build that had another ten minutes to go.

Raise the budget to 2700s. That is comfortably above a ~25 minute Rust release
build and under gongfoo's default 3600s absolute wall-clock cap, so we always
return a verdict of our own rather than being reaped mid-wait.

The heartbeat is now load-bearing rather than diagnostic: it is the signal
keeping a long job alive. Noted in CLAUDE.md so it does not get tidied away.
This commit is contained in:
2026-08-07 16:37:08 +03:00
parent 95973e731b
commit 8fa9802d58
3 changed files with 23 additions and 16 deletions

View File

@@ -42,14 +42,17 @@ inside a command substitution for twelve minutes and the job was killed five
minutes after COPR had reported success. Any new `copr-cli` invocation gets a
`timeout`.
**The runner kills a step at ~15 minutes.** Measured, not guessed: monsoon runs
56 and 60 stopped producing output at 14m29s and 14m30s after step start, with
their COPR builds still running and succeeding minutes later. The job is then
marked failed by a reaper that ticks at :05:50/:20:50/:35:50/:50:50. A monsoon
build takes ~25 minutes, so **waiting for it to finish is not possible here**.
The wait budget defaults to 720s and a build still running at expiry exits 0
with a note rather than a failure. Do not raise the budget past ~13 minutes
without first confirming the runner limit has changed.
**The heartbeat is the runner's aliveness signal — do not remove it.** gongfoo's
output-stall reaper used to sample the runner container's log, which never
carries step output, so any step over ~15 minutes was killed however much it
printed (gongfoo#4; monsoon runs 56 and 60 were cut at 14m29s and 14m30s with
their builds succeeding minutes later). That reaper now derives progress from
the Gitea job log, so the per-poll heartbeat is what keeps a long wait alive.
The wait budget is 2700s: comfortably above a ~25 minute build, and under
gongfoo's default 3600s absolute wall-clock cap so we always return a verdict
of our own instead of being reaped mid-wait. Raising it past ~50 minutes means
racing that cap.
**Never leave the step silent.** A long build produces no COPR output between
state transitions, and a silent step is killed by the runner's inactivity

View File

@@ -27,7 +27,7 @@ This action:
| Variable | Default | Purpose |
|---|---|---|
| `COPR_WAIT_BUDGET` | `720` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
| `COPR_WAIT_BUDGET` | `2700` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
| `COPR_POLL_INTERVAL` | `30` | Seconds between build-state polls. |
## Requirements

View File

@@ -12,13 +12,17 @@ set -o pipefail
# How long we are willing to wait for the build before handing control back to
# CI, and how often to ask COPR where it has got to.
#
# Deliberately well under the runner's step limit rather than the length of a
# build. Gitea's runner kills a step at ~15 minutes: two monsoon releases had
# their step killed at 14m29s and 14m30s with the build still running, and the
# job then reported a successful build as a failure. Waiting longer is simply
# not available to us, so past this budget the build is reported as still
# running and COPR is left to finish it.
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-720}}"
# Long enough to actually see a build through — a Rust release build runs ~25
# minutes — but under the runner's absolute wall-clock cap so we always exit
# with a verdict of our own rather than being killed mid-wait.
#
# This was briefly 720s, when a runner-side reaper sampled the container log
# for signs of life. Step output never reaches that log, so any step lasting
# over ~15 minutes was killed however much it printed, and successful builds
# were reported as failures. That reaper now derives progress from the Gitea
# job log, which does see our heartbeat, so waiting properly is possible again.
# The heartbeat below is what keeps the job alive: do not remove it.
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-2700}}"
POLL_INTERVAL="${COPR_POLL_INTERVAL:-30}"
# Every copr-cli invocation is bounded. They perform network calls with no
# internal timeout, and a hang in any of them stalls the whole job.