fix: wait for the build again now the runner reaper reads the job log
v1.0.5 cut the wait budget to 720s because gongfoo's output-stall reaper sampled the runner container's log, which never carries step output — any step over ~15 minutes was killed however much it printed, and successful builds were reported as failures. gongfoo#4 is fixed and deployed: the reaper now derives progress from the Gitea job log, which does see our heartbeat. So the short budget is now the wrong trade — it returns success while the build is still running, which is how monsoon run 66 reported green on a build that had another ten minutes to go. Raise the budget to 2700s. That is comfortably above a ~25 minute Rust release build and under gongfoo's default 3600s absolute wall-clock cap, so we always return a verdict of our own rather than being reaped mid-wait. The heartbeat is now load-bearing rather than diagnostic: it is the signal keeping a long job alive. Noted in CLAUDE.md so it does not get tidied away.
This commit is contained in:
19
CLAUDE.md
19
CLAUDE.md
@@ -42,14 +42,17 @@ inside a command substitution for twelve minutes and the job was killed five
|
||||
minutes after COPR had reported success. Any new `copr-cli` invocation gets a
|
||||
`timeout`.
|
||||
|
||||
**The runner kills a step at ~15 minutes.** Measured, not guessed: monsoon runs
|
||||
56 and 60 stopped producing output at 14m29s and 14m30s after step start, with
|
||||
their COPR builds still running and succeeding minutes later. The job is then
|
||||
marked failed by a reaper that ticks at :05:50/:20:50/:35:50/:50:50. A monsoon
|
||||
build takes ~25 minutes, so **waiting for it to finish is not possible here**.
|
||||
The wait budget defaults to 720s and a build still running at expiry exits 0
|
||||
with a note rather than a failure. Do not raise the budget past ~13 minutes
|
||||
without first confirming the runner limit has changed.
|
||||
**The heartbeat is the runner's aliveness signal — do not remove it.** gongfoo's
|
||||
output-stall reaper used to sample the runner container's log, which never
|
||||
carries step output, so any step over ~15 minutes was killed however much it
|
||||
printed (gongfoo#4; monsoon runs 56 and 60 were cut at 14m29s and 14m30s with
|
||||
their builds succeeding minutes later). That reaper now derives progress from
|
||||
the Gitea job log, so the per-poll heartbeat is what keeps a long wait alive.
|
||||
|
||||
The wait budget is 2700s: comfortably above a ~25 minute build, and under
|
||||
gongfoo's default 3600s absolute wall-clock cap so we always return a verdict
|
||||
of our own instead of being reaped mid-wait. Raising it past ~50 minutes means
|
||||
racing that cap.
|
||||
|
||||
**Never leave the step silent.** A long build produces no COPR output between
|
||||
state transitions, and a silent step is killed by the runner's inactivity
|
||||
|
||||
@@ -27,7 +27,7 @@ This action:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `COPR_WAIT_BUDGET` | `720` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
|
||||
| `COPR_WAIT_BUDGET` | `2700` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
|
||||
| `COPR_POLL_INTERVAL` | `30` | Seconds between build-state polls. |
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -12,13 +12,17 @@ set -o pipefail
|
||||
# How long we are willing to wait for the build before handing control back to
|
||||
# CI, and how often to ask COPR where it has got to.
|
||||
#
|
||||
# Deliberately well under the runner's step limit rather than the length of a
|
||||
# build. Gitea's runner kills a step at ~15 minutes: two monsoon releases had
|
||||
# their step killed at 14m29s and 14m30s with the build still running, and the
|
||||
# job then reported a successful build as a failure. Waiting longer is simply
|
||||
# not available to us, so past this budget the build is reported as still
|
||||
# running and COPR is left to finish it.
|
||||
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-720}}"
|
||||
# Long enough to actually see a build through — a Rust release build runs ~25
|
||||
# minutes — but under the runner's absolute wall-clock cap so we always exit
|
||||
# with a verdict of our own rather than being killed mid-wait.
|
||||
#
|
||||
# This was briefly 720s, when a runner-side reaper sampled the container log
|
||||
# for signs of life. Step output never reaches that log, so any step lasting
|
||||
# over ~15 minutes was killed however much it printed, and successful builds
|
||||
# were reported as failures. That reaper now derives progress from the Gitea
|
||||
# job log, which does see our heartbeat, so waiting properly is possible again.
|
||||
# The heartbeat below is what keeps the job alive: do not remove it.
|
||||
WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-2700}}"
|
||||
POLL_INTERVAL="${COPR_POLL_INTERVAL:-30}"
|
||||
# Every copr-cli invocation is bounded. They perform network calls with no
|
||||
# internal timeout, and a hang in any of them stalls the whole job.
|
||||
|
||||
Reference in New Issue
Block a user