fix: stop waiting past the runner's step limit and calling it a failure

The verdict logic was right; the waiting was impossible. Gitea's runner kills
a step at ~15 minutes. monsoon runs 56 and 60 stopped producing output at
14m29s and 14m30s after step start — within one second of each other, with
perfectly regular 31s heartbeats right up to the cut — while their COPR builds
carried on and succeeded at 10:15:20 and 11:34:55. The job was then marked
failed by a reaper ticking at :05:50/:20:50/:35:50/:50:50.

So the step was never going to survive a ~25 minute build, and every fix so
far addressed a real defect that was not this one.

Default the wait budget to 720s, comfortably inside the limit, and change what
expiry means: a build still running is reported with its URL and exits 0,
because it has not failed. A build that reaches failed or canceled inside the
budget still fails the job, which catches the early failures that make up most
build breakage. Skip the artifact download entirely when the build has not
finished, since there is nothing to fetch and no time to spend.

The trade is explicit: a build that fails after the budget will not be caught
by CI. That is strictly better than the previous behaviour of failing every
successful build and blocking dependent jobs.

CLAUDE.md

@@ -0,0 +1,115 @@
+# CLAUDE.md
+
+Context for future agent work in this repo. The README covers consumer-facing
+usage; this file covers things that aren't obvious from reading the code.
+
+## Where this action runs
+
+Hosted on a self-hosted Gitea at `git.lair.cafe` (remote name `origin`,
+SSH url `gitea@git.internal:actions/copr-publish.git`). Consumers reference
+it by fully-qualified URL (`uses: https://git.lair.cafe/actions/copr-publish@v1`)
+because Gitea's `DEFAULT_ACTIONS_URL` points at github.com.
+
+There is **no CI on this repo itself**. The action is tested by running it
+from a consumer repo (primarily `helexa/cortex`). When debugging, fetch job
+logs via the `gitea-mcp` tools against the consumer repo, not this one —
+e.g. `mcp__gitea-mcp__actions_run_read` with `owner=helexa`, `repo=cortex`.
+
+`tests/test-copr-build.sh` covers the parts that don't need a real COPR: it
+puts a stub `copr-cli` on `PATH` and drives `scripts/copr-build.sh` through
+succeeded / failed / canceled / skipped / never-settles, plus the case where
+`watch-build` hangs. Run it directly (`./tests/test-copr-build.sh`); it needs
+nothing but bash and takes a few seconds.
+
+## Why the verdict does not come from `watch-build`
+
+`copr-cli watch-build` holds a long-lived connection and can stop responding
+while the build carries on and finishes normally. Observed in `monsoon` run 48:
+COPR build 10835047 succeeded at 08:30:30, the watcher went silent after
+08:09:11, and the runner killed the step at 08:35:50 — reported as a failed
+build, which also blocked the dependent version-bump job.
+
+So the script polls `copr-cli status` for the authoritative state and treats
+`watch-build` purely as a progress stream whose exit code is ignored. If you
+are tempted to simplify this back into `if copr-cli watch-build; then`, don't.
+
+## Two rules learned the hard way
+
+**Bound every `copr-cli` call.** They make network requests with no internal
+timeout and will block forever. This bit us twice in different places —
+`watch-build` in run 48, then `status` in run 56, where the poll loop sat
+inside a command substitution for twelve minutes and the job was killed five
+minutes after COPR had reported success. Any new `copr-cli` invocation gets a
+`timeout`.
+
+**The runner kills a step at ~15 minutes.** Measured, not guessed: monsoon runs
+56 and 60 stopped producing output at 14m29s and 14m30s after step start, with
+their COPR builds still running and succeeding minutes later. The job is then
+marked failed by a reaper that ticks at :05:50/:20:50/:35:50/:50:50. A monsoon
+build takes ~25 minutes, so **waiting for it to finish is not possible here**.
+The wait budget defaults to 720s and a build still running at expiry exits 0
+with a note rather than a failure. Do not raise the budget past ~13 minutes
+without first confirming the runner limit has changed.
+
+**Never leave the step silent.** A long build produces no COPR output between
+state transitions, and a silent step is killed by the runner's inactivity
+timeout. The poll loop prints a heartbeat every interval; that is load-bearing,
+not decoration.
+
+## Consumers should pin an immutable tag
+
+The runner caches actions by ref, so moving the floating `v1` does **not**
+invalidate it — monsoon ran three releases against a stale cached copy after
+v1 had been moved to the fix. `v1` is still maintained for convenience, but
+consumers that need a specific fix must pin `vX.Y.Z`.
+
+## Tagging & release workflow
+
+We use a **floating major tag** alongside specific semver tags:
+
+- `v1.0.0`, `v1.0.1`, ... — immutable, annotated, per release.
+- `v1` — floating annotated tag, moved forward to the latest `v1.x.y` on
+  every v1 release. This is what `@v1` consumers resolve to.
+
+When cutting a patch/minor release within v1:
+
+```bash
+git tag -a v1.0.N -m "v1.0.N\n\n<summary>" <sha>
+git tag -a v1 -f -m "v1 (floating): latest v1.x release" <sha>
+git push origin main v1.0.N
+git push origin v1 --force     # floating tag move requires --force
+```
+
+The `--force` on the `v1` push is expected and authorized — that's how the
+floating tag works. Do **not** force-push `main` or immutable `vX.Y.Z` tags.
+
+A `v2` is reserved for the live-streaming behaviour change tracked in
+issue #1 (stdout timing differs enough that consumers should opt in). Do
+not quietly land that on `v1`.
+
+## The COPR builder-live.log gotcha
+
+The on-mirror file served by `copr-cli download-build` is
+`builder-live.log.gz` for completed builds — not plain `builder-live.log`.
+The script in `scripts/copr-build.sh` handles both, preferring `.gz` with
+`zcat`. If you add log handling for other COPR artifacts (`build.log`,
+`root.log`, `backend.log`), assume they are gzipped too.
+
+The **HTTP live endpoint** at
+`https://download.copr.fedorainfracloud.org/results/<owner>/<project>/<chroot>/<build_id>-<pkg>/builder-live.log`
+serves plaintext during and after the build — that's the path issue #1's
+live-streaming approach would use, sidestepping the `.gz`-on-disk issue.
+
+## Testing a change
+
+There is no local harness. To verify a change end-to-end:
+
+1. Commit + push + tag as above (or push a branch and reference it by
+   commit SHA from the consumer).
+2. Trigger a workflow in the consumer repo (e.g. push to `helexa/cortex`).
+3. Inspect the job log via `gitea-mcp` — note that job logs come back
+   base64-ish wrapped in a JSON envelope and with `\r`/`\n` escapes; pipe
+   through `jq -r '.[].text'` then `sed 's/\\r/\n/g; s/\\n/\n/g'` to get
+   something greppable.
+
+Syntax-only check locally: `bash -n scripts/copr-build.sh`.

README.md

@@ -13,10 +13,22 @@ This action:
 
 - Submits with `--nowait` and captures the build ID.
 - Prints a clickable `https://copr.fedorainfracloud.org/coprs/build/...` link so you can follow live.
-- Watches the build to completion (blocks, propagates exit status).
+- Streams status transitions from `copr-cli watch-build` while polling COPR for
+  the build's actual state, so the job's verdict comes from the build rather
+  than from the health of a long-lived connection.
 - On completion, fetches each chroot's `builder-live.log` via
   `copr-cli download-build` and emits them as `::group::` blocks.
-- Fails CI if the build fails, but always dumps logs first.
+- Fails CI if the build **fails**. A build merely still running when the wait
+  budget expires is reported, not failed.
+- Returns within `COPR_WAIT_BUDGET` whatever happens, so the step finishes
+  before a CI runner's step limit kills it.
+
+### Environment overrides
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `COPR_WAIT_BUDGET` | `720` | Seconds to wait for the build before returning. A build still running at expiry is reported as such and the step exits 0 — CI runners commonly kill a step long before a large build finishes. |
+| `COPR_POLL_INTERVAL` | `30` | Seconds between build-state polls. |
 
 ## Requirements
 

scripts/copr-build.sh

@@ -9,6 +9,23 @@
 
 set -o pipefail
 
+# How long we are willing to wait for the build before handing control back to
+# CI, and how often to ask COPR where it has got to.
+#
+# Deliberately well under the runner's step limit rather than the length of a
+# build. Gitea's runner kills a step at ~15 minutes: two monsoon releases had
+# their step killed at 14m29s and 14m30s with the build still running, and the
+# job then reported a successful build as a failure. Waiting longer is simply
+# not available to us, so past this budget the build is reported as still
+# running and COPR is left to finish it.
+WAIT_BUDGET="${COPR_WAIT_BUDGET:-${COPR_BUILD_TIMEOUT:-720}}"
+POLL_INTERVAL="${COPR_POLL_INTERVAL:-30}"
+# Every copr-cli invocation is bounded. They perform network calls with no
+# internal timeout, and a hang in any of them stalls the whole job.
+STATUS_TIMEOUT="${COPR_STATUS_TIMEOUT:-60}"
+SUBMIT_TIMEOUT="${COPR_SUBMIT_TIMEOUT:-600}"
+DOWNLOAD_TIMEOUT="${COPR_DOWNLOAD_TIMEOUT:-600}"
+
 PROJECT="$1"
 shift
 
@@ -17,8 +34,59 @@ if [ -z "$PROJECT" ] || [ "$#" -eq 0 ]; then
   exit 2
 fi
 
+# COPR states that mean the build has finished, one way or another. Anything
+# else (pending, starting, running, importing, waiting) is still in progress.
+is_terminal_state() {
+  case "$1" in
+    succeeded | failed | canceled | skipped) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+# Ask COPR for a build's state. Empty output means "could not tell", which the
+# caller treats as non-terminal and retries.
+#
+# Bounded, because copr-cli's network calls can block indefinitely. One release
+# hung here for over twelve minutes: the poll loop sat inside this command
+# substitution, heartbeats stopped, and the job was killed five minutes after
+# COPR had already reported success.
+build_state() {
+  timeout "$STATUS_TIMEOUT" copr-cli status "$1" 2>/dev/null | tail -n1 | tr -d '[:space:]'
+}
+
+# Poll COPR until the build settles, or until BUILD_TIMEOUT elapses. Echoes the
+# terminal state, or "unknown" if we ran out of patience.
+# Poll until the build settles or the budget runs out. Echoes the last state
+# seen, which the caller checks for terminality — a non-terminal state means we
+# ran out of patience, not that anything went wrong.
+wait_for_terminal_state() {
+  local build_id="$1"
+  local started deadline state now
+  started=$(date +%s)
+  deadline=$((started + WAIT_BUDGET))
+  state="$(build_state "$build_id")"
+  while ! is_terminal_state "$state"; do
+    now=$(date +%s)
+    if [ "$now" -ge "$deadline" ]; then
+      echo "${state:-unknown}"
+      return
+    fi
+    # Heartbeat on stderr (stdout carries the return value). A COPR build emits
+    # nothing between state transitions, so a long build leaves the step silent
+    # for its whole duration — and a silent step gets killed by the runner's
+    # inactivity timeout before it can finish. Two builds died this way after
+    # ~24-27 minutes of no output, one of them 66 seconds before COPR reported
+    # success. Printing every poll keeps the step alive and shows progress.
+    printf '  [%3dm %3ds] build %s: %s\n' \
+      $(((now - started) / 60)) $(((now - started) % 60)) "$build_id" "${state:-unknown}" >&2
+    sleep "$POLL_INTERVAL"
+    state="$(build_state "$build_id")"
+  done
+  echo "$state"
+}
+
 # Submit without waiting; capture the build ID from stdout.
-SUBMIT_OUT=$(copr-cli build --nowait "$PROJECT" "$@")
+SUBMIT_OUT=$(timeout "$SUBMIT_TIMEOUT" copr-cli build --nowait "$PROJECT" "$@")
 echo "$SUBMIT_OUT"
 BUILD_ID=$(echo "$SUBMIT_OUT" | grep -oP 'Created builds: \K[0-9]+' | head -n1)
 
@@ -32,18 +100,62 @@ echo "Build $BUILD_ID submitted to $PROJECT"
 echo "Follow live: https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID"
 echo
 
-# Watch the build; captures status transitions to stdout. Exit non-zero
-# on build failure, but defer propagating that until after we've fetched
-# logs so the CI output contains diagnostics either way.
-if copr-cli watch-build "$BUILD_ID"; then
-  STATUS=0
-else
-  STATUS=$?
-fi
+# Stream status transitions for the operator's benefit, in the background and
+# bounded so it cannot hang the job.
+#
+# Its exit status is deliberately ignored, and it is NOT what decides the
+# verdict. `watch-build` holds a long-lived connection to COPR, and it has been
+# seen to stop responding while the build carries on and finishes normally: in
+# one case the build succeeded at 08:30:30 while the watcher sat silent until
+# the runner killed the step at 08:35:50, which the job then reported as a
+# build failure. Conflating "the watcher lost its connection" with "the build
+# failed" turns a green release into a red one and blocks any dependent job.
+timeout "$BUILD_TIMEOUT" copr-cli watch-build "$BUILD_ID" &
+WATCH_PID=$!
+
+# The verdict comes from COPR itself. Polling also means we stop as soon as the
+# build settles, rather than waiting on the watcher to notice.
+BUILD_STATE="$(wait_for_terminal_state "$BUILD_ID")"
+
+kill "$WATCH_PID" 2>/dev/null || true
+wait "$WATCH_PID" 2>/dev/null || true
+
+echo
+echo "COPR build $BUILD_ID final state: $BUILD_STATE"
+
+case "$BUILD_STATE" in
+  succeeded)
+    STATUS=0
+    ;;
+  skipped)
+    # COPR already had this exact build; nothing was rebuilt, but nothing is
+    # wrong either.
+    echo "note: build $BUILD_ID was skipped (already built)"
+    STATUS=0
+    ;;
+  failed | canceled)
+    echo "error: COPR build $BUILD_ID finished with state '$BUILD_STATE'" >&2
+    STATUS=1
+    ;;
+  *)
+    # Still going, or we could not tell. Either way the build has not failed,
+    # and reporting it as a failure is what made every green release red.
+    echo "note: build $BUILD_ID has not finished within our ${WAIT_BUDGET}s wait budget"
+    echo "      (last state: ${BUILD_STATE:-unknown}). COPR will finish it on its own."
+    echo "      Follow: https://copr.fedorainfracloud.org/coprs/build/$BUILD_ID"
+    STATUS=0
+    ;;
+esac
 
 # Fetch per-chroot results (logs + rpms). Anonymous download — no auth needed.
+# Only meaningful once the build has finished; there is nothing to fetch for one
+# that is still running, and trying would burn the little time we have left.
+if ! is_terminal_state "$BUILD_STATE"; then
+  exit "$STATUS"
+fi
+
 LOG_DIR="$(mktemp -d -t copr-logs.XXXXXX)"
-copr-cli download-build --dest "$LOG_DIR" "$BUILD_ID" || {
+timeout "$DOWNLOAD_TIMEOUT" copr-cli download-build --dest "$LOG_DIR" "$BUILD_ID" || {
   echo "warning: failed to download build artifacts" >&2
 }
 

tests/test-copr-build.sh

@@ -0,0 +1,125 @@
+#!/bin/bash
+# Tests for scripts/copr-build.sh, driven by a stub `copr-cli` on PATH.
+#
+# The case that matters most is a watcher that hangs while the build succeeds
+# anyway: that is what made a green COPR release report as a failed CI job.
+#
+# Usage: tests/test-copr-build.sh
+
+set -o pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+SCRIPT="$HERE/../scripts/copr-build.sh"
+PASS=0
+FAIL=0
+
+# Build a stub copr-cli whose behaviour is driven by env vars, and put it first
+# on PATH. STUB_STATES is a space-separated list of states returned by
+# successive `status` calls; the last one repeats.
+make_stub() {
+  local dir="$1"
+  mkdir -p "$dir"
+  cat > "$dir/copr-cli" <<'STUB'
+#!/bin/bash
+case "$1" in
+  build)
+    echo "Uploading package test.src.rpm"
+    echo "Created builds: 12345"
+    ;;
+  watch-build)
+    if [ "${STUB_WATCH_HANGS:-0}" = "1" ]; then
+      # Mimic a watcher that streams a little then stops responding.
+      echo "  Build 12345: running"
+      sleep 3600
+    else
+      echo "  Build 12345: succeeded"
+    fi
+    ;;
+  status)
+    # Optionally hang, reproducing copr-cli blocking on a stalled network call.
+    if [ -n "${STUB_STATUS_HANGS_UNTIL:-}" ]; then
+      hidx_file="${STUB_IDX_FILE:-/tmp/stub_idx}.hang"
+      hidx=$(cat "$hidx_file" 2>/dev/null || echo 0)
+      echo $(( hidx + 1 )) > "$hidx_file"
+      if [ "$hidx" -lt "$STUB_STATUS_HANGS_UNTIL" ]; then sleep 3600; fi
+    fi
+    # Pop the next state; the final one repeats forever.
+    read -r -a states <<< "${STUB_STATES:-succeeded}"
+    idx_file="${STUB_IDX_FILE:-/tmp/stub_idx}"
+    idx=$(cat "$idx_file" 2>/dev/null || echo 0)
+    if [ "$idx" -ge "${#states[@]}" ]; then idx=$(( ${#states[@]} - 1 )); fi
+    echo "${states[$idx]}"
+    echo $(( idx + 1 )) > "$idx_file"
+    ;;
+  download-build)
+    exit 0
+    ;;
+esac
+STUB
+  chmod +x "$dir/copr-cli"
+}
+
+run_case() {
+  local name="$1" expected_rc="$2" expected_text="$3"
+  shift 3
+  local tmp out rc
+  tmp="$(mktemp -d)"
+  make_stub "$tmp/bin"
+  out="$(env "$@" STUB_IDX_FILE="$tmp/idx" PATH="$tmp/bin:$PATH" \
+    COPR_POLL_INTERVAL=1 COPR_BUILD_TIMEOUT="${CASE_TIMEOUT:-20}" COPR_STATUS_TIMEOUT="${COPR_STATUS_TIMEOUT:-10}" \
+    bash "$SCRIPT" owner/project test.src.rpm 2>&1)"
+  rc=$?
+  if [ "$rc" -eq "$expected_rc" ] && grep -q "$expected_text" <<< "$out"; then
+    echo "ok   - $name"
+    PASS=$((PASS + 1))
+  else
+    echo "FAIL - $name (rc=$rc want $expected_rc; wanted text: $expected_text)"
+    sed 's/^/       | /' <<< "$out"
+    FAIL=$((FAIL + 1))
+  fi
+  rm -rf "$tmp"
+}
+
+# The regression: watcher stops responding, build succeeded anyway.
+run_case "hung watcher + succeeded build exits 0" \
+  0 "final state: succeeded" STUB_WATCH_HANGS=1 STUB_STATES="running succeeded"
+
+# A genuine failure must still fail, hung watcher or not.
+run_case "hung watcher + failed build exits 1" \
+  1 "finished with state 'failed'" STUB_WATCH_HANGS=1 STUB_STATES="running failed"
+
+# Normal path.
+run_case "healthy watcher + succeeded build exits 0" \
+  0 "final state: succeeded" STUB_WATCH_HANGS=0 STUB_STATES="succeeded"
+
+run_case "canceled build exits 1" \
+  1 "finished with state 'canceled'" STUB_WATCH_HANGS=0 STUB_STATES="canceled"
+
+# Already-built is not an error.
+run_case "skipped build exits 0" \
+  0 "was skipped" STUB_WATCH_HANGS=0 STUB_STATES="skipped"
+
+# A build that takes a while must keep producing output, or the runner's
+# inactivity timeout kills the step before COPR finishes.
+CASE_TIMEOUT=6 run_case "slow build emits a heartbeat while waiting" \
+  0 "build 12345: running" STUB_WATCH_HANGS=1 STUB_STATES="running running running succeeded"
+
+# The v1.0.3 failure: copr-cli status blocks forever, so the poll loop stalls
+# inside the command substitution and heartbeats stop. Bounding the call lets
+# the loop recover and still reach the right verdict.
+CASE_TIMEOUT=30 run_case "a hung status call does not stall the poll loop" \
+  0 "final state: succeeded" STUB_WATCH_HANGS=1 STUB_STATES="running succeeded" \
+  STUB_STATUS_HANGS_UNTIL=1 COPR_STATUS_TIMEOUT=2
+
+# The step limit case: the build is still running when our wait budget expires.
+# This must NOT be reported as a failure — that is what turned every green
+# release red — and it must not try to fetch logs that do not exist yet.
+CASE_TIMEOUT=3 run_case "still-running build at budget expiry exits 0" \
+  0 "has not finished within" STUB_WATCH_HANGS=1 STUB_STATES="running"
+
+CASE_TIMEOUT=3 run_case "still-running build says so rather than claiming failure" \
+  0 "COPR will finish it" STUB_WATCH_HANGS=1 STUB_STATES="running"
+
+echo
+echo "passed: $PASS  failed: $FAIL"
+[ "$FAIL" -eq 0 ]