CI: replace inherited workflows with a Gitea fat-binary build and deploy-on-main #8

Closed
opened 2026-09-03 09:22:42 +00:00 by grenade · 4 comments
Owner

Part of #1. Outcome: every push to main here produces one binary that carries native code for every GPU architecture in the fleet, and lands it on each mining host, validated, with rollback. lair/quantus stops deploying the miner (lair/quantus#4) so exactly one workflow owns the deployed miners.

Revised 2026-09-03: an earlier version of this issue proposed one build per SM and an sm column per host. That is the right shape for tensor-core inference kernels, whose instruction sets differ per generation and whose libraries take too long to compile as fat binaries. It is the wrong shape here: the mining kernel is one file of 64-bit integer ALU work that Ampere, Ada and consumer Blackwell run with the same instructions, and compiling it three times takes seconds. One fat binary, same file on every host.

Fleet state (measured 2026-09-03 over ssh, not assumed)

Host GPU Compute cap Driver Driver CUDA ceiling Power limit (min / max) OS / kernel Local nvcc
beast 2x RTX 5090 12.0 580.178.04 13.0 400 W (400 / 600) both cards Fedora 43 WS, 7.1.10 13.0, 13.1
benjy 1x RTX 4090 8.9 580.173.02 13.0 250 W (150 / 600) Fedora 43 Server, 7.1.8 13.1, 13.2
quadbrat 1x RTX 3060 8.6 580.159.03 13.0 130 W (100 / 187) Fedora 43 WS, 7.0.8 13.0, 13

Consequences:

  • Build with the CUDA 13.0 toolkit, exactly. Every driver reports 13.0 as its ceiling. Cubins from a 13.1 or 13.2 toolkit would load under minor-version compatibility, but PTX from them would not JIT on a 13.0 driver, and we want the PTX fallback to actually work. The cuda-13.0 runner has the right toolkit; the newer toolkits sitting on benjy and beast are irrelevant to CI and must not be what a local build accidentally uses (scripts/local-cuda-build.sh pins CUDA_VERSIONS=(13.0) for this reason).
  • quadbrat is a 3060, not a 3090. Same sm_86 either way; noted so nobody expects 3090 numbers from it.
  • Drivers differ by point release across hosts. Irrelevant to cubins (the same SASS runs on all three) and one more reason to prefer cubins over PTX JIT: with JIT, the machine code would depend on each host's ptxas, and a driver bump would move hashrate with no commit to blame (#9).
  • All three cards are at or near their minimum power limit by policy (lair/quantus nvidia-power-limit.service). Any hashrate number in this repo's history has to carry the limit it was measured at.

What we inherited and do not need

.github/workflows/ from origin: ci.yml (GitHub-hosted ubuntu-latest, macOS and Windows targets), release-proposal.yml / release-publish.yml (GitHub Releases with four OS targets), dependency-cooldown*.yml. None of it runs here usefully, and worse, Gitea Actions also reads .github/workflows/, so these would sit queued forever waiting for a runner labelled ubuntu-latest on every push.

Delete the directory. On every origin merge the standing resolution for the resulting delete/modify conflicts is git rm -r .github/workflows; that is item 1 of the merge checklist in #1. Keep cuda-builder.cf and scripts/local-cuda-build.sh: they already encode the fleet's arch list and toolkit pin.

What we keep from origin's CI, translated to .gitea/workflows/ci.yml on the rust runner: fmt, taplo, clippy --all-features, test, doc. Those are the parity and lint gates and cost nothing.

Build: one fat binary

nvcc for the kernel in #3 is invoked once with

-gencode arch=compute_86,code=sm_86
-gencode arch=compute_89,code=sm_89
-gencode arch=compute_120,code=sm_120
-gencode arch=compute_120,code=compute_120

Three cubins plus one PTX, embedded in the binary. The driver picks the cubin matching the card at load; an unknown future card JIT-compiles the PTX. The build.rs from the deleted engine-gpu-cuda crate (see the comment on #3) takes a single CUDA_ARCH; extend it to a list, defaulting to the fleet set above. Per-arch tuning (threads per block, nonces per thread, unroll depth) is chosen at runtime from device properties, not at build time.

Until #3 lands, the same job builds the wgpu binary. Pipeline shape, artifact name and host mapping are in place on day one and #3 turns real by enabling one feature.

Watch: the RTX 5090 is compute capability 12.0 (sm_120), not datacenter Blackwell's 10.0 (sm_100). Cubins for the two are not interchangeable. The old build.rs normalises sm_120 correctly.

Workflow: .gitea/workflows/deploy.yaml

Triggers: push to main, and workflow_dispatch with mode: deploy|validate exactly as lair/quantus has today. concurrency: deploy-miner so two pushes cannot interleave restarts on a host.

build job

  • runs-on: cuda-13.0 (the only runner with nvcc; 8 CPU / 32 GiB, pinned to GPU hosts, architecture/gitea-runners.md). No matrix.
  • cargo build -p miner-cli --release --locked --features cuda.
  • Embed the commit: a build.rs in miner-cli that exports MINER_BUILD_SHA, MINER_CUDA_ARCHS (the gencode list), MINER_KERNEL_SET into --version output and into a miner_build_info gauge (#9 depends on this).
  • Upload artifact quantus-miner.
  • sccache against the fleet's S3 cache per the runner doc (RUSTC_WRAPPER=sccache and the SCCACHE_* secrets); a wgpu plus CUDA release build on every push is otherwise slow enough to hurt.

deploy job

  • runs-on: fedora-43, needs: build, fail-fast: false.
  • Host matrix moves here from lair/quantus, unchanged in shape:
- host: beast.hanzalova.internal
  node: bob.hanzalova.internal
  gpu_devices: "2"        # 2x RTX 5090, sm_120
- host: benjy.hanzalova.internal
  node: bob.hanzalova.internal
  gpu_devices: "1"        # 1x RTX 4090, sm_89
- host: quadbrat.hanzalova.internal
  node: bob.hanzalova.internal
  gpu_devices: "1"        # 1x RTX 3060, sm_86
  • Every host receives the same artifact.
  • The deploy body is lair/quantus's deploy miner step moved verbatim: sysusers, credential copy from the node host, miner.env render with the node IP resolved on the miner host, unit, restorecon, metrics firewalld rich rule, checksum-gated restart via rsync -ic. Every trap documented in lair/quantus's CLAUDE.md (pipefail SIGPIPE, ssh quoting, no-op deploys, netlink) still applies and that file's relevant sections come across with the code.
  • Assets move to deploy/ in this repo: quantus-miner.service, quantus-miner.sysusers.conf, quantus-miner-metrics.xml, miner.env.tmpl. Origin's examples/systemd/ stays untouched as origin's.
  • Rollback. Before pushing the new binary, copy the current one to /usr/local/bin/quantus-miner.prev on the host. If validate fails, restore it, restart, and fail the job loudly. Deploying from main means a bad commit reaches production; this is what makes that survivable. Needs two more sudoers lines.
  • Sudoers preflight stays, but reads the expected grants from this repo's deploy/infra-setup.sh (miner role only, moved from lair/quantus), so deployer and grant list cannot drift. Operators re-run it from here.

validate step

As today (unit active, GPU enumerated and drawing, miner_hashes_total advancing) plus:

  • quantus-miner --version on the host contains the deployed commit SHA, not a semver pin.
  • miner_build_info{commit} on the host's metrics matches the deployed commit.
  • The loaded kernel image is a cubin, not the PTX fallback. The engine logs which image it selected (the old crate's runtime did exactly this); export it as the kernel label on the per-device metric from #2 and assert kernel="cuda-sm_NN" with NN matching the card's compute capability. A 5090 silently running JIT-compiled PTX would otherwise pass every other check.
  • On a multi-card host, per-device hash counters (from #2) are all advancing, not just the aggregate.

Prerequisites

  • Drivers: all three hosts already support 13.0 (table above). Re-check this table if a host's driver is updated; the fleet has no policy pinning it.
  • The cuda-13.0 runner is host-pinned and scarce. One build per push is fine.
  • RSYNC_SSH_KEY secret must exist on this repo. It is the shared fleet runner key; copy the secret, never regenerate the key.

Cut-over

Runs in validate mode alongside lair/quantus for at least one cycle, then flips to deploy the same day lair/quantus's deploy-miner job is disabled (lair/quantus#4). Never both in deploy mode: lair/quantus pins a release version and this deploys main, so each run would flip the binary and restart the miner.

Origin coupling

.gitea/ and deploy/ are ours. miner-cli/build.rs is a new file. The .github/workflows deletion is the one recurring merge conflict and is mechanical.

Part of #1. Outcome: every push to `main` here produces one binary that carries native code for every GPU architecture in the fleet, and lands it on each mining host, validated, with rollback. lair/quantus stops deploying the miner (lair/quantus#4) so exactly one workflow owns the deployed miners. **Revised 2026-09-03:** an earlier version of this issue proposed one build per SM and an `sm` column per host. That is the right shape for tensor-core inference kernels, whose instruction sets differ per generation and whose libraries take too long to compile as fat binaries. It is the wrong shape here: the mining kernel is one file of 64-bit integer ALU work that Ampere, Ada and consumer Blackwell run with the same instructions, and compiling it three times takes seconds. One fat binary, same file on every host. ## Fleet state (measured 2026-09-03 over ssh, not assumed) | Host | GPU | Compute cap | Driver | Driver CUDA ceiling | Power limit (min / max) | OS / kernel | Local nvcc | | --- | --- | --- | --- | --- | --- | --- | --- | | beast | 2x RTX 5090 | 12.0 | 580.178.04 | 13.0 | 400 W (400 / 600) both cards | Fedora 43 WS, 7.1.10 | 13.0, 13.1 | | benjy | 1x RTX 4090 | 8.9 | 580.173.02 | 13.0 | 250 W (150 / 600) | Fedora 43 Server, 7.1.8 | 13.1, 13.2 | | quadbrat | 1x RTX 3060 | 8.6 | 580.159.03 | 13.0 | 130 W (100 / 187) | Fedora 43 WS, 7.0.8 | 13.0, 13 | Consequences: - **Build with the CUDA 13.0 toolkit, exactly.** Every driver reports 13.0 as its ceiling. Cubins from a 13.1 or 13.2 toolkit would load under minor-version compatibility, but PTX from them would not JIT on a 13.0 driver, and we want the PTX fallback to actually work. The `cuda-13.0` runner has the right toolkit; the newer toolkits sitting on benjy and beast are irrelevant to CI and must not be what a local build accidentally uses (`scripts/local-cuda-build.sh` pins `CUDA_VERSIONS=(13.0)` for this reason). - quadbrat is a **3060**, not a 3090. Same `sm_86` either way; noted so nobody expects 3090 numbers from it. - Drivers differ by point release across hosts. Irrelevant to cubins (the same SASS runs on all three) and one more reason to prefer cubins over PTX JIT: with JIT, the machine code would depend on each host's ptxas, and a driver bump would move hashrate with no commit to blame (#9). - All three cards are at or near their minimum power limit by policy (lair/quantus `nvidia-power-limit.service`). Any hashrate number in this repo's history has to carry the limit it was measured at. ## What we inherited and do not need `.github/workflows/` from origin: `ci.yml` (GitHub-hosted `ubuntu-latest`, macOS and Windows targets), `release-proposal.yml` / `release-publish.yml` (GitHub Releases with four OS targets), `dependency-cooldown*.yml`. None of it runs here usefully, and worse, **Gitea Actions also reads `.github/workflows/`**, so these would sit queued forever waiting for a runner labelled `ubuntu-latest` on every push. Delete the directory. On every origin merge the standing resolution for the resulting delete/modify conflicts is `git rm -r .github/workflows`; that is item 1 of the merge checklist in #1. Keep `cuda-builder.cf` and `scripts/local-cuda-build.sh`: they already encode the fleet's arch list and toolkit pin. What we keep from origin's CI, translated to `.gitea/workflows/ci.yml` on the `rust` runner: fmt, taplo, clippy `--all-features`, test, doc. Those are the parity and lint gates and cost nothing. ## Build: one fat binary `nvcc` for the kernel in #3 is invoked once with ``` -gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_120,code=sm_120 -gencode arch=compute_120,code=compute_120 ``` Three cubins plus one PTX, embedded in the binary. The driver picks the cubin matching the card at load; an unknown future card JIT-compiles the PTX. The `build.rs` from the deleted `engine-gpu-cuda` crate (see the comment on #3) takes a single `CUDA_ARCH`; extend it to a list, defaulting to the fleet set above. Per-arch tuning (threads per block, nonces per thread, unroll depth) is chosen at runtime from device properties, not at build time. Until #3 lands, the same job builds the wgpu binary. Pipeline shape, artifact name and host mapping are in place on day one and #3 turns real by enabling one feature. Watch: the RTX 5090 is compute capability **12.0** (`sm_120`), not datacenter Blackwell's 10.0 (`sm_100`). Cubins for the two are not interchangeable. The old `build.rs` normalises `sm_120` correctly. ## Workflow: `.gitea/workflows/deploy.yaml` Triggers: push to `main`, and `workflow_dispatch` with `mode: deploy|validate` exactly as lair/quantus has today. `concurrency: deploy-miner` so two pushes cannot interleave restarts on a host. ### `build` job - `runs-on: cuda-13.0` (the only runner with nvcc; 8 CPU / 32 GiB, pinned to GPU hosts, `architecture/gitea-runners.md`). No matrix. - `cargo build -p miner-cli --release --locked --features cuda`. - Embed the commit: a `build.rs` in `miner-cli` that exports `MINER_BUILD_SHA`, `MINER_CUDA_ARCHS` (the gencode list), `MINER_KERNEL_SET` into `--version` output and into a `miner_build_info` gauge (#9 depends on this). - Upload artifact `quantus-miner`. - `sccache` against the fleet's S3 cache per the runner doc (`RUSTC_WRAPPER=sccache` and the `SCCACHE_*` secrets); a wgpu plus CUDA release build on every push is otherwise slow enough to hurt. ### `deploy` job - `runs-on: fedora-43`, `needs: build`, `fail-fast: false`. - Host matrix moves here from lair/quantus, unchanged in shape: ```yaml - host: beast.hanzalova.internal node: bob.hanzalova.internal gpu_devices: "2" # 2x RTX 5090, sm_120 - host: benjy.hanzalova.internal node: bob.hanzalova.internal gpu_devices: "1" # 1x RTX 4090, sm_89 - host: quadbrat.hanzalova.internal node: bob.hanzalova.internal gpu_devices: "1" # 1x RTX 3060, sm_86 ``` - Every host receives the same artifact. - The deploy body is lair/quantus's `deploy miner` step moved verbatim: sysusers, credential copy from the node host, `miner.env` render with the node IP resolved on the miner host, unit, `restorecon`, metrics firewalld rich rule, checksum-gated restart via `rsync -ic`. Every trap documented in lair/quantus's `CLAUDE.md` (pipefail SIGPIPE, ssh quoting, no-op deploys, netlink) still applies and that file's relevant sections come across with the code. - Assets move to `deploy/` in this repo: `quantus-miner.service`, `quantus-miner.sysusers.conf`, `quantus-miner-metrics.xml`, `miner.env.tmpl`. Origin's `examples/systemd/` stays untouched as origin's. - **Rollback.** Before pushing the new binary, copy the current one to `/usr/local/bin/quantus-miner.prev` on the host. If validate fails, restore it, restart, and fail the job loudly. Deploying from `main` means a bad commit reaches production; this is what makes that survivable. Needs two more sudoers lines. - Sudoers preflight stays, but reads the expected grants from **this** repo's `deploy/infra-setup.sh` (miner role only, moved from lair/quantus), so deployer and grant list cannot drift. Operators re-run it from here. ### `validate` step As today (unit active, GPU enumerated and drawing, `miner_hashes_total` advancing) plus: - `quantus-miner --version` on the host contains the deployed commit SHA, not a semver pin. - `miner_build_info{commit}` on the host's metrics matches the deployed commit. - **The loaded kernel image is a cubin, not the PTX fallback.** The engine logs which image it selected (the old crate's runtime did exactly this); export it as the `kernel` label on the per-device metric from #2 and assert `kernel="cuda-sm_NN"` with NN matching the card's compute capability. A 5090 silently running JIT-compiled PTX would otherwise pass every other check. - On a multi-card host, per-device hash counters (from #2) are all advancing, not just the aggregate. ## Prerequisites - Drivers: all three hosts already support 13.0 (table above). Re-check this table if a host's driver is updated; the fleet has no policy pinning it. - The `cuda-13.0` runner is host-pinned and scarce. One build per push is fine. - `RSYNC_SSH_KEY` secret must exist on this repo. It is the shared fleet runner key; copy the secret, never regenerate the key. ## Cut-over Runs in `validate` mode alongside lair/quantus for at least one cycle, then flips to deploy the same day lair/quantus's `deploy-miner` job is disabled (lair/quantus#4). Never both in deploy mode: lair/quantus pins a release version and this deploys `main`, so each run would flip the binary and restart the miner. ## Origin coupling `.gitea/` and `deploy/` are ours. `miner-cli/build.rs` is a new file. The `.github/workflows` deletion is the one recurring merge conflict and is mechanical.
grenade added the origin-coupling label 2026-09-03 09:24:04 +00:00
grenade changed title from CI: replace inherited workflows with Gitea build-per-CUDA-arch and deploy-on-main to CI: replace inherited workflows with a Gitea fat-binary build and deploy-on-main 2026-09-03 09:31:45 +00:00
Author
Owner

Policy update (see #1): until the mainnet call, beast does not mine. The deploy matrix ships to benjy and quadbrat only; beast's entry stays in the file commented out with gpu_devices: "2" so enabling it is a one-line change on the day. SCRAPE_MINERS in lair/quantus follows the same list.

Prerequisites now met: RSYNC_SSH_KEY set on this repo, origin-main pushed at cb6deb9. Remaining unverified before the first run: that Actions is enabled on this repo, and the fleet's sccache secret names (optional).

Policy update (see #1): until the mainnet call, beast does not mine. The deploy matrix ships to **benjy and quadbrat only**; beast's entry stays in the file commented out with `gpu_devices: "2"` so enabling it is a one-line change on the day. `SCRAPE_MINERS` in lair/quantus follows the same list. Prerequisites now met: `RSYNC_SSH_KEY` set on this repo, `origin-main` pushed at `cb6deb9`. Remaining unverified before the first run: that Actions is enabled on this repo, and the fleet's sccache secret names (optional).
Author
Owner

Cut-over done (2026-09-03)

The miner on benjy and quadbrat now deploys from this repo's main (#12, #13). lair/quantus#6 retires the release-pinned deploy-miner there.

Runs, in order:

run mode outcome
actions/runs/13 validate (dispatch from the PR branch) green on both hosts against the lair/quantus-deployed v4.0.2
actions/runs/15 deploy (merge of #12) binary replaced and restarted correctly; validate sampled metrics before the miner had connected, failed, and the rollback then failed with Text file busy because cp writes in place. Net: the new binary stayed, both hosts kept mining.
actions/runs/19 deploy (merge of #13) green on both hosts: build 69 s, restart to connected in 3 s, validate confirmed miner_gpu_devices and an advancing miner_hashes_total, benjy job 1.5 min

Fixes from run 15, both in #13: validate waits for the miner to connect before judging (up to 90 s), and rollback restores with install (unlinks first) instead of cp. The install grant is in deploy/infra-setup.sh and applied on both hosts. The rollback path has now been executed for real once; its restore step has not, because the only failure so far was the validate bug, not a bad binary.

Still open from this issue: the CUDA fat-binary build (--features cuda, -gencode list) lands with #3; sccache is not wired (secret names unknown); beast stays commented out per #1.

Observed while doing this: Gitea's concurrency and cancel-in-progress are not honoured on this instance, and runner capacity is small enough that four lint jobs per push delay a deploy by a couple of minutes. Not a blocker; noted for #2's harness scheduling.

## Cut-over done (2026-09-03) The miner on benjy and quadbrat now deploys from this repo's `main` (#12, #13). lair/quantus#6 retires the release-pinned `deploy-miner` there. Runs, in order: | run | mode | outcome | | --- | --- | --- | | actions/runs/13 | validate (dispatch from the PR branch) | green on both hosts against the lair/quantus-deployed v4.0.2 | | actions/runs/15 | deploy (merge of #12) | binary replaced and restarted correctly; validate sampled metrics before the miner had connected, failed, and the rollback then failed with `Text file busy` because `cp` writes in place. Net: the new binary stayed, both hosts kept mining. | | actions/runs/19 | deploy (merge of #13) | green on both hosts: build 69 s, restart to connected in 3 s, validate confirmed `miner_gpu_devices` and an advancing `miner_hashes_total`, benjy job 1.5 min | Fixes from run 15, both in #13: validate waits for the miner to connect before judging (up to 90 s), and rollback restores with `install` (unlinks first) instead of `cp`. The `install` grant is in `deploy/infra-setup.sh` and applied on both hosts. The rollback path has now been executed for real once; its restore step has not, because the only failure so far was the validate bug, not a bad binary. Still open from this issue: the CUDA fat-binary build (`--features cuda`, `-gencode` list) lands with #3; `sccache` is not wired (secret names unknown); beast stays commented out per #1. Observed while doing this: Gitea's `concurrency` and `cancel-in-progress` are not honoured on this instance, and runner capacity is small enough that four lint jobs per push delay a deploy by a couple of minutes. Not a blocker; noted for #2's harness scheduling.
Author
Owner

Status: the cut-over is done and the CUDA fat binary now ships from the cuda-13.0 runner on every push (#16). Left open for two things only:

  1. sccache against the fleet S3 cache: not wired because the secret names are unknown. Builds are 70 s for the miner, so this is convenience, not a blocker.
  2. The beast matrix row, commented out per the operating policy in #1. Uncommenting it is the whole change when the mainnet call is made; validate already expects kernel: cuda and gpu_devices: "2".
Status: the cut-over is done and the CUDA fat binary now ships from the `cuda-13.0` runner on every push (#16). Left open for two things only: 1. `sccache` against the fleet S3 cache: not wired because the secret names are unknown. Builds are 70 s for the miner, so this is convenience, not a blocker. 2. The beast matrix row, commented out per the operating policy in #1. Uncommenting it is the whole change when the mainnet call is made; validate already expects `kernel: cuda` and `gpu_devices: "2"`.
Author
Owner

Closing: everything this issue asked for is in place and has run in anger.

  • Fat binary (sm_86, sm_89, sm_120 + compute_120 PTX) built with the CUDA 13.0 toolkit on the cuda-13.0 runner on every push to main, deployed with validate and rollback (#12, #13, #16); the validate step asserts the commit, the cuda kernel label and advancing per-device counters.
  • beast's matrix row was enabled for mainnet in #20 (2026-09-09); all three hosts have deployed from main since, most recently run 65 on 8b01387.
  • A tag-triggered release workflow now publishes the same binary publicly (#28, v4.0.2-lair.1).

The one item never done, sccache against the fleet cache, stays undone on purpose: the miner builds in about 70 s and the bench harness in 30 s incremental, so there is nothing to recover. If runner time ever becomes the bottleneck, open a fresh issue for it.

Closing: everything this issue asked for is in place and has run in anger. - Fat binary (sm_86, sm_89, sm_120 + compute_120 PTX) built with the CUDA 13.0 toolkit on the `cuda-13.0` runner on every push to `main`, deployed with validate and rollback (#12, #13, #16); the validate step asserts the commit, the `cuda` kernel label and advancing per-device counters. - beast's matrix row was enabled for mainnet in #20 (2026-09-09); all three hosts have deployed from `main` since, most recently run 65 on 8b01387. - A tag-triggered release workflow now publishes the same binary publicly (#28, `v4.0.2-lair.1`). The one item never done, `sccache` against the fleet cache, stays undone on purpose: the miner builds in about 70 s and the bench harness in 30 s incremental, so there is nothing to recover. If runner time ever becomes the bottleneck, open a fresh issue for it.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#8