CI: replace inherited workflows with a Gitea fat-binary build and deploy-on-main #8
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #1. Outcome: every push to
mainhere produces one binary that carries native code for every GPU architecture in the fleet, and lands it on each mining host, validated, with rollback. lair/quantus stops deploying the miner (lair/quantus#4) so exactly one workflow owns the deployed miners.Revised 2026-09-03: an earlier version of this issue proposed one build per SM and an
smcolumn per host. That is the right shape for tensor-core inference kernels, whose instruction sets differ per generation and whose libraries take too long to compile as fat binaries. It is the wrong shape here: the mining kernel is one file of 64-bit integer ALU work that Ampere, Ada and consumer Blackwell run with the same instructions, and compiling it three times takes seconds. One fat binary, same file on every host.Fleet state (measured 2026-09-03 over ssh, not assumed)
Consequences:
cuda-13.0runner has the right toolkit; the newer toolkits sitting on benjy and beast are irrelevant to CI and must not be what a local build accidentally uses (scripts/local-cuda-build.shpinsCUDA_VERSIONS=(13.0)for this reason).sm_86either way; noted so nobody expects 3090 numbers from it.nvidia-power-limit.service). Any hashrate number in this repo's history has to carry the limit it was measured at.What we inherited and do not need
.github/workflows/from origin:ci.yml(GitHub-hostedubuntu-latest, macOS and Windows targets),release-proposal.yml/release-publish.yml(GitHub Releases with four OS targets),dependency-cooldown*.yml. None of it runs here usefully, and worse, Gitea Actions also reads.github/workflows/, so these would sit queued forever waiting for a runner labelledubuntu-lateston every push.Delete the directory. On every origin merge the standing resolution for the resulting delete/modify conflicts is
git rm -r .github/workflows; that is item 1 of the merge checklist in #1. Keepcuda-builder.cfandscripts/local-cuda-build.sh: they already encode the fleet's arch list and toolkit pin.What we keep from origin's CI, translated to
.gitea/workflows/ci.ymlon therustrunner: fmt, taplo, clippy--all-features, test, doc. Those are the parity and lint gates and cost nothing.Build: one fat binary
nvccfor the kernel in #3 is invoked once withThree cubins plus one PTX, embedded in the binary. The driver picks the cubin matching the card at load; an unknown future card JIT-compiles the PTX. The
build.rsfrom the deletedengine-gpu-cudacrate (see the comment on #3) takes a singleCUDA_ARCH; extend it to a list, defaulting to the fleet set above. Per-arch tuning (threads per block, nonces per thread, unroll depth) is chosen at runtime from device properties, not at build time.Until #3 lands, the same job builds the wgpu binary. Pipeline shape, artifact name and host mapping are in place on day one and #3 turns real by enabling one feature.
Watch: the RTX 5090 is compute capability 12.0 (
sm_120), not datacenter Blackwell's 10.0 (sm_100). Cubins for the two are not interchangeable. The oldbuild.rsnormalisessm_120correctly.Workflow:
.gitea/workflows/deploy.yamlTriggers: push to
main, andworkflow_dispatchwithmode: deploy|validateexactly as lair/quantus has today.concurrency: deploy-minerso two pushes cannot interleave restarts on a host.buildjobruns-on: cuda-13.0(the only runner with nvcc; 8 CPU / 32 GiB, pinned to GPU hosts,architecture/gitea-runners.md). No matrix.cargo build -p miner-cli --release --locked --features cuda.build.rsinminer-clithat exportsMINER_BUILD_SHA,MINER_CUDA_ARCHS(the gencode list),MINER_KERNEL_SETinto--versionoutput and into aminer_build_infogauge (#9 depends on this).quantus-miner.sccacheagainst the fleet's S3 cache per the runner doc (RUSTC_WRAPPER=sccacheand theSCCACHE_*secrets); a wgpu plus CUDA release build on every push is otherwise slow enough to hurt.deployjobruns-on: fedora-43,needs: build,fail-fast: false.deploy minerstep moved verbatim: sysusers, credential copy from the node host,miner.envrender with the node IP resolved on the miner host, unit,restorecon, metrics firewalld rich rule, checksum-gated restart viarsync -ic. Every trap documented in lair/quantus'sCLAUDE.md(pipefail SIGPIPE, ssh quoting, no-op deploys, netlink) still applies and that file's relevant sections come across with the code.deploy/in this repo:quantus-miner.service,quantus-miner.sysusers.conf,quantus-miner-metrics.xml,miner.env.tmpl. Origin'sexamples/systemd/stays untouched as origin's./usr/local/bin/quantus-miner.prevon the host. If validate fails, restore it, restart, and fail the job loudly. Deploying frommainmeans a bad commit reaches production; this is what makes that survivable. Needs two more sudoers lines.deploy/infra-setup.sh(miner role only, moved from lair/quantus), so deployer and grant list cannot drift. Operators re-run it from here.validatestepAs today (unit active, GPU enumerated and drawing,
miner_hashes_totaladvancing) plus:quantus-miner --versionon the host contains the deployed commit SHA, not a semver pin.miner_build_info{commit}on the host's metrics matches the deployed commit.kernellabel on the per-device metric from #2 and assertkernel="cuda-sm_NN"with NN matching the card's compute capability. A 5090 silently running JIT-compiled PTX would otherwise pass every other check.Prerequisites
cuda-13.0runner is host-pinned and scarce. One build per push is fine.RSYNC_SSH_KEYsecret must exist on this repo. It is the shared fleet runner key; copy the secret, never regenerate the key.Cut-over
Runs in
validatemode alongside lair/quantus for at least one cycle, then flips to deploy the same day lair/quantus'sdeploy-minerjob is disabled (lair/quantus#4). Never both in deploy mode: lair/quantus pins a release version and this deploysmain, so each run would flip the binary and restart the miner.Origin coupling
.gitea/anddeploy/are ours.miner-cli/build.rsis a new file. The.github/workflowsdeletion is the one recurring merge conflict and is mechanical.CI: replace inherited workflows with Gitea build-per-CUDA-arch and deploy-on-mainto CI: replace inherited workflows with a Gitea fat-binary build and deploy-on-mainPolicy update (see #1): until the mainnet call, beast does not mine. The deploy matrix ships to benjy and quadbrat only; beast's entry stays in the file commented out with
gpu_devices: "2"so enabling it is a one-line change on the day.SCRAPE_MINERSin lair/quantus follows the same list.Prerequisites now met:
RSYNC_SSH_KEYset on this repo,origin-mainpushed atcb6deb9. Remaining unverified before the first run: that Actions is enabled on this repo, and the fleet's sccache secret names (optional).Cut-over done (2026-09-03)
The miner on benjy and quadbrat now deploys from this repo's
main(#12, #13). lair/quantus#6 retires the release-pinneddeploy-minerthere.Runs, in order:
Text file busybecausecpwrites in place. Net: the new binary stayed, both hosts kept mining.miner_gpu_devicesand an advancingminer_hashes_total, benjy job 1.5 minFixes from run 15, both in #13: validate waits for the miner to connect before judging (up to 90 s), and rollback restores with
install(unlinks first) instead ofcp. Theinstallgrant is indeploy/infra-setup.shand applied on both hosts. The rollback path has now been executed for real once; its restore step has not, because the only failure so far was the validate bug, not a bad binary.Still open from this issue: the CUDA fat-binary build (
--features cuda,-gencodelist) lands with #3;sccacheis not wired (secret names unknown); beast stays commented out per #1.Observed while doing this: Gitea's
concurrencyandcancel-in-progressare not honoured on this instance, and runner capacity is small enough that four lint jobs per push delay a deploy by a couple of minutes. Not a blocker; noted for #2's harness scheduling.Status: the cut-over is done and the CUDA fat binary now ships from the
cuda-13.0runner on every push (#16). Left open for two things only:sccacheagainst the fleet S3 cache: not wired because the secret names are unknown. Builds are 70 s for the miner, so this is convenience, not a blocker.kernel: cudaandgpu_devices: "2".Closing: everything this issue asked for is in place and has run in anger.
cuda-13.0runner on every push tomain, deployed with validate and rollback (#12, #13, #16); the validate step asserts the commit, thecudakernel label and advancing per-device counters.mainsince, most recently run 65 on8b01387.v4.0.2-lair.1).The one item never done,
sccacheagainst the fleet cache, stays undone on purpose: the miner builds in about 70 s and the bench harness in 30 s incremental, so there is nothing to recover. If runner time ever becomes the bottleneck, open a fresh issue for it.