rob thijssen
fcb1f3b033
The neuron CUDA builds SIGSEGV'd rustc in LLVM's SROA pass on candle-core whenever sccache was the wrapper, but compiled fine without it. Root cause: the workflow pre-starts the sccache server in a bare shell (`sccache --start-server`), so per sccache's documented behaviour the orphaned server creates its OWN jobserver with one slot per CPU core (24 on quadbrat), ignoring Cargo's CARGO_BUILD_JOBS=8. rustc then runs up to 24-way parallel LLVM codegen; on candle-core's huge cuda-feature functions the concurrent SROA passes overflow the default 8 MiB codegen-thread stack and kill rustc with SIGSEGV — the exact "set RUST_MIN_STACK" case rustc prints. Cargo's own 8-slot jobserver (the no-sccache path) stays within budget and compiles. Set RUST_MIN_STACK=32 MiB in the standalone Rust images (runner-rust, runner-cuda-13.0, runner-rust-ubuntu); the gtk3/ffmpeg/deb images inherit it. It's rustc's own recommended mitigation, virtual-only until used, and protects every repo regardless of how it drives sccache. Pair with raising the cuda runner image's mem_request_mb so the parallel codegen has headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
92 lines
4.6 KiB
Docker
92 lines
4.6 KiB
Docker
# Fedora 43 base, NOT runner-rust (Fedora 44): CUDA 13.0's nvcc rejects
|
|
# host compilers newer than gcc 15, and Fedora 44 ships gcc 16 with no
|
|
# compat package. Fedora 43 ships gcc 15 and matches the NVIDIA fedora43
|
|
# repo below. The rust toolchain is inlined here (same recipe as
|
|
# images/runner-rust) since we can't inherit it from the F44 chain.
|
|
FROM git.lair.cafe/gongfoo/runner-fedora-43:latest
|
|
|
|
ARG SCCACHE_VERSION=0.15.0
|
|
|
|
# C toolchain + dev libs. Rust via rustup, not dnf — see images/runner-rust
|
|
# for the strict-version-hash / musl rationale.
|
|
RUN dnf install -y --setopt=install_weak_deps=False \
|
|
gcc \
|
|
musl-gcc \
|
|
openssl-devel \
|
|
pkg-config \
|
|
&& dnf clean all
|
|
|
|
ENV RUSTUP_HOME=/usr/local/rustup \
|
|
CARGO_HOME=/usr/local/cargo \
|
|
PATH=/usr/local/cargo/bin:$PATH
|
|
|
|
# Codegen-thread stack headroom — see images/runner-rust/Containerfile. A
|
|
# pre-started sccache server (no Cargo jobserver) fans rustc codegen out to
|
|
# every CPU core; on candle-core (cuda features) the concurrent LLVM SROA
|
|
# passes overflowed the default 8 MiB codegen stack and SIGSEGV'd rustc (the
|
|
# "set RUST_MIN_STACK" case). 32 MiB is virtual-only until used.
|
|
ENV RUST_MIN_STACK=33554432
|
|
|
|
RUN curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \
|
|
| sh -s -- -y --no-modify-path --profile minimal \
|
|
--default-toolchain stable \
|
|
--component clippy,rustfmt \
|
|
--target x86_64-unknown-linux-musl \
|
|
&& chmod -R a+w "${RUSTUP_HOME}" "${CARGO_HOME}" \
|
|
&& printf 'fn main(){}' > /tmp/t.rs \
|
|
&& rustc --target x86_64-unknown-linux-musl /tmp/t.rs -o /tmp/t \
|
|
&& rm -f /tmp/t.rs /tmp/t
|
|
|
|
RUN curl -fsSL \
|
|
"https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
|
|
| tar xz -C /tmp \
|
|
&& install -m 755 /tmp/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl/sccache /usr/local/bin/sccache \
|
|
&& rm -rf /tmp/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl
|
|
|
|
ARG CUDA_REPO_FEDORA=43
|
|
ARG CUDA_REPO_RHEL=9
|
|
|
|
RUN dnf config-manager addrepo \
|
|
--from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora${CUDA_REPO_FEDORA}/x86_64/cuda-fedora${CUDA_REPO_FEDORA}.repo \
|
|
&& dnf config-manager addrepo \
|
|
--from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/rhel${CUDA_REPO_RHEL}/x86_64/cuda-rhel${CUDA_REPO_RHEL}.repo
|
|
|
|
RUN dnf config-manager setopt \
|
|
cuda-fedora${CUDA_REPO_FEDORA}-x86_64.exclude='nvidia-driver*,nvidia-modprobe,nvidia-persistenced,nvidia-settings,nvidia-libXNVCtrl,nvidia-xconfig,nvidia-open,xorg-x11-drv-nvidia*,xorg-x11-nvidia*,kmod-nvidia*,akmod-nvidia*,cuda-drivers*,nvidia-kmod-common,libnvidia-*' \
|
|
&& dnf config-manager setopt \
|
|
cuda-rhel${CUDA_REPO_RHEL}-x86_64.exclude='nvidia-driver*,nvidia-modprobe,nvidia-persistenced,nvidia-settings,nvidia-libXNVCtrl,nvidia-xconfig,nvidia-open,xorg-x11-drv-nvidia*,xorg-x11-nvidia*,kmod-nvidia*,akmod-nvidia*,cuda-drivers*,nvidia-kmod-common,libnvidia-*'
|
|
|
|
ARG CUDA_VERSION=13-0
|
|
RUN dnf install -y --setopt=install_weak_deps=False \
|
|
cuda-nvcc-${CUDA_VERSION} \
|
|
cuda-cudart-devel-${CUDA_VERSION} \
|
|
cuda-nvrtc-devel-${CUDA_VERSION} \
|
|
libcublas-devel-${CUDA_VERSION} \
|
|
libcusparse-devel-${CUDA_VERSION} \
|
|
libcurand-devel-${CUDA_VERSION} \
|
|
libcusolver-devel-${CUDA_VERSION} \
|
|
cuda-driver-devel-${CUDA_VERSION} \
|
|
cuda-profiler-api-${CUDA_VERSION} \
|
|
libcudnn9-devel-cuda-13 \
|
|
libnccl-devel \
|
|
cmake \
|
|
gcc-c++ \
|
|
ninja-build \
|
|
&& dnf clean all
|
|
|
|
# glibc 2.41+ (Fedora 42+) declares C23 rsqrt/rsqrtf in math.h with
|
|
# noexcept(true); CUDA 13.0's math_functions.h declares them without an
|
|
# exception specification and nvcc hard-errors on the mismatch. Align the
|
|
# CUDA declarations with glibc. Drop once NVIDIA ships fixed headers.
|
|
# The grep asserts the patch landed so a header change fails the build
|
|
# loudly instead of resurfacing as nvcc errors in CI jobs.
|
|
RUN sed -i \
|
|
-e 's/\(__device_builtin__ double[[:space:]]*rsqrt(double x)\);/\1 noexcept (true);/' \
|
|
-e 's/\(__device_builtin__ float[[:space:]]*rsqrtf(float x)\);/\1 noexcept (true);/' \
|
|
"/usr/local/cuda-${CUDA_VERSION/-/.}/targets/x86_64-linux/include/crt/math_functions.h" \
|
|
&& [ "$(grep -c 'noexcept (true)' "/usr/local/cuda-${CUDA_VERSION/-/.}/targets/x86_64-linux/include/crt/math_functions.h")" -eq 2 ]
|
|
|
|
ENV CUDA_HOME=/usr/local/cuda-${CUDA_VERSION/-/.}
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/targets/x86_64-linux/lib:${CUDA_HOME}/lib64:${LD_LIBRARY_PATH:-/usr/lib64}"
|