52 Commits

Author	SHA1	Message	Date
rob thijssen	452d7d9b3d	feat(B7): packaging + CI for helexa-upstream ... Ships the mesh authority as an RPM on the same prebuilt-binary pipeline as helexa-bench. - rpm/helexa-upstream-prerelease.spec: wraps the CI-built binary; installs the systemd unit, sysusers, firewalld service (tcp/8090), and /etc/helexa-upstream/helexa-upstream.toml (config noreplace). - data/helexa-upstream.{service,sysusers.conf,firewalld.xml}: Type=simple unit (serve --config), dedicated system user with a StateDirectory home, inbound 8090 (/authz/v1 + /web/v1). PostgreSQL is reached outbound; the schema migrates on startup. - build-prerelease.yml: build-upstream + package-upstream jobs with change-detection over crates/helexa-upstream/ (UPSTREAM_RE), gated into publish. SQLX_OFFLINE=true is set defensively — helexa-upstream uses the sqlx runtime query API (no compile-time macros), so it builds with no database and no .sqlx cache; DB integration tests stay gated behind UPSTREAM_TEST_DATABASE_URL. Validated: workflow YAML parses, rpmspec expands, and `SQLX_OFFLINE=true cargo build --release -p helexa-upstream` succeeds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 12:00:41 +03:00
rob thijssen	6088830e7d	feat(deploy): manage NEURON_MAX_PROMPT_TOKENS per host via model.conf drop-in ... Roll the per-model context cap into deploy.yml so it is deterministic per host and rolled out (with a restart) alongside the rest of the service config, rather than hand-edited in local.conf. The deploy now writes /etc/systemd/system/neuron.service.d/model.conf from a new per-host `max_prompt_tokens` matrix field, and restarts a neuron when the package OR the drop-in changes — so a cap change applies even with no new RPM. beast (Qwen3.6-27B, hybrid linear, 2x 32GB) -> 131072 (~128k); benjy and quadbrat (dense, VRAM-bound) stay at 16384 but become deploy-managed. Adds the scoped sudoers grant for the root-owned drop-in install, and doc/context-limits.md documenting the knob relationships and KV/VRAM math (refs #62 for the eventual /models-advertised source of truth, #65 for the length-aware text VRAM guard that gates pushing beyond 128k). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-16 18:48:19 +03:00
rob thijssen	b3dc835375	ci: bound job runtime + stop dropping sccache on rustc signal-death ... A neuron-blackwell build hung ~90 min (siblings finished in 2) and there was no job timeout to kill it, so it sat burning a runner. Root cause of the hang: the inline retry loop treated every failure identically and, on its final attempt, rebuilt with sccache disabled. When the real failure is a rustc SIGSEGV or an OOM-kill, an uncached rebuild does *more* work under the same memory pressure — turning one transient compiler crash into a wedged job. Two fixes: 1. timeout-minutes on every job in build-prerelease.yml and ci.yml (builds 25, neuron CUDA build/cuda-check 35, packaging 20, COPR 60, fast jobs 10-15). A hang now dies in minutes, not hours. 2. New script/ci-cargo-escalate.sh replaces the five (prerelease) + three (ci) inline escalation loops. It classifies the failure: - signal death (exit >=128, or cargo reporting `signal: N`/SIGSEGV/ SIGKILL) → compiler crash, NOT an sccache fault: keep the cache, one warm retry, then fail fast. Never escalate to uncached. - sccache fault (recognisable sccache error) → restart the server, retry, then one final uncached attempt. - deterministic compile/test error → fail fast (no wasteful retry). It also folds in the CUDA-image sccache probe the neuron/cuda-check jobs did inline. Classification verified locally against success, plain failure, exit-139, and the cargo-wrapped `signal: 11` form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 13:02:50 +03:00
rob thijssen	37c19aa985	feat(bench-ui): public hosting at https://bench.helexa.ai via gateway nginx ... nginx on the gateway serves the bench SPA and reverse-proxies /api to the bob bench API over WireGuard — public, auth-less, same-origin (no CORS), internal API stays private. - asset/nginx/bench.helexa.ai.conf (full TLS vhost: SPA + /api proxy) and a bootstrap http-only vhost for the initial ACME challenge. - infra-setup.sh: one-time gateway setup — webroot, Let's Encrypt cert (certbot webroot, idempotent), install + enable the vhost. - deploy.yml: deploy-bench-ui builds the SPA (setup-node) and rsyncs dist/ to /var/www/bench.helexa.ai every deploy; built same-origin so no VITE_API_BASE. - cortex-host.conf: scoped gitea_ci rsync grant for the webroot. - bench/README: production hosting notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:40:29 +03:00
rob thijssen	f50f5531cf	feat(bench): read-only JSON API on bob + bench/ React visualisation app ... Part A — helexa-bench read API: - [api] config (enabled, listen :13132); WAL on the store so API reads never block the sweep writer. - store read methods: summary, series (chronological per-build medians), runs (filtered), dimensions, run_count. - api.rs: axum /api/health|dimensions|summary|series|runs, permissive CORS (UI is a separate origin). The `run` daemon binds the API alongside the sweep; new `serve` subcommand serves API-only. - listener plumbing (bench gains a port): data/helexa-bench-firewalld.xml, spec install, deploy-bench /api/health probe + firewalld step, sudoers firewall-cmd grants, [api] in example + bob.toml. - 5 API tests + serve smoke. Part B — bench/ Vite + React-SWC-TS app (router, react-bootstrap, recharts): Overview (summary table), Trends (decode tok/s & TTFT across build SHAs), Runs (filterable explorer). Typed API client with VITE_API_BASE + dev proxy to bob. npm build/typecheck clean. Hosted separately from the API (per design); .gitignore excludes node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:26:55 +03:00
rob thijssen	66833890c0	ci(deploy): deploy helexa-bench to bob + enable all fleet services on boot ... Adds a deploy-bench job to deploy.yml that rolls helexa-bench onto bob (the bench host, also running Agent Zero), following the deploy-cortex pattern: manifest-gated skip-when-current, light "service stays active" validation (outbound-only, no listener/model to probe), journal capture. Runs alongside the cortex→neurons chain (no deploy-ordering dependency — the sweep loop is version-aware). Boot persistence: all systemd deployments now `systemctl enable --now` instead of bare `start`, so cortex / neuron / helexa-bench come back after a host reboot. Covers deploy.yml (all three services) and deploy-dev.yml (neuron fast path); sudoers gain the matching `enable --now <svc>` grant. infra-setup.sh handles bob: provisions gitea_ci, installs the bench-host sudoers, enables the lair-cafe-unstable repo (bob is a client host without it), pre-creates /etc/helexa-bench, and syncs asset/helexa-bench/bob.toml. New assets: bench-host.conf sudoers and bob.toml (three neuron targets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 09:10:07 +03:00
rob thijssen	42da25a37c	feat(bench): version-aware benchmark harness + neuron build metadata ... Adds automated, longitudinal performance tracking across neuron builds, replacing manual script/bench.py runs and hand edits to benchmarks.md. neuron build metadata + GET /version: - cortex-core: shared BuildInfo type (build_info.rs). - neuron build.rs captures git SHA (preferring injected HELEXA_BUILD_SHA, else git, else "unknown"), dirty flag, build timestamp, rustc version, profile, target, enabled cargo features, and best-effort candle-core version from Cargo.lock. - New GET /version endpoint (version.rs) + clap --version long form. - SHA injected in CI (build-neuron step) and helexa-neuron.spec (%{?helexa_commit}) so tarball RPMs report the real SHA. /version is now the canonical "which build is live" probe. helexa-bench crate: - Continuous daemon: hits each neuron directly on :13131, exercises each warm (status==loaded) model, records every run into a SQLite system-of-record stamped with the neuron's full BuildInfo. - Version-aware: skips any (target, build SHA, model, scenario) cell already at samples_per_version, so a steady fleet costs only cheap /version + /models polls until a new SHA ships. - Extensible Scenario trait; phase-1 chat-latency family ported verbatim from bench.py (synthetic 128/4096-tok prompts, /no_think, streamed TTFT + decode-window tok/s). `report` regenerates the benchmarks table. - kind="openai" comparison targets scaffolded, not yet wired. Packaging: data/helexa-bench.service (+ sysusers), prebuilt-binary RPM spec (outbound-only, no firewalld), and build/package/publish wiring in build-prerelease.yml with change detection. Tests: cortex-core BuildInfo round-trip, neuron GET /version integration, helexa-bench unit (prompt/SSE/config/store) + end-to-end sweep (record -> skip -> resume on new SHA). Docs updated (benchmarks.md, CLAUDE.md addendum). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-13 15:26:02 +03:00
rob thijssen	9a312098dd	fix(ci): drop the unused flash-attn feature from neuron builds (#42) ... The neuron fleet builds with `cuda cudnn flash-attn`, but nothing in neuron uses flash-attn: the qwen3_5 (27B) arch is hand-rolled, the candle-transformers qwen3 model has no flash path, llama is built with use_flash_attn=false, and `grep flash crates/neuron/src` is empty. The feature only pulls in candle-flash-attn's sm_80/sm_86 CUDA kernel sweep — which is exactly where ptxas SIGSEGVs/hangs in #42 (3 hits in one day, the last a ~4-hour hang that stalled the whole deploy behind the ampere job). Dropping the feature removes the #42 failure surface at the root (not a mitigation) and cuts the longest, most fragile part of each flavour build. No runtime change — nothing called those kernels. Removed from all three flavour builds in build-prerelease.yml and from deploy-dev.yml; ci.yml's cuda-check already used `--features cuda` only. Closes #42 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-13 09:43:14 +03:00
rob thijssen	8f6f1d3205	feat(deploy): validate neuron capability after every deploy ... A deploy previously went green the moment systemd reported the service started — a merge that broke model loading or inference itself would deploy "successfully" and only surface when a human noticed. Each neuron deploy now earns its green: 1. Wait for default models: poll /health until activation.state is ready, with per-host timeouts in the matrix (beast 900s for the 27B Q6K TP=2 cold-load, benjy/quadbrat 300s). Any entry in activation.failed fails the deploy with the per-model error — the structured equivalent of watching the journal for "loaded default model", plus failure detail the journal line can't carry. 2. LLM smoke probe: ask the first loaded model to reply with one specific word (max_tokens 512 so thinking models have room, temperature 0) and grep the response for it. Not a quality bar — just proof the deploy didn't lobotomize inference. Hosts whose package is already current still skip everything — the validation cost is only paid when a restart actually happened. The probe was dry-run against benjy's production neuron before landing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 15:28:20 +03:00
rob thijssen	112e4e124a	fix(ci): export RUSTC_WRAPPER in the build step itself — GITHUB_ENV doesn't propagate ... Run 375 proved the CUDA image ships sccache (probe step printed "sccache enabled") but the wrapper never reached cargo: the runner does not propagate GITHUB_ENV across steps, so the builds ran unwrapped (server stats: 4 compile requests for a ~600-crate build, durations unchanged). Probe and export inside the build step's own shell instead, in both build-neuron and ci.yml's cuda-check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 14:50:25 +03:00
rob thijssen	dc6feec6dc	fix(deploy): gate on the publish manifest, not unprivileged dnf check-update ... The f5fa840 deploy exposed both failure modes of gating with `dnf check-update` as the gitea_ci user in one run: it hung indefinitely on quadbrat (blocked process, 0 CPU, killed manually), and on benjy/beast it silently reported "no updates" two minutes after new RPMs were published — both hosts skipped a real (luckily binary-identical) update. Gate with data we own instead: fetch packages.json from rpm.lair.cafe (plain curl, no privileges, no dnf locks), take the newest release per package by buildTime, and skip the stop/upgrade/start cycle only when it exactly equals `rpm -q %{VERSION}-%{RELEASE}`. Unreachable or unparsable manifest fails open to a full deploy. The dnf transaction itself still runs under the scoped sudoers rules, unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 14:20:21 +03:00
rob thijssen	2dadea5d8d	ci: enable sccache on the build jobs (conditional on the CUDA image) ... The 3 CUDA flavour builds (10-14 min each, the critical path of every full run) and build-cortex compiled entirely uncached. With the gongfoo-side sccache hardening in place, wire them up: - build-cortex: full sccache env (rust image ships it) + the standard escalation loop (retry -> server restart -> uncached final attempt). - build-neuron: probe for sccache before enabling the wrapper — the CUDA image may not ship it, and a missing binary must degrade to an uncached build, not fail cargo at `sccache rustc -vV` (the original reason the wrapper was cleared here). rustc compilations are shared across all three flavours; candle-kernels' nvcc output stays uncached (build-script artifact). - ci.yml cuda-check: same probe pattern replaces the blanket env clear; also pins CUDA_COMPUTE_CAP=86 since the image no longer ships nvidia-smi for candle-kernels' fallback detection (mirrors 9bb9678 on the #20 branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 14:05:26 +03:00
rob thijssen	f5fa840dfb	ci: escalate sccache retries — restart server, then fall back uncached ... Run 361's Test job failed all 3 attempts with the sccache dead-server signature (sccache fatal error, ENOENT on its own tmp files under target/debug/deps). Retrying the same invocation only helps for transient races; against a wedged server every same-VM retry fails identically — and under the new pipeline that blocks publish and the deploy behind it. Escalate instead: attempt 1 plain, attempt 2 after an sccache server restart, attempt 3 with RUSTC_WRAPPER unset (uncached). A sick cache now costs build minutes, never the deploy. Applied to the lint/test jobs in build-prerelease.yml and ci.yml alike. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 13:24:02 +03:00
rob thijssen	7557c5e877	ci: cut iteration latency — change-aware builds, gated deploys, dev fast path ... Push-to-testable was ~20.5 min for every commit (measured on the 2026-06-08 green chain) plus a ~5 min 27B cold-load, regardless of what changed. Three structural fixes: - build-prerelease: a change-detection step in `prepare` diffs HEAD against the git sha embedded in the last *published* unstable RPM (per package, from packages.json) and skips builds whose inputs didn't change. Docs-only commits build nothing; gateway-only commits skip the 3 CUDA flavour builds. Detection failures fall open to a full build. - ci.yml no longer runs on pushes to main; fmt/clippy/test live in build-prerelease as parallel jobs gating publish. The two workflows previously queued against each other on the same runner labels, delaying the cortex build ~12 min. Branches, PRs, and tags keep the full ci.yml gate. - deploy: each host self-gates with `dnf check-update` and leaves the service untouched when the installed package is already current — no more neuron restarts (and 27B cold-loads) for commits that didn't change neuron. - deploy-dev (new): manual single-host fast path — build one CUDA flavour, scp the binary, restart the service. Skips packaging, signing, publish, and dnf entirely. Backed by a new exact-form sudoers rule in asset/sudoers.d/neuron-host.conf (already applied to all three hosts). Expected loop times when runners behave: docs ≈ 1 min (nothing deploys), gateway-only ≈ 6-8 min, single-neuron dev ≈ 8-10 min, full fleet ≈ 13-15 min. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 13:17:22 +03:00
rob thijssen	1a74cb0c56	chore: rename repo cortex -> helexa ... helexa is the project; cortex (per-operator control plane / LLM proxy) and neuron (per-host LLM harness) are its components. The Gitea repo is now helexa/helexa. Update repository URLs in Cargo metadata, RPM specs, and docs; make the CI changelog push URL rename-proof via the github.repository context; reframe README.md and CLAUDE.md around the project name. Binary, package, service, and config-path names are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 10:54:01 +03:00
rob thijssen	a26bb9f04b	feat(deploy): capture service startup journal after each restart ... After both `Start cortex.service` and `Start neuron.service`, sleep 10s and run `journalctl --unit <unit> -I --no-pager` to record the latest invocation's log in the workflow output. Step is guarded by `if: always()` so a failed start still leaves a usable trace. infra-setup.sh now adds gitea_ci to the systemd-journal group during user provisioning, so `journalctl` works without a sudoers entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-02 16:48:56 +03:00
rob thijssen	5c520c7e90	feat(deploy): gitea workflow for rolling RPM deploys + host bootstrap ... Replace operator-run script/deploy.sh with a CI-driven rolling deploy: - .gitea/workflows/deploy.yml fires on build-prerelease success (and is re-runnable via workflow_dispatch). Cortex upgrades first on hanzalova.internal; the three neuron hosts upgrade in parallel under fail-fast: false so one failing host doesn't sink the rest. Concurrency-grouped to serialize overlapping deploys, never cancelling in-flight runs (a half-applied dnf transaction is worse than a stale deploy). - asset/sudoers.d/{cortex,neuron}-host.conf are the canonical source for the scoped privileges gitea_ci needs on each host kind, installed as /etc/sudoers.d/helexa_gitea_ci. URLs and = signs are backslash-escaped per sudoers reserved-character rules. - script/infra-setup.sh idempotently provisions the gitea_ci user, installs the runner pubkey, drops in the appropriate sudoers fragment with visudo verification, and syncs cortex.toml / models.toml / per-host asset/neuron/<short>.toml — config still ships from operator workstations rather than CI because the first two are gitignored. The CI-only secret is RSYNC_SSH_KEY (already configured for the repo); the matching pubkey is ~/.ssh/id_gitea_ci.pub on the operator's box. script/deploy.sh and asset/manifest.yml are left in place until the first end-to-end deploy workflow run succeeds, then removed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-01 14:58:23 +03:00
rob thijssen	0af8c8d6e7	chore(ci): enable colored logs for readability	2026-06-01 09:06:28 +03:00
rob thijssen	2f387f33f8	ci: export CUDA paths in cuda-check so cudarc build.rs finds nvcc ... act launches step shells without sourcing /etc/profile, so the gitea_runner user's PATH lacks /usr/local/cuda-13.0/bin. cudarc's build.rs panics with ENOENT on `nvcc --version` under the neuron crate's cuda-version-from-build-system feature. build-prerelease.yml already does this export — mirror it here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-31 23:28:04 +03:00
rob thijssen	cad7552104	ci: clear sccache env on cuda-check so cargo doesn't try to wrap rustc ... CI run 255 job 3 (CUDA type-check) fails with: error: could not execute process `*** rustc -vV` (never executed) Caused by: No such file or directory (os error 2) The redacted `***` is `sccache`. The ci.yml workflow-level env block sets `RUSTC_WRAPPER: sccache` because the generic `rust` runner has sccache installed and routes the cache to caveman.kosherinata.internal. The new `cuda-check` job runs on `cuda-13.0` (where nvcc lives), and that runner doesn't carry sccache on PATH — so cargo's first action (`sccache rustc -vV` to probe the compiler version) fails before borrow-check even starts. `build-prerelease.yml`, which uses the same `cuda-13.0` runner for the actual release neuron builds, deliberately does NOT set RUSTC_WRAPPER. That's the pattern this commit applies. Fix: override `RUSTC_WRAPPER` (plus the SCCACHE_* and AWS_* env locally on the job. We lose caching on the cuda-check job (it's borrow-check-only and finishes in a couple minutes anyway), but the gate runs. The job's purpose — fail fast on `#[cfg(feature = "cuda")]` borrowck errors that the default-feature gate misses — is what matters, and that purpose was undermined by the env inheritance. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-31 13:55:18 +03:00
rob thijssen	957f704efa	feat(neuron): OpenAI Responses API + ci cuda-check runner label ... Step 2 of the Responses rollout: native `/v1/responses` endpoint on neuron that consumes the same InferenceEvent stream as `/v1/chat/completions` but emits it as the Responses API's named SSE event family. No gateway-side translation. ## Surface - `cortex-core::responses` envelope types: `ResponsesRequest`, `ResponsesInput` (text | items), `ResponsesInputItem` (message | function_call | function_call_output | reasoning), `ResponsesContentPart` (input_text | input_image | output_text), `ResponsesResponse`, `ResponsesOutputItem`, `ResponsesUsage`. Plus a `events::*` constant module so the projector and the wire shape stay in sync without string-typos. - `neuron::wire::openai_responses`: - `request_to_chat(req)` flattens Responses input + instructions into a `ChatCompletionRequest` the candle harness already understands. Text-only Parts collapse to a string; mixed text+image Parts go to chat's content-array shape; reasoning items drop; function_call / function_call_output round-trip via tool_calls / tool_call_id metadata so the surface is consistent for the day the harness emits tool calls. - `project_responses_stream(rx, meta)` reads InferenceEvents and emits the eight named events that compose a Responses stream: response.created → output_item.added → content_part.added → output_text.delta×N → output_text.done → content_part.done → output_item.done → response.completed. Synthesises start frames if the producer skips Start (poisoned model, early disconnect) so the stream stays coherent. - `build_response(meta, text, reason, usage)` for the non-streaming path. - `CandleHarness::inference_stream(req)` extracted from `chat_completion_stream`, returning a typed `InferenceStream` (event receiver + id/created/model_id metadata). Both `chat_completion_stream` and the new `responses_stream` are now thin wrappers that pick their wire projection. TP path got the same treatment (`chat_completion_tp_stream` → `inference_tp_stream`). - `POST /v1/responses` route on neuron. Non-streaming returns one buffered `ResponsesResponse`; streaming returns axum SSE with both event names and JSON data per frame (Responses, unlike chat completions, uses named `event:` lines). Reused `inference_error_response` helper hoisted out so the chat and responses handlers share the InferenceError → HTTP mapping. ## CI Also bundles the `cuda-check` runner-label fix from feedback on commit 1859777: `runs-on: rpm` doesn't ship the CUDA toolkit so cudarc's nvcc-version build script blew up. Switched to `runs-on: cuda-13.0` per the existing labels. ## Scope cuts (documented in the modules) - `previous_response_id` rejected at translate time with 400 (`code: chained_conversation_not_supported`) — stateful chained conversations need a persistence layer we haven't built. - Reasoning items dropped (no Qwen3 `<think>` routing yet). - Single output item per response (one `"message"` carrying text); `function_call` items reserved but not synthesised. - Streaming events cover the core set; `response.in_progress` and the web_search / image_generation event families are out-of-scope. 22 new tests: 5 in cortex-core (envelope round-trips), 13 in neuron::wire (request translator + projector + non-streaming builder), 4 in neuron's tests/api.rs (route surface — 503 when no candle, 400 on previous_response_id, 404 on missing model for both stream and non-stream). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-31 11:13:44 +03:00
rob thijssen	1859777332	ci: add cuda type-check job so CUDA-only borrowck errors fail fast ... Run 244 caught a use-of-moved-value in a `#[cfg(feature = "cuda")]` block that the default-feature workspace clippy/test gate had no chance of seeing. The error appeared only when the RPM build workflow compiled with `--features cuda` — 30+ minutes after push. Add a `cuda-check` job to ci.yml that runs `cargo check -p neuron --features cuda --all-targets` on the rpm runner (where nvcc / cudarc build deps live; the generic `rust` runner doesn't have them). Borrow-check only — we never run tests here, the runner has no GPU. Same retry pattern as clippy/test. Both SRPM jobs (`srpm-cortex`, `srpm-neuron`) now gate on `cuda-check` so a CUDA build break can't reach the release pipeline. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-31 09:49:51 +03:00
rob thijssen	6e1c1dd0fc	ci: retry clippy + test up to 3 times on spurious sccache failures ... sccache occasionally fails mid-compile with race-condition errors that clear on a re-run without any code changes. Rather than tracking that down right now, wrap the two affected steps in a bash loop that retries up to three times with a 5-second pause. Real failures still surface; they just take ~10s longer to fail. fmt is left as a single invocation — it's a one-shot syntactic check, not a build, and isn't subject to the same sccache races. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-27 12:55:18 +03:00
rob thijssen	60176e7c2e	ci: monotonic prerelease versions + serialize CI on shared runner ... Two CI hygiene fixes uncovered while validating against the live fleet. 1. Same-day prerelease packages were being ordered by RPM-vercmp's alpha-vs-digit precedence on the git SHA fragment, not by commit chronology. With release stamps like "0.1.${YYYYMMDD}git${SHA}", two commits on the same day produce the same numeric prefix and rpmvercmp falls back to comparing the alphanumeric SHA suffixes, where digit-leading SHAs are ranked above alpha-leading ones — completely unrelated to which commit landed first. Verified with rpmdev-vercmp: gitabc1234 < gitdef5678 (old scheme — purely lexicographic) Bumping the timestamp prefix to second-precision (%Y%m%d%H%M%S) makes the numeric prefix strictly monotonic for any chronologically- ordered commits, so the SHA fragment becomes a debug identifier only — never participates in version ordering. 2. ci.yml and build-prerelease.yml both target the `rust` runner label and both auto-trigger on push to main. The act-based runner reuses /root/.cache/act/<hash>/hostexecutor/ across concurrent jobs, so ci.yml's clippy and build-prerelease.yml's build-cortex were racing each other's checkout/cleanup steps and corrupting in-flight compile artifacts. Real fix is in gongfoo; workflow-level workaround is a shared concurrency group with cancel-in-progress=false so the two workflows queue sequentially on the same ref. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 13:36:53 +03:00
rob thijssen	e9d0a75dd5	ci(prerelease): auto-build on every push to main ... The build-prerelease workflow was workflow_dispatch-only, which meant every commit needed a manual run dispatch before any host could upgrade. That left rolling fixes (e.g. f9f5fa4's StateDirectory fix) sitting on main with no published RPM behind them, so deploy.sh silently fell back to an older prerelease. Add 'push: branches: [main]' alongside the existing workflow_dispatch trigger; the unstable channel now tracks head automatically. The concurrency group is keyed on ${{ github.ref }} with cancel-in-progress so successive rapid-fire pushes coalesce to one build (latest wins) rather than queueing every intermediate commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 13:13:36 +03:00
rob thijssen	57d7ef8d3c	chore: revert dnf. runner user has no system privs	2026-05-19 07:16:38 +03:00
rob thijssen	0e9671dd7d	fix(ci): drop sudo from dnf install (runner runs as root, no sudo) ... The act runner container has no sudo binary; the runner user already runs as root inside the container. Existing steps (rpmbuild, gpg, etc) already invoke privileged commands directly without sudo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 07:06:52 +03:00
rob thijssen	e29c9e35f0	fix(ci): ensure rust toolchain present on cuda-13.0 runner ... The currently-published runner-cuda-13.0 image (gongfoo) is missing rust/cargo despite inheriting from runner-rust. Build-neuron fails immediately with 'cargo: command not found' even though build-cortex on the bare 'rust' runner builds fine. Add a defensive `dnf install rust cargo clippy` step at the top of build-neuron. Idempotent — on a properly-built runner image this is a fast no-op; on the current broken image it installs the toolchain in a few seconds. The runner image itself should be rebuilt in gongfoo so this step becomes redundant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 07:04:57 +03:00
rob thijssen	5e17081fb4	ci(prerelease): drop redundant rustup install step ... The build-cortex and build-neuron jobs were running a copied-from- mistralrs rustup install step. Both jobs use runner images that already provide rust via dnf: - runner-rust installs rust/cargo/clippy/rustfmt directly. - runner-cuda-13.0 extends runner-rust. Running 'rustup update stable' on top would install a parallel rustup-managed toolchain and shadow the dnf one — confusing and unnecessary. The existing ci.yml already trusts the dnf toolchain without any install step, so match that behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:47:29 +03:00
rob thijssen	4a5211d830	ci(prerelease): add ampere flavour alongside ada and blackwell ... Adds ampere (CUDA compute capability sm_86) to both the build-neuron and package-neuron matrices, so helexa-neuron-ampere RPMs are built and published alongside helexa-neuron-ada and helexa-neuron-blackwell. The prerelease spec already lists ampere in its Conflicts: clause, so no spec change is needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:28:19 +03:00
rob thijssen	6d2dc5ff1a	fix(ci): give fmt/clippy/test distinct CARGO_TARGET_DIR to avoid races ... After the candle deps were added, cargo builds run long enough that the parallel fmt/clippy/test jobs (all on the `rust` runner label, which appears to use act in host-executor mode) start racing each other's intermediate temp files under /root/.cache/act/<hash>/hostexecutor/target/debug/deps/ Concretely the test job hit: error: No such file or directory at path "target/debug/deps/.tmprlicL7" Compiling unicode-ident because another job's cargo invocation cleaned up the temp file mid-compile. fmt and clippy happened to finish without their own target races landing fatally, so only test failed visibly. Set CARGO_TARGET_DIR=target-${{ github.job }} at the workflow level so each job writes to its own target directory. sccache still backs the actual rustc cache, so the rebuild penalty is just metadata not full recompiles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:26:29 +03:00
rob thijssen	b713dbe669	fix(ci): pass GPG secrets via env to avoid Gitea log leakage ... The previous "Import signing key" step inlined ${{ secrets.RPM_SIGNING_KEY }} and ${{ secrets.RPM_SIGNING_KEY_ID }} directly into the run: block. Template expansion writes the literal secret value into the rendered shell script, and Gitea logs the rendered script — Gitea's masker may not reliably scrub multi-line keys, so values can leak. Move both secrets into the step's env: block (the same pattern the "Set up SSH" step already uses) and reference $VARs in the script. The script body now contains only variable names; the secret values live in the process environment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:13:52 +03:00
rob thijssen	5c957d08ec	ci: add build-prerelease workflow for CUDA RPMs on rpm.lair.cafe ... Adds a manually-triggered workflow that builds CUDA-flavoured neuron binaries and a CPU cortex binary, packages them as Fedora RPMs, signs them, and rsyncs to the unstable channel at https://rpm.lair.cafe/fedora/43/x86_64/unstable/. Mirrors the build pipeline used by grenade/mistralrs-package. Pipeline: - prepare: derive {version,short_sha,commit_date} from the checkout; the prerelease Release stamp "0.1.YYYYMMDDgitSHORTSHA" sorts below the eventual "1" stable release. - build-cortex: cargo build --release -p cortex-cli on a rust runner. - build-neuron: matrix over ada (sm_89) and blackwell (sm_120) on cuda-13.0 runners; cargo build with features "cuda cudnn flash-attn" and CUDA_COMPUTE_CAP set per flavour. - package-{cortex,neuron}: rpmbuild on the rpm runner against the new prebuilt-binary specs in rpm/. - publish: import signing key, sign RPMs, rsync to oolon, createrepo_c --update, then regenerate packages.json for the UI. New specs are prebuilt-binary variants — they consume the artifact from the build job rather than running cargo at rpmbuild time. Each helexa-neuron-{flavour} package Conflicts with the other flavours and with helexa-neuron (the future source-build stable package) so one flavour is installed at a time on a given host. neuron crate gains cudnn and flash-attn feature flags forwarding to the corresponding candle features, so the CI build command compiles those kernels into the binary. sccache is intentionally NOT used in the prerelease jobs — CUDA compute cap isn't in its cache key, so flavours would mis-hit each other. Each prerelease build is a clean cargo build. Required Gitea secrets (already in place for cortex.spec / COPR workflow): - RPM_SIGNING_KEY, RPM_SIGNING_KEY_ID - RSYNC_SSH_KEY Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:01:35 +03:00
rob thijssen	7f797b0265	ci: parallelise fmt/clippy/test and drop sccache install step ... Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-11 13:55:17 +03:00
rob thijssen	5a0360c1d5	ci: use container runner labels for CI jobs ... Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-11 13:29:42 +03:00
rob thijssen	3e1fb60076	ci: drop actions/cache for cargo registry and target ... The cache round-trip (download + unpack) was consistently taking around 6 minutes, noticeably longer than the ~3 minute cold build it was meant to accelerate. Net-negative on CI time — remove it. sccache with the S3 backend still provides dep-level caching at a much lower overhead, so we keep the majority of the cache benefit without paying the actions/cache tarball cost. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 17:45:25 +03:00
rob thijssen	abe4ff7ccc	ci: publish both packages to a single helexa/helexa COPR project ... Consolidates the previous helexa/cortex and helexa/helexa-neuron COPR projects into one shared project. Hosts enable a single repo and get access to both packages — cortex for gateway hosts and helexa-neuron for GPU nodes. Reduces the "which copr do I enable on this host" friction, and makes it clear the two packages are parts of the same helexa project suite. CI keeps two independent publish jobs (copr-cortex and copr-neuron) running in parallel; they now both target helexa/helexa with their respective SRPMs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 16:37:47 +03:00
rob thijssen	7c3390a4e1	fix(rpm): rename neuron package to helexa-neuron ... Fedora's official repos ship a package named `neuron` — the NEURON neural-simulation environment from Yale (see https://src.fedoraproject.org/rpms/neuron). Having our own `neuron` in the helexa COPR caused dnf5 to silently no-op `dnf install neuron` because of the name collision, even with the COPR repo enabled and keys imported. The only workarounds were full NEVRA (`dnf install neuron-0.1.12-1.fc43.x86_64`) or a local file install — neither acceptable for end-users. Rename the RPM package to `helexa-neuron`. Keep binary (/usr/bin/neuron), systemd unit (neuron.service), system user (neuron), and config dir (/etc/neuron) unchanged — those are project-local contexts where the short name is unambiguous. Follows Fedora subpackage-style naming except with a vendor prefix rather than a parent-package prefix, because neuron is an independent package from cortex (installed on different hosts) and neither depends on the other. Changes: - neuron.spec -> helexa-neuron.spec (git rename) - Name: neuron -> helexa-neuron (with comment explaining why) - CI: srpm-neuron job now builds helexa-neuron-VERSION.tar.gz with the matching top-level dir prefix, publishes to helexa/helexa-neuron COPR - CI: bump-version job references helexa-neuron.spec - CLAUDE.md: install instructions updated Old helexa/neuron COPR project can be deleted after the first helexa/helexa-neuron build lands. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 16:37:47 +03:00
rob thijssen	2ff062da0e	ci: commit generated %changelog entries back to main ... Previously the srpm-* jobs generated a fresh %changelog entry and shipped it to COPR, but the version-stamped spec pushed back to main by the bump-version job only updated the Version: line — not the %changelog section. The result: SRPM and in-tree spec diverged and a fresh clone of the repo showed a perpetually empty changelog. Run the rpm-changelog action in bump-version too. Now the committed specs track the SRPMs: each release leaves a dated %changelog entry in main covering commits since the previous tag, visible in git log and in the repo's spec browser. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 16:37:03 +03:00
rob thijssen	1d90238b01	ci: migrate rpm changelog generation to reusable action ... Replace the local .gitea/scripts/generate-rpm-changelog.sh with the shared composite action at https://git.lair.cafe/actions/rpm-changelog@v1. Behaviour is identical — collect commits since the previous v* tag, filter bump-version and merge noise, prepend a dated entry to the spec — but the logic now lives in one place that other projects can consume. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 15:32:51 +03:00
rob thijssen	d99b25fb8a	ci: auto-generate rpm changelog entry per release ... On every tag push, build a %changelog entry from the git log since the previous v* tag and prepend it to each spec. Stops the initial entry from drifting further and catches bogus-date / stale-version warnings automatically since the generated date always matches the day the CI runs. The generator drops "chore: bump version" commits (bot-authored, noisy in user-facing changelogs) and merge commits. Author defaults to the gitea-actions identity but can be overridden via CHANGELOG_AUTHOR env var if a human release is desired. Requires fetch-depth: 0 on checkout so git describe can see prior tags and git log can reach them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 15:32:51 +03:00
rob thijssen	4a9a4fc775	ci: migrate copr publish to reusable action ... Replace the in-repo .gitea/scripts/copr-build.sh and per-job copr-cli configuration with the shared composite action at https://git.lair.cafe/actions/copr-publish@v1. Behaviour is identical — submit, watch, dump per-chroot logs — but the logic now lives in a single place that other projects can consume. Removes the actions/checkout step from both COPR jobs since the build script is no longer local to this repo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 12:34:39 +03:00
rob thijssen	5c7d63c658	ci: dump COPR per-chroot build logs to CI output ... Previously the COPR publish steps only surfaced copr-cli's status updates (pending/importing/running). When a build failed, diagnosing required clicking through to the COPR web UI. Now we submit with --nowait, watch the build, then use copr-cli download-build to fetch each chroot's builder-live.log and cat them as collapsible ::group:: blocks in the CI output. Logic is factored into .gitea/scripts/copr-build.sh so cortex and neuron jobs share it. Both COPR jobs now check out the repo to access the script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 12:06:05 +03:00
rob thijssen	15ded3a5bd	ci: cache target/, disable incremental, drop redundant build ... Three complementary tweaks to close the gap sccache alone can't: - CARGO_INCREMENTAL=0: reclaims the 17 incremental-mode cache misses per run and prevents cargo from writing incremental fingerprints that defeat sccache. Incremental mode is useless in CI anyway since each run starts from scratch. - actions/cache for ~/.cargo and target/: sidesteps sccache's structural limits (proc-macro non-cacheables, clippy-vs-rustc separate namespaces) by caching the whole build output keyed on Cargo.lock. Also caches ~/.cargo/bin so the installed sccache binary survives between runs. - Drop the separate 'cargo build' step: 'cargo test --workspace' builds everything anyway, so the standalone build was a full redundant workspace compile pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 09:44:45 +03:00
rob thijssen	7befa882d5	fix: yaml syntax	2026-04-16 09:25:02 +03:00
rob thijssen	d03fae960a	fix(ci): unset RUSTC_WRAPPER during sccache install ... The workflow-level env set RUSTC_WRAPPER=sccache for every step, including the install step itself. cargo install sccache then tried to invoke `sccache rustc -vV` to detect the toolchain before sccache existed on PATH, failing with "No such file or directory". Override RUSTC_WRAPPER to empty on the install step so cargo uses rustc directly; subsequent steps still inherit the wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-16 08:31:26 +03:00
rob thijssen	7b2235d56b	fix(ci): install sccache with S3 feature if missing ... The distro sccache package lacks S3 support. Install from cargo with --features s3 if the existing binary can't connect to the S3 backend. Skips install if already present and working. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 17:44:21 +03:00
rob thijssen	54f9f3dc36	ci: add sccache with MinIO backend for build caching ... All Rust compilation steps now use sccache backed by MinIO S3 at caveman.kosherinata.internal:9000. Credentials via repo secrets SCCACHE_S3_ACCESS_KEY and SCCACHE_S3_SECRET_KEY. Cache is shared across all bare metal runners. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 17:38:13 +03:00
rob thijssen	caee8bba11	fix(ci): use GITEA_TOKEN env var for push, not checkout ... Token is only needed for the authenticated push, not the public checkout. Set remote URL with token inline before pushing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 16:31:13 +03:00
rob thijssen	324dfa05c5	ci: add RPM packaging for cortex and neuron ... - cortex.spec: gateway binary, cortex.service systemd unit, cortex.toml + models.toml config files - neuron.spec: neuron binary, neuron.service systemd unit, neuron.toml config file - Parallel CI: srpm-cortex and srpm-neuron jobs build SRPMs concurrently, then publish to separate COPR repos (helexa/cortex and helexa/neuron) - bump-version job: after both COPR publishes succeed, stamps tag version into Cargo.toml, specs, Cargo.lock and pushes to main via GITEA_TOKEN - Shared cortex user/group across both packages - Example configs: cortex.example.toml, neuron.example.toml, models.example.toml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-15 16:28:31 +03:00