11 Commits

Author	SHA1	Message	Date
rob thijssen	6088830e7d	feat(deploy): manage NEURON_MAX_PROMPT_TOKENS per host via model.conf drop-in ... Roll the per-model context cap into deploy.yml so it is deterministic per host and rolled out (with a restart) alongside the rest of the service config, rather than hand-edited in local.conf. The deploy now writes /etc/systemd/system/neuron.service.d/model.conf from a new per-host `max_prompt_tokens` matrix field, and restarts a neuron when the package OR the drop-in changes — so a cap change applies even with no new RPM. beast (Qwen3.6-27B, hybrid linear, 2x 32GB) -> 131072 (~128k); benjy and quadbrat (dense, VRAM-bound) stay at 16384 but become deploy-managed. Adds the scoped sudoers grant for the root-owned drop-in install, and doc/context-limits.md documenting the knob relationships and KV/VRAM math (refs #62 for the eventual /models-advertised source of truth, #65 for the length-aware text VRAM guard that gates pushing beyond 128k). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-16 18:48:19 +03:00
rob thijssen	4ee7da4f97	feat(bench-ui): internal vhost bench.internal + step@ cert renewal ... Inside the WireGuard mesh, bench.helexa.ai dead-ends at the OPNsense LAN interface (only WAN :443 is port-forwarded), so add an internal path: - asset/nginx/bench.internal.conf — server_name bench.internal, internal "lair" CA cert, same SPA + /api→bob proxy. Mirrors the *.internal vhost convention on oolon.kosherinata.internal. - asset/systemd/step@.{service,timer} — replicate oolon's smallstep cert renewal (step ca renew via mTLS, every 15 min, reload nginx). - infra-setup.sh: install the step@ units + /etc/nginx/tls/{cert,key}, install the vhost + enable step@bench.timer once the cert exists; prints the one-time issuance command otherwise. Initial cert issuance (JWK provisioner) and bench.internal DNS are operator steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:34:38 +03:00
rob thijssen	37c19aa985	feat(bench-ui): public hosting at https://bench.helexa.ai via gateway nginx ... nginx on the gateway serves the bench SPA and reverse-proxies /api to the bob bench API over WireGuard — public, auth-less, same-origin (no CORS), internal API stays private. - asset/nginx/bench.helexa.ai.conf (full TLS vhost: SPA + /api proxy) and a bootstrap http-only vhost for the initial ACME challenge. - infra-setup.sh: one-time gateway setup — webroot, Let's Encrypt cert (certbot webroot, idempotent), install + enable the vhost. - deploy.yml: deploy-bench-ui builds the SPA (setup-node) and rsyncs dist/ to /var/www/bench.helexa.ai every deploy; built same-origin so no VITE_API_BASE. - cortex-host.conf: scoped gitea_ci rsync grant for the webroot. - bench/README: production hosting notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:40:29 +03:00
rob thijssen	f50f5531cf	feat(bench): read-only JSON API on bob + bench/ React visualisation app ... Part A — helexa-bench read API: - [api] config (enabled, listen :13132); WAL on the store so API reads never block the sweep writer. - store read methods: summary, series (chronological per-build medians), runs (filtered), dimensions, run_count. - api.rs: axum /api/health|dimensions|summary|series|runs, permissive CORS (UI is a separate origin). The `run` daemon binds the API alongside the sweep; new `serve` subcommand serves API-only. - listener plumbing (bench gains a port): data/helexa-bench-firewalld.xml, spec install, deploy-bench /api/health probe + firewalld step, sudoers firewall-cmd grants, [api] in example + bob.toml. - 5 API tests + serve smoke. Part B — bench/ Vite + React-SWC-TS app (router, react-bootstrap, recharts): Overview (summary table), Trends (decode tok/s & TTFT across build SHAs), Runs (filterable explorer). Typed API client with VITE_API_BASE + dev proxy to bob. npm build/typecheck clean. Hosted separately from the API (per design); .gitignore excludes node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:26:55 +03:00
rob thijssen	66833890c0	ci(deploy): deploy helexa-bench to bob + enable all fleet services on boot ... Adds a deploy-bench job to deploy.yml that rolls helexa-bench onto bob (the bench host, also running Agent Zero), following the deploy-cortex pattern: manifest-gated skip-when-current, light "service stays active" validation (outbound-only, no listener/model to probe), journal capture. Runs alongside the cortex→neurons chain (no deploy-ordering dependency — the sweep loop is version-aware). Boot persistence: all systemd deployments now `systemctl enable --now` instead of bare `start`, so cortex / neuron / helexa-bench come back after a host reboot. Covers deploy.yml (all three services) and deploy-dev.yml (neuron fast path); sudoers gain the matching `enable --now <svc>` grant. infra-setup.sh handles bob: provisions gitea_ci, installs the bench-host sudoers, enables the lair-cafe-unstable repo (bob is a client host without it), pre-creates /etc/helexa-bench, and syncs asset/helexa-bench/bob.toml. New assets: bench-host.conf sudoers and bob.toml (three neuron targets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 09:10:07 +03:00
rob thijssen	7557c5e877	ci: cut iteration latency — change-aware builds, gated deploys, dev fast path ... Push-to-testable was ~20.5 min for every commit (measured on the 2026-06-08 green chain) plus a ~5 min 27B cold-load, regardless of what changed. Three structural fixes: - build-prerelease: a change-detection step in `prepare` diffs HEAD against the git sha embedded in the last *published* unstable RPM (per package, from packages.json) and skips builds whose inputs didn't change. Docs-only commits build nothing; gateway-only commits skip the 3 CUDA flavour builds. Detection failures fall open to a full build. - ci.yml no longer runs on pushes to main; fmt/clippy/test live in build-prerelease as parallel jobs gating publish. The two workflows previously queued against each other on the same runner labels, delaying the cortex build ~12 min. Branches, PRs, and tags keep the full ci.yml gate. - deploy: each host self-gates with `dnf check-update` and leaves the service untouched when the installed package is already current — no more neuron restarts (and 27B cold-loads) for commits that didn't change neuron. - deploy-dev (new): manual single-host fast path — build one CUDA flavour, scp the binary, restart the service. Skips packaging, signing, publish, and dnf entirely. Backed by a new exact-form sudoers rule in asset/sudoers.d/neuron-host.conf (already applied to all three hosts). Expected loop times when runners behave: docs ≈ 1 min (nothing deploys), gateway-only ≈ 6-8 min, single-neuron dev ≈ 8-10 min, full fleet ≈ 13-15 min. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 13:17:22 +03:00
rob thijssen	ea1fdf8aa6	chore(deploy): drop deploy.sh and manifest.yml now that workflow runs ... First end-to-end run of the deploy workflow succeeded (gitea run #289), so the operator-run rolling-deploy script and its YAML manifest are no longer the source of truth — fleet topology lives in .gitea/workflows/deploy.yml and per-host config in script/infra-setup.sh. Per-host neuron config comments updated to point at the new sync path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-02 16:41:04 +03:00
rob thijssen	5c520c7e90	feat(deploy): gitea workflow for rolling RPM deploys + host bootstrap ... Replace operator-run script/deploy.sh with a CI-driven rolling deploy: - .gitea/workflows/deploy.yml fires on build-prerelease success (and is re-runnable via workflow_dispatch). Cortex upgrades first on hanzalova.internal; the three neuron hosts upgrade in parallel under fail-fast: false so one failing host doesn't sink the rest. Concurrency-grouped to serialize overlapping deploys, never cancelling in-flight runs (a half-applied dnf transaction is worse than a stale deploy). - asset/sudoers.d/{cortex,neuron}-host.conf are the canonical source for the scoped privileges gitea_ci needs on each host kind, installed as /etc/sudoers.d/helexa_gitea_ci. URLs and = signs are backslash-escaped per sudoers reserved-character rules. - script/infra-setup.sh idempotently provisions the gitea_ci user, installs the runner pubkey, drops in the appropriate sudoers fragment with visudo verification, and syncs cortex.toml / models.toml / per-host asset/neuron/<short>.toml — config still ships from operator workstations rather than CI because the first two are gitignored. The CI-only secret is RSYNC_SSH_KEY (already configured for the repo); the matching pubkey is ~/.ssh/id_gitea_ci.pub on the operator's box. script/deploy.sh and asset/manifest.yml are left in place until the first end-to-end deploy workflow run succeeds, then removed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-01 14:58:23 +03:00
rob thijssen	740299bd9d	chore(neuron/beast): switch default-model quant from q5k to q6k ... q5k produced NaN logits on Qwen/Qwen3.6-27B under candle TP=2 (sampler fell over with "logits unhealthy nan: 248320/248320"). q6k is the quant that worked well in production under mistral.rs on the same hardware, so it's the right baseline for verifying the mempool-trim fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-27 12:36:18 +03:00
rob thijssen	d3f2d50749	feat(deploy): per-host neuron config + pre-warm headline models ... Adds asset/neuron/{beast,benjy,quadbrat}.toml — per-host neuron.toml files keyed by the first dot-component of the host. deploy.sh now rsyncs the matching file to /etc/neuron/neuron.toml on each neuron and stops+starts the service so default_models is re-read. Headline model per host (drives /v1/models output immediately after a clean deploy): beast Qwen/Qwen3.6-27B (q5k, tp=2, devices=[0,1]) benjy Qwen/Qwen3-8B (bf16, devices=[0]) quadbrat Qwen/Qwen3-1.7B (bf16, devices=[0]) Removes the need to follow deploy.sh with `validate-neuron.sh beast Qwen/Qwen3.6-27B q5k 2` to surface the 27B in the catalogue — the neuron loads it itself on activation. The neuron loop now mirrors the cortex flow (stop → install/upgrade → sync config → start) so config-only changes pick up on subsequent deploys; previously a no-package-change deploy would silently leave the host on the old default_models. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 14:05:54 +03:00
rob thijssen	03bed93fee	add asset/manifest.yml describing fleet hosts and neuron flavours ... Adds a single source of truth for which hosts run cortex vs neuron and which CUDA compute-capability flavour each neuron host needs: cortex : hanzalova.internal neurons : beast → helexa-neuron-blackwell (2x RTX 5090, sm_120) benjy → helexa-neuron-ada (RTX 4090, sm_89) quadbrat → helexa-neuron-ampere (RTX 3060, sm_86) script/deploy.sh (gitignored, local-only) is updated locally to read hosts and flavours from this manifest and dnf install the correct helexa-neuron-<flavour> package per host. Using 'dnf install --refresh --allowerasing' lets it swap out the previous bare helexa-neuron RPM or a different flavour without manual intervention; the spec Conflicts: clauses keep at most one flavour resident. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:37:14 +03:00