267 Commits

Author	SHA1	Message	Date
rob thijssen	98d095099a	fix(neuron): context-limit observability + qwen3_next profile (#126) ... - derive_limit now logs every input and intermediate term (free VRAM, prefill rate, vram/throughput ceilings, hard clamp, result) at DEBUG so a surprising advertised limit is diagnosable from the journal — needed right now: beast's 27B advertises context=0 post-reboot and no by-hand input combination reproduces it. - profile_from_qwen3_5_config had the same too-narrow model_type gate as #124 and a plain serde parse that rejects the flat qwen3_next layout — Coder-Next has never had an advertised limit or the #65 length-aware preflight. Parse via Config::from_config_json and accept both types. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-06 02:11:44 +03:00
rob thijssen	687324745f	fix(neuron): qwen3_next models get snapshots, prefix cache + batch engine (#98) ... The snapshot-capability gates in both load paths checked model_type == "qwen3_5" only, silently denying the qwen3_next family (Coder-Next, 80B-A3B) prefix caching — and, since #98 keys engine activation off the same gate, batched decode. The qwen3_5 arch serves both model_types with identical snapshot support (the committed parity fixture IS qwen3_next). Surfaced live: with max_in_flight=8 on beast, the 27B spawned its engine but Coder-Next loaded without one (and, it turns out, has never had prefix caching). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 20:08:58 +03:00
rob thijssen	daeab6be12	fix(neuron): TP unload with Arc-shared pool mutex (#98) ... cfg(cuda)-only: the unload path moved the pool out of the (previously owned) mutex; with TpLoadedModel.pool behind Arc for the engine's owned guard, recover sole ownership via Arc::try_unwrap — Arc::try_unwrap(tp) succeeding already implies the engine is idle, so this is the expected case; the fallback (engine guard mid-release race) unloads through the lock and lets the pool drop with its last reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 17:04:43 +03:00
rob thijssen	2a366a1c17	feat(neuron): engine backend seam — TP models batch through the engine (#98) ... The engine loop now drives either serving substrate through a BackendConfig/ActiveSession seam: - Single (unchanged semantics): Arc'd clones of the LoadedModel pieces; active phase holds inference_lock. - Tp: a Weak<TpLoadedModel>, upgraded only while the batch is active — an idle engine never keeps an unloaded model alive, and an unload mid-batch completes when the batch drains. The active phase holds the pool mutex (TpLoadedModel.pool is now Arc<Mutex<WorkerPool>> so the guard can be owned), which is exactly the serialization the per-request TP path used — vision and non-streaming requests wait their turn as before. Backend ops (op_prefill/op_step/op_assemble/op_extract/ op_snapshot_seq/op_drop_snap) dispatch per substrate: the TP arms reuse restore_or_clear_tp/chunked_prefill_tp/store_prefix_snapshot_tp verbatim and mint pool-wide snapshot ids from next_snapshot_id so all ranks key identically. Wiring: TpLoadedModel.engine is a OnceLock set right after the Arc is built (the Weak needs the Arc to exist); spawn gated on the same conditions as single-GPU (max_in_flight > 1, snapshot-capable, NEURON_BATCHING != 0 — default config keeps everything off). inference_tp_stream routes text streams through the engine when present. Engine gold test (single backend) unchanged and green; the TP arms are cfg(cuda) — branch CUDA type-check + live smoke on beast are the gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 16:46:00 +03:00
rob thijssen	4c3a2735b0	feat(neuron): TP rank plumbing for batched decode (#98) ... TP mirror of the batched-decode primitives, spanning all three tiers: - arch: TpQwen3_5Model/ForCausalLM forward_batch_decode + batch_decode_mask (same per-row batch_cos_sin rope path and one-gap-per-row mask as single-GPU; the shared attention_context core already handles the batched shapes). - RPC: GenerateStepBatch (per-row tokens + broadcast geometry), AssembleKvBatch (pool-minted snapshot ids), ExtractKvRows (pre-minted ids so every rank keys identically); responses KvBatchAssembled { padded_len } / KvRowsExtracted { bytes }. No protocol version to bump — the RPC evolves by adding variants (subprocess is spawned from the same binary; unknown ops fail as bad_request). - subprocess worker: handlers derive per-row positions + padding mask locally and discard logits (the NCCL collectives are the point), and run assemble_batch/extract_row against the rank's own shard snapshots — the snapshot types and geometry are identical across ranks, only head counts shard. - leader: Job::{TpForwardLogitsBatch, TpAssembleKvBatch, TpExtractKvRows} + dispatch handlers against the TP slab + handle wrappers. - pool: WorkerPool::{generate_step_batch, assemble_kv_batch, extract_kv_rows} with the same fan-out → leader → always-drain shape as generate_step (incl. the #17 watchdog on the batched step); assembly asserts every rank agrees on padded_len. Nearly all of this is cfg(cuda) — the branch CUDA type-check is the real gate. Engine backend seam (TpLoadedModel wiring) follows in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 16:23:50 +03:00
rob thijssen	d3ef9513f8	fix(neuron): as_deref for Arc'd prefix_cache in cuda-only worker path (#98) ... candle.rs:2441 sits behind cfg(cuda) — the CPU build can't see it, and the CUDA type-check caught the Option<&Arc<Mutex<…>>> vs Option<&Mutex<…>> mismatch introduced by the LoadedModel Arc change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 15:24:00 +03:00
rob thijssen	ec7a5b750a	feat(neuron): lockstep batched decode engine (#98) ... Slice 3b: per-model engine task multiplexing concurrent text chat streams through one (B,1) forward per decode step, replacing the per-request inference_lock serialization when [admission] max_in_flight > 1 on a snapshot-capable worker-path model. - harness/engine.rs: engine loop (join → B=1 prefill via the existing chunked-prefill + prefix-cache paths → snapshot → rebatch via ExtractKvRows/AssembleKvBatch; lockstep ForwardLogitsBatch steps; per-slot CPU sampling with per-slot LogitsProcessor + repeat-penalty history). Per-slot router tasks own the incremental detokenizer and the reasoning/tool-call state machine (same logic as route_token!) and emit InferenceEvents — DecodeStream's tokenizer borrow lives inside the router's async block, and slow consumers decouple from the lockstep loop. - The engine holds the model's inference_lock while it has active slots, so vision and non-streaming requests (which keep the direct path) still serialize safely against the batch; the lock releases when the batch drains. - Fatal worker errors fail all active slots, mark the model poisoned on device-fault classification, and stop the engine (later submits fail fast). - LoadedModel: poisoned/inference_lock/prefix_cache/prefill_rate move behind Arc (shared with the engine without keeping the model alive); new engine: Option<EngineHandle> field. Engine activates only when max_in_flight > 1 (config default 1 = existing behaviour everywhere) and NEURON_BATCHING != 0 (kill switch). - Prefill helpers (restore_or_clear/chunked_prefill/store_prefix/ emit_delta) ungated from cfg(cuda) so the engine and its CPU tests compile everywhere. Gold test: three greedy requests submitted concurrently (ragged prompts, mid-flight joins, rebatching) produce byte-identical text, token counts, and finish reasons to the same requests run sequentially through the same engine, on the tiny fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 15:03:43 +03:00
rob thijssen	7f2fcc3527	feat(neuron): batch-row extraction for rebatch on join/leave (#98) ... Slice 3a: extract_row slices one row of a batched cache state back into a contiguous single-sequence snapshot — attention KV keeps the prefix [0, prefix_len) plus the lockstep decode columns [padded_len, padded_len + steps) and drops the padding gap; GDN rows are sliced whole and deep-copied. A join/leave rebatches by extracting every surviving row and re-running assemble_batch, preserving the one-gap-per-row invariant without any scatter kernels. Worker side: Job::ExtractKvRows captures the live state once and stores each extracted row in the snapshot slab; composes with AssembleKvBatch / DropKvSnapshot for the full rebatch. Tests: arch-level round-trip (batched decode → extract → solo B=1 continuation matches pure-sequential) and a worker end-to-end leave path (3-row batch → extract 2 survivors → re-assemble → continued lockstep decode matches the sequential reference). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 14:46:49 +03:00
rob thijssen	9ac1a1586c	feat(neuron): device-worker jobs for batched decode (#98) ... Slice 2: AssembleKvBatch + ForwardLogitsBatch route the batched-decode primitives through the per-device worker thread. - AssembleKvBatch assembles stored per-sequence KV snapshots into one batched (B,…) live state via snapshot::assemble_batch and installs it through the existing restore path; replies the padded KV length. - ForwardLogitsBatch runs one lockstep decode step: builds the (B,1) input on the worker's device, derives per-row positions and the padding mask from prefix_lens/padded_len/step, and replies one CPU [vocab] logits row per batch row — same tensors-never-escape contract as ForwardLogits. - ModelArch::forward_batch_decode / batch_decode_mask dispatch (qwen3_5 only; other archs error as defence in depth). - Dense safetensors loads now pick f32 on non-CUDA devices: candle's CPU backend has no bf16 matmul, so the CPU fallback (and the CPU test worker) was broken at first forward under the hardcoded BF16. End-to-end test: tiny qwen3_next fixture loaded through the worker, three ragged sequences prefilled+snapshotted, assembled, and every ForwardLogitsBatch row checked against the sequential ForwardLogits reference at each step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 14:42:06 +03:00
rob thijssen	05b6c70a3a	feat(neuron): batched lockstep decode groundwork for qwen3_5 (#98) ... Slice 1 of continuous batching (F4d): arch-level batched decode with per-row positions, assembled from per-sequence prefix snapshots. - rope: batch_cos_sin gathers cos/sin at arbitrary per-row positions ((B,1,half)); apply_cos_sin dispatches on cos rank — rank-3 takes a local GLM rotate-half broadcast path (rope_slow only broadcasts one position table across the batch). - snapshot: assemble_batch cats per-sequence KvCacheSnapshots into one (B,…) batched state — attention K/V right-padded along the sequence axis (keys are stored post-RoPE, so padding is position-inert), GDN conv/recurrent states cat on dim 0. Text-only: rope_delta must be 0. - model: forward_batch_decode((B,1), positions, mask) lockstep decode step returning (B,1,vocab); batch_decode_mask builds the (B,1,1,S) additive mask covering each row's padding gap [prefix_len, padded_len), None when lengths are uniform. Gold test: three ragged sequences decoded in one lockstep batch match the same sequences decoded sequentially, hidden-for-hidden at every step, on the tiny CPU fixture. Plus rope gather/uniform-equivalence and mask-geometry units. No serving-path changes yet — the engine loop (slice 3) and worker job plumbing (slice 2) build on these primitives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 14:34:31 +03:00
rob thijssen	cb6ddbbd98	perf(neuron): route only prefill through FlashAttention (#95) ... On-beast A/B (27B, 30,419-token prompt, 2x RTX 5090, e824ea3): prefill 24.8s eager → 22.1s flash (~11%, diluted by the hybrid's ~25% full-attention layer ratio), but decode REGRESSED ~20% (50 → 60 ms/token at 30k KV) — FA2 without flash-decoding is weak at query-length 1 and the per-step layout transposes add overhead. Greedy outputs byte-identical in both modes (parity gate passed, incl. chunked-prefill causal alignment). Restrict the flash dispatch to q_len > 1: prefill keeps the win, decode keeps the eager path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 12:15:03 +03:00
rob thijssen	9139b07d71	fix(neuron): candle-flash-attn must be a direct optional dependency (#95) ... The flash-attn feature only forwarded to candle-transformers, which enables flash inside ITS models — candle_flash_attn was not nameable from neuron's own attention core (E0433 in the CUDA type-check). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 11:31:28 +03:00
rob thijssen	4762537225	perf(neuron): FlashAttention on the qwen3_5 full-attention layers (#95) ... F4a. The hybrid's full-attention layers computed eager matmul→softmax→matmul with materialised GQA-repeated K/V and O(L²) score/mask tensors — the dominant term in prefill at agentic context sizes (measured: ~30 s TTFT at 16k tokens, the trigger for the opencode timeout spiral). - attention_context(): shared attention core for the single-GPU and TP paths. With the flash-attn feature on a CUDA device in f16/bf16 it dispatches to candle_flash_attn::flash_attn — GQA native (no repeated-K/V materialisation), causality as a kernel flag, no score matrix. Everything else (CPU, f32, feature off) keeps the eager fallback, byte-identical to the previous math — the qwen3_next HF parity fixture pins this in CI. - Chunked prefill correctness rides flash-attention v2.1+ causal semantics (bottom-right alignment for seqlen_q != seqlen_k); the invariant and the caller's mask contract are documented on the helper. On-beast flash-vs-eager greedy parity is the merge gate. - NEURON_FLASH_ATTN=0 forces eager at runtime — A/B lever + rollback. - Feature enabled for the blackwell flavour build only (beast carries the agentic prefill pain; ada/ampere follow once the win is measured). CUDA type-check now covers cuda,flash-attn; timeout raised for the first uncached kernel compile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-03 11:14:47 +03:00
rob thijssen	28aa06b768	fix(helexa-bench): thinking-model streams misread as dead + impossible decode rates (#117) ... The scenario stream reader only counted `content` deltas: reasoning models (Qwen3-Next-Thinking; Qwen3 family with thinking on) stream reasoning_content first — for their whole budget on small-max_tokens scenarios — so the harness saw a dead stream ("no content chunks received"), failed the cell, and re-retried it every sweep forever (measured=0 skipped=15 failed=27, for hours, live 2026-07-02). Meanwhile decode_tps divided usage.completion_tokens (reasoning INCLUSIVE) by the visible-content-only chunk window — reporting 244 tok/s for a 1.7B on a 3060 whose true rate is ~39. - Liveness now counts any generated delta (content or reasoning_content); a cell fails only when the stream is truly dead. ttft_s = first generated delta — unchanged semantics for non-thinking models (their first delta is content) and honest engine latency for thinking models. - decode_tps prefers the server-measured prefill/decode split (#85), which the reader was already capturing but not using; the client fallback divides the CHUNK count by the chunk window — one frame, never mixed. - Captured artifacts remain visible-content-only. Cells are keyed by the target neuron's build SHA, so shipping this mid-collection does not re-key the in-progress F3 rotation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 23:03:15 +03:00
rob thijssen	2df829ed73	fix(neuron): admission queue leaked slots when clients cancelled mid-wait ... AdmissionController::enter() incremented the pending + per-principal counts, then awaited the in-flight semaphore. That await is where a client disconnect lands: axum drops the request future, and neither the success path nor the timeout branch's rollback ever ran — every abandoned wait permanently leaked one pending slot and one per-principal count. Under a client retry storm (opencode re-sending after aborting), the leaks ratchet pending to max_pending, after which every request gets an instant QueueFull 429; the leaked per-principal counts alone can pin a principal at its fair-share cap forever. Observed live 2026-07-02 during the implementation-eval runs: queue_depth stuck at 1 on an idle model, ~70ms 429s at the gateway, 36 aborted attempts and 507k prompt tokens burned against 38k generated in one 40-minute window. The reservation is now a RAII PendingReservation taken before the await: admitted permits carry it for the request lifetime, and any other exit — timeout, or the future being dropped mid-queue — rolls the counts back in Drop. Regression test aborts two queued waiters and asserts both the queue depth and the principal count recover. Follow-up (separate issue): abandon chunked prefill early when the streaming client is gone — needs a clean sentinel distinct from the device-fault path so it never trips poison/auto-recovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 16:28:50 +03:00
rob thijssen	41ec05e40c	fix(neuron): truncated-mid-think non-streaming responses are all reasoning (#112) ... When the chat template force-opens the think block in the generation prompt (Qwen3-Next-80B-A3B-Thinking ends with '<|im_start|>assistant\n<think>\n') and the generation exhausts max_tokens before emitting </think>, the non-streaming split_off_reasoning had no close-marker anchor and returned the whole chain-of-thought as content. Thread the existing prompt_opens_reasoning result (already used by the streaming state machine) into the split: no close marker + prompt-opened block means content is empty and every generated token counts as reasoning. Closes #112 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 11:57:52 +03:00
rob thijssen	40fecb2cf7	fix(neuron): route the fused MoE block in the activation dtype (#92) ... The fused path fed f32 activations into the replicated bf16 router — a dtype-mismatched matmul that failed the first live fused request on the 80B and (correctly) tripped the poison/auto-recovery machinery. Route in the original activation dtype like route_scatter does; only the softmax and downstream weights are f32 (which the grouped-GEMM kernel requires for topk_weights anyway). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 07:27:43 +03:00
rob thijssen	29a7054c23	perf(neuron): fused grouped-GEMM dispatch for the qwen3_next MoE block (#92) ... F1 slice 4 — replaces the correctness-first scatter loop on CUDA: - TpExpertStore::Fused holds each projection as ONE stacked per-rank QTensor ([E, out/ws, in]) built at load: read every expert's rank slice, stack, ISQ the stack in a single parallel pass per projection (~0.5 GB transient bf16 per stack at 80B dims). - forward_fused ports candle-transformers' FusedMoeGGUF::forward onto the per-rank stacks via candle-nn's moe_gemm_gguf grouped-GEMM kernels: routing, index sort, and all expert GEMMs stay on-device — three kernel launches per layer regardless of top-k, no GPU→CPU routing sync. gate/up run unweighted (tokens×topk rows); the down GEMM folds the routing weights in-kernel; the (tokens, topk, hidden) view sums over topk. Output is the rank's partial; the shared expert and single block-end AllReduce are unchanged. - Store chosen at load: CUDA + ISQ with a kernel-supported GGML dtype (q2k-q6k, q8_0) + GGUF block alignment on both GEMM K dims; NEURON_MOE_FUSED=0 forces scatter as escape hatch and A/B lever. CPU/non-cuda builds keep scatter (the CI parity anchor). Motivation: the 80B-A3B live smoke measured 4.3 tok/s decode on the scatter path (host routing sync + ~30 tiny GEMV launches per layer per token) vs ~27 tok/s for the dense 27B. Target: close the ~6x gap. On-beast A/B (fused vs NEURON_MOE_FUSED=0 scatter, greedy-token equivalence + decode tok/s) follows this merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 06:46:06 +03:00
rob thijssen	a157c8fa27	fix(neuron): gate TP MoE load tests to non-cuda builds (#92) ... The CUDA type-check compiles test targets too, where TpQwen3_5MoeBlock::load takes an NCCL Comm the tests cannot construct (E0061). The CPU Test job runs the partial-sum parity test; the CUDA job type-checks the cuda load/reduce variants. Also drops an unused binding only the cuda test build flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 00:42:45 +03:00
rob thijssen	201cc54a7b	feat(neuron): TP expert sharding for the qwen3_next MoE block (#92) ... F1 slice 3 — the MoE FFN under tensor parallelism: - TpQwen3_5MoeBlock: per-expert sharding along moe_intermediate_size (gate/up column-sliced, down input-sliced) with forward_partial outputs, deliberately NOT composed from RowParallelLinear — that embeds an AllReduce per call, and top-10 routing would mean ten collectives per layer. Partial sums from routed experts and the shared expert add linearly, so the whole block does exactly ONE AllReduce at the end, preserving the one-reduce-per-FFN pattern. - Router + shared_expert_gate are replicated (tiny): every rank computes identical top-k assignments with zero communication, via the route_scatter helper now shared with the single-GPU block. - TpMlpKind { Dense, Moe } decoder dispatch on layer_uses_moe, mirroring the single-GPU MlpKind; per-rank ISQ applies to expert slices through the existing MaybeQuantLinear path. - TP fused-checkpoint loading: K- and V-heads shard uniformly together, so each rank owns a contiguous span of whole k-head groups of the fused in_proj_qkvz/in_proj_ba tensors — a uniform dim-0 shard is exactly the rank's groups, then the per-rank de-interleave (split_fused_qkvz/ba with per-rank head counts) restores the contiguous [Q|K|V]+Z / B+A layout. Auto-detected; Qwen3.6's separate-tensor path untouched. - text_weight_prefix threaded through the TP model load (qwen3_next keeps the text core at model.*); the slice-1 TP MoE guard removed. Validation (CPU, runs in CI): with the block-end AllReduce elided, rank-0 + rank-1 partial outputs at world_size=2 sum exactly to the single-GPU block's output (which itself has exact HF parity from slice 2) — pinning expert slicing, replicated routing, and shared-expert partial scaling. A second test pins the per-rank group-span de-interleave against row-slices of the full split. The cfg(cuda) load/reduce variants are validated by the CUDA type-check in CI; live TP-2 smoke lands with F2's checkpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 00:34:07 +03:00
grenade	781d92d9a3	Merge pull request 'feat(neuron): qwen3_next config + MoE FFN block, single-GPU path with exact HF parity (#92)' (#108) from feat/92-qwen3_5-moe into main	2026-07-01 21:23:23 +00:00
rob thijssen	9bf13f09dd	feat(neuron): qwen3_next MoE FFN block, single-GPU path + HF parity (#92) ... F1 slice 2 — the MoE block itself, CPU/single-GPU: - arch/qwen3_5/moe.rs: Qwen3_5MoeBlock — top-k router (upstream softmax-then-topk order, renorm iff norm_topk_prob), per-expert SwiGLU (reusing Qwen3_5MLP at moe_intermediate_size), and the always-on shared expert mixed via sigmoid(shared_expert_gate). Correctness-first host-side scatter dispatch; the fused grouped-GEMM path is slice 4 behind the same forward signature. - decoder.rs: MlpKind { Dense, Moe } dispatch on layer_uses_moe, mirroring AttentionKind. - linear_attn.rs: fused-checkpoint support — qwen3_next stores in_proj_qkvz / in_proj_ba interleaved per key-head group (upstream fix_query_key_value_ordering layout); split_fused_qkvz/ba de-interleave once at load into the contiguous [Q|K|V] + Z / B + A layout the forward path (incl. the conv channels) already uses. Auto-detected via contains_tensor, so Qwen3.6 checkpoints are untouched. - mod.rs: text_weight_prefix() — qwen3_next checkpoints put the text core at `model.*`, Qwen3.6 at `model.language_model.*`; the slice-1 single-GPU MoE guard is removed (TP guard stays until slice 3). Validation: - qwen3_next_parity integration test replays a committed tiny random-weight HF Qwen3NextForCausalLM checkpoint (generated by script/dump_qwen3_next_tiny.py on beast: transformers 5.9.0, torch 2.9.1) through neuron's full load path: max_abs 0.000000, cosine 1.00000000 at f32 — exact parity, pinning the config normalisation, weight prefix, qkvz/ba de-interleave, hybrid layer interleaving, and the whole MoE block against upstream. - Unit tests: scatter forward vs per-token dense reference, no-shared-expert/no-renorm behaviour, fused-split round-trip, and a flat-layout end-to-end structural load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-02 00:16:19 +03:00
rob thijssen	a1426f177c	feat(neuron): accept qwen3_next configs into the qwen3_5 arch (#92) ... F1 slice 1 — config + normalization, no kernel work yet: - TextConfig gains the MoE hyperparameters (num_experts, num_experts_per_tok, moe_intermediate_size, shared_expert_intermediate_size, decoder_sparse_step, mlp_only_layers, norm_topk_prob), all defaulting to the dense case so existing Qwen3.6 configs parse unchanged, plus layer_uses_moe() implementing the upstream sparse-layer selection. - Config::from_config_json normalises the flat qwen3_next layout (Qwen3-Next-80B-A3B family) into the nested qwen3_5 shape: wraps flat hyperparameters, nests flat rope fields into rope_parameters, forces attn_output_gate (unconditional in upstream qwen3_next), and derives layer_types from full_attention_interval via the upstream (i+1) % interval convention when absent. - All four config parse sites (dense CPU, dense CUDA worker, TP leader, TP worker subprocess) route through from_config_json; "qwen3_next" added to DENSE/TP_SUPPORTED_MODEL_TYPES and the dispatch match arms. - Explicit num_experts > 0 guards in Qwen3_5Model::load and TpQwen3_5Model::load so a premature 80B load fails with a clear "MoE not implemented yet (#92)" instead of a cryptic missing-tensor error. Slices 2/3 remove them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-01 23:59:44 +03:00
rob thijssen	869033d08e	feat(cortex): re-expose flat max_model_len/max_input_tokens/max_output_tokens on /v1/models (#78) ... Hermes Agent (and the wider vLLM-convention client ecosystem) probes /v1/models for flat context-window keys and cannot see helexa's limit.context — it fell back to a hardcoded catalogue guess that only matched by luck. Re-add the flat fields additively, derived from the settled `limit` at serialization time (cortex list_models and the router's federation aggregate), omitted when the window is genuinely unknown. `limit` stays the opencode-oriented source of truth. Flips the old regression guard that asserted max_model_len must not appear — the removal it guarded was based on the wrong assumption that the field had no consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx	2026-07-01 23:02:09 +03:00
rob thijssen	140f9124ee	feat(helexa-bench): capability probe scenario + quality scoring (#91) ... The six perf scenarios measure speed/resources; this measures the axis they miss — reasoning/planning quality — so the frontier A/B (F3) can pick on capability, not just throughput. Per the chosen approach: store the artifact always, with schema for BOTH a manual score and a future LLM-judge; start manual. - scenario: CapabilityScenario (capability:<name>) runs a fixed prompt and captures the full output text (stream_and_measure gains a capture_text path); opt-in via config.capability_probes (empty default — long outputs, deliberate). - store: three additive columns (artifact, quality_score, scorer); capability_runs(unscored_only) worklist + set_score(id, score, scorer). Drill-down RunRow omits the large artifact column. - cli: `helexa-bench score --id <n> --score <x> [--scorer ...]` (manual); `report --capability` (per-model median score + per-run artifact snippets); GET /api/capability. LLM-judge deferred (schema ready). - example config documents an implementation-planning probe. Tests: artifact storage + scoring lifecycle, capability scenario built from config, capability markdown (median + snippet). Part of the Performance observability epic (#83), O7 — completes the milestone. Feeds the F3 frontier A/B decision gate (#94). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 15:03:13 +03:00
rob thijssen	842caf5d4d	feat(helexa-bench): cold-load / model-swap cost measurement (#90) ... The vision cold-swap policy (F4e) needs real numbers for unload→reload latency and the cold first-request after a reload — not guesses. Adds a deliberate `swap-cost` subcommand (NOT the continuous sweep, since unloading takes the live model offline for the reload). For each neuron target's warm models: unload, time the synchronous reload, then time a cold first request. Recorded under scenario "swap" so it tracks per build like everything else. - client: unload_model, load_model, and spec_from_info (reconstructs a ModelSpec from /models — TP inferred from device count, quant left None for neuron to resolve). - sweep: Sweeper::swap_cost_once + measure_swap (build_record gains a swap-timing param; existing sweep call sites pass None). - scenario: cold_probe — a single timed request reusing the SSE core. - store: two additive columns (swap_unload_ms, swap_load_ms) + a swap_costs() pivot (reload latency + cold first-request medians). - report: `report --swap` view + render_swap_json; GET /api/swap. - cli: `helexa-bench swap-cost` with an explicit offline warning. Tests: swap_costs pivot + scenario exclusion, swap markdown render. Part of the Performance observability epic (#83), O6. Feeds the vision cold-swap policy (#99 / F4e). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 14:15:42 +03:00
rob thijssen	d472b6428a	feat(helexa-bench): context-length scaling view (#88) ... The chat:<n> cells already capture prefill & decode tok/s per context (via #85/#86); this pivots them into a per-(target,model) scaling curve and computes decode-flatness — decode tok/s at the largest context ÷ the smallest. ~1.0 confirms the Gated-DeltaNet O(1)-in-sequence-length decode; a sharp drop locates where the model stops scaling for free. - store: Store::scaling() pivots the latest-build chat:<n> report cells into ScalingCurve/ScalingPoint, ordered by context, with the flatness ratio (concurrency:<n> and other scenarios are excluded). - report: render_scaling_markdown (one block per model: prefill/decode tok/s vs ctx + a flatness verdict) and render_scaling_json. - cli: `helexa-bench report --scaling` selects the view. - api: GET /api/scaling for the bench UI. - example config documents widening prompt_sizes into a scaling ladder. No new request shape — reuses the chat-latency measurement points, so a denser curve is just more prompt_sizes entries (operators widen the ladder deliberately; large contexts cost more per sample). Tests: scaling pivot + flatness + scenario exclusion, markdown render. Part of the Performance observability epic (#83), O4. Validates the long-context property that makes the 80B-A3B frontier model (#84) viable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 13:56:07 +03:00
rob thijssen	209bf58207	feat(helexa-bench): concurrency / agentic-load scenario (#89) ... Batch-1 single-request timing can't characterize the real workload — a0/hermes/opencode each fan out many agentic requests per user turn. Add a ConcurrencyScenario (concurrency:<n>) that fires N simultaneous streams and records, per burst: aggregate node throughput (total tokens / burst window), within-burst TTFT p95 (the tail under load), median admission queue-wait — derived as TTFT − server prefill_ms (#85), no /health poll needed — and the count of streams shed by admission (429/503). On a batch-1 server, throughput stays flat while queue-wait and p95 inflate with N; that gap is the evidence for/against continuous batching (#98). - scenario: ConcurrencyScenario + shared chat_payload helper; reuses stream_and_measure per stream via join_all. - config: concurrency_levels (default empty — opt-in, a burst loads the fleet) + concurrency_prompt_tokens (512); example config documents it. - store: four additive columns (same migration pattern); aggregate surfaces concurrency, within-burst p95, queue-wait, rejected medians. - report: markdown gains conc / queue ms / rej columns; JSON the full set. Tests: median/percentile/admission-detect helpers, config builds concurrency scenarios, store surfaces burst metrics. Part of the Performance observability epic (#83), O5. Produces the evidence base for continuous batching (#98 / F4d). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 13:17:10 +03:00
rob thijssen	a415454962	feat(helexa-bench): per-run VRAM high-water + GPU util/temp from /health (#87) ... Bench had no measured VRAM headroom — only the anecdotal "~2/3 used". Sample neuron's GET /health (DeviceHealth: vram_used/free, util%, temp) right after each measured run and record node-sum VRAM used plus the hottest device's utilization and temperature. - client: fetch_health (neuron only; soft None on transport failure so a flaky /health never fails a measurement). - sweep: HealthAgg folds a /health snapshot to node totals; sampled post-run (the recent decode-VRAM peak) and threaded into the record. /health is ~5s-cached neuron-side, so this is a coarse high-water proxy, documented as such. - store: three additive columns (same PRAGMA-guarded migration); aggregate emits vram_used median + node vram_total (summed from the discovery devices in gpus_json) for real headroom, plus util/temp medians. - report: markdown gains a "VRAM (GB)" used/total column; JSON gains the used/total + util/temp medians. Tests: VRAM/headroom aggregation, gpu_total_vram summation. Part of the Performance observability epic (#83). Completes the p1-now trio (O1-O3); unblocks the enriched-27B baseline capture on beast. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 12:50:53 +03:00
rob thijssen	afc1f7a706	feat(helexa-bench): percentiles + prefill/decode split in store & report (#86) ... Bench reported only a single median per cell, hiding tail latency and unable to record the server-measured prefill/decode split now emitted on `usage.helexa_timing` (#85). - scenario: parse `usage.helexa_timing` into ScenarioMetrics (prefill_ms, decode_ms, prefill_tokens). - store: persist the three columns (additive PRAGMA-guarded migration via ensure_columns, so pre-#85 DBs backfill as NULL); aggregate now emits p50/p95/p99 for TTFT and total (nearest-rank) plus a prefill-tok/s median derived from the split. - report: markdown gains prefill tok/s, TTFT p95, total p95 columns; JSON gains p95/p99 + prefill_ms/decode_ms/prefill_tps medians. Tests: nearest-rank percentile, idempotent backfill migration, and a report cell asserting percentiles + prefill split. Part of the Performance observability epic (#83). Stacked on #85. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:58:36 +03:00
rob thijssen	6f956dfda3	fix(neuron): hoist TP prefill/decode timers out of 'work block (#85) ... The TP streaming producer builds its Finish event after the 'work labelled block exits (inside `if failure.is_none()`), but prefill_elapsed and decode_start were declared inside that block — so the CUDA type-check failed with E0425 (the CPU build doesn't compile this cfg(cuda) path). Hoist `prefill_ms_measured: u32` and `decode_start: Option<Instant>` above the block; populate them at the prefill→decode boundary inside; read them at the terminal Finish. The worker and local producers were unaffected (their timers already share the Finish scope). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:58:26 +03:00
rob thijssen	6e0f15c888	feat(neuron): server-measured prefill/decode timing on Finish (#85) ... The harness emitted only token counts on InferenceEvent::Finish; all timing was client-side SSE arrival, so bench "TTFT" conflated tokenize + prefill and decode tok/s was a window estimate. Add FinishTiming { prefill_ms, decode_ms, prefill_tokens } to the Finish event, populated by all three streaming producers (TP, worker, and local CPU paths), and surface it on the OpenAI chat `usage.helexa_timing` extension so helexa-bench can compute true prefill vs decode tok/s. cortex forwards usage verbatim, so the field survives proxying. Non-streaming and Responses paths carry None for now (bench reads the streaming chat path). Keystone for the Performance observability epic (#83). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq	2026-06-27 11:41:17 +03:00
rob thijssen	f96a2e7ed3	fix(neuron): surface reasoning_tokens in non-streaming /v1/responses usage ... The non-streaming responses handler hardcoded `output_tokens_details: None`, so the reasoning sub-count that the chat path now computes (via `split_off_reasoning`, which strips the `<think>` span and counts it into `completion_tokens_details.reasoning_tokens`) never reached the Responses-API usage object. The streaming responses path already emits it. Carry `chat usage.completion_tokens_details.reasoning_tokens` through to `ResponsesUsage.output_tokens_details.reasoning_tokens`, so streaming and non-streaming `/v1/responses` report reasoning accounting identically. `output_tokens` still counts every generated token (reasoning included); `reasoning_tokens` is the additive sub-count, per OpenAI's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 21:21:59 +03:00
rob thijssen	13daf95514	fix(neuron): strip <think> reasoning from non-streaming completions ... The non-streaming inference path (single-GPU `chat_completion` and the TP `chat_completion_tp_inner`) returned the model's full decode — reasoning preamble + `</think>` + answer — as the assistant `content`. The streaming path already drops reasoning (emits it as ReasoningDelta, which the chat and Responses projectors discard), so the two transports disagreed: a streaming client saw the clean answer, a non-streaming client saw the chain-of-thought glued to the front of it. This is exactly what broke agent-zero v2.0 on `/v1/responses` (non-streaming): the model produced a correct in-band JSON answer after `</think>`, but a0's parser saw the `<think>` preamble first and rejected the turn as "misformat, no valid tool request found". a0 v2.1 happens to tolerate the preamble, but any strict non-streaming client (and a0 v2.0) does not — and reasoning tokens leaking into `content` also misreport as visible output. Add `split_off_reasoning(generated_ids, reasoning_pair)`: if the model declares a reasoning marker pair and its close token (`</think>`) appears in the output, return only the tokens after the last close marker as content and count the rest as reasoning. The chat template injects the *opening* marker into the prompt, so the generated tokens carry the close marker but not the open one — splitting on the close-token id (not a decoded string) is robust to tokenizer byte-fallback. Non-reasoning models, thinking-disabled requests, and generations truncated mid-reasoning have no close token and pass through unchanged. Both non-streaming sites now decode only the answer span and populate `completion_tokens_details.reasoning_tokens` (the streaming-only accounting gap noted at the call sites, #64). Unit tests cover the strip, no-marker, no-pair, close-at-end, and multiple-close cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 20:39:41 +03:00
rob thijssen	6731adca51	fix(neuron): accept bare {role,content} input on /v1/responses (agent-zero) ... agent-zero (via litellm) drives the OpenAI Responses API and sends `input` items in the "easy input message" form — bare `{role, content}` objects with NO `type` field. Our `ResponsesInputItem` is internally tagged (`#[serde(tag="type")]`), so every such item failed the untagged `ResponsesInput` deserialize and axum's `Json` extractor returned 422: OpenAIException - Failed to deserialize the JSON body into the target type: data did not match any variant of untagged enum ResponsesInput This was a *total* failure for agent-zero (both the main model on beast and the utility model on benjy), confirmed by on-wire capture of 15 live requests: 36/36 input items were bare easy-messages. Other clients (/v1/chat/completions, /v1/messages) were unaffected — only the Responses path was exercised this strictly, for the first time. Make `input`-item parsing match OpenAI's real tolerance, mirroring the forward-compat `extra: Value` already at the top level of the request: - New `ResponsesInputElement` wraps the existing typed item enum with two more shapes: `EasyMessage { role, content }` (bare, no type; `content` optional so an assistant turn with `content: null` parses) and `Other(Value)` — a catch-all so a single unmodeled item can never again 422 the whole request. The typed enum is unchanged. - `ResponsesContentPart` gains a `#[serde(other)] Unknown` arm (e.g. `refusal`, audio) — dropped in translation, not rejected. - `FunctionCallOutput.output` is now `Value` (string OR array of content parts, per OpenAI) so a structured tool result isn't lost. - Translator handles all three element shapes; easy-messages translate exactly like typed messages, `Other` and unknown parts are dropped. Tests cover the bare-message, null-content, unknown-item, unknown-part, and array-tool-output shapes, validated against the 15 captured bodies. Tools forwarding + native function_call projection on the Responses path is deliberately a follow-up (Round 2), gated on observing how agent-zero consumes responses once unblocked (in-band JSON vs native tool calls). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLyKaJVFDAYnAGiLvVrK8K	2026-06-26 19:45:22 +03:00
rob thijssen	f4117224fc	feat(B6): served-usage ledger + reconciliation (#58) ... Operators are now metered for the tokens they serve on behalf of mesh accounts, and the upstream rolls that up for compensation. cortex-gateway: - served_usage.rs: an in-process per-(account,key,UTC-day) served-token counter, incremented in metering::usage_sink alongside spend/settle for every authenticated request. A flush task (spawned in run() when [upstream].enabled, mirroring poller/evictor) POSTs ABSOLUTE cumulative counters to upstream on [upstream].served_usage_report_interval_secs. - The no-limit infra key is the operator's local key with hard_cap=None (already supported) — it's metered for served-usage but never budget- refused and never hits upstream. helexa-upstream: - POST /authz/v1/served-usage: upserts rows keyed by (operator_id from the client bearer, account, key, period) with GREATEST(existing, incoming) — monotonic + idempotent, so re-sends, races, and a restarted cortex's lower counter never regress the total. - reconcile.rs + `helexa-upstream reconcile` CLI: rolls up unreconciled served_usage per operator/period (SUM::bigint), stamps reconciled_at, prints the totals. Payout mechanism out of scope. Validated against a throwaway Postgres: monotonic upsert (100→250, a 50 re-send stays 250, same-value idempotent) and reconcile rollup + stamp-once; cortex counter unit test for per-principal accumulation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 11:45:57 +03:00
rob thijssen	8600d4fbf2	Merge feat/B5-topup-codes: single-use top-up codes + mint CLI (B5, #59)	2026-06-23 11:01:26 +03:00
rob thijssen	bb0d1e51b8	Merge feat/B3-cortex-upstream-client: cortex upstream entitlement client + chain (B3, #57)	2026-06-23 10:51:03 +03:00
rob thijssen	2348cc2234	feat(B5): single-use top-up codes (redeem + mint CLI) ... Completes the hybrid allocation model — the flat free grant (B4) plus top-up codes that extend an account's allocation. - topup.rs: redeem(account, raw_code) — timing-safe, single-use. A conditional `UPDATE … WHERE redeemed_by IS NULL RETURNING value` claims the code atomically (concurrent double-redeem → exactly one winner), then raises accounts.allocation_total in the same tx. Only sha256(code) is stored; a not-found code and an already-redeemed code return the SAME generic error via the same path (no "valid but spent" oracle). Raising the total automatically lifts every percent-limited key's effective cap; hardcap keys stay pinned (by design). - mint(value, count, denomination) — inserts codes (hash-only), returns the raw codes once. Exposed as `helexa-upstream mint --value --count [--denomination]` (raw codes to stdout, one per line) — the seam the future faucet bot calls. Bot itself out of scope. - POST /web/v1/redeem (session-protected) → {allocation_total} | generic 400. Validated against Postgres: redeem raises the 1_000_000 free grant to 1_500_000, second redemption + unknown code both generic-400, and a concurrent race for one code yields exactly one HTTP 200. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:50:43 +03:00
rob thijssen	a9d7382be8	feat(B3): cortex upstream entitlement client (#57) + chained provider ... cortex can now validate locally-unrecognised bearer keys against the helexa-upstream authority and reserve/settle their budget there — mesh accounts work for real inference. The EntitlementProvider trait is the seam, so cortex's enforcement (auth.rs, metering.rs) is otherwise unchanged. - entitlements_upstream.rs: UpstreamEntitlementProvider over reqwest → B2's /authz/v1 (resolve/reserve/settle/release/snapshot), presenting the operator client bearer. Maps the wire contract back to the trait: granted → Reservation, rejected → BudgetError, 401 → InvalidKey. Fail-closed — unreachable resolve → AuthError::Unavailable (503, never 401); unreachable reserve → retryable BudgetError::RateLimited (refuse, never serve un-authorized). settle/release are best-effort (the upstream sweeper reaps a lost one). - entitlements_chain.rs: ChainedEntitlementProvider tries local first (operator + infra keys, no network), falls through to upstream for unknown keys, and dispatches reserve/settle/release/snapshot to whichever backend resolved each account (local treats unknown principals as uncapped, so it can't be the blind default). - cortex-core: AuthError::Unavailable{retry_after_secs}; [upstream] config (enabled/url/bearer/timeout). auth.rs maps Unavailable → 503 + Retry-After distinctly from InvalidKey → 401, regardless of require_auth. - state.rs wires the chain when [upstream].enabled, else stays purely local. Tests (upstream_chain.rs, 4): local key resolves without touching upstream; unknown key falls through to a mock upstream; unknown-everywhere → 401 InvalidKey; upstream-unreachable → Unavailable (503-mapped), with local keys still resolving. Existing gateway suites updated for the new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:41:48 +03:00
rob thijssen	d94c62c143	feat(B4): /web/v1 account API + silent fingerprint multi-account abuse ... The human-facing account surface the helexa.ai frontend (F4) consumes, on top of B2's authz surface. Email+password auth with JWT sessions (distinct from inference API keys); plain-JSON errors (the #63 envelope stays on the authz surface). - Auth lifecycle: register → email-verify → login → password-reset (request/confirm). argon2id passwords; verify/reset via single-use sha256-hashed email tokens; register and reset-request always return 202 (no account enumeration). Email via a pluggable EmailSender (lettre Smtp + dev Log transport). - API keys: create (sk-helexa-<base62(32 OsRng)>, raw shown once, stored as sha256 + non-secret prefix), list (prefix never the secret), archive, PATCH per-key limit (percent|hardcap). Protected by a JWT session middleware. - Account balance endpoint (allocation total/spent/reserved). - Silent fingerprint abuse: register captures the browser fingerprint; >= threshold (default 5) accounts sharing one fingerprint are silently deactivated + flagged — registration still returns a normal 202, and a deactivated account's key resolves as an ordinary 401 at the authz surface (no "banned" signal anywhere). - crypto: argon2 hash/verify + CSPRNG token/key minting (base62). config gains [auth] + [email]. CORS on the app for the browser SPA. Validated against a throwaway Postgres 16: verify-once, full lifecycle (register→verify→login→create key→account→list→authz resolve→archive→401), and 5-same-fingerprint → all accounts silently deactivated + no-clue 401. 8 unit + 11 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:28:20 +03:00
rob thijssen	46befde4cd	feat(B2): /authz/v1 authority surface + client-auth + reservation sweeper ... The machine surface cortex's UpstreamEntitlementProvider (#57) consumes, mirroring the EntitlementProvider trait 1:1 over the B1 ledger. - `authz.rs`: POST /authz/v1/{resolve,reserve,settle,release,snapshot}. resolve → {principal, snapshot} | 401 invalid_api_key (a deactivated account resolves as the SAME 401 — the silent-abuse no-clue property). reserve returns 200 whether granted ({reservation_id}) or budget-refused ({rejected:{kind,...}}) — a refusal is an authoritative answer, not a transport failure; non-2xx means "fail closed" to the client. settle/ release → 204 (idempotent). snapshot → {hard_cap,spent,reserved} | 404. Rejections use the shared #63 OpenAiError envelope (cortex-core dep). - Client auth: shared-bearer middleware (constant-time compare via subtle) maps a token → operator_id (stamped into request extensions for #58 served-usage); empty config = open dev surface (logged). mTLS deferred. - ledger gains resolve_key (sha256 lookup, account-active-gated), snapshot, and sweep_stale (one data-modifying-CTE statement releasing aged-out open reservations and folding their reserved tokens back into accounts+keys). - Sweeper task spawned in run(); [authz] ttl/interval + [client_auth] config; crypto::sha256 helper. Validated against a throwaway Postgres 16 (fresh schema): resolve→reserve→ settle→snapshot round-trip, over-cap → 200 insufficient_quota rejection (not retried away), deactivated account → 401 (no clue), missing/wrong client bearer → 401 before any DB hit. 5 unit + 8 gated integration tests; all skip cleanly without UPSTREAM_TEST_DATABASE_URL so CI stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 10:12:19 +03:00
rob thijssen	71106afaf1	feat(B1): helexa-upstream crate skeleton + Postgres schema + ledger ... First milestone of the mesh-level account/authorization authority (#59). New workspace crate `crates/helexa-upstream` (binary + lib), mirroring the helexa-router skeleton: axum `build_app`/`run`, clap `serve --config`, figment `UPSTREAM_`-prefixed config. - **Storage:** PostgreSQL via sqlx (runtime query API — builds in CI with no DB or offline cache; correctness covered by gated integration tests). `migrations/0001_init.sql` is the full schema: users (argon2 hash slot, email_verified, registration_fingerprint), email_tokens, accounts (allocation total/spent/reserved + `accounts_no_overshoot` CHECK, silent `status` deactivation flag, fingerprint_flagged), api_keys (sha256 hash, percent|hardcap limit, cap_window, per-key ledger), reservations (BIGSERIAL id → maps to cortex Reservation.id u64), top_up_codes, served_usage, sessions. Migrations run on startup. - **Ledger (no-overshoot core):** `ledger::{reserve,settle,release}` — reserve takes `SELECT … FOR UPDATE` on the account + key rows so concurrent reserves serialize and spent+reserved can never exceed the effective cap (= min(resolved key cap, remaining account allocation)); the CHECK is the DB backstop. Settle clamps actual to [0,reserved] and is idempotent; release is idempotent. `resolve_abs_cap` (percent/hardcap, i128 math) is pure + unit-tested. Balance semantics here; rolling-window sub-caps + RateLimited land with the authz API (B2). - `/health` does a DB round-trip. - Config + example TOML ([server]/[db]/[grant] free grant/[abuse] fingerprint threshold). Validated end-to-end against a throwaway Postgres 16: migration applies, 20 concurrent reserves of 100 against a 500 cap admit exactly 5 (reserved == 500, never over), settle/release idempotent, hardcap key sub-cap binds below the account, `/health` → db ok. CI runs the cap-math + config unit tests; the DB integration tests skip cleanly when UPSTREAM_TEST_DATABASE_URL is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-23 09:49:19 +03:00
rob thijssen	1115bb0942	feat(#74): verify downstream cortex TLS certs (outbound pinning) ... The router is a TLS client to cortexes; the router->cortex hop crosses the helexa->operator boundary carrying the client's bearer. This pins that hop to an enrolled cert. Trust mechanism (the open question): per-cortex enrolled trust anchor. Each [[cortexes]] entry gets an optional `tls_ca` — a PEM CA (or self-signed cert) the cortex's TLS cert must chain to. When set, the router builds a client that trusts ONLY that anchor (platform roots disabled), so the cortex must present the expected cert and a rogue endpoint with any other (even publicly-valid) cert is rejected at the handshake. Enrolment = the operator hands helexa the cortex's cert, referenced by path in router config. This is the natural model for self-hosted operators behind their own nginx/private CA, and reuses the reqwest public API (no custom rustls verifier, no new TLS backend). - `RouterState` now holds a per-cortex `reqwest::Client` map (`client_for`), replacing the single shared client; poller and dispatch use the per-cortex client. `build_client(tls_ca)` is the builder. - Fail closed: a `tls_ca` that can't load omits the cortex from the client map — it's never polled or routed to, rather than silently degrading to unpinned TLS. The poller treats a missing client (and a rejected handshake) as a failed poll, so #72's existing reachability debounce excludes it. Tests (`tls.rs`, 4): a live tokio-rustls HTTPS server proves a client enrolled with the server's cert is accepted (200) while clients pinned to a different cert — or using default roots — are rejected; the poller marks a wrong-cert cortex unreachable while a correctly-enrolled one is reachable; a missing pin file disables the cortex (fail closed); garbage PEM is rejected at build. Existing suites updated for the per-cortex client + new config field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:23:20 +03:00
rob thijssen	63f578cb15	feat(#75): aggregate /v1/models across operators (federation catalogue) ... The router's /v1/models is now the deduped union of every reachable cortex's catalogue, so an opencode client doing discovery against the router resolves the whole federation without knowing about operators or cortexes (resolves #61's "Router/discovery contract"). To preserve per-model limit/cost, the topology poller now retains each cortex's full `cortex_core::node::CortexModelEntry` (was distilled to a {loaded, feasible} bool). `entry_feasible()` replaces the dropped field; dispatch (#73) and `cortexes_serving` use it — no routing behaviour change. `catalogue.rs::aggregate_models`: - Dedupe by model id; a model served by >=1 reachable cortex appears once. - Merge availability: `loaded` OR across operators; only feasible (loaded-or-cold-loadable) entries surface — a catalogue-only model no neuron can host is hidden. - Re-tier to operator names: `feasible_on` becomes the cortexes that can serve it and `locations` the operators it's loaded on (node = cortex name), so the federation view doesn't leak each operator's neuron names or per-device VRAM. - Conflict resolution: `limit` → tightest (smallest context, so a client never overflows the most-constrained operator); `cost` → cheapest (the federation "from" price). Richer range/region policy couples to #68, noted as follow-up. Tests: 4 unit (dedupe+merge, unreachable excluded, infeasible hidden, tightest-limit+cheapest-cost) + 1 end-to-end (two mock cortexes overlapping on a model → GET /v1/models over HTTP asserts the merged union). dispatch/topology suites updated for the entry-storage change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 21:08:16 +03:00
rob thijssen	7984d27553	feat(#73): capacity-aware dispatch with region affinity + failover ... The router's data path. Wires the topology poller (#72) and the shared streaming proxy (#71) into real request routing. - `dispatch.rs`: `select_cortexes(model)` ranks reachable cortexes that can serve the model, best-first — loaded/warm before cold-loadable, region match before not, more healthy nodes before fewer, name for determinism. `dispatch()` extracts `model`, picks candidates, and forwards via `helexa_stream::forward_streaming` (bearer + bytes verbatim, SSE streamed back). Cortex's #63 rejections (429/400/…) pass through untouched; transport failures fail over to the next candidate; a genuine HTTP response — any status — is returned as-is, never retried away. - Router-originated rejections use the #63 envelope: 404 model_not_found (no operator serves it), 503 service_unavailable + Retry-After (known but all unreachable / all candidates failed to connect), 400 missing_model_field. `error.rs` is the router's envelope→axum adapter (mirrors cortex-gateway's). - `handlers.rs`: `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages` dispatch to the same path on a chosen cortex. The router holds zero entitlement logic — routes on capacity, not budget. - Config: optional `region` on the router and per-cortex for geo affinity. Tests (`dispatch.rs`): routes to a serving cortex + forwards the bearer; cortex 429 passes through and is NOT retried; transport failure fails over to a live cortex; unknown→404, known-but-unreachable→503, missing-model→400; ranking order (warm/region/headroom). 7 new, existing skeleton/topology suites unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:40:07 +03:00
rob thijssen	5fd7736abd	feat(#72): router↔cortex topology poller (multi-operator capacity map) ... Builds the live topology the dispatcher (#73) will route on — the same pattern as cortex↔neuron, one tier up. - `poller.rs`: background loop polls each configured cortex's `GET /v1/models` (deserialised straight into the shared `cortex_core::node::CortexModelEntry`) and `GET /health`, on a configurable `poll_interval_secs` (default 10). - `state.rs`: `RouterState` gains an `http_client`, `poll_interval`, and a `RwLock<HashMap<cortex_name, CortexTopology>>` pre-populated from config so the poller/handlers always find an entry. Per cortex: `reachable`, `consecutive_failures`, `last_poll`, healthy/total node counts, and a per-model `{loaded, feasible}` map (feasible = loaded OR cortex reports `feasible_on`, i.e. cold-loadable). `cortexes_serving(model)` returns the reachable cortexes that can serve a model — groundwork for #73. - Debounce: a cortex flips unreachable only after `POLL_FAILURE_THRESHOLD` (3) consecutive failed polls, and recovers on the next good poll — mirrors cortex's neuron-poll debounce so a blip can't yank a whole operator out of routing. `/health` poll is best-effort and never flips reachability on its own. - `lib.rs` spawns the poll loop in `run()`. `/health` now surfaces `cortexes.reachable`; `status` stays router-liveness (always `ok`). Tests (`topology.rs`): live-map build (loaded vs catalogue-only feasible, node counts, routing helper); unreachable→excluded→recovers across the debounce threshold; dead endpoint never panics. Skeleton tests unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 19:01:29 +03:00
rob thijssen	03fd4960c3	Merge fix/71-shared-streaming-proxy: shared helexa-stream SSE proxy (#71) ... # Conflicts: # Cargo.lock # Cargo.toml	2026-06-21 18:15:12 +03:00
rob thijssen	cabec1d08a	fix(#71): extract SSE streaming passthrough into shared helexa-stream ... The true-streaming SSE passthrough (Body::from_stream, no full-response buffering, with chunk-observation hooks) was cortex-only. helexa-router (#69) needs the same mechanism to proxy a chat-completions/messages stream verbatim to a selected cortex. Extract it once. New `crates/helexa-stream` owns the *mechanism* (kept HTTP-free cortex-core untouched — it would have forced axum/reqwest/futures onto every cortex-core consumer): - `forward_streaming(client, url, headers, body, observer)` — POST and stream the response back chunk-for-chunk; status-agnostic, so a non-2xx (e.g. cortex 429) is passed through with status+headers intact (the #69 backpressure-passthrough requirement). - `ChunkObserver` trait + `ObservedStream` wrapper — feeds each chunk to the observer, calls `finish` exactly once on clean end or on drop (client disconnect). - `BodyTail` (bounded tail accumulator) + `last_count_for` (trailing OpenAI `usage` extraction) — the reusable pieces an observer uses. cortex keeps its *policy*: `proxy.rs` now supplies a `CortexMetrics` observer (per-request token metrics + per-principal reservation settle), its logging contract, and the error envelope, driving the shared mechanism. `proxy::last_count_for` is re-exported so `handlers`/ `anthropic_sse` call sites are unchanged. No behaviour change — the existing cortex `streaming.rs` tests pass as-is. helexa-stream tests prove chunk-for-chunk incremental delivery, observer finish-once, usage extraction, and non-2xx passthrough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 18:05:35 +03:00
rob thijssen	881fc85a4c	feat(#70): helexa-router binary skeleton — plaintext axum server ... Foundation for epic #69 (public multi-operator ingress proxy). New `crates/helexa-router` workspace binary: a plaintext axum server that reuses cortex-core types and serves the two stub endpoints the rest of #69 builds on. - `[router] listen` + `[[cortexes]]` config via figment + `HELEXA_ROUTER_` env overrides, matching the cortex/neuron convention. - `GET /health` reports the configured downstream cortex count. - `GET /v1/models` returns an empty OpenAI list (real cross-operator aggregation is #75). - No inbound TLS listener (edge nginx terminates client TLS per #69's posture); no auth layer — the router forwards the client bearer to cortex and holds zero entitlement logic (#47 stays additive). - 3 tests: both endpoints over a real ephemeral-port server, plus TOML+env config load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh	2026-06-21 17:52:28 +03:00