321 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
rob thijssen	cdf87284af	feat(#47 phase 1d): budget enforcement — hard caps, reserve→settle, 429 ... Stage 1 complete: the A0 seatbelt (#52). Flips the metering-only reserve(0) from #51 to the request's real upper-bound cost and refuses over-cap requests *before* neuron is hit. - metering::reservation_estimate: prompt estimate (~4 chars/token over the body — cortex has no tokenizer, so a conservative over-estimate; neuron stays the exact context wall) + max output. Max output comes from max_completion_tokens / legacy max_tokens, else the model's advertised limit.output (#62), else FALLBACK_MAX_OUTPUT. Over-reserving is safe — settle reconciles to actual. - metering::reserve_or_reject: reserve the estimate; on BudgetError map to the #63 envelope and the caller refuses before dispatch — rolling window → 429 rate_limit_exceeded + Retry-After (until reset); hard balance → 429 insufficient_quota (no Retry-After). Never 402. - Wired into both the OpenAI proxy path (proxy_with_metrics) and the Anthropic path (estimate from the translated body). advertised_output_limit reads the loaded model's limit.output from fleet state. - Reservation prevents overshoot under concurrency: a successful reserve gates on spent+reserved+estimate ≤ cap, and settle records actual ≤ reserved, so spend can never exceed the hard cap. 4 integration tests with a hit-counting mock neuron: balance over-cap → 429 insufficient_quota (no Retry-After, not dispatched); rolling over-cap → 429 rate_limit_exceeded + Retry-After (not dispatched); within-cap served; **A0 repro** — a capped key's 20-request fan-out drains the cap, then is refused, neuron only saw the served ones, and spend never exceeds the cap. Plus 5 metering unit tests. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:35:04 +03:00
rob thijssen	4f16b8c541	feat(#47 phase 1c): per-request token metering + spend ledger ... Stage 1 accounting (#51): capture real per-request usage and feed it to the spend ledger + per-principal metrics. Establishes the reserve→settle lifecycle that budget enforcement (#52) will tighten. - cortex-gateway::metering: ReservationGuard makes reservation leaks impossible — settle() records actual spend + releases the remainder; dropping an un-settled guard releases the whole reservation, so any early return / error / dropped stream resolves it. UsageSink is the completion hook; principal_from_headers reconstructs the principal from the middleware-stamped headers (uniform across all proxy paths, no handler-signature churn); record_spend emits per-principal counters. - proxy::TokenMetrics gains an optional usage_sink, invoked exactly once in finish() with the observed (prompt, completion) — restructured so it always runs (even when no body/usage arrived → settle 0 → release), while preserving the existing per-model metric emissions unchanged. - All proxy paths metered: chat/completions/responses via proxy_with_metrics (reserve 0 → forward_request → settle in finish); Anthropic non-streaming settles from the buffered body; Anthropic streaming (anthropic_sse) now scans the upstream frames for the usage object (#48) — it captured none before — and settles at pump end. - This phase reserves 0 tokens (metering only, no enforcement); #52 flips the reserved amount to prompt+max_output and surfaces BudgetError. The settle/release plumbing is identical, so that change is localized. - New Prometheus counters: cortex_spend_tokens_total (+ prompt/completion splits), labelled by account/key. 2 integration tests: cumulative per-key spend after N requests with reservations settled to zero outstanding; anonymous requests record no spend. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:29:51 +03:00
rob thijssen	486d7e9a8f	feat(#47 phase 1b): API-key auth + principal resolution ... Stage 1 identity (#49): cortex now knows who a request is for. Identity rides standard bearer auth only (Authorization: Bearer <key>) — no custom required headers or body fields — which is what keeps every tier OpenAI-compatible by construction. - cortex-gateway::auth: `require_principal` axum middleware (from_fn_with_state), wired in build_app outer-to-inner as trace → CORS → auth → handlers (CORS outer so preflight short-circuits). It resolves the bearer key via the EntitlementProvider, inserts the typed Principal into request extensions (for metering #51 / enforcement #52), and stamps internal x-helexa-account-id / x-helexa-key-id headers so the principal reaches neuron, which trusts cortex over WireGuard (#54). - Anti-spoofing: client-supplied principal headers are stripped before the authoritative value is stamped — a client can never assert a principal it didn't authenticate as. - Rejection contract (#63): missing key under require_auth, or any present but unresolvable key, → 401 invalid_api_key in the #60 envelope. /health and / stay public. require_auth=false (default) allows anonymous through but still 401s a present-but-invalid key. - Header-name constants (HEADER_ACCOUNT_ID/KEY_ID) live in cortex-core so neuron (#54) shares them. The chat/completions/responses paths forward the stamped headers automatically via proxy::forward_request; the Anthropic streaming + non-streaming paths forward them explicitly via auth::forward_principal_headers (they build their own upstream requests). 5 integration tests: missing-key 401, invalid-key 401 (even when auth not required, not dispatched), valid key reaches neuron with principal headers + spoofed header stripped, anonymous allowed when not required, /health public. Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:07:10 +03:00
rob thijssen	bc74e0e95f	feat(#47 phase 1a): EntitlementProvider trait + local/static provider ... Stage 1's build seam (#50): the interface auth, metering, and budget enforcement all hang off, with a local/static provider so the A0 amplification fix can land before any upstream clearing house exists. The future helexa-upstream client (#57) is just another impl. - cortex-core::entitlements: Principal {account_id, key_id}, CapWindow (Balance | Rolling{seconds}), Reservation handle, BudgetSnapshot, AuthError/BudgetError, and the async EntitlementProvider trait (resolve / reserve / settle / release / snapshot). BudgetError carries the window semantics so callers pick the #63 code (rate_limit_exceeded + Retry-After vs insufficient_quota) without the provider touching HTTP. - cortex-core::config: [entitlements] section on GatewayConfig (require_auth + [[entitlements.keys]] with account_id, optional key_id, hard_cap, window). Additive + serde(default) — anonymous/uncapped when omitted, so existing setups are unaffected. - cortex-gateway::entitlements_local: LocalEntitlementProvider. Budget math serialized under one Mutex so spent+reserved can never exceed a hard cap under concurrency (the #52 guarantee); rolling windows reset lazily; uncapped keys (no hard_cap) always reserve but still meter. - CortexState gains Arc<dyn EntitlementProvider> + require_auth, built in from_config. Not yet consumed by the request path — auth middleware is 1b (#49), enforcement is 1d (#52). - cortex.example.toml documents the section; test GatewayConfig literals updated for the new field. 6 provider unit tests (resolve, unknown-key, round-trip, balance/rolling over-cap codes, uncapped infra key). Local fmt/clippy/test all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 19:00:05 +03:00
rob thijssen	f22d83df14	feat(#47 phase 0): centralize OpenAI error envelope + add Retry-After ... The rejection contract (#63) requires every "no" path to speak the OpenAI envelope with standard codes and, for retryable conditions, a Retry-After header. Two gaps remained despite #63 being closed: Retry-After was implemented nowhere, and the envelope was hand-built inline in four places (gateway handlers/proxy/router, neuron api) with no shared source of truth — exactly the inconsistency #63 set out to prevent, and a foundation every Stage 1-2 rejection (401/429/503) needs. - cortex-core: new `error_envelope::OpenAiError` — an axum-agnostic builder carrying status, type, code, message, param, optional retry_after, and diagnostic extras. Named constructors encode the #63 codes (invalid_api_key, rate_limit_exceeded, insufficient_quota, context_length_exceeded, service_unavailable) and which carry Retry-After. cortex-core stays a pure types crate; each HTTP crate owns a thin `envelope_response` adapter that sets the header. - cortex-gateway: route error_response, ProxyError, and RouteError through the shared builder; RouteError::retry_after_secs wires Retry-After on the transient NoHealthyNodes (5s) / ModelRecovering (2s) variants. - neuron: route inference_error_response through the shared builder; InsufficientVram (transient 503) now advertises Retry-After: 5. Behaviour for existing paths is unchanged (same status/type/code/extras); only the new Retry-After headers are added. Tests cover the builder wire shape and Retry-After presence/absence on both sides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 18:46:56 +03:00
rob thijssen	4b28a64b34	feat(#67 phase 5b): enforce the derived input as the prompt cap ... The request path now rejects prompts above the model's self-derived input budget, not the static NEURON_MAX_PROMPT_TOKENS — so a VRAM-tight host (where the VRAM ceiling binds below the static cap) rejects an over-budget prompt up front instead of accepting it and OOMing mid-prefill. - derived_input_cap: AtomicUsize on LoadedModel + TpLoadedModel; refreshed by LoadedHandle::derived_limit (runs on every /models poll). 0 = not derived yet. - effective_prompt_cap(): cached derived input when >0, else the static max_prompt_tokens() (cold-start / no-profile fallback). - validate_request takes the cap as a param; all 4 call sites (chat_completion, inference_stream, inference_tp_stream, TP chat_completion) pass the in-scope model's effective_prompt_cap(). - doc/context-limits.md: enforcement note updated from "remaining" to landed. Reads the cap lock-free from the sync validate path (no per-request VRAM query); the cap tracks live state via the poll-driven derivation. With this, advertise and enforce agree and both track the resident model. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:26:37 +03:00
rob thijssen	dd65eedb24	feat(#67 phase 5a): NEURON_MAX_PROMPT_TOKENS becomes a clamp-only backstop; docs ... Demotes the static per-host prompt cap from authority to an optional upper-bound clamp on the self-derived limit, and rewrites the context-limits doc around the computed model. - max_prompt_tokens_clamp(): reads NEURON_MAX_PROMPT_TOKENS directly so "explicitly set" is distinct from the 16384 default; returns None when unset (no clamp). Applied as derive_limit's hard_ceiling in LoadedHandle::derived_limit, so the advertised context is clamped only when an operator set a backstop — the derivation is otherwise authoritative and binds below it in practice. - doc/context-limits.md: intro + "After #62" rewritten as "After #67 — the neuron computes its own limit" (formula, live signals, config block, opencode note, NEURON_MAX_PROMPT_TOKENS demotion). Remaining (phase 5b, follow-up): enforce the *derived* input as the prompt cap (reject above computed input, not the static NEURON_MAX_PROMPT_TOKENS) so VRAM-tight hosts can't accept an OOM-inducing prompt. Needs a per-model cached cap read from the sync validate path; scoped separately. Until then the static cap remains the enforced backstop (advertised <= enforced holds when the env is set). fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:14:34 +03:00
rob thijssen	8b2e01a072	feat(#67 phase 4): advertise neuron-computed limit on /models; drop catalogue override ... The neuron now self-derives and advertises limit{context,input,output} per loaded model; cortex forwards it and stops consulting the operator-declared catalogue limit (which can't track hot-swapped models or live capacity). Operator-set `cost` still flows from the catalogue. neuron: - CandleHarness gains context_limit_cfg (from [harness.candle.context_limit]). - LoadedHandle::derived_limit(): profile + live tightest-card free VRAM (single: query_vram; TP: query_vram_tightest_free_mb) + prefill-rate EMA (bootstrap until first sample) → derive_limit. None for arches without a context profile. No operator clamp here (advertise the honest derived value; the clamp is an enforcement-side backstop). - list_models() fills ModelInfo.limit from derived_limit (was None). - derive_limit treats free_tightest_mb == 0 (unknown/CPU sentinel) as "no VRAM ceiling" instead of collapsing to zero. cortex: - ModelEntry gains `limit`, copied from ModelInfo.limit by the poller. - /v1/models: catalogue `limit` no longer flows (Pass 1 sets None); Pass 2 adopts the neuron's limit, taking the tightest across neurons via tightest_limit(). cost unchanged. - model_limits.rs rewritten: catalogue limit (999999) is ignored; the neuron's ModelEntry.limit is advertised; cost still from catalogue. - All ModelEntry literals updated with the new field. fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:10:20 +03:00
rob thijssen	464b6b0db9	feat(neuron): self-measured prefill tok/s EMA on streaming paths (#67 phase 3) ... Refs #67. Feeds the throughput ceiling a live, per-model prefill rate instead of only the configured bootstrap estimate, so the advertised limit tracks real prefill speed and rises automatically as prefix caching (#11) reduces effective prefill cost. - context_limit::PrefillRateEma: lock-free f64-bits EMA (alpha 0.3), ignores degenerate samples, None before the first sample. Unit-tested. - prefill_rate field on LoadedModel + TpLoadedModel. - Recorded as total-prompt-tokens / prefill-elapsed in the two streaming serving paths (TP: inference_tp_stream via tp_for_task; single-GPU: stream_inference_via_worker via a new &prefill_rate param threaded from loaded_for_task). Measuring total prompt (not just the divergent suffix) means a prefix-cache hit shrinks elapsed while the prompt stays large, so the effective rate — and the ceiling — rises toward the VRAM ceiling, exactly the #11 payoff. Per the agreed scope, non-streaming + CPU paths fall back to the bootstrap estimate (opencode streams; those paths rarely carry the fleet). fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 14:02:02 +03:00
rob thijssen	f2e05d96ec	feat(neuron): capture ContextProfile at load + per-rank VRAM fan-out (#67 phase 2) ... Refs #67. Captures the per-model context physics at load and adds the live free-VRAM signal the derivation needs — the tightest card across TP ranks, not just the leader. - ContextProfile captured at load: - single-GPU dense CUDA path (world_size 1) via context_limit::profile_from_qwen3_5_config(config_path, ..); - TP path (world_size = tp_size) at TpLoadedModel construction. GGUF/CPU/non-qwen3_5 → None (fall back to the static prompt cap). New `context_profile` field on LoadedModel + TpLoadedModel. - profile_from_qwen3_5_config(): reads config.json (mirrors VisionMeta::from_config_path), counts full_attention layers (layer_types authoritative, full_attention_interval fallback), builds the per-card KV cost via the shared helper. - Folded the inline per-rank KV-bytes math in tp_qwen3.rs (both cuda/non-cuda log_construction_complete) and tp_qwen3_5.rs onto context_limit::kv_bytes_per_token + KV_CACHE_DTYPE_BYTES. - Per-rank VRAM fan-out (tightest card): - WorkerRequest::QueryVram + WorkerResponse::VramInfo { free_mb, total_mb }; - worker.rs handle_query_vram (cuda: mem_get_info; non-cuda: error); - WorkerPool::query_vram_tightest_free_mb fans out to every rank (leader via its device worker, subprocess ranks via RPC) → min free; - TpLoadedModel::query_vram_tightest_free_mb convenience wrapper. No advertise/enforce yet (phases 4/5). fmt/clippy/test green; CUDA paths type-checked in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 13:18:27 +03:00
rob thijssen	4f05a87449	feat(neuron): self-derived context-limit core — physics + policy (#67 phase 1) ... Refs #67. The correct limit{context,input,output} for a deployment is a computed function of model architecture + live free VRAM + a coherence/throughput trade-off, not an operator-declared static fact that goes stale on model swap. This lands the arch-agnostic derivation core; later phases capture per-model physics at load, measure throughput, and advertise/enforce the computed limit. - crates/neuron/src/harness/context_limit.rs (new): - kv_bytes_per_token(): shared per-card KV cost (counts only full-attention layers; sharded by TP world size). The TP load paths' inline math folds onto this in phase 2. - ContextProfile: per-model physics snapshot (max_position_embeddings, kv_bytes_per_token_per_card, world_size). - derive_limit(): context = min(max_pos, vram_ceiling, throughput_ceiling) clamped by an optional backstop; input = context − output; rounded to 1024. 6 unit tests. - config.rs: [harness.candle.context_limit] block (mirrors prefix_cache): target_prefill_latency_secs, bootstrap_prefill_tok_per_sec, activation_headroom_mb, min_free_floor_mb, output_reserve_tokens. - neuron.example.toml: documented the new block. No runtime behaviour change yet. fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 13:00:52 +03:00
rob thijssen	2f67d17ec7	feat(neuron): emit reasoning_tokens usage details on streaming ... Closes #64. opencode meters reasoning tokens separately via the OpenAI-standard detail objects, which neuron's usage structs didn't expose. Add them additively so older clients ignore them. - cortex-core: Usage gains completion_tokens_details/prompt_tokens_details; ResponsesUsage gains output_tokens_details/input_tokens_details. Optional + skip_serializing_if, so the wire shape is unchanged for non-reasoning models. cached_tokens fields are defined but always None until prompt caching lands (#11). - candle.rs: count tokens generated while in_reasoning across all three streaming paths (TP, worker, CPU); carry the count on InferenceEvent::Finish. - chat projector: populate completion_tokens_details.reasoning_tokens. - responses projector: wire up base usage emission on the streaming path (it emitted none before) and add output_tokens_details.reasoning_tokens. - non-streaming paths leave details None (they don't track in_reasoning). reasoning_tokens is a sub-count of completion/output tokens (OpenAI semantics) — not added into total_tokens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 12:04:05 +03:00
rob thijssen	11b2e6f78c	fix(cortex): default models_config to the packaged absolute path ... cortex resolved the catalogue path "models.toml" relative to the service's working directory, so the systemd-launched binary never found /etc/cortex/models.toml and ran with an EMPTY catalogue in production — limits, cost, pinning, aliases and feasibility were all silent no-ops, with models surfacing only via the neuron poller. Tests never caught it because they pass models_config explicitly; only the defaulted, packaged path was broken. Default to the absolute /etc/cortex/models.toml (where cortex.spec installs it) and document the override in cortex.example.toml. Restores the #62 limit/cost advertisement (the catalogue is now actually read) along with pinning/aliases/feasibility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 10:04:29 +03:00
rob thijssen	8a636c687f	feat(cortex): per-model limit + cost on /v1/models; remove max_model_len ... Resolves #62. opencode's helexa provider discovers a model's serving budget from /v1/models and uses it to size context, trigger compaction, and show spend with no hand-configuration. Each model entry now carries: - limit { context, input?, output } — operator-declared in models.toml - cost { input, output, cache_read?, cache_write? } — USD per 1M tokens - tool_call / reasoning — runtime-detected by the candle harness and OR-ed in from each serving neuron Composition: the catalogue profile supplies limit/cost (Pass 1); the poller carries the neuron's detected tool_call/reasoning into ModelEntry, which the gateway unions onto the entry (Pass 2); aliases propagate every field (Pass 4). Wire types extend ModelInfo / ModelProfile / CortexModelEntry additively (serde default + skip_serializing_if), so older neurons and clients are unaffected. helexa-bench's ModelInfo constructor and the gateway test fixtures are updated for the new fields. Adds tests/model_limits.rs asserting /v1/models surfaces limit + cost (catalogue) and tool_call + reasoning (runtime), and that max_model_len is gone. Removes max_model_len. It was write-only with no consumer — opencode's source references it nowhere and it is not an OpenAI /v1/models field — and doubly misleading: vLLM's max_model_len means total sequence length, but cortex populated it from NEURON_MAX_PROMPT_TOKENS, a prompt-only cap. The limit{} contract replaces it. The neuron's max_prompt_tokens remains the enforced prompt cap (neuron-side); cortex just stops re-advertising a derived, mis-named copy. Closes #66 — its stale-max_model_len premise is moot once the field is gone. limit/cost are operator-declared (catalogue) per #62's design; auto- deriving the advertised budget from each neuron's reported cap is a tracked follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 09:26:55 +03:00
rob thijssen	6088830e7d	feat(deploy): manage NEURON_MAX_PROMPT_TOKENS per host via model.conf drop-in ... Roll the per-model context cap into deploy.yml so it is deterministic per host and rolled out (with a restart) alongside the rest of the service config, rather than hand-edited in local.conf. The deploy now writes /etc/systemd/system/neuron.service.d/model.conf from a new per-host `max_prompt_tokens` matrix field, and restarts a neuron when the package OR the drop-in changes — so a cap change applies even with no new RPM. beast (Qwen3.6-27B, hybrid linear, 2x 32GB) -> 131072 (~128k); benjy and quadbrat (dense, VRAM-bound) stay at 16384 but become deploy-managed. Adds the scoped sudoers grant for the root-owned drop-in install, and doc/context-limits.md documenting the knob relationships and KV/VRAM math (refs #62 for the eventual /models-advertised source of truth, #65 for the length-aware text VRAM guard that gates pushing beyond 128k). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-16 18:48:19 +03:00
rob thijssen	04f798ec23	feat(cortex-gateway): enhance error responses with structured data ... fixes #63 Standardize error messages by adding type, code, and param fields to align with OpenAI API format. Updates include: - Structured error envelopes with broad type categorization (invalid_request_error/api_error) - Specific machine-readable codes (model_not_found/service_unavailable) - Null param field as required by OpenAI specification - Consistent error response formatting across handlers, proxy, and routing layers New tests verify correct error envelope structure for various failure scenarios. Co-Authored-By: Helexa (Qwen3.6-27B, 48k context) <noreply@helexa.ai>	2026-06-16 17:51:04 +03:00
rob thijssen	6f3e9276cd	docs: add AGENTS.md with project architecture, build commands, and conventions	2026-06-16 14:15:32 +03:00
rob thijssen	8f9e956d17	fix(neuron): emit OpenAI-standard nested error envelopes (#60) ... InferenceError responses were a flat `{"error": "..."}` string. OpenAI clients (opencode, the openai SDK) reach into `error.type`/`error.code` to drive behaviour — most importantly `code == "context_length_exceeded"` triggers auto-compaction + retry instead of a hard failure. A flat string is invisible to that logic. Rewrite `inference_error_response` to emit the nested envelope `{"error": {"message","type","code","param", ...diagnostics}}` and map: - ModelNotLoaded → 404 invalid_request_error / model_not_found - PromptTooLong → 400 invalid_request_error / context_length_exceeded (message: "maximum context length is N tokens", + prompt_len/max) - InsufficientVram → 503 api_error / insufficient_vram - VisionUnsupported→ 400 invalid_request_error / vision_unsupported - TemplateRenderFailed → 422 invalid_request_error / template_render_failed - Other → 500 api_error / null code Diagnostic extras ride inside the error object so the envelope shape is stable. Both inline match blocks in the chat-completions handler (streaming + non-streaming) now defer to the shared helper, which the responses handler already used — one source of truth. Adds 4 unit tests covering the envelope shape and codes. Also fixes a pre-existing clippy lint (cloned_ref_to_slice_refs) in qwen3_5 snapshot test surfaced by a newer clippy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 20:42:14 +03:00
rob thijssen	cb758d4706	feat(neuron): emit usage on the streaming path so clients can track context ... The deeper reason opencode showed "Context: 0 tokens / 0% used" and flew into a 400: streaming responses carried NO `usage`. Clients track context (and trigger compaction) from the `usage` field; the legacy candle streaming path set `usage: None` on every chunk, so a streaming client had no token count at all — `max_model_len` alone is a denominator with no numerator. InferenceEvent::Finish now carries prompt_tokens + completion_tokens (the streaming loops already have both: prompt_tokens.len() and the generated all_tokens.len()). The openai_chat projector emits an OpenAI-style trailing usage chunk (empty `choices`, populated `usage`) after the finish chunk. cortex's Anthropic stream translator already reads chunk.usage, so this fixes context tracking on BOTH the OpenAI (opencode) and Anthropic (Claude Code) paths. Also harden the max_model_len plumbing's sibling: cortex re-polls /discovery while a neuron's max_prompt_tokens is still 0 (unknown), so a rolling-deploy race where cortex caches discovery before the neuron has the field self-heals instead of pinning max_model_len to None until a manual cortex restart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 19:43:59 +03:00
rob thijssen	a2d2dbd006	feat: advertise max_model_len on /v1/models so clients can compact ... opencode (and any OpenAI/Anthropic client) couldn't size or compact its context against helexa because /v1/models never advertised a context window — opencode showed "0 tokens / 0% used" and flew straight into a 400 PromptTooLong once a conversation + a fetched 64KB log overflowed the 49152-token cap. Compaction is the client's job, but the client needs to know the limit to do it. neuron now reports its effective prompt cap (NEURON_MAX_PROMPT_TOKENS) in GET /discovery (`max_prompt_tokens`). cortex surfaces it on /v1/models as `max_model_len` (vLLM / OpenAI-compatible convention) per model — the smallest cap among the neurons that can serve it (feasible_on ∪ locations), so the advertised limit holds wherever the request routes. A neuron reporting 0 predates the field and is treated as unknown (skipped); models with no reporting neuron omit the field. helexa still rejects over-limit prompts with a clean 400 — this just gives clients the number to compact *before* hitting it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 19:11:13 +03:00
rob thijssen	544214d0f8	fix(neuron): normalize OpenAI string tool-call arguments before rendering ... opencode (OpenAI path, /v1/chat/completions passthrough) hit the same chat_template:120 failure Claude Code did — "cannot convert value into pairs" — because the OpenAI wire format carries tool_calls[].function.arguments as a JSON *string*, while Qwen3.6's template iterates it as a dict (`arguments | items`). The Anthropic-side fix (8880b2f) only covered cortex's translation; the OpenAI path reaches neuron unchanged. render_chat_template now normalizes string-form tool-call arguments to objects across all messages before building the Jinja context, so OpenAI and Anthropic clients both render. Object args (Anthropic path) pass through untouched; a string that doesn't parse is left as-is and the render fails loudly (422 TemplateRenderFailed, a94dd55) rather than silently dropping tools. The loud-fail change earned out immediately here: opencode got a clean 422 with the exact `chat_template:120` cause instead of a degraded session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 18:13:36 +03:00
rob thijssen	a94dd55ab8	feat(neuron): fail loud (422) when a tools-bearing request can't render ... Three of this session's bugs (system-message position, tool_call argument shape, and the original tool rendering) all hid behind the same silent behaviour: chat_template render fails → neuron falls back to format_qwen3_prompt, which drops every tool → the request still returns 200 with degraded, tool-less output. Each cost real debugging time because the failure was invisible on the wire. build_prompt_for_request now returns Result. On a render failure it checks whether the request carried tools: if so it returns the new InferenceError::TemplateRenderFailed (mapped to 422 with a template_render_failed code and the underlying Jinja error), instead of silently degrading. A render failure with no tools still falls back quietly — there's nothing to lose, and `format_qwen3_prompt` is a reasonable text-only prompt. The four prompt-build call sites propagate with `?`. Now the next client/template incompatibility surfaces as a loud 422 the operator sees immediately, not a mysteriously-degraded session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 17:48:31 +03:00
rob thijssen	8880b2f8a6	fix(cortex): emit tool_call arguments as an object so Qwen3.6 can chain tools ... Verified live via the rendered-prompt trace: once a tool call is in the conversation history, the Qwen3.6 chat template fails to render — render chat_template: invalid operation: cannot convert value into pairs (in chat_template:120) because line 120 iterates `tool_call.arguments | items` (treats arguments as a dict), while cortex emitted the OpenAI-standard JSON *string*. On that render error neuron silently falls back to a tool-less prompt, so the model loses every tool the moment it makes one call — it can make the first tool call, read the result, then can only narrate ("now let me check the runs") and stop, because the next turn has no tools. That's the "drops the ball a little later" symptom: the CC trace shows the get_me turn rendering 42653 tokens (tools present) and every subsequent tool-history turn falling back to ~6k tokens (tools gone). anthropic_to_openai now passes `function.arguments` as the parsed object rather than stringifying it. Tests updated to expect the object form. This is the same silent-fallback failure class as the system-message merge (295b10c) — which is why making neuron's template-render fallback LOUD (4xx on a tools-bearing request instead of a degraded 200) is now clearly worth doing: it would have surfaced both in seconds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 16:43:17 +03:00
rob thijssen	4e8f4e0d04	fix(neuron): don't generate <think> reasoning when the client drops it ... Verified live: Qwen/Qwen3.6-27B with a simple prompt and max_tokens=400 generated 400 tokens, finish_reason=length, and 0 visible characters — the model spent the ENTIRE budget on <think> reasoning, which we then drop for OpenAI/Anthropic clients (include_thinking=false), starving the visible answer. This is why Claude Code "dropped the ball": empty or truncated responses. A/B confirms the cause — same prompt with chat_template_kwargs.enable_thinking=false yields a full 545-char answer. The earlier prompt_opens_reasoning fix stopped the reasoning *leaking* as text but left it consuming the token budget. Couple the two: when the caller isn't going to see the reasoning (include_thinking=false, the default), default chat_template_kwargs.enable_thinking to false so the model doesn't generate it. An explicit client enable_thinking wins; thinking-aware clients (helexa-acp, x-include-thinking: true) keep reasoning on. Tests cover the default (false), surfacing (true), explicit override, and preservation of other kwargs. Note: only the /v1/chat/completions path (what Claude Code uses via cortex /v1/messages); /v1/responses could get the same defaulting as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 15:00:50 +03:00
rob thijssen	295b10c103	fix(cortex): merge all system content into one leading system message ... Verified live via neuron trace: Claude Code's real requests carry a top-level `system` AND a `role:"system"` turn inside `messages`. cortex passed the latter through at a non-first position, and Qwen3.6's chat template hard-rejects it: WARN chat_template render failed; falling back to format_qwen3_prompt error=... invalid operation: System message must be at the beginning. On that render error neuron silently falls back to a template that renders NO tools, so the model got zero tool-format guidance and improvised an unparseable `<tool><name>…` syntax — tool calling broke entirely for real CC traffic, even though synthetic single-system probes (and the earlier translation/parse fixes) worked. anthropic_to_openai now accumulates the top-level `system` plus every `role:"system"` conversation turn and emits a single system message at index 0, with the non-system turns following in order. Reproduced the trigger (system-role message at index>0 → fallback) and the fix (merged → template renders tools). Test covers the merge + ordering. Secondary hardening worth a follow-up: neuron's silent template fallback drops tools without surfacing it to the client — a render failure on a tools-bearing request should arguably 4xx rather than degrade invisibly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 14:09:08 +03:00
rob thijssen	1c485aedce	feat(neuron): trace the fully rendered chat-template prompt ... Debugging tool-call format drift (Qwen3.6-27B emitting wrapper-less <tool><name>…> under Claude Code's real system prompt + 120-tool list, which neuron's <tool_call> detector can't parse) needs ground truth on what the model actually sees. neuron logged nothing about the rendered prompt. Add a trace! in build_prompt_for_request emitting the full rendered prompt + char count + tool count, so we can see whether the chat template's <tool_call> format instruction survives a large system prompt and how the tools render. Gated at trace (the prompt can be tens of KB): RUST_LOG=neuron::harness::candle=trace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 13:38:51 +03:00
rob thijssen	b3dc835375	ci: bound job runtime + stop dropping sccache on rustc signal-death ... A neuron-blackwell build hung ~90 min (siblings finished in 2) and there was no job timeout to kill it, so it sat burning a runner. Root cause of the hang: the inline retry loop treated every failure identically and, on its final attempt, rebuilt with sccache disabled. When the real failure is a rustc SIGSEGV or an OOM-kill, an uncached rebuild does *more* work under the same memory pressure — turning one transient compiler crash into a wedged job. Two fixes: 1. timeout-minutes on every job in build-prerelease.yml and ci.yml (builds 25, neuron CUDA build/cuda-check 35, packaging 20, COPR 60, fast jobs 10-15). A hang now dies in minutes, not hours. 2. New script/ci-cargo-escalate.sh replaces the five (prerelease) + three (ci) inline escalation loops. It classifies the failure: - signal death (exit >=128, or cargo reporting `signal: N`/SIGSEGV/ SIGKILL) → compiler crash, NOT an sccache fault: keep the cache, one warm retry, then fail fast. Never escalate to uncached. - sccache fault (recognisable sccache error) → restart the server, retry, then one final uncached attempt. - deterministic compile/test error → fail fast (no wasteful retry). It also folds in the CUDA-image sccache probe the neuron/cuda-check jobs did inline. Classification verified locally against success, plain failure, exit-139, and the cargo-wrapped `signal: 11` form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 13:02:50 +03:00
rob thijssen	746d84c0fb	fix(neuron): seed in_reasoning from the prompt so Qwen3.6 thinking isn't leaked ... Qwen3.6's chat template injects the opening <think> into the generation prompt, so generation begins mid-thought and the open marker is never sampled. The streaming loops flipped in_reasoning to true only on a *generated* open token, so they stayed in text mode and streamed the model's reasoning out as visible text — verified live: a tool request returned a 255-char text block of chain-of-thought ("The user wants to know the weather… I will construct the function call now.") ahead of the tool_use block, with the trailing </think> stripped (close token recognised) but no opening <think>. Each streaming loop now seeds in_reasoning by replaying the prompt's reasoning markers (new `prompt_opens_reasoning`): if the prompt ends inside an open <think>, the loop starts in reasoning mode, the thinking routes to ReasoningDelta (dropped by the chat projector's default include_thinking=false, which is what cortex uses), and the model's </think> flips back to visible text for the answer/tool call. Template- agnostic and self-correcting: a prompt that doesn't open reasoning (no think injection, enable_thinking off, non-reasoning model) starts false, preserving current behaviour. Thinking is hidden, not disabled, so answer quality is unaffected. Applied to all three streaming loops (inference_tp_stream, stream_inference_via_worker, run_inference_streaming). Test covers open/close replay, multi-turn closed state, reopen-at-tail, and the no-pair pass-through. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 11:03:26 +03:00
rob thijssen	f15b9e2848	fix(neuron): parse Qwen-XML tool calls + emit tool_use stop_reason ... Verified live (commit d662fa2 logs): cortex now delivers OpenAI-shaped tools to neuron correctly, but Qwen3.6-27B emits tool calls in the Qwen-XML form inside the <tool_call> markers — <tool_call> <function=get_weather> <parameter=city> Brno </parameter> </function> </tool_call> — while parse_tool_call_body only did serde_json::from_str expecting {"name":…,"arguments":…}. It returned None, the dispatch re-emitted the raw block as a text delta, and clients saw the markup as prose. cortex logged upstream_tool_calls=false finish_reason="stop". parse_tool_call_body is now format-tolerant: JSON first (Qwen3-Instruct / Hermes), then a Qwen-XML parser (Qwen3-Coder / Qwen3.6). Each <parameter> value is coerced to its declared JSON type using a new ToolSchemas map built from the request's tools (string stays string, integer/number/boolean/object/array coerced, mistyped values fall back to string so an argument is never dropped). build_tool_schemas is threaded into all three streaming loops (inference_tp_stream, stream_inference_via_worker, run_inference_streaming). Each loop also tracks emitted_tool_call and promotes the terminal finish_reason from Stop to ToolCalls when a call parsed, so the OpenAI chunk carries finish_reason:"tool_calls" and cortex maps it to Anthropic stop_reason:"tool_use" — without which an Anthropic agent (Claude Code) sees a tool_use block but stop_reason:end_turn and may not run the tool. FinishReason::ToolCalls drops its dead_code allow. Tests: JSON form still parses; Qwen-XML multi-param parse with schema-driven string/integer/boolean coercion; no-schema type sniffing; type-mismatch string fallback; unparseable body returns None. Known gap (separate): the non-streaming run_inference paths have no tool-call handling at all; Claude Code streams, so the streaming loops are the ones that matter here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 10:39:38 +03:00
rob thijssen	d662fa20ef	fix(cortex): translate Anthropic tools to OpenAI shape + wire-debug logging ... Claude Code (ANTHROPIC_BASE_URL -> cortex) hits POST /v1/messages, but anthropic_to_openai forwarded the request's `tools` array verbatim via the flattened `extra`. neuron feeds that straight into the HF chat template, which iterates the OpenAI shape (tool.function.name/.parameters). Anthropic-shaped tools ({name, description, input_schema}) rendered as broken/empty definitions, the model improvised an unparseable <tool_use_name>...</tool_use_name> tool-call format, neuron's <tool_call>{json}</tool_call> detector missed it, and the markup fell through as plain assistant text — so CC never received a structured tool_use and the agent loop died. Request-side translation now reshapes: - tool definitions: {name, description, input_schema} -> {type:"function", function:{name, description, parameters}} - tool_choice: auto->"auto", any->"required", none->"none", tool->{type:"function",function:{name}} - assistant tool_use blocks -> OpenAI assistant.tool_calls (arguments JSON-stringified) — fixes multi-turn - user tool_result blocks -> standalone role:"tool" messages keyed by tool_call_id - system content blocks flatten to text instead of being JSON-serialised into the prompt; best-effort image-block -> image_url part Wire-debug instrumentation (tracing levels only; cortex/neuron ship at info, operator infra runs at debug): - every handler emits a debug! "inbound request" line tagging the wire surface (anthropic | openai-chat | openai-responses | openai-completions) plus model/stream/tools and, for Anthropic, tool_history/system - response side reports upstream_tool_calls + finish_reason, streaming and non-streaming - full inbound + translated-upstream bodies at trace! (UTF-8-safe, capped) Tests: 8 request-side unit tests + an end-to-end gateway test asserting the upstream neuron receives OpenAI-shaped tools and a user->assistant(+tool_calls)->tool->user history. Also tighten script/infra-log-verbosity.sh: independent cortex/neuron RUST_LOG args, cortex-only by default (neuron restart behind --with-neuron so we don't needlessly cold-reload models), mkdir -p the drop-in dir, symmetric RUST_LOG cleanup, and set -euo pipefail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 09:58:25 +03:00
rob thijssen	d04f4ad704	feat(bench): show GPUs as the resource name instead of hostnames ... Public visitors don't know the hostnames, so surface each host's GPU(s) as the resource name across the UI. - store: gpu_label() turns the stored gpus_json into a compact label ("2× RTX 5090", "RTX 4090"); add `gpu` to ReportRow + RunRow and `host_gpus`/`model_gpus` maps to /api/dimensions (from each one's latest run). render_json gains gpu too. - UI: Overview + Runs show a "GPU" column (gpu, fallback host); Runs' filter is now GPU-labelled (still filters by host underneath); Trends shows a "Measured on <gpu>" line for the selected model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 16:29:13 +03:00
rob thijssen	e3879f093a	feat(bench-ui): drop host selector from Trends; resolve host server-side ... Public visitors don't know the hostnames or per-host hardware, so the host picker on Trends was confusing. Select by model + scenario only; /api/series now takes host as optional and resolves it to the host serving that (model, scenario) — coherent since each model maps to one host today. Runs (drill-down) keeps its host filter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 16:19:09 +03:00
rob thijssen	e4b9b88de0	feat(bench-ui): mark the baseline↔live regime boundary on Trends ... Add a dashed vertical ReferenceLine at the first live build (labelled "bench.py → helexa-bench") so the intentional gap between the gateway baseline and the direct-to-neuron series reads as a deliberate measurement-regime change, not missing data. The two series stay unconnected by design (different regimes, not directly comparable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 16:13:34 +03:00
rob thijssen	21db334e37	feat(bench-ui): overlay pre-helexa-bench baseline on Trends ... Option C: a curated static baseline (bench/src/baseline.ts), transcribed from doc/benchmarks.md (8f6f1d3 + a1952a4 post-#11), overlaid on the Trends charts as a dashed, clearly-labelled historical series ahead of the bench era. Host inferred from model via the doc's fleet table; ordered by snapshot time so it anchors the timeline. Kept deliberately separate from the live series (no DB/API change) — the baseline is a different regime (bench.py through the cortex gateway, medians only) so it's never merged into the direct-to-neuron line; a caption spells out the distinction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 16:02:43 +03:00
rob thijssen	7dd1ddcfba	fix(infra-setup): stat LE live dir via sudo; rsync provisioner secret for bench.internal issuance ... - cert_present() must `sudo test -d /etc/letsencrypt/live/...` (root-only 0700); without sudo it falsely reported "no cert" and downgraded the bench.helexa.ai vhost to the http-only bootstrap (dropping its 443 server). Now correctly keeps the full TLS vhost. - bench.internal initial cert: rsync the operator's JWK 'lair' provisioner password to the host transiently (root, 0600), issue via step ca certificate, then remove it (trap + belt-and-suspenders rm). Verified: bench.helexa.ai (LE) and bench.internal (lair CA) both serve the SPA + /api→bob; step@bench.timer renews; secret removed from host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:40:38 +03:00
rob thijssen	4ee7da4f97	feat(bench-ui): internal vhost bench.internal + step@ cert renewal ... Inside the WireGuard mesh, bench.helexa.ai dead-ends at the OPNsense LAN interface (only WAN :443 is port-forwarded), so add an internal path: - asset/nginx/bench.internal.conf — server_name bench.internal, internal "lair" CA cert, same SPA + /api→bob proxy. Mirrors the *.internal vhost convention on oolon.kosherinata.internal. - asset/systemd/step@.{service,timer} — replicate oolon's smallstep cert renewal (step ca renew via mTLS, every 15 min, reload nginx). - infra-setup.sh: install the step@ units + /etc/nginx/tls/{cert,key}, install the vhost + enable step@bench.timer once the cert exists; prints the one-time issuance command otherwise. Initial cert issuance (JWK provisioner) and bench.internal DNS are operator steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:34:38 +03:00
rob thijssen	db3cb95cbf	fix(infra-setup): provision bench.helexa.ai cert via Cloudflare DNS-01 (ecdsa) ... The webroot/http-01 approach needed nginx serving :80, but the gateway's nginx was dormant. Switch to the host's established convention — certbot --dns-cloudflare --key-type ecdsa with /root/.certbot-internal — which needs neither nginx nor :80, so the cert provisions independently of the vhost being served. Also restorecon the webroot (SELinux enforcing → nginx 403 without httpd_sys_content_t), and only ever install the full TLS vhost once the cert exists (http-only bootstrap otherwise) so `nginx -t` always passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:54:24 +03:00
rob thijssen	37c19aa985	feat(bench-ui): public hosting at https://bench.helexa.ai via gateway nginx ... nginx on the gateway serves the bench SPA and reverse-proxies /api to the bob bench API over WireGuard — public, auth-less, same-origin (no CORS), internal API stays private. - asset/nginx/bench.helexa.ai.conf (full TLS vhost: SPA + /api proxy) and a bootstrap http-only vhost for the initial ACME challenge. - infra-setup.sh: one-time gateway setup — webroot, Let's Encrypt cert (certbot webroot, idempotent), install + enable the vhost. - deploy.yml: deploy-bench-ui builds the SPA (setup-node) and rsyncs dist/ to /var/www/bench.helexa.ai every deploy; built same-origin so no VITE_API_BASE. - cortex-host.conf: scoped gitea_ci rsync grant for the webroot. - bench/README: production hosting notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:40:29 +03:00
rob thijssen	f50f5531cf	feat(bench): read-only JSON API on bob + bench/ React visualisation app ... Part A — helexa-bench read API: - [api] config (enabled, listen :13132); WAL on the store so API reads never block the sweep writer. - store read methods: summary, series (chronological per-build medians), runs (filtered), dimensions, run_count. - api.rs: axum /api/health|dimensions|summary|series|runs, permissive CORS (UI is a separate origin). The `run` daemon binds the API alongside the sweep; new `serve` subcommand serves API-only. - listener plumbing (bench gains a port): data/helexa-bench-firewalld.xml, spec install, deploy-bench /api/health probe + firewalld step, sudoers firewall-cmd grants, [api] in example + bob.toml. - 5 API tests + serve smoke. Part B — bench/ Vite + React-SWC-TS app (router, react-bootstrap, recharts): Overview (summary table), Trends (decode tok/s & TTFT across build SHAs), Runs (filterable explorer). Typed API client with VITE_API_BASE + dev proxy to bob. npm build/typecheck clean. Hosted separately from the API (per design); .gitignore excludes node_modules/dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 11:26:55 +03:00
rob thijssen	5999c8a5a3	Merge branch 'feat/deploy-bench-on-bob' into main ... ci(deploy): deploy helexa-bench to bob + enable all fleet services on boot Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 09:17:11 +03:00
rob thijssen	66833890c0	ci(deploy): deploy helexa-bench to bob + enable all fleet services on boot ... Adds a deploy-bench job to deploy.yml that rolls helexa-bench onto bob (the bench host, also running Agent Zero), following the deploy-cortex pattern: manifest-gated skip-when-current, light "service stays active" validation (outbound-only, no listener/model to probe), journal capture. Runs alongside the cortex→neurons chain (no deploy-ordering dependency — the sweep loop is version-aware). Boot persistence: all systemd deployments now `systemctl enable --now` instead of bare `start`, so cortex / neuron / helexa-bench come back after a host reboot. Covers deploy.yml (all three services) and deploy-dev.yml (neuron fast path); sudoers gain the matching `enable --now <svc>` grant. infra-setup.sh handles bob: provisions gitea_ci, installs the bench-host sudoers, enables the lair-cafe-unstable repo (bob is a client host without it), pre-creates /etc/helexa-bench, and syncs asset/helexa-bench/bob.toml. New assets: bench-host.conf sudoers and bob.toml (three neuron targets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 09:10:07 +03:00
rob thijssen	7bb20241a6	Merge branch 'feat/version-metadata-and-bench' into main ... feat(bench): version-aware benchmark harness + neuron build metadata Adds GET /version build metadata to neuron and the helexa-bench crate — a continuous, version-aware harness that records fleet benchmarks into SQLite keyed by neuron build SHA, replacing manual bench.py runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-13 15:33:33 +03:00
rob thijssen	42da25a37c	feat(bench): version-aware benchmark harness + neuron build metadata ... Adds automated, longitudinal performance tracking across neuron builds, replacing manual script/bench.py runs and hand edits to benchmarks.md. neuron build metadata + GET /version: - cortex-core: shared BuildInfo type (build_info.rs). - neuron build.rs captures git SHA (preferring injected HELEXA_BUILD_SHA, else git, else "unknown"), dirty flag, build timestamp, rustc version, profile, target, enabled cargo features, and best-effort candle-core version from Cargo.lock. - New GET /version endpoint (version.rs) + clap --version long form. - SHA injected in CI (build-neuron step) and helexa-neuron.spec (%{?helexa_commit}) so tarball RPMs report the real SHA. /version is now the canonical "which build is live" probe. helexa-bench crate: - Continuous daemon: hits each neuron directly on :13131, exercises each warm (status==loaded) model, records every run into a SQLite system-of-record stamped with the neuron's full BuildInfo. - Version-aware: skips any (target, build SHA, model, scenario) cell already at samples_per_version, so a steady fleet costs only cheap /version + /models polls until a new SHA ships. - Extensible Scenario trait; phase-1 chat-latency family ported verbatim from bench.py (synthetic 128/4096-tok prompts, /no_think, streamed TTFT + decode-window tok/s). `report` regenerates the benchmarks table. - kind="openai" comparison targets scaffolded, not yet wired. Packaging: data/helexa-bench.service (+ sysusers), prebuilt-binary RPM spec (outbound-only, no firewalld), and build/package/publish wiring in build-prerelease.yml with change detection. Tests: cortex-core BuildInfo round-trip, neuron GET /version integration, helexa-bench unit (prompt/SSE/config/store) + end-to-end sweep (record -> skip -> resume on new SHA). Docs updated (benchmarks.md, CLAUDE.md addendum). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-13 15:26:02 +03:00
grenade	30d50d6215	Merge pull request 'fix(ci): drop the unused flash-attn feature from neuron builds (#42)' (#46) from fix/42-drop-flash-attn into main	2026-06-13 07:15:43 +00:00
rob thijssen	9a312098dd	fix(ci): drop the unused flash-attn feature from neuron builds (#42) ... The neuron fleet builds with `cuda cudnn flash-attn`, but nothing in neuron uses flash-attn: the qwen3_5 (27B) arch is hand-rolled, the candle-transformers qwen3 model has no flash path, llama is built with use_flash_attn=false, and `grep flash crates/neuron/src` is empty. The feature only pulls in candle-flash-attn's sm_80/sm_86 CUDA kernel sweep — which is exactly where ptxas SIGSEGVs/hangs in #42 (3 hits in one day, the last a ~4-hour hang that stalled the whole deploy behind the ampere job). Dropping the feature removes the #42 failure surface at the root (not a mitigation) and cuts the longest, most fragile part of each flavour build. No runtime change — nothing called those kernels. Removed from all three flavour builds in build-prerelease.yml and from deploy-dev.yml; ci.yml's cuda-check already used `--features cuda` only. Closes #42 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-13 09:43:14 +03:00
grenade	98e9749f22	Merge pull request 'feat(neuron): speculative decoding — acceptance core + config (#25, phase 1)' (#45) from feat/25-speculative-decoding into main	2026-06-13 06:39:49 +00:00
rob thijssen	ec764a2cac	feat(neuron): speculative decoding — acceptance core + config (#25, phase 1) ... First phase of speculative decoding: the pure, state-free acceptance logic and per-target config, unit-tested in isolation before the draft/verify loop and GDN-state rollback wire it into the generation path. greedy_accept walks the drafter's K proposed tokens against the target's greedy token at each of the K+1 positions, accepting the longest matching prefix and always committing one bonus token on top (the target's correction at the first mismatch, or a free extra token when the whole draft matched). So a round commits 1..=K+1 tokens — never zero, guaranteeing forward progress even with a useless drafter. Greedy is exact for temperature-0 (the fleet probe + #22 bench regime); stochastic acceptance is a later phase. SpeculativeConfig carries the drafter id (must share the target's tokenizer — Qwen3.5-0.8B for the Qwen3.6-27B target, both qwen3_5, byte-identical tokenizer, confirmed on beast) and the draft length K. 6 unit tests: full accept, partial accept, zero accept (progress guarantee), last-position mismatch, single-token draft, config gating. Not yet wired into the decode path — phase 2 (single-GPU draft/verify) follows. Design + phasing on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-13 08:30:21 +03:00
grenade	4c1bdba31d	Merge pull request 'feat(neuron): chunk the single-GPU vision prefill (parity with TP) (#18)' (#44) from feat/18-single-gpu-vision-chunked into main	2026-06-13 05:23:07 +00:00
rob thijssen	988ef5afc2	feat(neuron): chunk the single-GPU vision prefill (parity with TP) (#18) ... The single-GPU vision path was still single-shot: a long vision-bearing prompt to a single-GPU-loaded qwen3_5 had the OOM exposure the TP path shed in fa01350 (it was only guard-rejected, never served). Mirror TpQwen3_5ForCausalLM::prefill_with_images_chunked onto the single-GPU Qwen3_5ForCausalLM: encode the image(s) once, walk the pre-expanded prompt in prefill_chunk_tokens() windows splicing the per-chunk <|image_pad|> rows, accumulate KV + GDN state across chunks via the growing offset, keep the last chunk's logits. Interleaved M-RoPE positions are computed once over the whole prompt and sliced per chunk (an image compresses the position space, so per-chunk offset arithmetic would be wrong) — so Qwen3_5Model::forward_inner gains an explicit position_ids path alongside the internal-from-grids (single-shot) and plain (text/decode) paths, plus a forward_with_positions entry point. The device-worker ForwardLogitsWithImages handler now calls the chunked method; chunk size comes from prefill_chunk_tokens() on the worker thread, so the Job/handle surface and the callers are unchanged. The shared validate_vision_prefill VRAM/KV backstop stays (TP keeps it too) — chunking bounds activation memory, not the accumulating KV cache, so the guard still does useful work. Verified on real weights (Qwen3.5-0.8B): extended the #15 vision reference test to also run the chunked path with chunk_size=64 over the 217-token prompt (4 chunks; the ~196-token image-pad run spans them). Chunked vs single-shot logits: cosine 1.000000, max_abs 0.0001; argmax matches the HF reference. The test covers all three forward_inner branches (text plain / single-shot vision / chunked vision) on a real single-GPU qwen3_5 load. Closes #18 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-13 08:17:11 +03:00
grenade	a1450789d2	Merge pull request 'docs(learnings): source-control P1 + P2 sprint learnings' (#43) from docs/learnings-p1-p2 into main	2026-06-13 04:21:11 +00:00