rob thijssen
140f9124ee
All checks were successful
CI / Format (push) Successful in 43s
CI / CUDA type-check (push) Successful in 2m35s
CI / Clippy (push) Successful in 2m46s
CI / Test (push) Successful in 5m54s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 43s
CI / CUDA type-check (pull_request) Successful in 1m53s
CI / Clippy (pull_request) Successful in 2m57s
CI / Test (pull_request) Successful in 7m42s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
The six perf scenarios measure speed/resources; this measures the axis they miss — reasoning/planning quality — so the frontier A/B (F3) can pick on capability, not just throughput. Per the chosen approach: store the artifact always, with schema for BOTH a manual score and a future LLM-judge; start manual. - scenario: CapabilityScenario (capability:<name>) runs a fixed prompt and captures the full output text (stream_and_measure gains a capture_text path); opt-in via config.capability_probes (empty default — long outputs, deliberate). - store: three additive columns (artifact, quality_score, scorer); capability_runs(unscored_only) worklist + set_score(id, score, scorer). Drill-down RunRow omits the large artifact column. - cli: `helexa-bench score --id <n> --score <x> [--scorer ...]` (manual); `report --capability` (per-model median score + per-run artifact snippets); GET /api/capability. LLM-judge deferred (schema ready). - example config documents an implementation-planning probe. Tests: artifact storage + scoring lifecycle, capability scenario built from config, capability markdown (median + snippet). Part of the Performance observability epic (#83), O7 — completes the milestone. Feeds the F3 frontier A/B decision gate (#94). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrJ4i3pfLRSTM76o3ofnVq