helexa/helexa
1
0
Fork 0
Code Issues 143 Pull Requests Actions Packages Projects Releases Wiki Activity

Vision: numerical validation against transformers reference #15

New Issue
Closed
opened 2026-06-01 13:18:45 +00:00 by grenade · 1 comment
grenade commented 2026-06-01 13:18:45 +00:00
Owner
Copy Link
Copy Source

Context

Deferred during planning of the initial vision capability (umbrella:
#3). Stage A–D ships with "loose" validation — coherent
image-grounded responses, a per-image quality benchmark — but no
rigorous numerical-correctness check against the reference Python
implementation. Refs: ~/.claude/plans/foamy-twirling-catmull.md.

Problem

The qwen3_5 arch module's own doc-comment already notes "numerical
correctness vs the reference Python is not yet validated." Adding a
vision tower stacks more hand-rolled tensor math on top of that.
Without a rigorous comparison fixture, a subtle numerical bug
(wrong RoPE base for vision attention, off-by-one in patch
position embedding, projector bias missing, etc.) could go
unnoticed and surface as gradual quality degradation rather than a
crash — exactly the failure mode that's hardest to debug.

Scope

  • Small Python harness in script/ that loads the same model via
    transformers, encodes a known image, and dumps:
    • Vision-tower output (pre-projector) for the first patch.
    • Projector output for the first patch.
    • LM-side input embeddings at image-token positions.
    • Final logits at the last text-token position for a fixed prompt.
  • Companion Rust test in crates/neuron/tests/vision_numerical.rs
    that loads the same model, replays the same image+prompt, and
    asserts the produced tensors match within 1e-3 (tunable).
  • Capture the reference dumps once and check them into a
    crates/neuron/tests/fixtures/vision/ directory rather than
    running Python in CI; document how to regenerate.

Acceptance

  • A reference fixture exists for the iteration model and for
    Qwen3.6-27B (the latter once deployment lands).
  • The Rust numerical test passes against the iteration model and
    surfaces a clear failure when a deliberately-mutated weight or
    off-by-one is introduced.
  • Documented procedure for capturing new reference dumps when the
    model is retrained or the iteration target changes.

Blocked by

Stage A–C of the vision plan. The Stage D quality benchmark is a
coarser substitute; this issue tracks the rigorous version.

References

## Context Deferred during planning of the initial vision capability (umbrella: #3). Stage A–D ships with "loose" validation — coherent image-grounded responses, a per-image quality benchmark — but no rigorous numerical-correctness check against the reference Python implementation. Refs: `~/.claude/plans/foamy-twirling-catmull.md`. ## Problem The `qwen3_5` arch module's own doc-comment already notes "numerical correctness vs the reference Python is not yet validated." Adding a vision tower stacks more hand-rolled tensor math on top of that. Without a rigorous comparison fixture, a subtle numerical bug (wrong RoPE base for vision attention, off-by-one in patch position embedding, projector bias missing, etc.) could go unnoticed and surface as gradual quality degradation rather than a crash — exactly the failure mode that's hardest to debug. ## Scope - Small Python harness in `script/` that loads the same model via `transformers`, encodes a known image, and dumps: - Vision-tower output (pre-projector) for the first patch. - Projector output for the first patch. - LM-side input embeddings at image-token positions. - Final logits at the last text-token position for a fixed prompt. - Companion Rust test in `crates/neuron/tests/vision_numerical.rs` that loads the same model, replays the same image+prompt, and asserts the produced tensors match within `1e-3` (tunable). - Capture the reference dumps once and check them into a `crates/neuron/tests/fixtures/vision/` directory rather than running Python in CI; document how to regenerate. ## Acceptance - A reference fixture exists for the iteration model and for Qwen3.6-27B (the latter once deployment lands). - The Rust numerical test passes against the iteration model and surfaces a clear failure when a deliberately-mutated weight or off-by-one is introduced. - Documented procedure for capturing new reference dumps when the model is retrained or the iteration target changes. ## Blocked by Stage A–C of the vision plan. The Stage D quality benchmark is a coarser substitute; this issue tracks the rigorous version. ## References - Plan: `~/.claude/plans/foamy-twirling-catmull.md` - Umbrella: #3 - Existing arch doc-comment note: `crates/neuron/src/harness/arch/qwen3_5/mod.rs:1-65`
grenade commented 2026-06-12 08:58:14 +00:00
Author
Owner
Copy Link
Copy Source

Posture note (2026-06-12 reframing): this issue's priority rose. The README now publicly claims every served architecture is "implemented in this repository, ported against the HuggingFace reference" — this validation harness is what makes that claim verifiable rather than aspirational. Since filing, the surface to validate has grown (M-RoPE stages 1–4, dynamic resolution #14, TP vision #12 — all landed), which strengthens the case for the fixture-based comparison described here. Natural companion to the #22 benchmark harness: #22 proves speed, this proves fidelity.

Posture note (2026-06-12 reframing): this issue's priority rose. The README now publicly claims every served architecture is "implemented in this repository, ported against the HuggingFace reference" — this validation harness is what makes that claim verifiable rather than aspirational. Since filing, the surface to validate has grown (M-RoPE stages 1–4, dynamic resolution #14, TP vision #12 — all landed), which strengthens the case for the fixture-based comparison described here. Natural companion to the #22 benchmark harness: #22 proves speed, this proves fidelity.
grenade added the p2-next label 2026-06-12 09:01:46 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No labels p2-next
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#15