rob thijssen
7df84fed8f
All checks were successful
CI / CUDA type-check (push) Successful in 32s
build-prerelease / Resolve version stamps (push) Successful in 30s
CI / Format (push) Successful in 28s
CI / Clippy (push) Successful in 2m35s
build-prerelease / Build cortex binary (push) Successful in 5m13s
build-prerelease / Build neuron-blackwell (push) Successful in 6m23s
build-prerelease / Build neuron-ampere (push) Successful in 7m56s
CI / Test (push) Successful in 7m11s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Package cortex RPM (push) Successful in 1m19s
build-prerelease / Build neuron-ada (push) Successful in 5m30s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 2m56s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m45s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 4m25s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 1m1s
Stage A of the vision implementation plan (doc/vision-qwen3_6-spec.md). Builds the vision tower scaffolding that today's silent-drop failure mode (issue #3) needs — the Qwen3.6 ViT loads from `model.visual.*`, runs forward producing post-merger LM-side image embeddings, and routes through the device worker via a new `Job::EncodeImage`. No LM splice yet — that's Stage B. Refs #3 (umbrella). Deferred sub-stages tracked as #12 (TP-vision), #13 (27B production deploy), #14 (dynamic resolution), #15 (numerical validation). What landed: - **A0 — investigation**: pulled config.json, preprocessor_config.json, chat_template.jinja, and safetensors index from beast's local Qwen3.6-27B cache. Documented in doc/vision-qwen3_6-spec.md with exact tensor shapes for every `model.visual.*` weight. Confirms 27-block ViT with `hidden_size=1152`, `patch_size=16`, `spatial_merge_size=2`, `out_hidden_size=5120`. Vision tower lives in 2 of the 15 safetensors shards. - **A1 — deps + scaffolding**: added `image = "0.25"` (default- features off, PNG/JPEG/WebP/BMP/GIF) and `base64 = "0.22"` to crates/neuron/Cargo.toml. Created `harness::preprocess` and `harness::arch::qwen3_5::vision` modules. - **A2 — preprocess.rs**: `decode_data_uri` strips `data:image/...;base64,...` → image bytes → `image::DynamicImage` (rejecting `http(s)://` URLs to avoid SSRF/recursion); `preprocess` resizes to a fixed `PreprocessProfile::qwen3_6()` (448×448), normalises to `[-1, 1]` per the model's mean/std=0.5, emits row-major `(3, H, W)` f32. 9 unit tests covering data URI parse, decode failure paths, grayscale-to-RGB promotion, and the exact-value normalisation contract. - **A3 — vision.rs**: `VisionTower` struct with `patch_embed: Conv2d`, learned `pos_embed: Embedding`, 27 `VisionBlock`s (pre-LN + multi-head self-attention with fused QKV + GELU-tanh MLP + residuals), and `VisionMerger` (LayerNorm → 2×2 spatial concat → linear_fc1 → GELU-tanh → linear_fc2 to LM hidden_size). Includes the Conv3d→Conv2d fold trick documented at the top of the file — the published patch_embed.proj.weight is 5D `(1152, 3, 2, 16, 16)` but candle 0.10 has no Conv3d; for static images we sum-collapse the temporal axis. Video would need real Conv3d. 5 unit tests including the exact `gelu_pytorch_tanh` reference values from PyTorch. - **A4 — wire vision into Qwen3_5ForCausalLM**: extended `Config` with optional `vision_config: Option<VisionConfig>` and `image_token_id`; `Qwen3_5ForCausalLM::new` now loads the vision tower when present, exposes `has_vision()` and `vision()` so the HTTP layer can advertise capability and so the encode path can reach it. - **A5 — device worker `Job::EncodeImage`**: new job variant carrying CPU-side `(C, H, W)` pixels. Dispatch handler reconstructs the tensor on the worker's device, calls `arch.encode_image(image)`, copies the result back to CPU as flat `Vec<f32>`. Keeps the "tensors don't escape the worker" invariant. Poisoned-worker drain path handles the new variant. - **A6 — dispatch round-trip test**: `encode_image_routes_to_dispatch_ and_errors_on_unknown_handle` proves the channel/dispatch wiring works end-to-end via the CPU device worker (errors on unknown ArchHandle, which is the expected behaviour without a loaded model — real-weights validation happens in Stage B when the LM splice path exists). CI gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (all 28 test groups ok, zero failures). New test counts: +9 in preprocess, +5 in vision, +1 in device_worker. Out of scope (deferred): - LM-side splice of image embeddings at `<|image_pad|>` positions → Stage B. - Streaming SSE for vision-bearing chat completions → Stage C. - Reject `image_url` with HTTP 400 for non-vision models / advertise `capabilities` in /v1/models → Stage C. - TP-vision (#12), 27B production deploy (#13), dynamic resolution (#14), numerical validation (#15). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
115 lines
4.1 KiB
TOML
115 lines
4.1 KiB
TOML
[package]
|
|
name = "neuron"
|
|
version.workspace = true
|
|
edition.workspace = true
|
|
license.workspace = true
|
|
|
|
[lib]
|
|
name = "neuron"
|
|
path = "src/lib.rs"
|
|
|
|
[[bin]]
|
|
name = "neuron"
|
|
path = "src/main.rs"
|
|
|
|
[features]
|
|
default = []
|
|
# Enables CUDA acceleration in candle and the cudarc/nccl bindings the
|
|
# TP worker pool uses. Without this feature, candle compiles for CPU
|
|
# only, Device::new_cuda calls fall back to CPU, and TP Init/sanity
|
|
# requests return Error{kind="cuda_feature_not_enabled"}.
|
|
cuda = [
|
|
"candle-core/cuda",
|
|
"candle-core/nccl",
|
|
"candle-nn/cuda",
|
|
"candle-transformers/cuda",
|
|
"dep:cudarc",
|
|
"dep:half",
|
|
"dep:cudaforge",
|
|
]
|
|
# Use cuDNN for convolution / attention kernels. Requires CUDA.
|
|
cudnn = [
|
|
"cuda",
|
|
"candle-core/cudnn",
|
|
"candle-nn/cudnn",
|
|
"candle-transformers/cudnn",
|
|
]
|
|
# FlashAttention kernels. Requires CUDA.
|
|
flash-attn = [
|
|
"cuda",
|
|
"candle-transformers/flash-attn",
|
|
]
|
|
# Reserved for GPU-only integration tests in later stages.
|
|
cuda-integration = ["cuda"]
|
|
|
|
[dependencies]
|
|
cortex-core.workspace = true
|
|
tokio.workspace = true
|
|
axum.workspace = true
|
|
serde.workspace = true
|
|
serde_json.workspace = true
|
|
reqwest.workspace = true
|
|
tracing.workspace = true
|
|
tracing-subscriber.workspace = true
|
|
anyhow.workspace = true
|
|
async-trait.workspace = true
|
|
clap.workspace = true
|
|
thiserror.workspace = true
|
|
futures.workspace = true
|
|
tokio-stream.workspace = true
|
|
figment.workspace = true
|
|
toml.workspace = true
|
|
|
|
# candle for in-process inference. CUDA support is gated behind the
|
|
# crate's `cuda` feature (default off) so the workspace builds on
|
|
# non-CUDA hosts and CI runners.
|
|
candle-core = "0.10.2"
|
|
candle-nn = "0.10.2"
|
|
candle-transformers = "0.10.2"
|
|
# Direct dep on cudarc (matching candle's transitive version) so the
|
|
# TP worker pool can call cudarc::nccl::{Comm, Id} directly. Gated on
|
|
# the `cuda` feature; same toolchain requirement as candle's CUDA path.
|
|
cudarc = { version = "0.19", optional = true, default-features = false, features = ["nccl", "cuda-version-from-build-system"] }
|
|
# Used by the AllReduce CustomOp1 to type-dispatch on bf16/f16 candle
|
|
# storages. Matches candle-core's pinned major version to avoid double-
|
|
# compiling the `half` crate at conflicting versions.
|
|
half = { version = "2.5", optional = true }
|
|
tokenizers = { version = "0.22", default-features = false, features = ["onig"] }
|
|
hf-hub = { version = "0.4", features = ["tokio"] }
|
|
# Jinja-compatible template renderer for the model's
|
|
# `tokenizer_config.json::chat_template`. Hugging Face's chat
|
|
# templates use a strict subset of Jinja2 that minijinja supports
|
|
# out of the box. ~80KB compiled; pure Rust, no async surface.
|
|
# Features: `builtins` for the `is defined` / `default` filters HF
|
|
# templates use; `json` for `tojson` (some Qwen3 templates emit
|
|
# tool definitions via tojson); `serde` so we can hand it a
|
|
# serde_json::Value as the context.
|
|
minijinja = { version = "2", features = ["builtins", "json", "serde"] }
|
|
# Direct dep on `safetensors` (re-exported by candle but its `TensorView`
|
|
# / `slice::IndexOp` types are public-but-not-re-exported). Used by the
|
|
# tp `fused_load` module to read per-rank slices of fused QKV tensors
|
|
# without materialising the full tensor on device.
|
|
safetensors = "0.7"
|
|
# Vision capability for Qwen3.6 (Stage A of the vision plan in
|
|
# doc/vision-qwen3_6-spec.md). `image` decodes PNG/JPEG/etc from
|
|
# the bytes embedded in `data:image/...;base64,...` content parts;
|
|
# `base64` does the URI decode. Default-features off on `image` to
|
|
# avoid pulling in audio/video formats we don't need.
|
|
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "webp", "bmp", "gif"] }
|
|
base64 = "0.22"
|
|
|
|
[dev-dependencies]
|
|
tokio = { workspace = true, features = ["test-util"] }
|
|
reqwest.workspace = true
|
|
tempfile = "3"
|
|
|
|
[build-dependencies]
|
|
# Used by `build.rs` to compile `src/cuda/*.cu` into `libneuroncuda.a`
|
|
# under the `cuda` feature. Matches mistralrs's upstream build setup
|
|
# (their `mistralrs-core/build.rs` uses the same constructor).
|
|
cudaforge = { version = "0.1", optional = true }
|
|
|
|
[package.metadata.docs.rs]
|
|
# Skip the CUDA path on docs.rs (it lacks nvcc).
|
|
no-default-features = true
|