rob thijssen
61adff347a
Some checks failed
CI / CUDA type-check (push) Successful in 31s
CI / Format (push) Successful in 30s
build-prerelease / Resolve version stamps (push) Successful in 48s
CI / Test (push) Failing after 1m10s
CI / Clippy (push) Successful in 2m49s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 4m25s
build-prerelease / Build neuron-blackwell (push) Successful in 5m53s
build-prerelease / Package cortex RPM (push) Successful in 1m20s
build-prerelease / Build neuron-ampere (push) Successful in 8m0s
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
Phase 2 of plan-source-aware-loader-preflight. Adds a one-RTT
placement feasibility check that runs before any device allocation,
NCCL handshake, or weight fetch. Replaces today's opaque
"fetch config.json … 404" failure mode (when an operator points
`tensor_parallel = 2` at a GGUF-only repo) with a structured
error that names the failure class and points at the fix.
What lands:
- `crates/neuron/src/harness/preflight.rs` — new module. Classifies
a repo's siblings listing into `SourceFormat` (Gguf | DenseSafetensors
| Mixed | Empty), applies the tp/quant feasibility table, returns a
`PlacementPlan` on success or a typed `PreflightError` on rejection.
`PreflightError` is `serde::Serialize` so the HTTP layer can emit
the structured shape verbatim; it's `thiserror::Error` so log lines
get a single-line Display when downcasting from anyhow. Includes
best-effort Levenshtein-nearest suggestion for malformed quant names
(the second sharp edge the HauhauCS scenario surfaced — operator
writes `q6k` against filenames containing `Q6_K_P`, and today's
matcher just says "no GGUF file matching quant").
- `CandleHarness::load_model` — calls `preflight(...)` first thing
after the "already loaded" guard, before any `ensure_device_worker`
or `resolve_*`. Failure wraps the typed error in `anyhow::Error` so
the existing trait surface is unchanged; the HTTP handler and the
startup logger downcast to recover the structured form.
- `crates/neuron/src/api.rs::load_model` handler — maps `PreflightError`
to 422 Unprocessable Entity with `{"error": {"kind": "...",
"model_id": "...", "suggestion": "..." }}`. Other failures keep
the existing 400 + free-form `format!("{e:#}")` shape.
- `crates/neuron/src/startup.rs::load_default_models` — when the
failure is a preflight rejection, log as `reason=<kind> detail=<msg>`
instead of the opaque `error=<chain>`, so journalctl on beast will
now show `reason=tp_requires_safetensors detail="repo is GGUF-only
(8 .gguf files); TP requires dense safetensors..."` instead of
`error=fetch config.json from HauhauCS/...: 404 Not Found`.
Tests:
- 18 unit tests in `harness/preflight.rs` covering classifier,
quant matching, Levenshtein, error serialization, and the full
feasibility table (gguf+tp rejected, gguf+bad-quant suggests
nearest, gguf+good-quant ok, dense+tp ok, empty rejected, mixed
prefers safetensors).
- 7 integration tests in `tests/preflight.rs` exercising the
network path through an axum mock that serves hf-hub-compatible
`/api/models/{org}/{name}/revision/main` payloads. Adds `tempfile`
as a dev-dependency for per-test cache dirs.
Out of scope (deferred to subsequent phases):
- Phase 1 (source-aware loader plumbing — `scheme:org/name` parsing,
per-scheme `SourceConfig`, cache disambiguation). Preflight runs
against the single configured HuggingFace source today; the scheme
threading lands cleanly when Phase 1 ships.
- Phase 3 (cortex catalogue source field).
- GGUF tensor-parallel loading. Preflight rejects this combination
with `TpRequiresSafetensors`; the underlying loader gap is the
separate `Helexa` curated-registry / heretic-rs conversation.
Refs #4-#9 architectural follow-up; no specific issue closed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
108 lines
3.6 KiB
TOML
108 lines
3.6 KiB
TOML
[package]
|
|
name = "neuron"
|
|
version.workspace = true
|
|
edition.workspace = true
|
|
license.workspace = true
|
|
|
|
[lib]
|
|
name = "neuron"
|
|
path = "src/lib.rs"
|
|
|
|
[[bin]]
|
|
name = "neuron"
|
|
path = "src/main.rs"
|
|
|
|
[features]
|
|
default = []
|
|
# Enables CUDA acceleration in candle and the cudarc/nccl bindings the
|
|
# TP worker pool uses. Without this feature, candle compiles for CPU
|
|
# only, Device::new_cuda calls fall back to CPU, and TP Init/sanity
|
|
# requests return Error{kind="cuda_feature_not_enabled"}.
|
|
cuda = [
|
|
"candle-core/cuda",
|
|
"candle-core/nccl",
|
|
"candle-nn/cuda",
|
|
"candle-transformers/cuda",
|
|
"dep:cudarc",
|
|
"dep:half",
|
|
"dep:cudaforge",
|
|
]
|
|
# Use cuDNN for convolution / attention kernels. Requires CUDA.
|
|
cudnn = [
|
|
"cuda",
|
|
"candle-core/cudnn",
|
|
"candle-nn/cudnn",
|
|
"candle-transformers/cudnn",
|
|
]
|
|
# FlashAttention kernels. Requires CUDA.
|
|
flash-attn = [
|
|
"cuda",
|
|
"candle-transformers/flash-attn",
|
|
]
|
|
# Reserved for GPU-only integration tests in later stages.
|
|
cuda-integration = ["cuda"]
|
|
|
|
[dependencies]
|
|
cortex-core.workspace = true
|
|
tokio.workspace = true
|
|
axum.workspace = true
|
|
serde.workspace = true
|
|
serde_json.workspace = true
|
|
reqwest.workspace = true
|
|
tracing.workspace = true
|
|
tracing-subscriber.workspace = true
|
|
anyhow.workspace = true
|
|
async-trait.workspace = true
|
|
clap.workspace = true
|
|
thiserror.workspace = true
|
|
futures.workspace = true
|
|
tokio-stream.workspace = true
|
|
figment.workspace = true
|
|
toml.workspace = true
|
|
|
|
# candle for in-process inference. CUDA support is gated behind the
|
|
# crate's `cuda` feature (default off) so the workspace builds on
|
|
# non-CUDA hosts and CI runners.
|
|
candle-core = "0.10.2"
|
|
candle-nn = "0.10.2"
|
|
candle-transformers = "0.10.2"
|
|
# Direct dep on cudarc (matching candle's transitive version) so the
|
|
# TP worker pool can call cudarc::nccl::{Comm, Id} directly. Gated on
|
|
# the `cuda` feature; same toolchain requirement as candle's CUDA path.
|
|
cudarc = { version = "0.19", optional = true, default-features = false, features = ["nccl", "cuda-version-from-build-system"] }
|
|
# Used by the AllReduce CustomOp1 to type-dispatch on bf16/f16 candle
|
|
# storages. Matches candle-core's pinned major version to avoid double-
|
|
# compiling the `half` crate at conflicting versions.
|
|
half = { version = "2.5", optional = true }
|
|
tokenizers = { version = "0.22", default-features = false, features = ["onig"] }
|
|
hf-hub = { version = "0.4", features = ["tokio"] }
|
|
# Jinja-compatible template renderer for the model's
|
|
# `tokenizer_config.json::chat_template`. Hugging Face's chat
|
|
# templates use a strict subset of Jinja2 that minijinja supports
|
|
# out of the box. ~80KB compiled; pure Rust, no async surface.
|
|
# Features: `builtins` for the `is defined` / `default` filters HF
|
|
# templates use; `json` for `tojson` (some Qwen3 templates emit
|
|
# tool definitions via tojson); `serde` so we can hand it a
|
|
# serde_json::Value as the context.
|
|
minijinja = { version = "2", features = ["builtins", "json", "serde"] }
|
|
# Direct dep on `safetensors` (re-exported by candle but its `TensorView`
|
|
# / `slice::IndexOp` types are public-but-not-re-exported). Used by the
|
|
# tp `fused_load` module to read per-rank slices of fused QKV tensors
|
|
# without materialising the full tensor on device.
|
|
safetensors = "0.7"
|
|
|
|
[dev-dependencies]
|
|
tokio = { workspace = true, features = ["test-util"] }
|
|
reqwest.workspace = true
|
|
tempfile = "3"
|
|
|
|
[build-dependencies]
|
|
# Used by `build.rs` to compile `src/cuda/*.cu` into `libneuroncuda.a`
|
|
# under the `cuda` feature. Matches mistralrs's upstream build setup
|
|
# (their `mistralrs-core/build.rs` uses the same constructor).
|
|
cudaforge = { version = "0.1", optional = true }
|
|
|
|
[package.metadata.docs.rs]
|
|
# Skip the CUDA path on docs.rs (it lacks nvcc).
|
|
no-default-features = true
|