feat(neuron): render the model's chat_template with chat_template_kwargs
Some checks failed
CI / CUDA type-check (push) Failing after 58s
build-prerelease / Resolve version stamps (push) Successful in 39s
CI / Format (push) Successful in 40s
build-prerelease / Build neuron-ampere (push) Failing after 1s
CI / Clippy (push) Successful in 2m37s
build-prerelease / Build cortex binary (push) Successful in 4m47s
CI / Test (push) Successful in 6m13s
build-prerelease / Build neuron-blackwell (push) Failing after 5m34s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
build-prerelease / Package cortex RPM (push) Successful in 1m27s
build-prerelease / Build neuron-ada (push) Failing after 7m20s
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped

Closes #9.

Replaces the hardcoded `format_qwen3_prompt` ChatML glue with
`minijinja`-driven rendering of the model's own `chat_template`
from `tokenizer_config.json`. The request's `chat_template_kwargs`
flow into the Jinja context so model-specific levers
(Qwen3's `enable_thinking: false`, etc.) actually take effect.

## Implementation

- New `harness::chat_template` module with three entry points:
  - `load_chat_template_alongside(tokenizer_json_path)` — probes
    `tokenizer_config.json` in the same hf-hub snapshot directory.
    Supports both the canonical string-form `chat_template` and
    the array-form some tokenizers ship (multi-template models).
  - `render_chat_template(template, messages, tools, kwargs)` —
    renders via `minijinja`. Messages flatten into the
    `[{role, content}]` shape HF templates iterate, with
    per-message extras (`tool_calls`, `tool_call_id`) preserved.
    `tools` and `kwargs` add into the Jinja context so templates
    that reference them work without us interpreting their shape.
  - `chat_templates_enabled()` reads `NEURON_USE_CHAT_TEMPLATE`
    (default true). Falsy values force the fallback path
    everywhere — a kill switch for emergency rollback without a
    rebuild.

- `LoadedModel.chat_template: Option<String>` and the TP
  equivalent are populated once at load time. `None` (no
  tokenizer_config.json, parse error, missing field) routes the
  fallback path silently; logs go through `tracing::debug`/`warn`
  per condition.

- New `build_prompt_for_request(chat_template, request)` wraps
  the decision: when both the template is present AND the kill
  switch is off, render with kwargs from `request.extra` (looks
  up `chat_template_kwargs` and `tools` lazily). On render error
  → warn + fallback to `format_qwen3_prompt`. Wired into all four
  current prompt-build sites (single-GPU stream + non-stream, TP
  stream + non-stream).

## Dependency

`minijinja = "2"` with the `builtins`, `json`, and `serde`
features. Pure-Rust Jinja2 implementation, ~80KB compiled. Used
internally by HF's `tokenizers-rs` for its own chat templating;
the API surface we touch (`Environment::add_template` +
`Template::render(serde_value)`) is stable.

## Validation strategy

I can't byte-compare the new path's output against
`format_qwen3_prompt` for live models without GPU (CI doesn't
have one). The fallback path and kill switch are the mitigations
— a deploy can flip `NEURON_USE_CHAT_TEMPLATE=false` in the
neuron service env if the chat template renders surprisingly on
Qwen3-8B in production. The legacy formatter stays the
fail-closed default.

## Scope cuts (documented in module header)

- Tool-definition lifting from helexa-acp's system-prompt
  injection into the chat_template's native tools block is
  deferred. Today the request's `tools` array threads into the
  Jinja context, but helexa-acp continues to inject Hermes-format
  tool descriptions into the system prompt for backwards-compat
  with non-cortex endpoints.

## Tests

9 unit tests in `chat_template`: kill-switch matrix (truthy /
falsy / unset), template loading (string form, array form,
missing file, unparseable JSON, missing field), rendering
(basic conversation threading, kwargs forwarding, message-extras
threading for tool_calls).

215 workspace tests pass; clippy + fmt clean across all workspace
features (default).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-31 23:43:11 +03:00
parent 44008358c5
commit cb303832bc
5 changed files with 517 additions and 4 deletions

18
Cargo.lock generated
View File

@@ -2379,6 +2379,12 @@ dependencies = [
"stable_deref_trait",
]
[[package]]
name = "memo-map"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38d1115007560874e373613744c6fba374c17688327a71c1476d1a5954cc857b"
[[package]]
name = "metrics"
version = "0.24.3"
@@ -2432,6 +2438,17 @@ version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "minijinja"
version = "2.20.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2929e494b2280e1e18959bb2e121da03347ae896896fdfaceaab43c88a02803f"
dependencies = [
"memo-map",
"serde",
"serde_json",
]
[[package]]
name = "minimal-lexical"
version = "0.2.1"
@@ -2516,6 +2533,7 @@ dependencies = [
"futures",
"half",
"hf-hub",
"minijinja",
"reqwest",
"safetensors 0.7.0",
"serde",