Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3241680bfd
|
||
|
|
4878924b9b
|
||
|
|
4661557ed7
|
||
|
|
01d034df08
|
||
|
|
3d28e016e4
|
||
|
|
fdfdb19c64
|
||
|
|
9853c5b08a
|
||
|
|
1e3d3f76b7
|
||
|
|
8b0ed91380
|
||
|
|
c9f7383d91
|
||
|
|
18683d904d
|
||
|
|
46fe3d525c
|
||
|
|
fec5b33195
|
||
|
|
5ce4d8481e
|
||
|
|
c62cfc6a50
|
||
|
|
adebbd429a
|
||
|
|
8f7edd7d00
|
||
|
|
4fd23c5322
|
||
|
|
eca4251054
|
||
|
|
9e1a0fa231
|
||
|
|
af7a1e578b
|
||
|
|
e4a45f00ca
|
||
|
|
3c1dececeb
|
@@ -105,6 +105,11 @@ jobs:
|
||||
echo "commit_timestamp=${COMMIT_TIMESTAMP}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- id: changes
|
||||
env:
|
||||
# The push's own range. Empty on workflow_dispatch and all
|
||||
# zeroes for a new branch; `decide` treats both as unusable and
|
||||
# falls back to the full gate.
|
||||
PUSH_BEFORE: ${{ github.event.before }}
|
||||
run: |
|
||||
set -ux
|
||||
# Default: build everything. Detection only ever narrows
|
||||
@@ -189,9 +194,30 @@ jobs:
|
||||
BUILD_TOOLS=$(decide "$TOOLS_BASE" "$TOOLS_RE")
|
||||
BUILD_ANGELS=$(decide "$ANGELS_BASE" "$ANGELS_RE")
|
||||
if [ "$BUILD_CORTEX" = "true" ] || [ "$BUILD_NEURON" = "true" ] || [ "$BUILD_BENCH" = "true" ] || [ "$BUILD_UPSTREAM" = "true" ] || [ "$BUILD_ROUTER" = "true" ] || [ "$BUILD_TOOLS" = "true" ] || [ "$BUILD_ANGELS" = "true" ]; then
|
||||
# Anything we are about to build gets linted and tested.
|
||||
# This is the invariant that matters: no artefact reaches
|
||||
# the fleet unchecked, however the branch below behaves.
|
||||
CHECK_RUST=true
|
||||
else
|
||||
CHECK_RUST=$(decide "$CORTEX_BASE" "$RUST_RE")
|
||||
# Nothing is being built, so nothing will be published.
|
||||
# The only question left is whether THIS PUSH touched
|
||||
# Rust — and the honest base for that is the push's own
|
||||
# range, not a package's publication lag.
|
||||
#
|
||||
# This used to ask "has anything Rust changed since
|
||||
# cortex was last published". Cortex is only rebuilt when
|
||||
# cortex paths change, so after a run of neuron-only work
|
||||
# its base sits days behind and the diff from it sweeps in
|
||||
# every unrelated Rust commit since. A one-character
|
||||
# whitespace fix to a doc then re-ran fmt, clippy and the
|
||||
# full test suite over a delta that had already been
|
||||
# linted on the branch that introduced it (2026-08-18,
|
||||
# base fec5b33).
|
||||
#
|
||||
# `decide` returns true for an unusable base, so a new
|
||||
# branch (all-zero before-sha) or a force-push still gets
|
||||
# the full gate.
|
||||
CHECK_RUST=$(decide "$PUSH_BEFORE" "$RUST_RE")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
+46
-9
@@ -39,24 +39,34 @@ env:
|
||||
CARGO_TARGET_DIR: target-${{ github.job }}
|
||||
|
||||
jobs:
|
||||
# Two independent questions: does this change need the Rust gate, and
|
||||
# does it need the web gate.
|
||||
#
|
||||
# The SPA under helexa.ai/ is built and shipped by its own deploy job and
|
||||
# is not read by any crate, build script or spec file — so a change
|
||||
# confined to it cannot alter a Rust result. Waiting up to 70 minutes for
|
||||
# a CUDA type-check to prove that is pure latency, and the wait is what
|
||||
# tempts people to merge front-end work unchecked.
|
||||
# tempts people to merge front-end work unchecked. Documentation is inert
|
||||
# the same way, and more obviously: no crate `include_str!`s a markdown
|
||||
# file, no spec packages `doc/`, no build script reads one.
|
||||
#
|
||||
# The classification is deliberately one-directional: it only ever
|
||||
# narrows the gate when *every* changed path is front-end, and every
|
||||
# Conversely a Rust-only change cannot break the SPA, so it need not
|
||||
# spend three minutes on `npm ci` + lint + typecheck + build.
|
||||
#
|
||||
# The classification stays deliberately one-directional: it only ever
|
||||
# narrows the gate when *every* changed path is known-inert, and every
|
||||
# other outcome — a tag, an unknown base, an empty diff, a mixed
|
||||
# changeset — falls through to the full gate. A new top-level directory
|
||||
# is therefore covered on the day it appears, without anyone
|
||||
# remembering to add it here.
|
||||
# remembering to add it here. Adding an inert pattern is the only way to
|
||||
# widen what gets skipped, which is the property worth keeping.
|
||||
changes:
|
||||
name: Classify changes
|
||||
timeout-minutes: 10
|
||||
runs-on: fedora-43
|
||||
outputs:
|
||||
rust: ${{ steps.classify.outputs.rust }}
|
||||
web: ${{ steps.classify.outputs.web }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -64,7 +74,12 @@ jobs:
|
||||
- id: classify
|
||||
run: |
|
||||
set -u
|
||||
full() { echo "→ full gate: $1"; echo "rust=true" >> "$GITHUB_OUTPUT"; exit 0; }
|
||||
full() {
|
||||
echo "→ full gate: $1"
|
||||
echo "rust=true" >> "$GITHUB_OUTPUT"
|
||||
echo "web=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Tags build the SRPMs from the whole tree, and the srpm jobs
|
||||
# gate on fmt/clippy/test/cuda-check. Never narrow there.
|
||||
@@ -86,11 +101,31 @@ jobs:
|
||||
echo "changed files:"
|
||||
printf '%s\n' "${files}" | sed 's/^/ /'
|
||||
|
||||
other="$(printf '%s\n' "${files}" | grep -v '^helexa\.ai/' || true)"
|
||||
[ -z "${other}" ] || full "changes outside helexa.ai/"
|
||||
# Inert for Rust: the SPA, and documentation. Markdown is
|
||||
# matched only at the repo root and under doc/ — a .md inside
|
||||
# crates/ is left to fall through, since `include_str!` could
|
||||
# make one load-bearing without this file knowing.
|
||||
rust_relevant="$(printf '%s\n' "${files}" \
|
||||
| grep -v '^helexa\.ai/' \
|
||||
| grep -v '^doc/' \
|
||||
| grep -vE '^[^/]+\.md$' \
|
||||
|| true)"
|
||||
if [ -n "${rust_relevant}" ]; then
|
||||
echo "rust=true" >> "$GITHUB_OUTPUT"
|
||||
echo "→ Rust gate: required"
|
||||
else
|
||||
echo "rust=false" >> "$GITHUB_OUTPUT"
|
||||
echo "→ Rust gate: skipped (front-end and/or docs only)"
|
||||
fi
|
||||
|
||||
echo "→ front-end only: skipping the Rust gate"
|
||||
echo "rust=false" >> "$GITHUB_OUTPUT"
|
||||
# The web gate is needed only when the SPA itself changed.
|
||||
if printf '%s\n' "${files}" | grep -q '^helexa\.ai/'; then
|
||||
echo "web=true" >> "$GITHUB_OUTPUT"
|
||||
echo "→ web gate: required"
|
||||
else
|
||||
echo "web=false" >> "$GITHUB_OUTPUT"
|
||||
echo "→ web gate: skipped (nothing under helexa.ai/)"
|
||||
fi
|
||||
|
||||
fmt:
|
||||
name: Format
|
||||
@@ -106,6 +141,8 @@ jobs:
|
||||
name: Web (lint + typecheck + i18n + build)
|
||||
timeout-minutes: 20
|
||||
runs-on: fedora-43
|
||||
needs: changes
|
||||
if: needs.changes.outputs.web == 'true'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: helexa.ai
|
||||
|
||||
@@ -109,6 +109,40 @@ pub struct ModelLoad {
|
||||
/// (#54/#137). `#[serde(default)]` for back-compat.
|
||||
#[serde(default)]
|
||||
pub rejected_per_principal: u64,
|
||||
/// Anonymous requests currently holding a seat (#262). Read against
|
||||
/// `in_flight` it says how much of this model's load is unattributable;
|
||||
/// against `anon_max_in_flight`, how close that traffic is to the
|
||||
/// ceiling that keeps it from starving identified callers.
|
||||
#[serde(default)]
|
||||
pub anon_in_flight: usize,
|
||||
/// Seats anonymous traffic may hold at once (#262); `0` = refused
|
||||
/// outright. `#[serde(default)]` (→ 0) for pre-#262 neurons, which
|
||||
/// applied no ceiling at all — cortex skips the gauge rather than
|
||||
/// publish a 0 that reads as the opposite of the truth.
|
||||
#[serde(default)]
|
||||
pub anon_max_in_flight: usize,
|
||||
/// Cumulative anonymous requests refused for want of leftover capacity
|
||||
/// (#262). Rising here with a flat `rejected_timeout` means the
|
||||
/// reservation is working, not that the model is overloaded.
|
||||
#[serde(default)]
|
||||
pub rejected_anon_yield: u64,
|
||||
/// Cumulative requests that waited for KV budget and never got it
|
||||
/// (#257) — the model is saturated by long-context sequences rather
|
||||
/// than by request count. `#[serde(default)]` for back-compat.
|
||||
#[serde(default)]
|
||||
pub rejected_kv_timeout: u64,
|
||||
/// Cumulative requests whose KV reservation exceeded the model's entire
|
||||
/// budget (#257) — prompts too long for this node at any load, not a
|
||||
/// load-shedding signal. `#[serde(default)]` for back-compat.
|
||||
#[serde(default)]
|
||||
pub rejected_kv_unservable: u64,
|
||||
/// This model's total KV budget in MiB and how much of it is currently
|
||||
/// unreserved (#257). `kv_budget_mb == 0` means the gate is disabled
|
||||
/// (CPU load, or an arch with no context profile).
|
||||
#[serde(default)]
|
||||
pub kv_budget_mb: u64,
|
||||
#[serde(default)]
|
||||
pub kv_available_mb: u64,
|
||||
/// Live prefill throughput EMA in tokens/sec (#137) — prompt tokens
|
||||
/// processed per second. `0.0` before the first sample. `#[serde(default)]`
|
||||
/// for back-compat with pre-#137 neurons.
|
||||
@@ -148,8 +182,15 @@ mod health_load_tests {
|
||||
max_in_flight: 8,
|
||||
max_queue_depth: 8,
|
||||
rejected_queue_full: 0,
|
||||
rejected_kv_timeout: 0,
|
||||
rejected_kv_unservable: 0,
|
||||
kv_budget_mb: 0,
|
||||
kv_available_mb: 0,
|
||||
rejected_timeout: 0,
|
||||
rejected_per_principal: 0,
|
||||
anon_in_flight: 2,
|
||||
anon_max_in_flight: 7,
|
||||
rejected_anon_yield: 0,
|
||||
tok_s_prefill: 0.0,
|
||||
tok_s_decode: 0.0,
|
||||
}],
|
||||
@@ -161,6 +202,8 @@ mod health_load_tests {
|
||||
assert_eq!(back.models[0].queue_depth, 3);
|
||||
assert_eq!(back.models[0].max_in_flight, 8);
|
||||
assert_eq!(back.models[0].max_queue_depth, 8);
|
||||
assert_eq!(back.models[0].anon_in_flight, 2);
|
||||
assert_eq!(back.models[0].anon_max_in_flight, 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -172,6 +215,10 @@ mod health_load_tests {
|
||||
assert_eq!(m.in_flight, 2);
|
||||
assert_eq!(m.max_in_flight, 0);
|
||||
assert_eq!(m.max_queue_depth, 0);
|
||||
// Likewise a pre-#262 neuron, which enforced no anonymous ceiling.
|
||||
assert_eq!(m.anon_in_flight, 0);
|
||||
assert_eq!(m.anon_max_in_flight, 0);
|
||||
assert_eq!(m.rejected_anon_yield, 0);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -18,8 +18,16 @@ pub struct ChatCompletionRequest {
|
||||
pub temperature: Option<f64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub top_p: Option<f64>,
|
||||
/// The deprecated spelling of the output cap. Still what most
|
||||
/// clients send; see [`ChatCompletionRequest::effective_max_tokens`].
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub max_tokens: Option<u64>,
|
||||
/// OpenAI's current spelling of the output cap, which deprecated
|
||||
/// `max_tokens`. Newer SDKs and clients send only this one — it has
|
||||
/// to be a named field rather than falling into `extra`, or the cap
|
||||
/// is silently ignored and generation runs to the server default.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub max_completion_tokens: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub stream: Option<bool>,
|
||||
/// All other fields (tools, response_format, backend extensions, etc.)
|
||||
@@ -27,20 +35,49 @@ pub struct ChatCompletionRequest {
|
||||
pub extra: Value,
|
||||
}
|
||||
|
||||
impl ChatCompletionRequest {
|
||||
/// The output cap the caller asked for, under either spelling.
|
||||
///
|
||||
/// `max_completion_tokens` wins when both are present — it is the
|
||||
/// current OpenAI field, and cortex's metering already reserves
|
||||
/// against it (`metering::requested_max_output`), so preferring it
|
||||
/// here keeps what we bill and what we generate in agreement.
|
||||
pub fn effective_max_tokens(&self) -> Option<u64> {
|
||||
self.max_completion_tokens.or(self.max_tokens)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ChatMessage {
|
||||
pub role: String,
|
||||
/// Absent on an assistant turn that carries only `tool_calls` —
|
||||
/// see [`MessageContent::Null`]. `#[serde(default)]` so a message
|
||||
/// with no `content` key at all deserializes rather than being
|
||||
/// rejected.
|
||||
#[serde(default)]
|
||||
pub content: MessageContent,
|
||||
#[serde(flatten)]
|
||||
pub extra: Value,
|
||||
}
|
||||
|
||||
/// Content can be a simple string or an array of content parts (for vision).
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
/// Content can be a simple string, an array of content parts (for
|
||||
/// vision), or absent.
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
#[serde(untagged)]
|
||||
pub enum MessageContent {
|
||||
Text(String),
|
||||
Parts(Vec<Value>),
|
||||
/// `"content": null`, or no `content` key at all.
|
||||
///
|
||||
/// This is the OpenAI-canonical shape for an assistant turn whose
|
||||
/// only payload is `tool_calls`, and agentic clients replay the
|
||||
/// assistant turn verbatim on the follow-up request that carries
|
||||
/// the tool result — so any client doing OpenAI-native tool
|
||||
/// calling sends it on its second turn. HF chat templates model it
|
||||
/// the same way (`content is none` renders as empty), so it maps
|
||||
/// straight through to the prompt.
|
||||
#[default]
|
||||
Null,
|
||||
}
|
||||
|
||||
// ── Chat completion response (non-streaming) ─────────────────────────
|
||||
@@ -170,3 +207,133 @@ pub struct ModelObject {
|
||||
#[serde(flatten)]
|
||||
pub extra: Value,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The follow-up request an OpenAI-native agentic client sends
|
||||
/// after running a tool: it replays its own assistant turn, which
|
||||
/// OpenAI emits with `"content": null` because the payload was
|
||||
/// only `tool_calls`. Rejecting this shape breaks every such
|
||||
/// client on its second turn.
|
||||
#[test]
|
||||
fn assistant_tool_call_turn_with_null_content_deserializes() {
|
||||
let req: ChatCompletionRequest = serde_json::from_str(
|
||||
r#"{
|
||||
"model": "m",
|
||||
"messages": [
|
||||
{"role": "user", "content": "list the files"},
|
||||
{"role": "assistant", "content": null, "tool_calls": [
|
||||
{"id": "call_0", "type": "function",
|
||||
"function": {"name": "bash", "arguments": "{\"command\":\"ls\"}"}}
|
||||
]},
|
||||
{"role": "tool", "content": "a.txt", "tool_call_id": "call_0"}
|
||||
]
|
||||
}"#,
|
||||
)
|
||||
.expect("null assistant content is a valid OpenAI request");
|
||||
|
||||
assert!(matches!(req.messages[1].content, MessageContent::Null));
|
||||
// The tool calls survive in `extra`, where the chat template
|
||||
// reads them from.
|
||||
assert_eq!(
|
||||
req.messages[1].extra["tool_calls"][0]["function"]["name"],
|
||||
"bash"
|
||||
);
|
||||
assert_eq!(req.messages[2].extra["tool_call_id"], "call_0");
|
||||
}
|
||||
|
||||
/// Some clients omit the key entirely rather than sending null.
|
||||
#[test]
|
||||
fn assistant_turn_with_no_content_key_deserializes() {
|
||||
let msg: ChatMessage = serde_json::from_str(
|
||||
r#"{"role": "assistant", "tool_calls": [
|
||||
{"id": "c", "type": "function",
|
||||
"function": {"name": "bash", "arguments": "{}"}}]}"#,
|
||||
)
|
||||
.expect("absent content is a valid OpenAI message");
|
||||
assert!(matches!(msg.content, MessageContent::Null));
|
||||
}
|
||||
|
||||
/// Absent content round-trips back onto the wire as null, so a
|
||||
/// translated or re-serialized request stays OpenAI-shaped.
|
||||
#[test]
|
||||
fn null_content_round_trips() {
|
||||
let msg = ChatMessage {
|
||||
role: "assistant".into(),
|
||||
content: MessageContent::Null,
|
||||
extra: Value::Null,
|
||||
};
|
||||
let v = serde_json::to_value(&msg).expect("serialize");
|
||||
assert_eq!(v["content"], Value::Null);
|
||||
assert!(v.get("content").is_some(), "content key must be present");
|
||||
}
|
||||
|
||||
/// Newer OpenAI SDKs send only `max_completion_tokens`; the cap
|
||||
/// must survive as a named field rather than falling into `extra`,
|
||||
/// where every generation loop would miss it.
|
||||
#[test]
|
||||
fn max_completion_tokens_is_the_effective_cap() {
|
||||
let req: ChatCompletionRequest = serde_json::from_str(
|
||||
r#"{"model": "m", "max_completion_tokens": 240,
|
||||
"messages": [{"role": "user", "content": "hi"}]}"#,
|
||||
)
|
||||
.expect("deserialize");
|
||||
assert_eq!(req.max_completion_tokens, Some(240));
|
||||
assert_eq!(req.max_tokens, None);
|
||||
assert_eq!(req.effective_max_tokens(), Some(240));
|
||||
assert!(
|
||||
req.extra.get("max_completion_tokens").is_none(),
|
||||
"must be a named field, not swept into extra"
|
||||
);
|
||||
}
|
||||
|
||||
/// The legacy spelling keeps working on its own.
|
||||
#[test]
|
||||
fn legacy_max_tokens_still_caps() {
|
||||
let req: ChatCompletionRequest = serde_json::from_str(
|
||||
r#"{"model": "m", "max_tokens": 128,
|
||||
"messages": [{"role": "user", "content": "hi"}]}"#,
|
||||
)
|
||||
.expect("deserialize");
|
||||
assert_eq!(req.effective_max_tokens(), Some(128));
|
||||
}
|
||||
|
||||
/// Clients that send both for compatibility must not be rejected,
|
||||
/// and the current field wins — matching what cortex meters
|
||||
/// against, so billed and generated caps agree.
|
||||
#[test]
|
||||
fn both_spellings_resolve_to_the_current_field() {
|
||||
let req: ChatCompletionRequest = serde_json::from_str(
|
||||
r#"{"model": "m", "max_tokens": 99, "max_completion_tokens": 256,
|
||||
"messages": [{"role": "user", "content": "hi"}]}"#,
|
||||
)
|
||||
.expect("sending both must not be an error");
|
||||
assert_eq!(req.effective_max_tokens(), Some(256));
|
||||
}
|
||||
|
||||
/// Neither spelling present leaves the cap to the server default.
|
||||
#[test]
|
||||
fn absent_cap_is_none() {
|
||||
let req: ChatCompletionRequest = serde_json::from_str(
|
||||
r#"{"model": "m", "messages": [{"role": "user", "content": "hi"}]}"#,
|
||||
)
|
||||
.expect("deserialize");
|
||||
assert_eq!(req.effective_max_tokens(), None);
|
||||
}
|
||||
|
||||
/// The string and array forms keep working unchanged.
|
||||
#[test]
|
||||
fn text_and_parts_content_still_deserialize() {
|
||||
let text: ChatMessage =
|
||||
serde_json::from_str(r#"{"role": "user", "content": "hi"}"#).expect("text");
|
||||
assert!(matches!(text.content, MessageContent::Text(ref t) if t == "hi"));
|
||||
|
||||
let parts: ChatMessage = serde_json::from_str(
|
||||
r#"{"role": "user", "content": [{"type": "text", "text": "hi"}]}"#,
|
||||
)
|
||||
.expect("parts");
|
||||
assert!(matches!(parts.content, MessageContent::Parts(ref p) if p.len() == 1));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,6 +102,8 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
|
||||
temperature: req.temperature,
|
||||
top_p: req.top_p,
|
||||
max_tokens: Some(req.max_tokens),
|
||||
// Anthropic has one spelling; it maps onto the legacy field.
|
||||
max_completion_tokens: None,
|
||||
stream: req.stream,
|
||||
extra,
|
||||
}
|
||||
@@ -387,6 +389,7 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
|
||||
let text = match c.message.content {
|
||||
MessageContent::Text(t) => t,
|
||||
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
|
||||
MessageContent::Null => String::new(),
|
||||
};
|
||||
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
|
||||
(text, stop)
|
||||
|
||||
@@ -111,7 +111,15 @@ fn describe_metrics() {
|
||||
);
|
||||
metrics::describe_counter!(
|
||||
"cortex_model_rejections_total",
|
||||
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal — the load-shedding signal (#137)"
|
||||
"Admission rejections per neuron:model by reason: queue_full / wait_timeout / per_principal / anon_yield — the load-shedding signal (#137, #262)"
|
||||
);
|
||||
metrics::describe_gauge!(
|
||||
"cortex_model_anon_in_flight",
|
||||
"Anonymous (unattributable) requests holding a seat on a neuron:model (#262)"
|
||||
);
|
||||
metrics::describe_gauge!(
|
||||
"cortex_model_anon_max_in_flight",
|
||||
"Seats anonymous traffic may hold at once, so it cannot starve identified callers (#262)"
|
||||
);
|
||||
metrics::describe_gauge!(
|
||||
"cortex_model_tok_s_decode",
|
||||
|
||||
@@ -380,6 +380,21 @@ fn export_health_metrics(node: &str, h: &HealthResponse) {
|
||||
counter!("cortex_model_rejections_total",
|
||||
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "per_principal")
|
||||
.absolute(m.rejected_per_principal);
|
||||
counter!("cortex_model_rejections_total",
|
||||
"node" => node.to_string(), "model" => m.id.clone(), "reason" => "anon_yield")
|
||||
.absolute(m.rejected_anon_yield);
|
||||
// How much of this model's load is unattributable (#262), and the
|
||||
// ceiling that keeps it from starving identified callers. 0 for a
|
||||
// pre-#262 neuron, which had no ceiling — skip rather than publish
|
||||
// a 0 that reads as "anonymous is refused" when it means the
|
||||
// opposite.
|
||||
gauge!("cortex_model_anon_in_flight", "node" => node.to_string(), "model" => m.id.clone())
|
||||
.set(m.anon_in_flight as f64);
|
||||
if m.anon_max_in_flight > 0 {
|
||||
gauge!("cortex_model_anon_max_in_flight",
|
||||
"node" => node.to_string(), "model" => m.id.clone())
|
||||
.set(m.anon_max_in_flight as f64);
|
||||
}
|
||||
}
|
||||
for d in &h.devices {
|
||||
let device = d.index.to_string();
|
||||
|
||||
@@ -71,15 +71,46 @@ pub async fn forward_request(
|
||||
})?;
|
||||
|
||||
if !response.status().is_success() {
|
||||
// Streaming body — can't snippet without breaking the stream
|
||||
// pass-through. Log status + URL; the client still gets the
|
||||
// upstream status, just without the leaked body.
|
||||
tracing::warn!(
|
||||
node = %route.node_name,
|
||||
url = %url,
|
||||
status = response.status().as_u16(),
|
||||
"proxy: upstream returned non-2xx"
|
||||
);
|
||||
// Buffer the error body so the journal records WHY, then hand the
|
||||
// same bytes on. Streaming is what a success body needs; an error
|
||||
// body is a few hundred bytes and streaming it only guarantees
|
||||
// that nobody ever sees the reason.
|
||||
//
|
||||
// This was a real dead end: neuron answered 503 `insufficient_vram`
|
||||
// with the deficit in the body, and cortex logged only
|
||||
// `status=503`. Neither journal held the cause, and the guard
|
||||
// responsible could only be found by reading neuron's source
|
||||
// (#257). The body is upstream-generated (our own #63 envelope),
|
||||
// not caller content, and it is truncated in the log.
|
||||
const MAX_ERROR_BODY: usize = 64 * 1024;
|
||||
let status = response.status();
|
||||
let (parts, body) = response.into_parts();
|
||||
return match axum::body::to_bytes(body, MAX_ERROR_BODY).await {
|
||||
Ok(bytes) => {
|
||||
let snippet: String = String::from_utf8_lossy(&bytes).chars().take(600).collect();
|
||||
tracing::warn!(
|
||||
node = %route.node_name,
|
||||
url = %url,
|
||||
status = status.as_u16(),
|
||||
body = %snippet,
|
||||
"proxy: upstream returned non-2xx"
|
||||
);
|
||||
Ok(Response::from_parts(parts, axum::body::Body::from(bytes)))
|
||||
}
|
||||
// Over the cap or a read failure: the body is consumed and
|
||||
// cannot be handed on. Say so rather than passing an empty
|
||||
// body off as the upstream's answer.
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
node = %route.node_name,
|
||||
url = %url,
|
||||
status = status.as_u16(),
|
||||
error = %e,
|
||||
"proxy: upstream returned non-2xx (body unreadable, dropped)"
|
||||
);
|
||||
Ok(Response::from_parts(parts, axum::body::Body::empty()))
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
Ok(response)
|
||||
|
||||
@@ -49,8 +49,15 @@ async fn seed_loaded(fleet: &CortexState, node: &str, in_flight: usize, queue_de
|
||||
max_in_flight: 8,
|
||||
max_queue_depth: 8,
|
||||
rejected_queue_full: 0,
|
||||
rejected_kv_timeout: 0,
|
||||
rejected_kv_unservable: 0,
|
||||
kv_budget_mb: 0,
|
||||
kv_available_mb: 0,
|
||||
rejected_timeout: 0,
|
||||
rejected_per_principal: 0,
|
||||
anon_in_flight: 0,
|
||||
anon_max_in_flight: 7,
|
||||
rejected_anon_yield: 0,
|
||||
tok_s_prefill: 0.0,
|
||||
tok_s_decode: 0.0,
|
||||
},
|
||||
|
||||
@@ -27,7 +27,15 @@ use sqlx::Row;
|
||||
use uuid::Uuid;
|
||||
|
||||
pub fn router(state: &AppState) -> Router<AppState> {
|
||||
let protected = Router::new()
|
||||
// Session-only: identity management and the full account view.
|
||||
//
|
||||
// Key management stays here deliberately (#264). If an inference key
|
||||
// could mint keys, one leaked credential would become a foothold that
|
||||
// SURVIVES revoking the original — a spending problem turning into an
|
||||
// eviction problem. `/account` stays too: it carries `angel_access`,
|
||||
// and whatever field is added to it next would otherwise be exposed to
|
||||
// inference credentials by default.
|
||||
let session_only = Router::new()
|
||||
.route("/web/v1/account", get(account))
|
||||
.route("/web/v1/keys", get(list_keys).post(create_key))
|
||||
.route("/web/v1/keys/{id}/archive", post(archive_key))
|
||||
@@ -35,11 +43,23 @@ pub fn router(state: &AppState) -> Router<AppState> {
|
||||
"/web/v1/keys/{id}/limit",
|
||||
axum::routing::patch(update_key_limit),
|
||||
)
|
||||
.layer(axum::middleware::from_fn_with_state(
|
||||
state.clone(),
|
||||
require_session,
|
||||
));
|
||||
|
||||
// Session **or** API key: reading and refilling the allocation.
|
||||
//
|
||||
// A service that watches its own balance should not have to hold its
|
||||
// owner's password — the credential that can mint and revoke keys — to
|
||||
// read a number it is already entitled to spend against.
|
||||
let account_auth = Router::new()
|
||||
.route("/web/v1/allocation", get(allocation))
|
||||
.route("/web/v1/redeem", post(redeem))
|
||||
.route("/web/v1/topup/request", post(request_topup))
|
||||
.layer(axum::middleware::from_fn_with_state(
|
||||
state.clone(),
|
||||
require_session,
|
||||
require_account,
|
||||
));
|
||||
|
||||
Router::new()
|
||||
@@ -49,7 +69,8 @@ pub fn router(state: &AppState) -> Router<AppState> {
|
||||
.route("/web/v1/login", post(login))
|
||||
.route("/web/v1/password-reset/request", post(reset_request))
|
||||
.route("/web/v1/password-reset/confirm", post(reset_confirm))
|
||||
.merge(protected)
|
||||
.merge(session_only)
|
||||
.merge(account_auth)
|
||||
}
|
||||
|
||||
// ── errors ──────────────────────────────────────────────────────────
|
||||
@@ -142,6 +163,71 @@ async fn require_session(State(state): State<AppState>, mut req: Request, next:
|
||||
}
|
||||
}
|
||||
|
||||
/// Authenticated **account**, injected by [`require_account`]. Unlike
|
||||
/// [`AuthUser`] this says nothing about *who* is calling — only which
|
||||
/// account the call bills to — because an API key identifies an account
|
||||
/// without identifying a person.
|
||||
#[derive(Clone, Copy)]
|
||||
struct AuthAccount(Uuid);
|
||||
|
||||
/// Accept **either** a web session or an inference API key, and resolve
|
||||
/// both to an account id (#264).
|
||||
///
|
||||
/// Routes behind this may read and refill an allocation. They may not
|
||||
/// touch keys: a service watching its own balance should not have to
|
||||
/// hold the credential that can mint and revoke credentials. See the
|
||||
/// router for why key management stays session-only.
|
||||
///
|
||||
/// The two token shapes are unambiguous — API keys carry the
|
||||
/// `sk-helexa-` prefix (`crypto::generate_api_key`) and JWTs cannot — so
|
||||
/// this dispatches on the prefix rather than trying to decode a session
|
||||
/// and falling back, which would log a decode failure for every
|
||||
/// perfectly valid key.
|
||||
async fn require_account(State(state): State<AppState>, mut req: Request, next: Next) -> Response {
|
||||
let Some(token) = req
|
||||
.headers()
|
||||
.get(header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|v| v.strip_prefix("Bearer "))
|
||||
.map(str::trim)
|
||||
.map(str::to_string)
|
||||
else {
|
||||
return WebError::Unauthorized.into_response();
|
||||
};
|
||||
|
||||
let account = if token.starts_with("sk-helexa-") {
|
||||
// `resolve_key` already requires both key and account to be
|
||||
// `active`, so an archived key or a deactivated account
|
||||
// authenticates nothing — no extra check needed here.
|
||||
match crate::ledger::resolve_key(&state.pool, &sha256(&token)).await {
|
||||
Ok(Some(principal)) => principal.account_id,
|
||||
Ok(None) => return WebError::Unauthorized.into_response(),
|
||||
Err(e) => {
|
||||
tracing::error!(error = %e, "key resolve failed on account route");
|
||||
return WebError::Internal.into_response();
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let uid = decode::<Claims>(
|
||||
&token,
|
||||
&DecodingKey::from_secret(state.config.auth.jwt_secret.as_bytes()),
|
||||
&Validation::default(),
|
||||
)
|
||||
.ok()
|
||||
.and_then(|d| Uuid::parse_str(&d.claims.sub).ok());
|
||||
let Some(uid) = uid else {
|
||||
return WebError::Unauthorized.into_response();
|
||||
};
|
||||
match account_id_for(&state, uid).await {
|
||||
Ok(a) => a,
|
||||
Err(e) => return e.into_response(),
|
||||
}
|
||||
};
|
||||
|
||||
req.extensions_mut().insert(AuthAccount(account));
|
||||
next.run(req).await
|
||||
}
|
||||
|
||||
/// The caller's single account id.
|
||||
async fn account_id_for(state: &AppState, user_id: Uuid) -> WebResult<Uuid> {
|
||||
let row = sqlx::query("SELECT id FROM accounts WHERE owner_user_id = $1")
|
||||
@@ -635,6 +721,46 @@ async fn account(
|
||||
.into_response())
|
||||
}
|
||||
|
||||
/// `GET /web/v1/allocation` — the account's token balance, readable with
|
||||
/// **either** a session or an API key (#264).
|
||||
///
|
||||
/// Deliberately narrower than [`account`]: balance and top-up
|
||||
/// availability, nothing else. `/account` also returns `angel_access`,
|
||||
/// and whatever field is added to it next would otherwise be handed to
|
||||
/// every inference key by default. A separate projection means widening
|
||||
/// the account view can never widen what a key can see.
|
||||
async fn allocation(
|
||||
State(state): State<AppState>,
|
||||
Extension(AuthAccount(acct)): Extension<AuthAccount>,
|
||||
) -> WebResult<Response> {
|
||||
let row = sqlx::query(
|
||||
"SELECT allocation_total, allocation_spent, allocation_reserved FROM accounts WHERE id = $1",
|
||||
)
|
||||
.bind(acct)
|
||||
.fetch_one(&state.pool)
|
||||
.await?;
|
||||
// Same display-only eligibility the dashboard uses, so an automated
|
||||
// caller can check before asking and avoid provoking a 409. The grant
|
||||
// path re-checks, so a stale read cannot conjure a top-up.
|
||||
let (topup_available, topup_reason) =
|
||||
match crate::topup::auto_eligibility(&state.pool, acct).await {
|
||||
Ok(Ok(())) => (true, None),
|
||||
Ok(Err(reason)) => (false, Some(reason.to_string())),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "top-up eligibility check failed");
|
||||
(false, None)
|
||||
}
|
||||
};
|
||||
Ok(Json(json!({
|
||||
"allocation_total": row.get::<i64, _>("allocation_total"),
|
||||
"allocation_spent": row.get::<i64, _>("allocation_spent"),
|
||||
"allocation_reserved": row.get::<i64, _>("allocation_reserved"),
|
||||
"topup_available": topup_available,
|
||||
"topup_reason": topup_reason,
|
||||
}))
|
||||
.into_response())
|
||||
}
|
||||
|
||||
/// Whether this user holds any investor-portal grant.
|
||||
///
|
||||
/// A **boolean, and deliberately nothing more**. helexa.ai is a static
|
||||
@@ -680,10 +806,9 @@ async fn has_angel_access(state: &AppState, user_id: Uuid) -> bool {
|
||||
/// caller is authenticated and asking about their own account.
|
||||
async fn request_topup(
|
||||
State(state): State<AppState>,
|
||||
Extension(user): Extension<AuthUser>,
|
||||
Extension(AuthAccount(acct)): Extension<AuthAccount>,
|
||||
) -> WebResult<Response> {
|
||||
use crate::topup::AutoTopUpError;
|
||||
let acct = account_id_for(&state, user.0).await?;
|
||||
match crate::topup::auto_grant(&state.pool, acct).await {
|
||||
Ok(grant) => {
|
||||
tracing::info!(
|
||||
@@ -859,10 +984,9 @@ struct RedeemReq {
|
||||
/// or already-redeemed code (no oracle).
|
||||
async fn redeem(
|
||||
State(state): State<AppState>,
|
||||
Extension(user): Extension<AuthUser>,
|
||||
Extension(AuthAccount(acct)): Extension<AuthAccount>,
|
||||
Json(req): Json<RedeemReq>,
|
||||
) -> WebResult<Response> {
|
||||
let acct = account_id_for(&state, user.0).await?;
|
||||
match crate::topup::redeem(&state.pool, acct, &req.code).await {
|
||||
Ok(new_total) => Ok(Json(json!({ "allocation_total": new_total })).into_response()),
|
||||
Err(crate::topup::TopUpError::Invalid) => {
|
||||
|
||||
@@ -424,3 +424,157 @@ async fn topup_concurrent_double_redeem_one_winner() {
|
||||
.count();
|
||||
assert_eq!(wins, 1, "exactly one redemption wins the single-use code");
|
||||
}
|
||||
|
||||
/// An API key may read its own allocation and refill it, but must never
|
||||
/// reach key management (#264).
|
||||
///
|
||||
/// The second half is the assertion that matters. If an inference key
|
||||
/// could mint keys, one leaked credential would become a foothold that
|
||||
/// survives revoking the original — a spending problem turning into an
|
||||
/// eviction problem. The first half is the point of the change: a service
|
||||
/// watching its own balance should not need the account password.
|
||||
#[tokio::test]
|
||||
async fn api_key_reads_allocation_but_cannot_touch_keys() {
|
||||
let Some((base, pool)) = spawn_or_skip("api_key_reads_allocation_but_cannot_touch_keys").await
|
||||
else {
|
||||
return;
|
||||
};
|
||||
let client = reqwest::Client::new();
|
||||
let email = unique_email();
|
||||
post(
|
||||
format!("{base}/web/v1/register"),
|
||||
json!({"email": email, "password": "password123"}),
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
pool.execute(
|
||||
sqlx::query("UPDATE users SET email_verified = true WHERE email = $1").bind(&email),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let session = post(
|
||||
format!("{base}/web/v1/login"),
|
||||
json!({"email": email, "password": "password123"}),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.json::<Value>()
|
||||
.await
|
||||
.unwrap()["token"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.to_string();
|
||||
|
||||
let created = post(
|
||||
format!("{base}/web/v1/keys"),
|
||||
json!({"label": "service"}),
|
||||
Some(&session),
|
||||
)
|
||||
.await
|
||||
.json::<Value>()
|
||||
.await
|
||||
.unwrap();
|
||||
let raw_key = created["key"].as_str().unwrap().to_string();
|
||||
let key_id = created["id"].as_str().unwrap().to_string();
|
||||
|
||||
// ── the key CAN read its own allocation ──────────────────────────
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/allocation"))
|
||||
.bearer_auth(&raw_key)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
r.status(),
|
||||
200,
|
||||
"an API key must be able to read allocation"
|
||||
);
|
||||
let body: Value = r.json().await.unwrap();
|
||||
assert!(body["allocation_total"].is_i64());
|
||||
assert!(body["allocation_spent"].is_i64());
|
||||
assert!(body["allocation_reserved"].is_i64());
|
||||
assert!(body["topup_available"].is_boolean());
|
||||
// Narrower than /account on purpose: nothing about the human.
|
||||
assert!(
|
||||
body.get("angel_access").is_none(),
|
||||
"allocation must not leak account-holder fields to an inference key: {body}"
|
||||
);
|
||||
|
||||
// A session reaches the same route.
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/allocation"))
|
||||
.bearer_auth(&session)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(r.status(), 200, "a session must still work here");
|
||||
|
||||
// ── the key CANNOT manage keys ───────────────────────────────────
|
||||
let r = post(
|
||||
format!("{base}/web/v1/keys"),
|
||||
json!({"label": "minted-by-a-key"}),
|
||||
Some(&raw_key),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
r.status(),
|
||||
401,
|
||||
"an API key must not be able to mint another key"
|
||||
);
|
||||
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/keys"))
|
||||
.bearer_auth(&raw_key)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(r.status(), 401, "an API key must not be able to list keys");
|
||||
|
||||
let r = post(
|
||||
format!("{base}/web/v1/keys/{key_id}/archive"),
|
||||
json!({}),
|
||||
Some(&raw_key),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
r.status(),
|
||||
401,
|
||||
"an API key must not be able to archive a key"
|
||||
);
|
||||
|
||||
// /account stays session-only — it carries account-holder fields.
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/account"))
|
||||
.bearer_auth(&raw_key)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(r.status(), 401, "/account must remain session-only");
|
||||
|
||||
// ── an archived key authenticates nothing ────────────────────────
|
||||
let r = post(
|
||||
format!("{base}/web/v1/keys/{key_id}/archive"),
|
||||
json!({}),
|
||||
Some(&session),
|
||||
)
|
||||
.await;
|
||||
assert!(r.status().is_success(), "archive via session should work");
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/allocation"))
|
||||
.bearer_auth(&raw_key)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(r.status(), 401, "an archived key must authenticate nothing");
|
||||
|
||||
// Garbage bearers are refused on both shapes.
|
||||
for bogus in ["sk-helexa-nonsense", "not-a-jwt"] {
|
||||
let r = client
|
||||
.get(format!("{base}/web/v1/allocation"))
|
||||
.bearer_auth(bogus)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(r.status(), 401, "bogus bearer {bogus} must be refused");
|
||||
}
|
||||
}
|
||||
|
||||
+128
-9
@@ -97,7 +97,7 @@ async fn list_models(State(state): State<Arc<NeuronState>>) -> impl IntoResponse
|
||||
|
||||
async fn load_model(
|
||||
State(state): State<Arc<NeuronState>>,
|
||||
Json(spec): Json<ModelSpec>,
|
||||
ApiJson(spec): ApiJson<ModelSpec>,
|
||||
) -> impl IntoResponse {
|
||||
// Driver/library mismatch preflight (#19): every CUDA load is
|
||||
// guaranteed to fail until the host reboots. Reject up front with
|
||||
@@ -168,7 +168,7 @@ fn preflight_kind(err: &PreflightError) -> &'static str {
|
||||
|
||||
async fn unload_model(
|
||||
State(state): State<Arc<NeuronState>>,
|
||||
Json(body): Json<Value>,
|
||||
ApiJson(body): ApiJson<Value>,
|
||||
) -> impl IntoResponse {
|
||||
let model_id = match body.get("model_id").and_then(|v| v.as_str()) {
|
||||
Some(id) => id.to_string(),
|
||||
@@ -284,7 +284,7 @@ fn principal_key(headers: &axum::http::HeaderMap) -> Option<String> {
|
||||
async fn chat_completions(
|
||||
State(state): State<Arc<NeuronState>>,
|
||||
headers: axum::http::HeaderMap,
|
||||
Json(mut req): Json<ChatCompletionRequest>,
|
||||
ApiJson(mut req): ApiJson<ChatCompletionRequest>,
|
||||
) -> impl IntoResponse {
|
||||
let Some(candle) = state.candle.as_ref().map(Arc::clone) else {
|
||||
return (
|
||||
@@ -294,6 +294,13 @@ async fn chat_completions(
|
||||
.into_response();
|
||||
};
|
||||
|
||||
// Collapse the two spellings of the output cap onto the one field
|
||||
// the generation loops read. Clients that send only
|
||||
// `max_completion_tokens` (OpenAI's current field — the DeepSeek
|
||||
// Harness among them) would otherwise have their cap silently
|
||||
// ignored and generate to the server default.
|
||||
req.max_tokens = req.effective_max_tokens();
|
||||
|
||||
// Reasoning-content opt-in. Off by default → naïve clients
|
||||
// (Zed's commit-message generator, vanilla OpenAI clients)
|
||||
// never see `<think>` blocks. On when the caller sends
|
||||
@@ -359,7 +366,7 @@ async fn chat_completions(
|
||||
async fn responses(
|
||||
State(state): State<Arc<NeuronState>>,
|
||||
headers: axum::http::HeaderMap,
|
||||
Json(req): Json<ResponsesRequest>,
|
||||
ApiJson(req): ApiJson<ResponsesRequest>,
|
||||
) -> impl IntoResponse {
|
||||
let Some(candle) = state.candle.as_ref().map(Arc::clone) else {
|
||||
return (
|
||||
@@ -431,11 +438,11 @@ async fn responses(
|
||||
.first()
|
||||
.map(|c| match &c.message.content {
|
||||
MessageContent::Text(t) => t.clone(),
|
||||
MessageContent::Parts(_) => {
|
||||
MessageContent::Parts(_) | MessageContent::Null => {
|
||||
// Candle output is always text today;
|
||||
// a Parts response would be surprising.
|
||||
// Empty-string fallback is safer than
|
||||
// a panic.
|
||||
// a Parts or absent response would be
|
||||
// surprising. Empty-string fallback is
|
||||
// safer than a panic.
|
||||
String::new()
|
||||
}
|
||||
})
|
||||
@@ -483,7 +490,7 @@ async fn responses(
|
||||
async fn images_generations(
|
||||
State(state): State<Arc<NeuronState>>,
|
||||
headers: axum::http::HeaderMap,
|
||||
Json(req): Json<cortex_core::images::ImagesGenerationRequest>,
|
||||
ApiJson(req): ApiJson<cortex_core::images::ImagesGenerationRequest>,
|
||||
) -> impl IntoResponse {
|
||||
use cortex_core::error_envelope::OpenAiError;
|
||||
use cortex_core::images as wire;
|
||||
@@ -644,6 +651,24 @@ fn inference_error_response(err: InferenceError) -> axum::response::Response {
|
||||
.with_retry_after(5)
|
||||
.with_extra("free_mb", json!(free_mb))
|
||||
.with_extra("required_mb", json!(required_mb)),
|
||||
// Not backpressure: an idle node would refuse this prompt too, so
|
||||
// there is nothing to retry (#257). 413 rather than 400 — the
|
||||
// request is well-formed, it is the payload that is too large.
|
||||
InferenceError::PromptTooLongForVram {
|
||||
required_mb,
|
||||
budget_mb,
|
||||
} => OpenAiError::new(
|
||||
413,
|
||||
"invalid_request_error",
|
||||
"prompt_too_long_for_vram",
|
||||
format!(
|
||||
"prompt needs {required_mb} MiB of KV cache but this model's \
|
||||
entire budget is {budget_mb} MiB. Shorten the conversation \
|
||||
or use a model with more headroom."
|
||||
),
|
||||
)
|
||||
.with_extra("required_mb", json!(required_mb))
|
||||
.with_extra("budget_mb", json!(budget_mb)),
|
||||
InferenceError::VisionUnsupported { model_id } => OpenAiError::new(
|
||||
400,
|
||||
"invalid_request_error",
|
||||
@@ -704,8 +729,37 @@ fn inference_error_response(err: InferenceError) -> axum::response::Response {
|
||||
"too many concurrent requests for this key; retry shortly",
|
||||
)
|
||||
.with_retry_after(retry_after_secs),
|
||||
// Draining (#256). 503 + Retry-After, like any transient capacity
|
||||
// condition: the node is coming back, and a well-behaved client
|
||||
// should retry rather than surface a failure. Deliberately not
|
||||
// `rate_limit_exceeded` — nothing about this is the caller's
|
||||
// fault or their rate.
|
||||
InferenceError::ShuttingDown => OpenAiError::new(
|
||||
503,
|
||||
"api_error",
|
||||
"shutting_down",
|
||||
"neuron is shutting down; retry shortly",
|
||||
)
|
||||
.with_retry_after(15),
|
||||
InferenceError::Other(e) => OpenAiError::without_code(500, "api_error", format!("{e:#}")),
|
||||
};
|
||||
// Every failed request says why, once, here (#63). Before this the
|
||||
// envelope went to the client and nothing reached the journal: a 503
|
||||
// `insufficient_vram` was returned in 225 ms with no server-side
|
||||
// trace, and the guard responsible could only be identified by
|
||||
// reading the source and reproducing its arithmetic (#257).
|
||||
//
|
||||
// `extra` carries the numbers that make a rejection actionable —
|
||||
// free_mb/required_mb, prompt_len/max, retry_after — so the log line
|
||||
// is self-contained rather than a pointer to go and look elsewhere.
|
||||
tracing::warn!(
|
||||
status = env.status,
|
||||
error_type = %env.error_type,
|
||||
code = env.code.as_deref().unwrap_or("-"),
|
||||
detail = %env.message,
|
||||
extra = %serde_json::to_string(&env.extra).unwrap_or_default(),
|
||||
"request rejected"
|
||||
);
|
||||
envelope_response(env)
|
||||
}
|
||||
|
||||
@@ -938,3 +992,68 @@ mod error_envelope_tests {
|
||||
assert!(error["message"].as_str().unwrap().contains("kaboom"));
|
||||
}
|
||||
}
|
||||
|
||||
// ── Request-body extraction (#63) ────────────────────────────────────
|
||||
|
||||
/// `Json<T>` that fails like the rest of our API instead of like axum.
|
||||
///
|
||||
/// axum's own extractor answers a malformed body with a bare string —
|
||||
/// `"Failed to deserialize the JSON body into the target type: …"` —
|
||||
/// at 400/422, *before any handler runs*, so nothing logs it either.
|
||||
/// Both halves of that hurt: no OpenAI client can parse the body (they
|
||||
/// look for `error.code` / `error.type`), and the operator sees nothing
|
||||
/// at all in the journal.
|
||||
///
|
||||
/// That combination cost a real session (#253): a client sent a
|
||||
/// spec-legal shape our types didn't model, got 422 with an unparseable
|
||||
/// body, surfaced nothing to its user, and neuron logged not one line —
|
||||
/// the cause was findable only by reading the wire types.
|
||||
///
|
||||
/// So: same #63 envelope as every other error, plus a `WARN` naming the
|
||||
/// endpoint and the deserialization detail.
|
||||
pub struct ApiJson<T>(pub T);
|
||||
|
||||
impl<T, S> axum::extract::FromRequest<S> for ApiJson<T>
|
||||
where
|
||||
T: serde::de::DeserializeOwned,
|
||||
S: Send + Sync,
|
||||
{
|
||||
type Rejection = axum::response::Response;
|
||||
|
||||
async fn from_request(req: axum::extract::Request, state: &S) -> Result<Self, Self::Rejection> {
|
||||
use axum::extract::rejection::JsonRejection;
|
||||
let (method, uri) = (req.method().clone(), req.uri().clone());
|
||||
match Json::<T>::from_request(req, state).await {
|
||||
Ok(Json(value)) => Ok(ApiJson(value)),
|
||||
Err(rejection) => {
|
||||
// Distinguish "shape we don't accept" (422 — the body
|
||||
// parsed as JSON but not as this request) from "not JSON
|
||||
// at all" (400) and a wrong content-type (415), because
|
||||
// they need different fixes from the caller.
|
||||
let (status, code) = match &rejection {
|
||||
JsonRejection::JsonDataError(_) => (422, "invalid_request_body"),
|
||||
JsonRejection::JsonSyntaxError(_) => (400, "malformed_json"),
|
||||
JsonRejection::MissingJsonContentType(_) => (415, "unsupported_media_type"),
|
||||
_ => (400, "invalid_request_body"),
|
||||
};
|
||||
let detail = rejection.body_text();
|
||||
tracing::warn!(
|
||||
method = %method,
|
||||
uri = %uri,
|
||||
status,
|
||||
code,
|
||||
detail = %detail,
|
||||
"request body rejected before dispatch"
|
||||
);
|
||||
Err(envelope_response(
|
||||
cortex_core::error_envelope::OpenAiError::new(
|
||||
status,
|
||||
"invalid_request_error",
|
||||
code,
|
||||
detail,
|
||||
),
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -180,9 +180,44 @@ pub struct AdmissionConfig {
|
||||
/// for any single principal (resolved from the `x-helexa-*` headers
|
||||
/// cortex stamps), so one client can't monopolize the queue while others
|
||||
/// wait. Over-cap → `429 rate_limit_exceeded` + `Retry-After`. `0`
|
||||
/// disables the cap; anonymous requests are always exempt.
|
||||
/// disables the cap.
|
||||
///
|
||||
/// Anonymous requests carry no principal and so cannot be capped by it.
|
||||
/// They are bounded instead by `anon_max_in_flight` / `anon_max_pending`
|
||||
/// below (#262).
|
||||
#[serde(default = "default_admission_max_per_principal")]
|
||||
pub max_per_principal: usize,
|
||||
/// Ceiling on **anonymous** requests holding an in-flight seat (#262).
|
||||
///
|
||||
/// Anonymous traffic is served from capacity left over once identified
|
||||
/// traffic is satisfied: it never takes a seat while an identified
|
||||
/// request is waiting, and it may never hold every seat at once — which
|
||||
/// is what this bounds, because a request already running cannot be
|
||||
/// preempted. Reserving one seat is what caps how long a newly arrived
|
||||
/// identified request waits: one request's duration, not all of them.
|
||||
///
|
||||
/// Unset (the default) derives `max_in_flight - 1`, floored at 1 so a
|
||||
/// single-seat model still serves anonymous callers. `0` refuses
|
||||
/// anonymous traffic outright.
|
||||
#[serde(default)]
|
||||
pub anon_max_in_flight: Option<usize>,
|
||||
/// Ceiling on anonymous requests in flight **plus** queued (#262).
|
||||
///
|
||||
/// The queue needs the same reservation as the seats: without it an
|
||||
/// anonymous flood fills the queue and identified callers are refused at
|
||||
/// the door, before priority can help them.
|
||||
///
|
||||
/// Unset derives `max_in_flight + max_queue_depth - 1`, floored at 1.
|
||||
#[serde(default)]
|
||||
pub anon_max_pending: Option<usize>,
|
||||
/// Maximum seconds a request waits for enough **KV budget** to free
|
||||
/// before it is refused (#257). Deliberately far longer than
|
||||
/// `max_wait_secs`: that one waits for a slot to turn over, this one
|
||||
/// waits for a whole long-context sequence to finish, which on a 27B at
|
||||
/// 64k context is minutes rather than seconds. Too short and the queue
|
||||
/// merely converts a fast rejection into a slow one.
|
||||
#[serde(default = "default_admission_kv_max_wait_secs")]
|
||||
pub kv_max_wait_secs: u64,
|
||||
}
|
||||
|
||||
impl Default for AdmissionConfig {
|
||||
@@ -192,6 +227,9 @@ impl Default for AdmissionConfig {
|
||||
max_queue_depth: default_admission_max_queue_depth(),
|
||||
max_wait_secs: default_admission_max_wait_secs(),
|
||||
max_per_principal: default_admission_max_per_principal(),
|
||||
anon_max_in_flight: None,
|
||||
anon_max_pending: None,
|
||||
kv_max_wait_secs: default_admission_kv_max_wait_secs(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -328,6 +366,13 @@ fn default_bootstrap_prefill_tok_per_sec() -> f64 {
|
||||
800.0
|
||||
}
|
||||
|
||||
/// Long enough for a multi-minute long-context turn ahead of us to finish
|
||||
/// and return its KV, but inside cortex's 300 s proxy timeout so the
|
||||
/// caller is still listening when we answer.
|
||||
fn default_admission_kv_max_wait_secs() -> u64 {
|
||||
240
|
||||
}
|
||||
|
||||
fn default_activation_headroom_mb() -> u64 {
|
||||
2048
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -227,6 +227,10 @@ pub fn render_chat_template(
|
||||
let content_value = match &m.content {
|
||||
MessageContent::Text(s) => Value::String(s.clone()),
|
||||
MessageContent::Parts(parts) => Value::Array(parts.clone()),
|
||||
// Rendered as JSON null so the template's own
|
||||
// `content is none` branch handles it — that is how HF
|
||||
// templates model a tool-calls-only assistant turn.
|
||||
MessageContent::Null => Value::Null,
|
||||
};
|
||||
let mut obj = serde_json::Map::new();
|
||||
obj.insert("role".into(), Value::String(m.role.clone()));
|
||||
@@ -417,6 +421,56 @@ mod tests {
|
||||
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
|
||||
}
|
||||
|
||||
/// The second turn of an OpenAI-native agentic loop: the client
|
||||
/// replays its own assistant turn, which carries `tool_calls` and
|
||||
/// `"content": null`. That must render as the tool-call block —
|
||||
/// not as the literal string "none", and not as a render error
|
||||
/// (which would silently drop back to the no-tools fallback
|
||||
/// template and break tool calling for the rest of the session).
|
||||
#[test]
|
||||
fn real_template_renders_null_content_assistant_tool_call_turn() {
|
||||
let template = include_str!("testdata/qwen3_6_chat_template.jinja");
|
||||
let mut assistant_extras = serde_json::Map::new();
|
||||
assistant_extras.insert(
|
||||
"tool_calls".into(),
|
||||
json!([{
|
||||
"id": "call_0", "type": "function",
|
||||
"function": {"name": "bash", "arguments": "{\"command\":\"ls\"}"}
|
||||
}]),
|
||||
);
|
||||
let mut tool_extras = serde_json::Map::new();
|
||||
tool_extras.insert("tool_call_id".into(), json!("call_0"));
|
||||
|
||||
let messages = vec![
|
||||
user_msg("list the files"),
|
||||
ChatMessage {
|
||||
role: "assistant".into(),
|
||||
content: MessageContent::Null,
|
||||
extra: Value::Object(assistant_extras),
|
||||
},
|
||||
ChatMessage {
|
||||
role: "tool".into(),
|
||||
content: MessageContent::Text("a.txt".into()),
|
||||
extra: Value::Object(tool_extras),
|
||||
},
|
||||
];
|
||||
|
||||
let out = render_chat_template(template, &messages, &Value::Null, &Value::Null)
|
||||
.expect("null assistant content must render, not error");
|
||||
assert!(
|
||||
out.contains("<function=bash>"),
|
||||
"tool call must survive the null-content turn; rendered:\n{out}"
|
||||
);
|
||||
assert!(
|
||||
out.contains("<tool_response>\na.txt"),
|
||||
"tool result must render; rendered:\n{out}"
|
||||
);
|
||||
assert!(
|
||||
!out.contains("none"),
|
||||
"null content must render as empty, not the literal \"none\"; rendered:\n{out}"
|
||||
);
|
||||
}
|
||||
|
||||
fn user_msg(text: &str) -> ChatMessage {
|
||||
ChatMessage {
|
||||
role: "user".into(),
|
||||
|
||||
@@ -850,7 +850,7 @@ fn load_gguf_inner(
|
||||
tracing::info!(model = %model_id, path = ?gguf_path, "loading GGUF");
|
||||
let mut file = std::fs::File::open(gguf_path).context("open GGUF file")?;
|
||||
let content =
|
||||
gguf_file::Content::read(&mut file).map_err(|e| anyhow::anyhow!("parse GGUF: {e}"))?;
|
||||
gguf_file::Content::read(&mut file).map_err(|e| anyhow::anyhow!("parse GGUF: {e:#}"))?;
|
||||
|
||||
let architecture = content
|
||||
.metadata
|
||||
@@ -865,7 +865,7 @@ fn load_gguf_inner(
|
||||
match architecture.as_str() {
|
||||
"qwen3" => {
|
||||
let weights = QuantizedQwen3Weights::from_gguf(content, &mut file, device)
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf qwen3: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf qwen3: {e:#}"))?;
|
||||
Ok(ModelArch::Qwen3Quantized(weights))
|
||||
}
|
||||
"qwen3moe" => {
|
||||
@@ -874,12 +874,12 @@ fn load_gguf_inner(
|
||||
// accumulation precision and gives the best tokens/sec on
|
||||
// consumer cards.
|
||||
let weights = GGUFQWenMoE::from_gguf(content, &mut file, device, DType::F16)
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf qwen3_moe: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf qwen3_moe: {e:#}"))?;
|
||||
Ok(ModelArch::Qwen3MoeQuantized(weights))
|
||||
}
|
||||
"llama" => {
|
||||
let weights = QuantizedLlamaWeights::from_gguf(content, &mut file, device)
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf llama: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("from_gguf llama: {e:#}"))?;
|
||||
Ok(ModelArch::LlamaQuantized(weights))
|
||||
}
|
||||
other => anyhow::bail!(
|
||||
@@ -943,14 +943,14 @@ fn load_dense_inner(
|
||||
let cfg: qwen3_dense::Config =
|
||||
serde_json::from_str(&cfg_text).context("parse Qwen3 config.json")?;
|
||||
let model = qwen3_dense::ModelForCausalLM::new(&cfg, vb)
|
||||
.map_err(|e| anyhow::anyhow!("build Qwen3 dense model: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("build Qwen3 dense model: {e:#}"))?;
|
||||
Ok(ModelArch::Qwen3Dense(model))
|
||||
}
|
||||
"qwen3_moe" => {
|
||||
let cfg: qwen3_moe_dense::Config =
|
||||
serde_json::from_str(&cfg_text).context("parse Qwen3 MoE config.json")?;
|
||||
let model = qwen3_moe_dense::ModelForCausalLM::new(&cfg, vb)
|
||||
.map_err(|e| anyhow::anyhow!("build Qwen3 MoE dense model: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("build Qwen3 MoE dense model: {e:#}"))?;
|
||||
Ok(ModelArch::Qwen3MoeDense(model))
|
||||
}
|
||||
"llama" => {
|
||||
@@ -960,7 +960,7 @@ fn load_dense_inner(
|
||||
let cache = llama_dense::Cache::new(true, dtype, &config, device)
|
||||
.context("build Llama Cache")?;
|
||||
let model = llama_dense::Llama::load(vb, &config)
|
||||
.map_err(|e| anyhow::anyhow!("build Llama dense model: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("build Llama dense model: {e:#}"))?;
|
||||
Ok(ModelArch::LlamaDense(Box::new(
|
||||
crate::harness::candle::LlamaDense::from_parts(
|
||||
model,
|
||||
|
||||
@@ -170,7 +170,7 @@ async fn op_step(
|
||||
(BackendConfig::Single { worker, handle, .. }, ActiveSession::Single { .. }) => worker
|
||||
.forward_logits_batch(*handle, tokens, prefix_lens, padded_len, step)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("batched decode step {step}: {e}")),
|
||||
.map_err(|e| anyhow::anyhow!("batched decode step {step}: {e:#}")),
|
||||
#[cfg(feature = "cuda")]
|
||||
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => pool
|
||||
.generate_step_batch(
|
||||
@@ -182,7 +182,7 @@ async fn op_step(
|
||||
step,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("TP batched decode step {step}: {e}")),
|
||||
.map_err(|e| anyhow::anyhow!("TP batched decode step {step}: {e:#}")),
|
||||
#[cfg(feature = "cuda")]
|
||||
_ => anyhow::bail!("engine backend/session mismatch"),
|
||||
}
|
||||
@@ -204,13 +204,13 @@ async fn op_assemble(
|
||||
worker
|
||||
.assemble_kv_batch(*handle, seqs)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("assemble_kv_batch: {e}"))
|
||||
.map_err(|e| anyhow::anyhow!("assemble_kv_batch: {e:#}"))
|
||||
}
|
||||
#[cfg(feature = "cuda")]
|
||||
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => pool
|
||||
.assemble_kv_batch(&tp.model_id, tp.leader_handle, seqs)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("TP assemble_kv_batch: {e}")),
|
||||
.map_err(|e| anyhow::anyhow!("TP assemble_kv_batch: {e:#}")),
|
||||
#[cfg(feature = "cuda")]
|
||||
_ => anyhow::bail!("engine backend/session mismatch"),
|
||||
}
|
||||
@@ -229,7 +229,7 @@ async fn op_extract(
|
||||
(BackendConfig::Single { worker, handle, .. }, ActiveSession::Single { .. }) => Ok(worker
|
||||
.extract_kv_rows(*handle, rows, padded_len, steps)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("extract_kv_rows: {e}"))?
|
||||
.map_err(|e| anyhow::anyhow!("extract_kv_rows: {e:#}"))?
|
||||
.into_iter()
|
||||
.map(|(id, _bytes)| id.0)
|
||||
.collect()),
|
||||
@@ -247,7 +247,7 @@ async fn op_extract(
|
||||
ids.clone(),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("TP extract_kv_rows: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("TP extract_kv_rows: {e:#}"))?;
|
||||
Ok(ids)
|
||||
}
|
||||
#[cfg(feature = "cuda")]
|
||||
@@ -262,13 +262,13 @@ async fn op_snapshot_seq(cfg: &EngineConfig, session: &mut ActiveSession) -> Res
|
||||
.snapshot_kv(*handle)
|
||||
.await
|
||||
.map(|(id, _bytes)| id.0)
|
||||
.map_err(|e| anyhow::anyhow!("snapshot after prefill: {e}")),
|
||||
.map_err(|e| anyhow::anyhow!("snapshot after prefill: {e:#}")),
|
||||
#[cfg(feature = "cuda")]
|
||||
(BackendConfig::Tp { .. }, ActiveSession::Tp { tp, pool }) => {
|
||||
let id = tp.next_snapshot_id.fetch_add(1, Ordering::Relaxed);
|
||||
pool.snapshot_kv_cache(&tp.model_id, tp.leader_handle, id)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("TP snapshot after prefill: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("TP snapshot after prefill: {e:#}"))?;
|
||||
Ok(id)
|
||||
}
|
||||
#[cfg(feature = "cuda")]
|
||||
|
||||
@@ -30,6 +30,10 @@ pub mod worker;
|
||||
use anyhow::{Context, Result};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Stdio;
|
||||
#[cfg(any(feature = "cuda", test))]
|
||||
use std::sync::Arc;
|
||||
#[cfg(any(feature = "cuda", test))]
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, Lines};
|
||||
use tokio::process::{Child, ChildStdin, ChildStdout, Command};
|
||||
|
||||
@@ -319,6 +323,87 @@ pub struct WorkerPool {
|
||||
leader_comm: Option<nccl_state::SendComm>,
|
||||
}
|
||||
|
||||
/// How long the watchdog waits for `ncclCommAbort` to return before
|
||||
/// concluding the process is unrecoverable (#265). Generous relative to a
|
||||
/// healthy abort (milliseconds) and short relative to a human noticing.
|
||||
#[cfg(any(feature = "cuda", test))]
|
||||
fn tp_abort_deadline() -> std::time::Duration {
|
||||
let secs = std::env::var("NEURON_TP_ABORT_DEADLINE_S")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.filter(|s| *s > 0)
|
||||
.unwrap_or(60);
|
||||
std::time::Duration::from_secs(secs)
|
||||
}
|
||||
|
||||
/// Wait for the abort thread to signal completion. `true` if it returned
|
||||
/// in time (recovery proceeds normally), `false` if the deadline elapsed
|
||||
/// (the process is unrecoverable).
|
||||
///
|
||||
/// Split out from the deadman thread so the decision is testable without
|
||||
/// a GPU: the branch that ends the process is the one most worth
|
||||
/// exercising, and the one hardest to reach by accident.
|
||||
#[cfg(any(feature = "cuda", test))]
|
||||
fn wait_for_abort(done: &AtomicBool, deadline: std::time::Duration) -> bool {
|
||||
let start = std::time::Instant::now();
|
||||
while start.elapsed() < deadline {
|
||||
if done.load(Ordering::Acquire) {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(25));
|
||||
}
|
||||
// One last look: the abort may have landed inside the final sleep.
|
||||
done.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
/// Exit non-zero so systemd's `Restart=on-failure` restarts us (#265).
|
||||
///
|
||||
/// Called only when the process cannot recover on its own: the NCCL
|
||||
/// collective is wedged and the abort that would unblock it has itself
|
||||
/// hung. The daemon then serves nothing — on beast 2026-08-17 it sat in
|
||||
/// that state for 70 minutes with GPUs idle, weights resident, 27
|
||||
/// connections queued on the listener, and `systemctl` reporting it
|
||||
/// active. A process that knows it is unrecoverable should not wait for a
|
||||
/// human.
|
||||
///
|
||||
/// **Everything diagnostic is logged first.** A restart destroys the
|
||||
/// state that explains the hang, so trading a diagnosable outage for an
|
||||
/// undiagnosable self-healing one would be a poor bargain.
|
||||
///
|
||||
/// `process::exit` deliberately skips destructors and the graceful-drain
|
||||
/// path. That is normally the wrong instinct, but draining a wedged
|
||||
/// process cannot work — and `TimeoutStopSec=120s` plus
|
||||
/// `TimeoutStopFailureMode=abort` would add two more minutes of outage on
|
||||
/// top of the wedge (#256).
|
||||
#[cfg(feature = "cuda")]
|
||||
fn exit_unrecoverable(model_id: &str, reason: &str) -> ! {
|
||||
// Read straight from the driver rather than any cached value: the
|
||||
// cache is refreshed by the very machinery that is wedged.
|
||||
let vram = std::process::Command::new("nvidia-smi")
|
||||
.args([
|
||||
"--query-gpu=index,memory.used,memory.free,utilization.gpu",
|
||||
"--format=csv,noheader",
|
||||
])
|
||||
.output()
|
||||
.ok()
|
||||
.map(|o| String::from_utf8_lossy(&o.stdout).replace('\n', " | "))
|
||||
.unwrap_or_else(|| "unavailable".into());
|
||||
tracing::error!(
|
||||
model = %model_id,
|
||||
reason = %reason,
|
||||
vram = %vram,
|
||||
pid = std::process::id(),
|
||||
"tp watchdog: UNRECOVERABLE — exiting non-zero so systemd restarts this daemon (#265). \
|
||||
The GPU state above is from immediately before the exit; the wedged collective and its \
|
||||
request id are in the preceding watchdog lines."
|
||||
);
|
||||
// Give the logging layer a moment to flush before the process dies —
|
||||
// an exit that eats its own explanation is the failure mode this
|
||||
// whole change exists to avoid.
|
||||
std::thread::sleep(std::time::Duration::from_millis(500));
|
||||
std::process::exit(70) // EX_SOFTWARE
|
||||
}
|
||||
|
||||
/// Per-step deadline for a TP forward (#17 Stage 2). A healthy decode
|
||||
/// step or chunked prefill completes in well under a second; a wedged
|
||||
/// NCCL collective never returns. Generous default so no legitimate step
|
||||
@@ -349,23 +434,66 @@ impl WorkerPool {
|
||||
"tp watchdog: leader forward exceeded deadline — NCCL collective wedged; \
|
||||
aborting comm to unblock the leader thread for auto-recovery"
|
||||
);
|
||||
match &self.leader_comm {
|
||||
Some(c) => match c.0.abort() {
|
||||
Ok(()) => tracing::error!(
|
||||
model = %model_id,
|
||||
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
|
||||
failing the step so the model auto-recovers (unload+reload)"
|
||||
),
|
||||
Err(e) => tracing::error!(
|
||||
model = %model_id, error = ?e,
|
||||
"tp watchdog: ncclCommAbort failed — recovery may stall until a process restart"
|
||||
),
|
||||
},
|
||||
None => tracing::error!(
|
||||
let Some(comm) = self.leader_comm.as_ref().map(|c| c.0.clone()) else {
|
||||
tracing::error!(
|
||||
model = %model_id,
|
||||
"tp watchdog: no cached leader comm handle — cannot abort; recovery will rely \
|
||||
on a process restart"
|
||||
),
|
||||
"tp watchdog: no cached leader comm handle — cannot abort; exiting so \
|
||||
systemd restarts us (#265)"
|
||||
);
|
||||
exit_unrecoverable(model_id, "no leader comm handle to abort");
|
||||
};
|
||||
|
||||
// `ncclCommAbort` can itself block forever — observed on beast
|
||||
// 2026-08-17, where neither the success nor the failure branch below
|
||||
// was ever reached and the daemon sat wedged for 70 minutes with 27
|
||||
// connections queued on its listener (#265). So the abort runs on its
|
||||
// own thread and the deadline below does NOT depend on it returning:
|
||||
// an abort that hangs must not take the supervisor with it.
|
||||
let done = Arc::new(AtomicBool::new(false));
|
||||
let signal = Arc::clone(&done);
|
||||
let owned_model = model_id.to_string();
|
||||
std::thread::Builder::new()
|
||||
.name("tp-watchdog-abort".into())
|
||||
.spawn(move || {
|
||||
match comm.abort() {
|
||||
Ok(()) => tracing::error!(
|
||||
model = %owned_model,
|
||||
"tp watchdog: ncclCommAbort succeeded — wedged collective unblocked; \
|
||||
failing the step so the model auto-recovers (unload+reload)"
|
||||
),
|
||||
Err(e) => tracing::error!(
|
||||
model = %owned_model, error = ?e,
|
||||
"tp watchdog: ncclCommAbort failed — the collective is still wedged"
|
||||
),
|
||||
}
|
||||
// Returning at all is the signal, success or failure: a
|
||||
// returned abort leaves the process responsive, so
|
||||
// auto-recovery can run and we must not exit under it.
|
||||
signal.store(true, Ordering::Release);
|
||||
})
|
||||
.map(|_| ())
|
||||
.unwrap_or_else(|e| {
|
||||
// Cannot even spawn a thread — nothing here can recover.
|
||||
tracing::error!(model = %model_id, error = %e, "tp watchdog: abort thread spawn failed");
|
||||
exit_unrecoverable(model_id, "could not spawn the abort thread");
|
||||
});
|
||||
|
||||
// Deadman. Disarmed only by the abort returning.
|
||||
let deadline = tp_abort_deadline();
|
||||
let owned_model = model_id.to_string();
|
||||
if let Err(e) = std::thread::Builder::new()
|
||||
.name("tp-watchdog-deadman".into())
|
||||
.spawn(move || {
|
||||
if wait_for_abort(&done, deadline) {
|
||||
return; // abort came back; the normal path owns recovery
|
||||
}
|
||||
exit_unrecoverable(
|
||||
&owned_model,
|
||||
"ncclCommAbort did not return within the deadline",
|
||||
);
|
||||
})
|
||||
{
|
||||
tracing::error!(model = %model_id, error = %e, "tp watchdog: deadman spawn failed");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -497,7 +625,7 @@ impl WorkerPool {
|
||||
.leader_worker
|
||||
.nccl_init(leader_cfg, comm_id.clone())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("leader NCCL init via device worker: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("leader NCCL init via device worker: {e:#}"))?;
|
||||
match leader_resp {
|
||||
rpc::WorkerResponse::InitOk => {}
|
||||
rpc::WorkerResponse::Error { kind, message } => {
|
||||
@@ -569,7 +697,7 @@ impl WorkerPool {
|
||||
.leader_worker
|
||||
.nccl_sanity()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("leader NCCL sanity via device worker: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("leader NCCL sanity via device worker: {e:#}"))?;
|
||||
|
||||
let expected = self.world_size;
|
||||
let leader_sum = match leader_resp {
|
||||
@@ -694,7 +822,7 @@ impl WorkerPool {
|
||||
world_size,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("leader TP shard load via device worker: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("leader TP shard load via device worker: {e:#}"))?;
|
||||
|
||||
// 3. Collect worker confirmations. Anything other than
|
||||
// LoadDenseShardOk aborts the whole load — the leader's
|
||||
@@ -1250,7 +1378,7 @@ impl WorkerPool {
|
||||
.leader_worker
|
||||
.query_vram()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("leader query_vram: {e}"))?;
|
||||
.map_err(|e| anyhow::anyhow!("leader query_vram: {e:#}"))?;
|
||||
let mut frees = vec![leader_free_mb];
|
||||
let worker_errors = drain_workers(&mut self.workers, |r| match r {
|
||||
WorkerResponse::VramInfo { free_mb, .. } => {
|
||||
@@ -1442,3 +1570,73 @@ impl WorkerPool {
|
||||
&self.exe
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod watchdog_tests {
|
||||
use super::*;
|
||||
use std::time::Duration;
|
||||
|
||||
/// An abort that returns disarms the deadman — the normal path then
|
||||
/// owns recovery and the process must NOT exit.
|
||||
#[test]
|
||||
fn abort_returning_disarms_the_deadman() {
|
||||
let done = Arc::new(AtomicBool::new(false));
|
||||
let signal = Arc::clone(&done);
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
signal.store(true, Ordering::Release);
|
||||
});
|
||||
assert!(
|
||||
wait_for_abort(&done, Duration::from_secs(5)),
|
||||
"a returning abort must disarm the deadman"
|
||||
);
|
||||
}
|
||||
|
||||
/// The #265 case: `ncclCommAbort` never returns. The deadline must
|
||||
/// elapse and report unrecoverable, which is what triggers the exit.
|
||||
#[test]
|
||||
fn abort_that_never_returns_trips_the_deadline() {
|
||||
let done = Arc::new(AtomicBool::new(false)); // never set
|
||||
let start = std::time::Instant::now();
|
||||
assert!(
|
||||
!wait_for_abort(&done, Duration::from_millis(200)),
|
||||
"an abort that never returns must trip the deadline"
|
||||
);
|
||||
assert!(
|
||||
start.elapsed() >= Duration::from_millis(200),
|
||||
"must actually wait the deadline rather than failing fast"
|
||||
);
|
||||
}
|
||||
|
||||
/// An abort landing in the last instant still counts as returned —
|
||||
/// exiting on a race would restart a process that had just recovered.
|
||||
#[test]
|
||||
fn abort_landing_at_the_deadline_still_counts() {
|
||||
let done = Arc::new(AtomicBool::new(false));
|
||||
let signal = Arc::clone(&done);
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(Duration::from_millis(180));
|
||||
signal.store(true, Ordering::Release);
|
||||
});
|
||||
assert!(
|
||||
wait_for_abort(&done, Duration::from_millis(200)),
|
||||
"an abort landing at the deadline must not be treated as a hang"
|
||||
);
|
||||
}
|
||||
|
||||
/// The deadline is operator-tunable but never zero — a zero deadline
|
||||
/// would exit the moment any watchdog fired.
|
||||
#[test]
|
||||
fn deadline_defaults_sanely_and_rejects_zero() {
|
||||
assert_eq!(tp_abort_deadline(), Duration::from_secs(60));
|
||||
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "0") };
|
||||
assert_eq!(
|
||||
tp_abort_deadline(),
|
||||
Duration::from_secs(60),
|
||||
"zero must fall back, not exit instantly on every watchdog"
|
||||
);
|
||||
unsafe { std::env::set_var("NEURON_TP_ABORT_DEADLINE_S", "15") };
|
||||
assert_eq!(tp_abort_deadline(), Duration::from_secs(15));
|
||||
unsafe { std::env::remove_var("NEURON_TP_ABORT_DEADLINE_S") };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -186,6 +186,9 @@ async fn daemon(args: Args) -> Result<()> {
|
||||
let registry = HarnessRegistry::from_configs(&cfg.harnesses, &bind_url, &cfg.harness);
|
||||
discovery_result.harnesses = registry.names();
|
||||
let candle = registry.candle();
|
||||
// A second handle for the shutdown future: it must close admission
|
||||
// before the drain begins, and `candle` is moved into NeuronState.
|
||||
let shutdown_candle = candle.clone();
|
||||
|
||||
let health_cache = Arc::new(health::HealthCache::new());
|
||||
health_cache
|
||||
@@ -246,7 +249,7 @@ async fn daemon(args: Args) -> Result<()> {
|
||||
}
|
||||
|
||||
axum::serve(listener, app)
|
||||
.with_graceful_shutdown(startup::shutdown_signal())
|
||||
.with_graceful_shutdown(startup::shutdown_signal(shutdown_candle))
|
||||
.await?;
|
||||
|
||||
// Deactivation: serve has returned (graceful shutdown signal
|
||||
|
||||
@@ -9,6 +9,7 @@ use crate::activation::ActivationTracker;
|
||||
use crate::harness::HarnessRegistry;
|
||||
use crate::harness::preflight::PreflightError;
|
||||
use cortex_core::harness::ModelSpec;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::signal;
|
||||
|
||||
@@ -165,7 +166,18 @@ fn preflight_kind(err: &PreflightError) -> &'static str {
|
||||
/// Wired into `axum::serve(...).with_graceful_shutdown(shutdown_signal())`
|
||||
/// so the HTTP listener stops accepting new connections, lets in-flight
|
||||
/// requests drain, and then yields control back to main for cleanup.
|
||||
pub async fn shutdown_signal() {
|
||||
///
|
||||
/// **Before returning, it closes admission and wakes every waiter (#256).**
|
||||
/// Order matters: axum's graceful shutdown waits for *all* in-flight
|
||||
/// requests, and a request queued behind the TP pool lock is in-flight as
|
||||
/// far as axum is concerned. Those waits are unbounded — held by whichever
|
||||
/// request is mid-forward — so without this the drain cannot finish and
|
||||
/// systemd SIGKILLs at `TimeoutStopSec`. Observed 2026-08-15: SIGTERM at
|
||||
/// 13:03:14, one queued request logging "still waiting on pool lock" every
|
||||
/// two seconds, `Failed with result 'timeout'` at 13:05:24. The same node
|
||||
/// drained in about a second when idle, which is what identified the
|
||||
/// queued waiter as the cause.
|
||||
pub async fn shutdown_signal(candle: Option<Arc<crate::harness::candle::CandleHarness>>) {
|
||||
let ctrl_c = async {
|
||||
signal::ctrl_c().await.ok();
|
||||
};
|
||||
@@ -179,6 +191,13 @@ pub async fn shutdown_signal() {
|
||||
_ = ctrl_c => tracing::info!("received SIGINT, shutting down"),
|
||||
_ = terminate => tracing::info!("received SIGTERM, shutting down"),
|
||||
}
|
||||
// Refuse new admissions and abandon the unbounded waits, so the drain
|
||||
// that starts the moment this future resolves has a bounded amount of
|
||||
// work left to wait for.
|
||||
crate::harness::candle::begin_shutdown();
|
||||
if let Some(candle) = candle {
|
||||
candle.close_all_admission().await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Unload every model currently registered. Called from `main.rs` after
|
||||
|
||||
@@ -157,6 +157,9 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
|
||||
temperature: req.temperature,
|
||||
top_p: req.top_p,
|
||||
max_tokens: req.max_output_tokens,
|
||||
// Responses spells the cap `max_output_tokens`, already
|
||||
// resolved onto the legacy field above.
|
||||
max_completion_tokens: None,
|
||||
stream: Some(req.stream),
|
||||
extra: Value::Object(extra),
|
||||
})
|
||||
|
||||
@@ -709,3 +709,71 @@ async fn test_images_chat_at_image_endpoint_is_not_found_for_text_model() {
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 404);
|
||||
}
|
||||
|
||||
/// A malformed body must come back as the #63 envelope, not axum's bare
|
||||
/// string (#253). An OpenAI client reads `error.code` / `error.type`; a
|
||||
/// plain string leaves it with nothing to act on, which is exactly how a
|
||||
/// harness ended up quitting silently mid-session.
|
||||
#[tokio::test]
|
||||
async fn malformed_body_returns_a_parseable_error_envelope() {
|
||||
let base = spawn_neuron(fake_discovery()).await;
|
||||
let client = reqwest::Client::new();
|
||||
|
||||
// Valid JSON, wrong shape for the request type → 422.
|
||||
let resp = client
|
||||
.post(format!("{base}/v1/chat/completions"))
|
||||
.json(&json!({"model": "m", "messages": "not-an-array"}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 422);
|
||||
let body: serde_json::Value = resp.json().await.expect("body must be JSON");
|
||||
assert_eq!(body["error"]["type"], "invalid_request_error");
|
||||
assert_eq!(body["error"]["code"], "invalid_request_body");
|
||||
assert!(
|
||||
body["error"]["message"]
|
||||
.as_str()
|
||||
.is_some_and(|m| !m.is_empty()),
|
||||
"message must say what was wrong: {body}"
|
||||
);
|
||||
|
||||
// Not JSON at all → 400, distinguished from the above so the caller
|
||||
// knows whether to fix its serializer or its schema.
|
||||
let resp = client
|
||||
.post(format!("{base}/v1/chat/completions"))
|
||||
.header("content-type", "application/json")
|
||||
.body("{not json")
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 400);
|
||||
let body: serde_json::Value = resp.json().await.expect("body must be JSON");
|
||||
assert_eq!(body["error"]["code"], "malformed_json");
|
||||
}
|
||||
|
||||
/// The shape from #253 itself: an assistant turn carrying only
|
||||
/// `tool_calls`. It must be *accepted* now — this guards the envelope
|
||||
/// work from quietly re-breaking what #253 fixed.
|
||||
#[tokio::test]
|
||||
async fn null_assistant_content_is_not_a_body_rejection() {
|
||||
let base = spawn_neuron(fake_discovery()).await;
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!("{base}/v1/chat/completions"))
|
||||
.json(&json!({
|
||||
"model": "m",
|
||||
"messages": [
|
||||
{"role": "user", "content": "hi"},
|
||||
{"role": "assistant", "content": null, "tool_calls": [
|
||||
{"id": "c1", "type": "function",
|
||||
"function": {"name": "b", "arguments": "{}"}}]},
|
||||
{"role": "tool", "content": "ok", "tool_call_id": "c1"}
|
||||
]
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
// 503 (no candle harness in this fixture) — the point is that it got
|
||||
// past body extraction rather than being rejected as malformed.
|
||||
assert_ne!(resp.status(), 422, "null content must not be a body error");
|
||||
assert_ne!(resp.status(), 400, "null content must not be a body error");
|
||||
}
|
||||
|
||||
@@ -2,6 +2,22 @@
|
||||
Description=Neuron — per-node GPU discovery and harness daemon for cortex
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Start-limit bound for the self-restart path (#265). When a TP forward
|
||||
# wedges and the NCCL abort that would unblock it also hangs, neuron
|
||||
# exits 70 so Restart=on-failure recovers it in seconds instead of the
|
||||
# 70-minute outage that needed a human on 2026-08-17.
|
||||
#
|
||||
# That only helps if the wedge is transient. A deterministic one — a bad
|
||||
# build, a sick card, a model that wedges on load — would otherwise
|
||||
# restart forever, and a flapping node is WORSE than a down one: cortex
|
||||
# marks it healthy between restarts and keeps routing to it, so every
|
||||
# client sees intermittent failures instead of one node cleanly out.
|
||||
#
|
||||
# Four failures in ten minutes stops the unit and leaves it `failed`,
|
||||
# where `systemctl status` and an alert can see it. Visibly broken beats
|
||||
# invisibly cycling.
|
||||
StartLimitIntervalSec=600
|
||||
StartLimitBurst=4
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
@@ -24,6 +40,11 @@ TimeoutStartSec=1800s
|
||||
# On stop, neuron drains in-flight requests then unloads every model
|
||||
# to release CUDA contexts cleanly. Allow generous time for big-model
|
||||
# unloads; systemd will SIGKILL after this bound.
|
||||
#
|
||||
# Note this bound does NOT apply to the self-restart path (#265): a
|
||||
# wedged process exits directly rather than draining, precisely because
|
||||
# draining cannot work when the collective is stuck — and waiting this
|
||||
# out would add two minutes to an outage that is already total.
|
||||
TimeoutStopSec=120s
|
||||
KillSignal=SIGTERM
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ gateway.
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
clients ─────▶│ cortex │ :31313 API, :31314 metrics
|
||||
clients ────▶│ cortex │ :31313 API, :31314 metrics
|
||||
└──┬────┬────┬─┘
|
||||
│ │ │
|
||||
┌────▼┐ ┌─▼──┐ ┌▼────┐
|
||||
|
||||
@@ -108,6 +108,43 @@ KV cache for several concurrent long contexts is itself substantial.
|
||||
`max_per_principal` is what stops one busy client consuming the whole
|
||||
model.
|
||||
|
||||
### Anonymous callers
|
||||
|
||||
Fair share is keyed on the account and key cortex resolves from the
|
||||
bearer token. A caller that sends no credential — or one that does not
|
||||
resolve, which `require_auth = false` deliberately tolerates — has no
|
||||
key to be shared out, so `max_per_principal` cannot bind it.
|
||||
|
||||
Anonymous traffic is therefore served from the capacity left over once
|
||||
identified traffic is satisfied. It never takes a seat while an
|
||||
authenticated request is waiting for one, whoever arrived first, and it
|
||||
cannot hold every seat at once:
|
||||
|
||||
```toml
|
||||
anon_max_in_flight = 7 # unset: max_in_flight - 1, floored at 1
|
||||
anon_max_pending = 15 # unset: max_in_flight + max_queue_depth - 1
|
||||
```
|
||||
|
||||
The defaults hold back one seat and one queue place, which is what
|
||||
bounds how long an authenticated request can be made to wait: a request
|
||||
already running cannot be preempted, so without a reserved seat an
|
||||
anonymous burst arriving during an idle moment locks the model for as
|
||||
long as all of it takes to finish. On a single-seat model the floor
|
||||
keeps anonymous callers served — priority alone decides who wins there.
|
||||
|
||||
Set `anon_max_in_flight = 0` to refuse anonymous traffic outright, or
|
||||
`require_auth = true` on the gateway to make attribution universal.
|
||||
Anonymous callers still contend with each other, and under sustained
|
||||
authenticated load they are refused rather than served slowly — the
|
||||
refusal is a retryable `503`, so a client that backs off gets served
|
||||
when the load passes.
|
||||
|
||||
Watch it with `cortex_model_anon_in_flight` against
|
||||
`cortex_model_in_flight` (how much of a model's load is unattributable)
|
||||
and `cortex_model_rejections_total{reason="anon_yield"}`. That last
|
||||
counter rising while `reason="wait_timeout"` stays flat means the
|
||||
reservation is doing its job, not that the model is overloaded.
|
||||
|
||||
## VRAM guards
|
||||
|
||||
neuron refuses work it cannot finish rather than crashing partway:
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
Reference in New Issue
Block a user