Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bc7f4a301d
|
||
|
|
318dc5f4e1
|
||
|
|
3b0b3a2227
|
||
|
|
2d1a4ebb2f
|
||
|
|
d87327c3a2
|
||
|
|
a17b3acb70
|
||
|
|
c6c6fc8a50
|
||
|
|
192030dff5
|
||
|
|
24b8de7f82
|
||
|
|
432f8009ce
|
||
|
|
98ba45cbb4
|
||
|
|
522682282c
|
||
|
|
176ce47349
|
||
|
|
00acd283b6
|
||
|
|
eb783c345e
|
||
|
|
72d6a37986
|
||
|
|
039a353adf
|
||
|
|
4e4ffd9b60
|
||
|
|
7ad9f48cad
|
@@ -526,6 +526,161 @@ jobs:
|
||||
# dependency on cortex or the neurons (the sweep loop is version-aware
|
||||
# and picks up whatever each neuron reports whenever). So it runs
|
||||
# alongside the cortex→neurons chain rather than after it.
|
||||
# The vhost sync gets its own job, with NO `needs`, because it is not
|
||||
# part of deploying the gateway binary and must not be able to stop
|
||||
# one.
|
||||
#
|
||||
# It was briefly a step inside deploy-cortex, and that broke the fleet
|
||||
# on its first run (2026-08-27, run 1284): the step hard-fails until an
|
||||
# operator has run script/infra-setup.sh to install its sudoers grant,
|
||||
# a failed step fails the job, and `deploy-neurons` has
|
||||
# `needs: [deploy-cortex]` — so a missing nginx permission silently
|
||||
# skipped the rollout of all three neuron flavours. The fleet stayed
|
||||
# two commits behind while the run reported only "deploy-cortex:
|
||||
# failure".
|
||||
#
|
||||
# #287 names this ordering trap. The lesson is narrower than "order the
|
||||
# steps": a step whose prerequisite is installed by hand does not
|
||||
# belong in a job that something else depends on, because its failure
|
||||
# is not a statement about that job. Moving it later within
|
||||
# deploy-cortex would not have helped — the job still ends red and the
|
||||
# dependents still skip.
|
||||
#
|
||||
# Failing hard here is still right: drift between the repo and the live
|
||||
# file is a real defect, and this job going red says that and nothing
|
||||
# else.
|
||||
deploy-nginx:
|
||||
runs-on: fedora-43
|
||||
if: >-
|
||||
${{
|
||||
github.event_name == 'workflow_dispatch'
|
||||
|| github.event.workflow_run.conclusion == 'success'
|
||||
}}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: SSH init
|
||||
run: |
|
||||
mkdir -p ~/.ssh
|
||||
echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
|
||||
chmod 600 ~/.ssh/id_ed25519
|
||||
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
|
||||
gitea_ci@hanzalova.internal 'hostname -f'
|
||||
|
||||
# nginx vhost config for the internal site. Version-controlled in
|
||||
# `asset/nginx/` since the beginning, but until now only ever
|
||||
# installed by `script/infra-setup.sh` — a one-time, run-by-hand
|
||||
# script. So the repo copy and the live file could diverge with
|
||||
# nothing to detect it, which is the #252 shape: two sources of
|
||||
# truth for one thing and nothing forcing agreement.
|
||||
#
|
||||
# Writes to sites-available and (re)points sites-enabled at it as
|
||||
# a symlink. On this host sites-enabled was a *regular file*, not a
|
||||
# symlink — two independent copies, of which nginx reads one. An
|
||||
# operator editing "the config" would reasonably edit
|
||||
# sites-available, watch `nginx -t` pass, reload, and see nothing
|
||||
# change. Making the link canonical is part of the sync rather
|
||||
# than a separate chore, so the layout cannot drift back.
|
||||
#
|
||||
# `nginx -t` gates the reload: a config that fails validation is
|
||||
# left un-reloaded with the previous one still serving.
|
||||
- name: Sync internal nginx vhost
|
||||
run: |
|
||||
src="asset/nginx/helexa.internal.conf"
|
||||
|
||||
# The sudoers drop-in that permits this is installed by
|
||||
# script/infra-setup.sh, which is run by hand, so the live copy
|
||||
# can lag the repo by however long it has been since anyone ran
|
||||
# it. Check EVERY grant the step needs, not a representative
|
||||
# one: the first version of this preflight tested `nginx -t`
|
||||
# alone, which had existed for ages, so it passed and the job
|
||||
# then died on the two helexa-ratelimit.conf grants added in the
|
||||
# same commit as the check. A preflight that names a remedy
|
||||
# while probing the wrong permission is worse than none — it
|
||||
# tells you the sudoers are fine seconds before they are not.
|
||||
#
|
||||
# Each probe runs the command in a harmless form that still
|
||||
# matches its sudoers pattern (sha256sum reads; nginx -t and
|
||||
# `systemctl show` do not mutate). rsync and ln have no such
|
||||
# form, so they are checked with `sudo -nl`, which reports
|
||||
# whether a command is permitted without running it.
|
||||
missing=""
|
||||
probe() { # probe <description> <command...>
|
||||
local what="$1"; shift
|
||||
ssh gitea_ci@hanzalova.internal "$@" >/dev/null 2>&1 \
|
||||
|| missing="${missing}
|
||||
- ${what}"
|
||||
}
|
||||
probe "nginx -t" \
|
||||
'sudo -n /usr/sbin/nginx -t'
|
||||
probe "sha256sum /etc/nginx/sites-available/helexa.internal.conf" \
|
||||
'sudo -n /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf'
|
||||
probe "sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf" \
|
||||
'sudo -n /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf'
|
||||
probe "rsync -> /etc/nginx/sites-available/helexa.internal.conf" \
|
||||
'sudo -nl /usr/bin/rsync -a x /etc/nginx/sites-available/helexa.internal.conf'
|
||||
probe "rsync -> /etc/nginx/conf.d/helexa-ratelimit.conf" \
|
||||
'sudo -nl /usr/bin/rsync -a x /etc/nginx/conf.d/helexa-ratelimit.conf'
|
||||
probe "ln -sfn sites-available -> sites-enabled" \
|
||||
'sudo -nl /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf'
|
||||
probe "systemctl reload nginx" \
|
||||
'sudo -nl /usr/bin/systemctl reload nginx'
|
||||
|
||||
if [ -n "${missing}" ]; then
|
||||
echo "FAIL: gitea_ci is missing nginx sudoers grants on hanzalova:${missing}" >&2
|
||||
echo "" >&2
|
||||
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
|
||||
echo " install the current asset/sudoers.d/cortex-host.conf. Having run" >&2
|
||||
echo " it before is not enough — grants are added to that file as deploy" >&2
|
||||
echo " steps grow, and the host keeps whatever revision it last saw." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rl="asset/nginx/helexa-ratelimit.conf"
|
||||
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
|
||||
local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
|
||||
remote_sum=$(ssh gitea_ci@hanzalova.internal \
|
||||
'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
|
||||
remote_rl=$(ssh gitea_ci@hanzalova.internal \
|
||||
'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
|
||||
link_ok=$(ssh gitea_ci@hanzalova.internal \
|
||||
'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
|
||||
|
||||
if [ "${local_sum}" = "${remote_sum}" ] \
|
||||
&& [ "${local_rl}" = "${remote_rl}" ] \
|
||||
&& [ "${link_ok}" = "yes" ]; then
|
||||
echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Order matters: the vhost references names defined at http
|
||||
# context by helexa-ratelimit.conf (`limit_req_zone`s and the
|
||||
# `helexa_noquery` log format). Ship that first, or a vhost
|
||||
# referencing a not-yet-installed name fails `nginx -t` and the
|
||||
# deploy stops with the two files out of step.
|
||||
echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
|
||||
rsync --archive --compress \
|
||||
--chown root:root --chmod 0644 \
|
||||
--rsync-path 'sudo rsync' \
|
||||
"${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
|
||||
|
||||
echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
|
||||
rsync --archive --compress \
|
||||
--chown root:root --chmod 0644 \
|
||||
--rsync-path 'sudo rsync' \
|
||||
"${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
|
||||
|
||||
ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
|
||||
set -eu
|
||||
sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
|
||||
/etc/nginx/sites-enabled/helexa.internal.conf
|
||||
if ! sudo /usr/sbin/nginx -t; then
|
||||
echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
|
||||
exit 1
|
||||
fi
|
||||
sudo /usr/bin/systemctl reload nginx
|
||||
echo "nginx reloaded"
|
||||
NGINX
|
||||
|
||||
# Roll helexa-router + helexa-upstream onto the federation service host
|
||||
# (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
|
||||
# two services are independent daemons on one host, so one job handles
|
||||
|
||||
+12
-8
@@ -7,10 +7,10 @@
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# neuron.toml for beast.hanzalova.internal
|
||||
#
|
||||
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.6-27B with
|
||||
# q5k ISQ across both GPUs at activation, matching the validate-neuron
|
||||
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.8-27B with
|
||||
# q6k ISQ across both GPUs at activation, matching the validate-neuron
|
||||
# invocation: `validate-neuron.sh beast.hanzalova.internal
|
||||
# Qwen/Qwen3.6-27B q5k 2`.
|
||||
# Qwen/Qwen3.8-27B q6k 2`.
|
||||
#
|
||||
# Synced to /etc/neuron/neuron.toml by script/infra-setup.sh. Edits
|
||||
# take effect after the next deploy workflow run restarts the service
|
||||
@@ -83,11 +83,15 @@ max_in_flight = 2
|
||||
budget_mb = 2048
|
||||
|
||||
[[default_models]]
|
||||
# Qwen3.8-27B is the resident flagship (#251). Architecturally identical
|
||||
# to the 3.6-27B it replaced — same model_type, layer counts and vision
|
||||
# tower — so the load spec is unchanged. The 3.6-27B stays catalogued at
|
||||
# the same residency class, so it can still be cold-loaded on demand and
|
||||
# the two displace each other in either direction.
|
||||
# Qwen3.8-27B is the resident flagship (#251). It replaced the
|
||||
# 3.6-27B — architecturally identical, same model_type, layer counts and
|
||||
# vision tower — so the load spec is unchanged.
|
||||
#
|
||||
# The 3.6-27B is no longer catalogued (retired 2026-08-27). While both
|
||||
# were listed at the same residency class they could displace each other
|
||||
# in either direction, which is what let a stale `helexa/balanced` alias
|
||||
# evict this model on every authenticated web-chat turn. See models.toml
|
||||
# for the incident that ended the arrangement.
|
||||
model_id = "Qwen/Qwen3.8-27B"
|
||||
harness = "candle"
|
||||
quant = "q6k"
|
||||
|
||||
@@ -24,3 +24,24 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
|
||||
limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
|
||||
limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
|
||||
limit_req_status 429;
|
||||
|
||||
# Access-log format for the search locations, which must not record what
|
||||
# people searched for.
|
||||
#
|
||||
# The default `main` format logs `$request` — method, full URI *and
|
||||
# query string*. For a search proxy that means every query a user types
|
||||
# is written to /var/log/nginx/access.log next to their IP, for
|
||||
# authenticated and anonymous visitors alike. SearXNG's own web UI
|
||||
# defaults to POST partly to avoid exactly this.
|
||||
#
|
||||
# `$uri` is the path with the query stripped, so the operational signal
|
||||
# survives — who called, when, which endpoint, what status, how big, how
|
||||
# long — and the query itself does not. `$request_method` is kept
|
||||
# explicit since `$uri` drops it.
|
||||
#
|
||||
# Rate-limit debugging and abuse detection both work from this: they
|
||||
# need counts, statuses and client addresses, never the search terms.
|
||||
log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
|
||||
'"$request_method $uri $server_protocol" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" $request_time';
|
||||
|
||||
@@ -73,6 +73,9 @@ server {
|
||||
location = /tools/fetch {
|
||||
limit_req zone=helexa_tools burst=10 nodelay;
|
||||
limit_except GET { deny all; }
|
||||
# The `url=` parameter is as revealing as a search query: it is
|
||||
# the page a user is having read to them. Same treatment.
|
||||
access_log /var/log/nginx/access.log helexa_noquery;
|
||||
proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
@@ -83,7 +86,40 @@ server {
|
||||
# federation service host. GET only; the SPA requests format=json.
|
||||
location = /tools/web_search {
|
||||
limit_req zone=helexa_tools burst=10 nodelay;
|
||||
limit_except GET { deny all; }
|
||||
# GET and POST: SearXNG accepts both, and a client that POSTs
|
||||
# keeps its query out of any intermediary's request line. Other
|
||||
# methods have no meaning here.
|
||||
limit_except GET POST { deny all; }
|
||||
# A search query is small. Bounding the body stops this becoming
|
||||
# a way to push arbitrary bytes at the upstream.
|
||||
client_max_body_size 8k;
|
||||
# Never log the query — see helexa_noquery in
|
||||
# conf.d/helexa-ratelimit.conf.
|
||||
access_log /var/log/nginx/access.log helexa_noquery;
|
||||
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_read_timeout 30s;
|
||||
}
|
||||
|
||||
# The same upstream under SearXNG's own path, for clients that speak
|
||||
# the SearXNG API rather than our app's.
|
||||
#
|
||||
# `/tools/web_search` is our name for it, and a generic SearXNG
|
||||
# client cannot guess it: they take a base URL and append `/search`,
|
||||
# because that is what every SearXNG instance serves. Without this
|
||||
# block such a request falls through to `location /` and is answered
|
||||
# by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
|
||||
# client. A 404 would at least be honest; 200-with-HTML sends people
|
||||
# looking for a bug in their own parser.
|
||||
#
|
||||
# Same upstream, same rate-limit zone, same GET-only restriction:
|
||||
# this is an alias, not a second, laxer door.
|
||||
location = /search {
|
||||
limit_req zone=helexa_tools burst=10 nodelay;
|
||||
limit_except GET POST { deny all; }
|
||||
client_max_body_size 8k;
|
||||
access_log /var/log/nginx/access.log helexa_noquery;
|
||||
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
|
||||
@@ -13,6 +13,18 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
|
||||
# deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
|
||||
# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
|
||||
# sites-enabled a symlink to it, validate, reload. Scoped to the one
|
||||
# file and the one service — `nginx -t` is included because a reload
|
||||
# must never happen on a config nginx has rejected.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
|
||||
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service
|
||||
|
||||
@@ -276,6 +276,15 @@ pub enum ResponsesOutputItem {
|
||||
#[serde(default = "default_item_status")]
|
||||
status: String,
|
||||
},
|
||||
/// A think block, as its own output item ahead of the message —
|
||||
/// the same shape the streaming projector emits, so a client sees
|
||||
/// one contract whether or not it streams (#300).
|
||||
Reasoning {
|
||||
id: String,
|
||||
summary: Vec<ResponsesSummaryPart>,
|
||||
#[serde(default = "default_item_status")]
|
||||
status: String,
|
||||
},
|
||||
/// Reserved for the day tool-call extraction lands. The wire
|
||||
/// shape mirrors `ResponsesInputItem::FunctionCall`.
|
||||
FunctionCall {
|
||||
@@ -288,6 +297,24 @@ pub enum ResponsesOutputItem {
|
||||
},
|
||||
}
|
||||
|
||||
/// One `summary_text` part of a reasoning output item.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ResponsesSummaryPart {
|
||||
/// Always `"summary_text"`.
|
||||
#[serde(rename = "type")]
|
||||
pub part_type: String,
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
impl ResponsesSummaryPart {
|
||||
pub fn summary_text(text: impl Into<String>) -> Self {
|
||||
Self {
|
||||
part_type: "summary_text".into(),
|
||||
text: text.into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn default_item_status() -> String {
|
||||
"completed".into()
|
||||
}
|
||||
|
||||
@@ -415,17 +415,26 @@ fn anthropic_content_to_text(content: AnthropicContent) -> String {
|
||||
pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
|
||||
let choice = resp.choices.into_iter().next();
|
||||
|
||||
let (content_text, stop_reason) = match choice {
|
||||
let (content_text, reasoning_text, stop_reason) = match choice {
|
||||
Some(c) => {
|
||||
let text = match c.message.content {
|
||||
MessageContent::Text(t) => t,
|
||||
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
|
||||
MessageContent::Null => String::new(),
|
||||
};
|
||||
// The model's think block, which the harness returns on
|
||||
// `message.reasoning_content` (#300).
|
||||
let reasoning = c
|
||||
.message
|
||||
.extra
|
||||
.get("reasoning_content")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(str::to_string)
|
||||
.filter(|t| !t.is_empty());
|
||||
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
|
||||
(text, stop)
|
||||
(text, reasoning, stop)
|
||||
}
|
||||
None => (String::new(), None),
|
||||
None => (String::new(), None, None),
|
||||
};
|
||||
|
||||
let usage = resp.usage.unwrap_or(Usage {
|
||||
@@ -441,10 +450,32 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
|
||||
id: resp.id,
|
||||
response_type: "message".into(),
|
||||
role: "assistant".into(),
|
||||
content: vec![ContentBlock {
|
||||
block_type: "text".into(),
|
||||
data: json!({ "text": content_text }),
|
||||
}],
|
||||
// A `thinking` block ahead of the text, which is Anthropic's
|
||||
// own ordering. This path accepted `thinking.budget_tokens` on
|
||||
// the way in (see `anthropic_to_openai`) and returned nothing on
|
||||
// the way out: the caller spent the tokens, could not see them,
|
||||
// and could not replay the model's reasoning next turn (#300).
|
||||
//
|
||||
// Emitted without a `signature`. Anthropic signs its thinking
|
||||
// blocks and a client that round-trips one may expect the field;
|
||||
// we cannot mint a valid signature and inventing one would be
|
||||
// worse than omitting it. If a real client rejects an unsigned
|
||||
// block, the fix is to gate this behind a flag — not to go back
|
||||
// to silently deleting the model's work.
|
||||
content: {
|
||||
let mut blocks = Vec::with_capacity(2);
|
||||
if let Some(thinking) = reasoning_text {
|
||||
blocks.push(ContentBlock {
|
||||
block_type: "thinking".into(),
|
||||
data: json!({ "thinking": thinking }),
|
||||
});
|
||||
}
|
||||
blocks.push(ContentBlock {
|
||||
block_type: "text".into(),
|
||||
data: json!({ "text": content_text }),
|
||||
});
|
||||
blocks
|
||||
},
|
||||
model: resp.model,
|
||||
stop_reason,
|
||||
usage: AnthropicUsage {
|
||||
@@ -501,6 +532,7 @@ pub struct AnthropicStreamTranslator {
|
||||
enum OpenBlock {
|
||||
Text,
|
||||
ToolUse,
|
||||
Thinking,
|
||||
}
|
||||
|
||||
impl AnthropicStreamTranslator {
|
||||
@@ -540,6 +572,32 @@ impl AnthropicStreamTranslator {
|
||||
}
|
||||
|
||||
for choice in &chunk.choices {
|
||||
// The model's think block. Dropped entirely until #300: this
|
||||
// translator read `content` and `tool_calls` and nothing
|
||||
// else, so an Anthropic client asked for extended thinking
|
||||
// (`thinking.budget_tokens`, which `anthropic_to_openai`
|
||||
// accepts), was billed for the tokens, and received a stream
|
||||
// with no thinking in it — every turn, silently. Fixing only
|
||||
// the non-streaming translation would have missed this,
|
||||
// because the clients that ask for thinking stream.
|
||||
if let Some(text) = choice
|
||||
.delta
|
||||
.get("reasoning_content")
|
||||
.and_then(Value::as_str)
|
||||
&& !text.is_empty()
|
||||
{
|
||||
self.ensure_thinking_block(&mut out);
|
||||
let index = self.open_block.as_ref().map(|(i, _)| *i).unwrap_or(0);
|
||||
out.push((
|
||||
"content_block_delta".to_string(),
|
||||
json!({
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": { "type": "thinking_delta", "thinking": text }
|
||||
}),
|
||||
));
|
||||
}
|
||||
|
||||
if let Some(text) = choice.delta.get("content").and_then(Value::as_str)
|
||||
&& !text.is_empty()
|
||||
{
|
||||
@@ -653,6 +711,32 @@ impl AnthropicStreamTranslator {
|
||||
out
|
||||
}
|
||||
|
||||
/// Open a `thinking` block, closing whatever preceded it.
|
||||
///
|
||||
/// Emitted without a `signature`. Anthropic signs its thinking
|
||||
/// blocks; we cannot mint a valid one, and inventing a value would
|
||||
/// be worse than omitting it. If a client rejects an unsigned block
|
||||
/// the fix is a flag — not a return to deleting the model's work.
|
||||
fn ensure_thinking_block(&mut self, out: &mut Vec<(String, Value)>) {
|
||||
match &self.open_block {
|
||||
Some((_, OpenBlock::Thinking)) => {}
|
||||
_ => {
|
||||
self.close_open_block(out);
|
||||
let index = self.next_index;
|
||||
self.next_index += 1;
|
||||
self.open_block = Some((index, OpenBlock::Thinking));
|
||||
out.push((
|
||||
"content_block_start".to_string(),
|
||||
json!({
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": { "type": "thinking", "thinking": "" }
|
||||
}),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn ensure_text_block(&mut self, out: &mut Vec<(String, Value)>) {
|
||||
match &self.open_block {
|
||||
Some((_, OpenBlock::Text)) => {}
|
||||
@@ -1062,4 +1146,115 @@ mod request_tests {
|
||||
let tools = openai.extra.get("tools").and_then(Value::as_array).unwrap();
|
||||
assert_eq!(tools[0]["function"]["name"], "x");
|
||||
}
|
||||
|
||||
/// The Anthropic path must return the model's thinking (#300).
|
||||
///
|
||||
/// It accepted `thinking.budget_tokens` on the way in and returned
|
||||
/// a bare text block on the way out: the caller spent the tokens,
|
||||
/// never saw them, and could not replay the model's reasoning on
|
||||
/// the next turn. Taking a parameter and silently discarding what
|
||||
/// it buys is the defect, not the missing field.
|
||||
#[test]
|
||||
fn anthropic_response_carries_a_thinking_block() {
|
||||
let mut extra = serde_json::Map::new();
|
||||
extra.insert("reasoning_content".into(), json!("weighing the options"));
|
||||
let resp = ChatCompletionResponse {
|
||||
id: "id".into(),
|
||||
object: "chat.completion".into(),
|
||||
created: 0,
|
||||
model: "m".into(),
|
||||
choices: vec![crate::openai::ChatCompletionChoice {
|
||||
index: 0,
|
||||
message: ChatMessage {
|
||||
role: "assistant".into(),
|
||||
content: MessageContent::Text("the answer".into()),
|
||||
extra: Value::Object(extra),
|
||||
},
|
||||
finish_reason: Some("stop".into()),
|
||||
extra: Value::Null,
|
||||
}],
|
||||
usage: None,
|
||||
extra: Value::Null,
|
||||
};
|
||||
let a = openai_to_anthropic(resp);
|
||||
assert_eq!(a.content.len(), 2, "expected thinking + text");
|
||||
assert_eq!(a.content[0].block_type, "thinking", "thinking comes first");
|
||||
assert_eq!(a.content[0].data["thinking"], "weighing the options");
|
||||
assert_eq!(a.content[1].block_type, "text");
|
||||
assert_eq!(a.content[1].data["text"], "the answer");
|
||||
}
|
||||
|
||||
/// No thinking means the unchanged single text block.
|
||||
#[test]
|
||||
fn anthropic_response_without_thinking_is_unchanged() {
|
||||
let resp = ChatCompletionResponse {
|
||||
id: "id".into(),
|
||||
object: "chat.completion".into(),
|
||||
created: 0,
|
||||
model: "m".into(),
|
||||
choices: vec![crate::openai::ChatCompletionChoice {
|
||||
index: 0,
|
||||
message: ChatMessage {
|
||||
role: "assistant".into(),
|
||||
content: MessageContent::Text("plain".into()),
|
||||
extra: Value::Object(Default::default()),
|
||||
},
|
||||
finish_reason: Some("stop".into()),
|
||||
extra: Value::Null,
|
||||
}],
|
||||
usage: None,
|
||||
extra: Value::Null,
|
||||
};
|
||||
let a = openai_to_anthropic(resp);
|
||||
assert_eq!(a.content.len(), 1);
|
||||
assert_eq!(a.content[0].block_type, "text");
|
||||
}
|
||||
|
||||
/// Streaming Anthropic must carry thinking too (#300).
|
||||
///
|
||||
/// Fixing only the non-streaming translation would have missed this
|
||||
/// entirely — and missed it for the clients that matter, since the
|
||||
/// ones asking for extended thinking stream.
|
||||
#[test]
|
||||
fn anthropic_stream_emits_thinking_deltas() {
|
||||
use crate::openai::{ChatCompletionChunk, ChunkChoice};
|
||||
let mut t = AnthropicStreamTranslator::new();
|
||||
let chunk = |delta: Value| ChatCompletionChunk {
|
||||
id: "c".into(),
|
||||
object: "chat.completion.chunk".into(),
|
||||
created: 0,
|
||||
model: "m".into(),
|
||||
choices: vec![ChunkChoice {
|
||||
index: 0,
|
||||
delta,
|
||||
finish_reason: None,
|
||||
extra: Value::Null,
|
||||
}],
|
||||
usage: None,
|
||||
extra: Value::Null,
|
||||
};
|
||||
|
||||
let ev = t.on_chunk(&chunk(json!({ "reasoning_content": "weighing it" })));
|
||||
let start = ev
|
||||
.iter()
|
||||
.find(|(n, _)| n == "content_block_start")
|
||||
.expect("a block must open");
|
||||
assert_eq!(start.1["content_block"]["type"], "thinking");
|
||||
let delta = ev
|
||||
.iter()
|
||||
.find(|(n, _)| n == "content_block_delta")
|
||||
.expect("a delta must follow");
|
||||
assert_eq!(delta.1["delta"]["type"], "thinking_delta");
|
||||
assert_eq!(delta.1["delta"]["thinking"], "weighing it");
|
||||
|
||||
// Visible text opens a separate block, after thinking closes.
|
||||
let ev = t.on_chunk(&chunk(json!({ "content": "the answer" })));
|
||||
let names: Vec<&str> = ev.iter().map(|(n, _)| n.as_str()).collect();
|
||||
assert!(
|
||||
names.contains(&"content_block_stop"),
|
||||
"the thinking block must close first: {names:?}"
|
||||
);
|
||||
let start = ev.iter().find(|(n, _)| n == "content_block_start").unwrap();
|
||||
assert_eq!(start.1["content_block"]["type"], "text");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -80,7 +80,7 @@ The fastest path: env-var single-endpoint config.
|
||||
|
||||
```sh
|
||||
export HELEXA_ACP_BASE_URL=http://hanzalova.internal:31313/v1
|
||||
export HELEXA_ACP_MODEL=Qwen/Qwen3.6-27B
|
||||
export HELEXA_ACP_MODEL=Qwen/Qwen3.8-27B
|
||||
helexa-acp # speaks ACP over stdin/stdout; not interactive
|
||||
```
|
||||
|
||||
@@ -117,7 +117,7 @@ default_endpoint = "helexa"
|
||||
name = "helexa"
|
||||
base_url = "http://hanzalova.internal:31313/v1"
|
||||
wire_api = "openai-chat"
|
||||
default_model = "Qwen/Qwen3.6-27B"
|
||||
default_model = "Qwen/Qwen3.8-27B"
|
||||
max_tokens = 8192
|
||||
context_window = 32768
|
||||
|
||||
@@ -138,7 +138,7 @@ default_model = "claude-opus-4"
|
||||
|
||||
Restart Zed. The model dropdown lists every model from every
|
||||
configured endpoint with the `endpoint:model` selector
|
||||
(`helexa:Qwen/Qwen3.6-27B`, `openrouter:anthropic/claude-opus-4`,
|
||||
(`helexa:Qwen/Qwen3.8-27B`, `openrouter:anthropic/claude-opus-4`,
|
||||
…). Switch mid-session; the next prompt routes to the new endpoint.
|
||||
|
||||
When only one endpoint is configured the prefix is dropped (model
|
||||
@@ -166,7 +166,7 @@ Copy-pasteable blocks. Mix and match.
|
||||
name = "helexa"
|
||||
base_url = "http://hanzalova.internal:31313/v1"
|
||||
wire_api = "openai-chat"
|
||||
default_model = "Qwen/Qwen3.6-27B"
|
||||
default_model = "Qwen/Qwen3.8-27B"
|
||||
max_tokens = 8192
|
||||
context_window = 32768
|
||||
```
|
||||
@@ -241,7 +241,7 @@ vLLM.
|
||||
name = "lan"
|
||||
base_url = "http://hanzalova.internal:31313/v1"
|
||||
wire_api = "openai-chat"
|
||||
default_model = "Qwen/Qwen3.6-27B"
|
||||
default_model = "Qwen/Qwen3.8-27B"
|
||||
|
||||
[[endpoints]]
|
||||
name = "cloud"
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
use crate::activation::ActivationTracker;
|
||||
use crate::harness::HarnessRegistry;
|
||||
use crate::harness::candle::WireSurface;
|
||||
use crate::harness::candle::{CandleHarness, InferenceError};
|
||||
use crate::harness::preflight::PreflightError;
|
||||
use crate::health::HealthCache;
|
||||
@@ -362,7 +363,7 @@ async fn chat_completions(
|
||||
|
||||
if req.stream.unwrap_or(false) {
|
||||
match candle
|
||||
.chat_completion_stream_with(req, chat_config, principal)
|
||||
.chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
|
||||
.await
|
||||
{
|
||||
Ok(rx) => {
|
||||
@@ -381,7 +382,10 @@ async fn chat_completions(
|
||||
Err(e) => inference_error_response(e),
|
||||
}
|
||||
} else {
|
||||
match candle.chat_completion(req, principal).await {
|
||||
match candle
|
||||
.chat_completion(req, principal, WireSurface::Chat)
|
||||
.await
|
||||
{
|
||||
Ok(resp) => Json(resp).into_response(),
|
||||
Err(e) => inference_error_response(e),
|
||||
}
|
||||
@@ -434,7 +438,13 @@ async fn responses(
|
||||
|
||||
if stream_requested {
|
||||
match candle
|
||||
.responses_stream(chat_req, response_id, message_item_id, principal)
|
||||
.responses_stream(
|
||||
chat_req,
|
||||
response_id,
|
||||
message_item_id,
|
||||
principal,
|
||||
WireSurface::Responses,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(rx) => {
|
||||
@@ -458,7 +468,10 @@ async fn responses(
|
||||
// and translate the result. We don't currently re-tokenise
|
||||
// to compute usage; the harness returns it via the chat
|
||||
// response and we pass it through.
|
||||
match candle.chat_completion(chat_req, principal).await {
|
||||
match candle
|
||||
.chat_completion(chat_req, principal, WireSurface::Responses)
|
||||
.await
|
||||
{
|
||||
Ok(chat_resp) => {
|
||||
// Extract the assistant text (chat completions
|
||||
// always emits one choice on the candle path).
|
||||
@@ -486,9 +499,9 @@ async fn responses(
|
||||
input_tokens: u.prompt_tokens,
|
||||
output_tokens: u.completion_tokens,
|
||||
total_tokens: u.prompt_tokens + u.completion_tokens,
|
||||
// Carry the reasoning sub-count through from the chat
|
||||
// usage — the non-streaming path now splits off the
|
||||
// `<think>` span and counts it (see `split_off_reasoning`).
|
||||
// Carry the reasoning sub-count through from the
|
||||
// chat usage: the collector counts the `<think>`
|
||||
// span and returns its text separately.
|
||||
output_tokens_details: u.completion_tokens_details.as_ref().map(|d| {
|
||||
OutputTokensDetails {
|
||||
reasoning_tokens: d.reasoning_tokens,
|
||||
@@ -502,8 +515,18 @@ async fn responses(
|
||||
model_id,
|
||||
message_item_id: mint_message_item_id(),
|
||||
};
|
||||
let _ = chat_resp; // make the borrow-checker happy if `text` consumed it
|
||||
let resp = openai_responses::build_response(&meta, text, finish, usage);
|
||||
// The model's think block, which the chat collector now
|
||||
// returns on `message.reasoning_content` (#300). Without
|
||||
// this the caller is billed for `reasoning_tokens` it
|
||||
// never sees, and cannot replay the model's own
|
||||
// reasoning on the next turn.
|
||||
let reasoning = chat_resp
|
||||
.choices
|
||||
.first()
|
||||
.and_then(|c| c.message.extra.get("reasoning_content"))
|
||||
.and_then(|v| v.as_str())
|
||||
.map(str::to_string);
|
||||
let resp = openai_responses::build_response(&meta, text, finish, usage, reasoning);
|
||||
Json(resp).into_response()
|
||||
}
|
||||
Err(e) => inference_error_response(e),
|
||||
|
||||
@@ -669,6 +669,34 @@ fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::Reque
|
||||
}
|
||||
}
|
||||
|
||||
/// Which API surface a request arrived on.
|
||||
///
|
||||
/// Carried into the tracing span because neuron translates
|
||||
/// `/v1/responses` into a chat request at the door
|
||||
/// (`openai_responses::request_to_chat`) and every line after that
|
||||
/// point — spans, finish lines, rendered prompts — is emitted by the
|
||||
/// shared chat path. Without this the journal reads as though every
|
||||
/// client used `/v1/chat/completions`, which is not a missing detail
|
||||
/// but an actively wrong one: it says something false about the caller.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum WireSurface {
|
||||
/// `POST /v1/chat/completions`
|
||||
Chat,
|
||||
/// `POST /v1/responses`
|
||||
Responses,
|
||||
}
|
||||
|
||||
impl WireSurface {
|
||||
/// The value recorded as `wire=` on every span, matching the names
|
||||
/// cortex already logs via `log_inbound` so the two journals join.
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Chat => "openai-chat",
|
||||
Self::Responses => "openai-responses",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct LoadedModel {
|
||||
pub model_id: String,
|
||||
/// What the model published in `generation_config.json` (#272).
|
||||
@@ -2879,8 +2907,9 @@ impl CandleHarness {
|
||||
&self,
|
||||
request: ChatCompletionRequest,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<ChatCompletionResponse, InferenceError> {
|
||||
let stream = self.inference_stream(request, principal).await?;
|
||||
let stream = self.inference_stream(request, principal, wire).await?;
|
||||
wire_chat::collect_chat_completion(
|
||||
stream.events,
|
||||
stream.id,
|
||||
@@ -2907,11 +2936,13 @@ impl CandleHarness {
|
||||
&self,
|
||||
request: ChatCompletionRequest,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
|
||||
self.chat_completion_stream_with(
|
||||
request,
|
||||
wire_chat::ChatProjectionConfig::default(),
|
||||
principal,
|
||||
wire,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -2925,8 +2956,9 @@ impl CandleHarness {
|
||||
request: ChatCompletionRequest,
|
||||
mut config: wire_chat::ChatProjectionConfig,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
|
||||
let stream = self.inference_stream(request, principal).await?;
|
||||
let stream = self.inference_stream(request, principal, wire).await?;
|
||||
// Fill in the model's reasoning markers if the caller
|
||||
// didn't pre-populate them — they're a property of the
|
||||
// loaded model (which the HTTP handler doesn't reach into
|
||||
@@ -2954,9 +2986,10 @@ impl CandleHarness {
|
||||
response_id: String,
|
||||
message_item_id: String,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<mpsc::Receiver<crate::wire::openai_responses::ResponseStreamFrame>, InferenceError>
|
||||
{
|
||||
let stream = self.inference_stream(request, principal).await?;
|
||||
let stream = self.inference_stream(request, principal, wire).await?;
|
||||
let meta = crate::wire::openai_responses::ResponseMeta {
|
||||
response_id,
|
||||
created_at: stream.created,
|
||||
@@ -2978,6 +3011,7 @@ impl CandleHarness {
|
||||
&self,
|
||||
request: ChatCompletionRequest,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<InferenceStream, InferenceError> {
|
||||
let handle = {
|
||||
let models = self.models.read().await;
|
||||
@@ -3002,7 +3036,7 @@ impl CandleHarness {
|
||||
LoadedHandle::Single(m) => m,
|
||||
#[cfg(feature = "cuda")]
|
||||
LoadedHandle::Tp(m) => {
|
||||
return self.inference_tp_stream(m, request, principal).await;
|
||||
return self.inference_tp_stream(m, request, principal, wire).await;
|
||||
}
|
||||
LoadedHandle::Image(_) => {
|
||||
return Err(InferenceError::WrongModality {
|
||||
@@ -3140,7 +3174,12 @@ impl CandleHarness {
|
||||
// executor so we capture the span explicitly and re-enter it
|
||||
// inside the closure to keep the req_id on every emitted line.
|
||||
let req_id = new_req_id();
|
||||
let span = tracing::info_span!("chat_stream", req_id = %req_id, model = %model_id);
|
||||
let span = tracing::info_span!(
|
||||
"chat_stream",
|
||||
req_id = %req_id,
|
||||
model = %model_id,
|
||||
wire = wire.as_str()
|
||||
);
|
||||
let prompt_len = prompt_tokens.len();
|
||||
let req_start = std::time::Instant::now();
|
||||
// Cloned `Arc<LoadedModel>` so the spawned task can mark the
|
||||
@@ -4714,6 +4753,7 @@ impl CandleHarness {
|
||||
tp: Arc<TpLoadedModel>,
|
||||
request: ChatCompletionRequest,
|
||||
principal: Option<String>,
|
||||
wire: WireSurface,
|
||||
) -> Result<InferenceStream, InferenceError> {
|
||||
if tp.poisoned.load(Ordering::Acquire) {
|
||||
return Err(self.trigger_recovery(&request.model).await);
|
||||
@@ -4865,14 +4905,19 @@ impl CandleHarness {
|
||||
// of this inference; concurrent requests against the same TP
|
||||
// model serialise behind it.
|
||||
//
|
||||
// Tagged with the same req_id span as the non-streaming path
|
||||
// so the journal can be reconstructed regardless of which API
|
||||
// surface the client hit.
|
||||
// Tagged with the same req_id span as the non-streaming path,
|
||||
// and with the surface the request actually arrived on. Both
|
||||
// matter: neuron translates `/v1/responses` into a chat request
|
||||
// at the door, so without `wire` every line here reads as
|
||||
// though the caller used `/v1/chat/completions`. The value
|
||||
// matches what cortex logs via `log_inbound`, so the two
|
||||
// journals join on it.
|
||||
let req_id = new_req_id();
|
||||
let span = tracing::info_span!(
|
||||
"tp_chat_stream",
|
||||
req_id = %req_id,
|
||||
model = %model_id
|
||||
model = %model_id,
|
||||
wire = wire.as_str()
|
||||
);
|
||||
let req_start = std::time::Instant::now();
|
||||
let (vram_free_mb, vram_total_mb) = tp.query_vram().await;
|
||||
@@ -7573,6 +7618,26 @@ mod tests {
|
||||
assert_eq!(effort_in(&out), None);
|
||||
}
|
||||
|
||||
/// The wire surface must serialise as the string cortex logs, so a
|
||||
/// journal search joins the two.
|
||||
///
|
||||
/// neuron translates `/v1/responses` into a chat request at the
|
||||
/// door, and every line after that is emitted by the shared chat
|
||||
/// path. Before `wire` was recorded the journal showed
|
||||
/// `chat_completion (stream): done` for every request whatever the
|
||||
/// surface — worse than omitting the detail, because it asserts
|
||||
/// something false about the caller. It cost a real investigation
|
||||
/// into why a Responses-configured client "was using chat
|
||||
/// completions". It was not.
|
||||
#[test]
|
||||
fn the_wire_surface_matches_what_cortex_logs() {
|
||||
// These strings are load-bearing: `cortex-gateway`'s
|
||||
// `log_inbound` emits the same two, so `journalctl | grep
|
||||
// openai-responses` spans both services.
|
||||
assert_eq!(WireSurface::Chat.as_str(), "openai-chat");
|
||||
assert_eq!(WireSurface::Responses.as_str(), "openai-responses");
|
||||
}
|
||||
|
||||
/// Effort must not select a *rung's* budget — that was the
|
||||
/// guillotine (#290): the model was told to reason at `xhigh` and
|
||||
/// cut off at `medium`'s 12,288. The reserve below is not that; it
|
||||
|
||||
@@ -438,6 +438,70 @@ mod tests {
|
||||
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
|
||||
}
|
||||
|
||||
/// A file read back through a tool result must reach the model
|
||||
/// byte-for-byte.
|
||||
///
|
||||
/// The reported symptom: the model writes a file correctly, reads it
|
||||
/// back, and sees errors that are not in the file — consistently,
|
||||
/// after every write. Anything that mangles `<`, `&`, `"` or
|
||||
/// newlines on this path would produce exactly that, and would make
|
||||
/// the model burn its budget reconciling a file against a corrupted
|
||||
/// view of itself.
|
||||
#[test]
|
||||
fn a_file_read_back_through_a_tool_result_is_byte_exact() {
|
||||
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
|
||||
// The characters most likely to be mangled, in the shapes a real
|
||||
// web project contains them.
|
||||
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n</head>\n<body>\n <script>\n if (a && b || c < d) { g.fillStyle = '#b8905a'; }\n const s = `rgba(${r},${g},0.5)`;\n\n // a comment & an ampersand\n </script>\n</body>\n</html>\n";
|
||||
|
||||
let mut tool_extras = serde_json::Map::new();
|
||||
tool_extras.insert("tool_call_id".into(), json!("call_0"));
|
||||
let mut assistant_extras = serde_json::Map::new();
|
||||
assistant_extras.insert(
|
||||
"tool_calls".into(),
|
||||
json!([{"id":"call_0","type":"function",
|
||||
"function":{"name":"read","arguments":"{\"path\":\"/tmp/index.html\"}"}}]),
|
||||
);
|
||||
|
||||
let messages = vec![
|
||||
user_msg("read the file back"),
|
||||
ChatMessage {
|
||||
role: "assistant".into(),
|
||||
content: MessageContent::Null,
|
||||
extra: Value::Object(assistant_extras),
|
||||
},
|
||||
ChatMessage {
|
||||
role: "tool".into(),
|
||||
content: MessageContent::Text(file.to_string()),
|
||||
extra: Value::Object(tool_extras),
|
||||
},
|
||||
user_msg("is it correct?"),
|
||||
];
|
||||
let out =
|
||||
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
|
||||
|
||||
for probe in [
|
||||
"<!DOCTYPE html>",
|
||||
"<html lang=\"en\">",
|
||||
"if (a && b || c < d)",
|
||||
"g.fillStyle = '#b8905a';",
|
||||
"`rgba(${r},${g},0.5)`",
|
||||
"// a comment & an ampersand",
|
||||
] {
|
||||
assert!(
|
||||
out.contains(probe),
|
||||
"the model would not see {probe:?} verbatim — rendered prompt mangles it"
|
||||
);
|
||||
}
|
||||
// And nothing HTML-escaped anywhere.
|
||||
for bad in ["<", ">", "&", """, "'", "'"] {
|
||||
assert!(
|
||||
!out.contains(bad),
|
||||
"found {bad:?} in the rendered prompt: content is being HTML-escaped"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Blank lines inside a replayed tool-call argument must survive
|
||||
/// rendering intact.
|
||||
///
|
||||
|
||||
@@ -36,6 +36,28 @@ use crate::config::ContextLimitConfig;
|
||||
/// few turns.
|
||||
const RATE_EMA_ALPHA: f64 = 0.3;
|
||||
|
||||
/// Smallest prompt whose prefill time says anything about throughput.
|
||||
///
|
||||
/// A prefill has a fixed per-request cost — tokenize, launch, sync —
|
||||
/// that dominates a short prompt completely. The deploy's own smoke
|
||||
/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
|
||||
/// host that measures ~1,400. Folding that in does not make the
|
||||
/// estimate slightly worse; it replaces it.
|
||||
///
|
||||
/// That mattered because the fallback it displaces is the whole
|
||||
/// cold-start guard: with no sample at all, `derive_limit` uses
|
||||
/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
|
||||
/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
|
||||
/// the host advertised **17,408** — a sixth of what it can serve —
|
||||
/// until real traffic happened to warm it (#295).
|
||||
///
|
||||
/// 512 excludes the smoke probe and the short auxiliary calls agent
|
||||
/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
|
||||
/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
|
||||
/// so fixed overhead is roughly a tenth of it — informative rather than
|
||||
/// dominant.
|
||||
const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
|
||||
|
||||
/// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
|
||||
/// `bits`. No-op for degenerate inputs so a probe request or a clock blip
|
||||
/// can't poison the average.
|
||||
@@ -99,7 +121,15 @@ impl ThroughputEma {
|
||||
|
||||
/// Fold one prefill measurement (`prompt_tokens` processed in
|
||||
/// `elapsed`) into the prefill EMA.
|
||||
///
|
||||
/// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
|
||||
/// measure fixed per-request cost rather than throughput, and the
|
||||
/// value they displace — the bootstrap estimate — is better than
|
||||
/// they are.
|
||||
pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
|
||||
if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
|
||||
return;
|
||||
}
|
||||
fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
|
||||
}
|
||||
|
||||
@@ -356,6 +386,60 @@ pub fn derive_limit(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
|
||||
/// The deploy's smoke probe must not set the advertised context.
|
||||
///
|
||||
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
|
||||
/// ~1,400. Before this guard that one sample displaced the
|
||||
/// bootstrap estimate and the host advertised 17,408 instead of
|
||||
/// ~96,000 — until real traffic happened along (#295).
|
||||
#[test]
|
||||
fn a_smoke_probe_does_not_poison_the_prefill_rate() {
|
||||
let ema = ThroughputEma::new();
|
||||
ema.record(19, Duration::from_millis(130));
|
||||
assert_eq!(
|
||||
ema.get(),
|
||||
None,
|
||||
"a 19-token prefill must leave the EMA unset so the bootstrap value stands"
|
||||
);
|
||||
}
|
||||
|
||||
/// The short auxiliary calls agent harnesses make are the same
|
||||
/// shape as the probe and equally uninformative — observed at 143
|
||||
/// and 469 tokens against prompts three orders of magnitude larger.
|
||||
#[test]
|
||||
fn short_auxiliary_prompts_are_ignored_too() {
|
||||
let ema = ThroughputEma::new();
|
||||
for tokens in [143usize, 469] {
|
||||
ema.record(tokens, Duration::from_millis(200));
|
||||
}
|
||||
assert_eq!(ema.get(), None);
|
||||
}
|
||||
|
||||
/// A real prompt still records, and still tracks a shift.
|
||||
#[test]
|
||||
fn representative_prefills_still_measure() {
|
||||
let ema = ThroughputEma::new();
|
||||
ema.record(1400, Duration::from_secs(1));
|
||||
let first = ema.get().expect("a 1400-token sample must record");
|
||||
assert!((first - 1400.0).abs() < 1.0, "got {first}");
|
||||
|
||||
// A slower host is tracked, not ignored.
|
||||
for _ in 0..20 {
|
||||
ema.record(700, Duration::from_secs(1));
|
||||
}
|
||||
let after = ema.get().unwrap();
|
||||
assert!(after < first && after > 600.0, "got {after}");
|
||||
}
|
||||
|
||||
/// Exactly at the threshold counts — the boundary is inclusive so
|
||||
/// the constant means what it says.
|
||||
#[test]
|
||||
fn the_threshold_itself_is_a_valid_sample() {
|
||||
let ema = ThroughputEma::new();
|
||||
ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
|
||||
assert!(ema.get().is_some());
|
||||
}
|
||||
use super::*;
|
||||
|
||||
/// The ceiling a client is told to plan against must not be the
|
||||
|
||||
@@ -468,11 +468,21 @@ fn usage_chunk(
|
||||
/// where the same wire object could be assembled differently — the
|
||||
/// shape of defect #252 and #272 both were.
|
||||
///
|
||||
/// Semantics deliberately preserved from the path this replaces:
|
||||
/// reasoning text is **dropped** from `content` (it is counted in
|
||||
/// `usage.completion_tokens_details.reasoning_tokens` but never
|
||||
/// rendered), and `finish_reason` becomes `"tool_calls"` whenever any
|
||||
/// tool call was parsed.
|
||||
/// Reasoning is kept out of `content` — it is not the assistant's
|
||||
/// reply — but it is **returned** on `message.reasoning_content`, the
|
||||
/// same field the streaming path emits as
|
||||
/// `choice.delta.reasoning_content`.
|
||||
///
|
||||
/// It used to be discarded here (#300). That was inherited from the
|
||||
/// path this replaced and was wrong on both counts: the tokens are
|
||||
/// metered in `usage.completion_tokens_details.reasoning_tokens`, so a
|
||||
/// caller paid for text it could never see; and a client cannot replay
|
||||
/// what it never received, so on the next turn the model saw its own
|
||||
/// prior turn as a bare assertion with the reasoning that produced it
|
||||
/// deleted.
|
||||
///
|
||||
/// `finish_reason` becomes `"tool_calls"` whenever any tool call was
|
||||
/// parsed.
|
||||
///
|
||||
/// Two fields that the old path hardcoded to `None` now populate,
|
||||
/// because the events carry them and the struct already had the slots:
|
||||
@@ -488,6 +498,7 @@ pub async fn collect_chat_completion(
|
||||
model_id: String,
|
||||
) -> Result<ChatCompletionResponse, String> {
|
||||
let mut content = String::new();
|
||||
let mut reasoning = String::new();
|
||||
let mut tool_calls: Vec<serde_json::Value> = Vec::new();
|
||||
let mut finish: Option<(FinishReason, u32, u32, u32, u32, Option<FinishTiming>)> = None;
|
||||
|
||||
@@ -495,8 +506,8 @@ pub async fn collect_chat_completion(
|
||||
match event {
|
||||
InferenceEvent::Start => {}
|
||||
InferenceEvent::TextDelta(t) => content.push_str(&t),
|
||||
// Counted, never rendered — see the doc comment.
|
||||
InferenceEvent::ReasoningDelta(_) => {}
|
||||
// Kept out of `content`, returned on `reasoning_content`.
|
||||
InferenceEvent::ReasoningDelta(t) => reasoning.push_str(&t),
|
||||
InferenceEvent::ToolCall {
|
||||
index,
|
||||
id: call_id,
|
||||
@@ -545,11 +556,16 @@ pub async fn collect_chat_completion(
|
||||
} else {
|
||||
"tool_calls".to_string()
|
||||
};
|
||||
let message_extra = if tool_calls.is_empty() {
|
||||
serde_json::Value::Object(Default::default())
|
||||
} else {
|
||||
json!({ "tool_calls": tool_calls })
|
||||
};
|
||||
let mut message_extra = serde_json::Map::new();
|
||||
if !tool_calls.is_empty() {
|
||||
message_extra.insert("tool_calls".into(), json!(tool_calls));
|
||||
}
|
||||
// Absent rather than empty when the model did not think: a
|
||||
// non-reasoning model must keep the unchanged message shape.
|
||||
if !reasoning.is_empty() {
|
||||
message_extra.insert("reasoning_content".into(), json!(reasoning));
|
||||
}
|
||||
let message_extra = serde_json::Value::Object(message_extra);
|
||||
|
||||
Ok(ChatCompletionResponse {
|
||||
id,
|
||||
@@ -588,6 +604,86 @@ pub async fn collect_chat_completion(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
|
||||
/// The model's think block must come back on the non-streaming
|
||||
/// path too (#300).
|
||||
///
|
||||
/// It used to be dropped here while the streaming path emitted it
|
||||
/// as `choice.delta.reasoning_content`. The tokens were metered
|
||||
/// either way, so a non-streaming caller paid for text it could
|
||||
/// never see — and, worse, could not replay the model's reasoning
|
||||
/// on the next turn, leaving the model to face its own prior turn
|
||||
/// as a bare assertion.
|
||||
#[tokio::test]
|
||||
async fn non_streaming_returns_the_models_reasoning() {
|
||||
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
|
||||
tx.send(InferenceEvent::Start).await.unwrap();
|
||||
tx.send(InferenceEvent::ReasoningDelta(
|
||||
"weighing the options".into(),
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
tx.send(InferenceEvent::TextDelta("the answer".into()))
|
||||
.await
|
||||
.unwrap();
|
||||
tx.send(InferenceEvent::Finish {
|
||||
reason: FinishReason::Stop,
|
||||
prompt_tokens: 5,
|
||||
completion_tokens: 7,
|
||||
reasoning_tokens: 3,
|
||||
cached_tokens: 0,
|
||||
timing: None,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
drop(tx);
|
||||
|
||||
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
|
||||
.await
|
||||
.expect("collect");
|
||||
let msg = &r.choices[0].message;
|
||||
assert_eq!(
|
||||
msg.extra["reasoning_content"], "weighing the options",
|
||||
"reasoning must be returned, not discarded"
|
||||
);
|
||||
// And it must NOT be folded into the visible reply.
|
||||
assert!(
|
||||
matches!(&msg.content, MessageContent::Text(t) if t == "the answer"),
|
||||
"reasoning leaked into content"
|
||||
);
|
||||
}
|
||||
|
||||
/// A non-reasoning model must keep the unchanged message shape —
|
||||
/// absent, not an empty string.
|
||||
#[tokio::test]
|
||||
async fn no_reasoning_means_no_reasoning_field() {
|
||||
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
|
||||
tx.send(InferenceEvent::Start).await.unwrap();
|
||||
tx.send(InferenceEvent::TextDelta("plain".into()))
|
||||
.await
|
||||
.unwrap();
|
||||
tx.send(InferenceEvent::Finish {
|
||||
reason: FinishReason::Stop,
|
||||
prompt_tokens: 1,
|
||||
completion_tokens: 1,
|
||||
reasoning_tokens: 0,
|
||||
cached_tokens: 0,
|
||||
timing: None,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
drop(tx);
|
||||
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
|
||||
.await
|
||||
.expect("collect");
|
||||
assert!(
|
||||
r.choices[0]
|
||||
.message
|
||||
.extra
|
||||
.get("reasoning_content")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
use super::*;
|
||||
|
||||
// ── Non-streaming collector (#285) ────────────────────────────
|
||||
|
||||
@@ -1249,28 +1249,49 @@ fn finish_to_status(reason: FinishReason) -> &'static str {
|
||||
/// Collect a chat-completions response into a non-streaming
|
||||
/// [`ResponsesResponse`]. Used by the `/v1/responses` handler when
|
||||
/// the request doesn't set `stream: true`.
|
||||
///
|
||||
/// `reasoning` is the model's think block, when it produced one. It
|
||||
/// becomes its own output item ahead of the message — the same shape
|
||||
/// and ordering the streaming projector emits. Omitting it (as this did
|
||||
/// until #300) meant a non-streaming caller was billed for
|
||||
/// `reasoning_tokens` it could never see, and could not replay the
|
||||
/// model's own thinking on the next turn.
|
||||
pub fn build_response(
|
||||
meta: &ResponseMeta,
|
||||
full_text: String,
|
||||
reason: FinishReason,
|
||||
usage: Option<ResponsesUsage>,
|
||||
reasoning: Option<String>,
|
||||
) -> ResponsesResponse {
|
||||
let status = finish_to_status(reason).to_string();
|
||||
let mut output: Vec<ResponsesOutputItem> = Vec::with_capacity(2);
|
||||
if let Some(text) = reasoning.filter(|t| !t.is_empty()) {
|
||||
output.push(ResponsesOutputItem::Reasoning {
|
||||
id: reasoning_item_id(meta),
|
||||
summary: vec![cortex_core::responses::ResponsesSummaryPart::summary_text(
|
||||
text,
|
||||
)],
|
||||
status: "completed".into(),
|
||||
});
|
||||
}
|
||||
ResponsesResponse {
|
||||
id: meta.response_id.clone(),
|
||||
object: "response".into(),
|
||||
created_at: meta.created_at,
|
||||
status: status.clone(),
|
||||
model: meta.model_id.clone(),
|
||||
output: vec![ResponsesOutputItem::Message {
|
||||
id: meta.message_item_id.clone(),
|
||||
role: "assistant".into(),
|
||||
content: vec![ResponsesOutputContent::OutputText {
|
||||
text: full_text,
|
||||
annotations: vec![],
|
||||
}],
|
||||
status,
|
||||
}],
|
||||
output: {
|
||||
output.push(ResponsesOutputItem::Message {
|
||||
id: meta.message_item_id.clone(),
|
||||
role: "assistant".into(),
|
||||
content: vec![ResponsesOutputContent::OutputText {
|
||||
text: full_text,
|
||||
annotations: vec![],
|
||||
}],
|
||||
status,
|
||||
});
|
||||
output
|
||||
},
|
||||
// Same contract as the streaming shell: a truncated response
|
||||
// says so, or the caller cannot tell it from a fault.
|
||||
incomplete_details: matches!(reason, FinishReason::Length)
|
||||
@@ -1284,6 +1305,99 @@ mod tests {
|
||||
use super::*;
|
||||
use cortex_core::openai::MessageContent;
|
||||
|
||||
/// The non-streaming Responses body must carry a reasoning item
|
||||
/// (#300), matching the item the streaming projector emits.
|
||||
///
|
||||
/// Observed before the fix: a response reporting
|
||||
/// `output_tokens_details.reasoning_tokens: 4000` with an empty
|
||||
/// `output` — 4,000 tokens generated, metered, and invisible.
|
||||
#[test]
|
||||
fn non_streaming_response_carries_a_reasoning_item() {
|
||||
let r = build_response(
|
||||
&meta(),
|
||||
"the answer".into(),
|
||||
FinishReason::Stop,
|
||||
None,
|
||||
Some("weighing the options".into()),
|
||||
);
|
||||
let v = serde_json::to_value(&r).unwrap();
|
||||
let items = v["output"].as_array().expect("output array");
|
||||
assert_eq!(items.len(), 2, "expected a reasoning item and a message");
|
||||
assert_eq!(items[0]["type"], "reasoning", "reasoning comes first");
|
||||
assert_eq!(items[0]["summary"][0]["type"], "summary_text");
|
||||
assert_eq!(items[0]["summary"][0]["text"], "weighing the options");
|
||||
assert_eq!(items[1]["type"], "message");
|
||||
assert_eq!(items[1]["content"][0]["text"], "the answer");
|
||||
}
|
||||
|
||||
/// A model that did not think produces no reasoning item — the
|
||||
/// unchanged single-item shape.
|
||||
#[test]
|
||||
fn no_reasoning_means_no_reasoning_item() {
|
||||
let r = build_response(&meta(), "plain".into(), FinishReason::Stop, None, None);
|
||||
let v = serde_json::to_value(&r).unwrap();
|
||||
let items = v["output"].as_array().unwrap();
|
||||
assert_eq!(items.len(), 1);
|
||||
assert_eq!(items[0]["type"], "message");
|
||||
}
|
||||
|
||||
/// A file read back must survive the *whole* chain: Responses
|
||||
/// `function_call_output` -> chat message -> rendered prompt.
|
||||
///
|
||||
/// The reported symptom is a model that writes a file correctly,
|
||||
/// reads it back, and consistently sees errors that are not there.
|
||||
/// This walks the real path rather than a hand-built chat message,
|
||||
/// because the translation is the part in between.
|
||||
#[test]
|
||||
fn a_file_read_back_survives_the_whole_translation() {
|
||||
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n\n <script>\n if (a && b || c < d) { s = '#b8905a'; }\n </script>\n</html>\n";
|
||||
let req: ResponsesRequest = serde_json::from_value(serde_json::json!({
|
||||
"model": "m",
|
||||
"input": [
|
||||
{"type": "message", "role": "user",
|
||||
"content": [{"type": "input_text", "text": "read it back"}]},
|
||||
{"type": "function_call", "call_id": "c1", "name": "read",
|
||||
"arguments": "{\"path\":\"/tmp/index.html\"}"},
|
||||
// pi sends the tool result as a plain string (its
|
||||
// `convertToolResultOutput` only produces an array when
|
||||
// images are involved).
|
||||
{"type": "function_call_output", "call_id": "c1", "output": file},
|
||||
{"type": "message", "role": "user",
|
||||
"content": [{"type": "input_text", "text": "any errors?"}]}
|
||||
]
|
||||
}))
|
||||
.unwrap();
|
||||
let chat = request_to_chat(req).expect("translate");
|
||||
|
||||
// The tool message must carry the file verbatim.
|
||||
let tool = chat
|
||||
.messages
|
||||
.iter()
|
||||
.find(|m| m.role == "tool")
|
||||
.expect("a tool message");
|
||||
let got = match &tool.content {
|
||||
MessageContent::Text(t) => t.clone(),
|
||||
other => panic!("tool content is not text: {other:?}"),
|
||||
};
|
||||
assert_eq!(got, file, "translation altered the file content");
|
||||
|
||||
// And it must survive the render too.
|
||||
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
|
||||
let out = crate::harness::chat_template::render_chat_template(
|
||||
template,
|
||||
&chat.messages,
|
||||
&Value::Null,
|
||||
&Value::Null,
|
||||
)
|
||||
.expect("render");
|
||||
for probe in ["<!DOCTYPE html>", "if (a && b || c < d)", "s = '#b8905a';"] {
|
||||
assert!(out.contains(probe), "rendered prompt lost {probe:?}");
|
||||
}
|
||||
for bad in ["<", "&", """] {
|
||||
assert!(!out.contains(bad), "content HTML-escaped: {bad:?}");
|
||||
}
|
||||
}
|
||||
|
||||
/// One agentic step must render as ONE assistant turn.
|
||||
///
|
||||
/// Responses splits a turn into `reasoning` + `message` +
|
||||
@@ -2670,6 +2784,7 @@ mod tests {
|
||||
output_tokens_details: None,
|
||||
input_tokens_details: None,
|
||||
}),
|
||||
None,
|
||||
);
|
||||
assert_eq!(r.status, "completed");
|
||||
match &r.output[0] {
|
||||
@@ -2695,7 +2810,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn build_response_length_yields_incomplete_status() {
|
||||
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None);
|
||||
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None, None);
|
||||
assert_eq!(r.status, "incomplete");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -35,7 +35,7 @@ default_endpoint = "helexa"
|
||||
name = "helexa"
|
||||
base_url = "http://hanzalova.internal:31313/v1"
|
||||
wire_api = "openai-chat"
|
||||
default_model = "Qwen/Qwen3.6-27B"
|
||||
default_model = "Qwen/Qwen3.8-27B"
|
||||
max_tokens = 8192
|
||||
# Compaction kicks in when the rolling history grows past this token
|
||||
# budget. Set to your model's context window. Disable by removing
|
||||
|
||||
@@ -23,7 +23,7 @@ listen = "0.0.0.0:8088"
|
||||
#
|
||||
# [aliases]
|
||||
# "helexa/small" = "Qwen/Qwen3-8B"
|
||||
# "helexa/balanced" = "Qwen/Qwen3.6-27B"
|
||||
# "helexa/balanced" = "Qwen/Qwen3.8-27B"
|
||||
|
||||
# -- Downstream cortexes -------------------------------------------------
|
||||
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
|
||||
|
||||
@@ -78,7 +78,7 @@ Clients should not hardcode model ids. Aliases let them ask for a role:
|
||||
[aliases]
|
||||
"helexa/small" = "Qwen/Qwen3-1.7B"
|
||||
"helexa/balanced" = "Qwen/Qwen3-8B"
|
||||
"helexa/large" = "Qwen/Qwen3.6-27B"
|
||||
"helexa/large" = "Qwen/Qwen3.8-27B"
|
||||
"helexa/image" = "Tongyi-MAI/Z-Image-Turbo"
|
||||
```
|
||||
|
||||
|
||||
@@ -58,6 +58,32 @@ Two things that bite:
|
||||
- **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
|
||||
very different experience from the same model on NVMe.
|
||||
|
||||
### Where weights come from
|
||||
|
||||
By default a bare `org/name` id is fetched from HuggingFace. Operators
|
||||
running a mirror, or serving models from more than one place, can
|
||||
declare sources explicitly:
|
||||
|
||||
```toml
|
||||
[harness.candle]
|
||||
default_source = "huggingface"
|
||||
|
||||
[harness.candle.sources.huggingface]
|
||||
endpoint = "https://huggingface.co"
|
||||
auth_env = "HF_TOKEN" # env var name, never the token itself
|
||||
cache_dir = "/archive/llm" # optional; per-source cache
|
||||
```
|
||||
|
||||
`auth_env` names an environment variable rather than holding a token, so
|
||||
credentials stay out of the config file — worth keeping to even when the
|
||||
file is not obviously shared, because config is exactly what gets copied
|
||||
between hosts.
|
||||
|
||||
Give a source its own `cache_dir` when two sources can serve the same
|
||||
`org/name`. The cache tree is keyed on the model id alone, so sharing a
|
||||
directory between sources that disagree about what `org/name` means will
|
||||
collide on disk.
|
||||
|
||||
## Default models
|
||||
|
||||
Models listed as defaults load when the daemon activates, so the host
|
||||
@@ -66,7 +92,7 @@ load cost:
|
||||
|
||||
```toml
|
||||
[[default_models]]
|
||||
model_id = "Qwen/Qwen3.6-27B"
|
||||
model_id = "Qwen/Qwen3.8-27B"
|
||||
harness = "candle"
|
||||
quant = "q6k"
|
||||
tensor_parallel = 2
|
||||
@@ -89,12 +115,19 @@ Each loaded model has bounded admission control:
|
||||
|
||||
```toml
|
||||
[harness.candle.admission]
|
||||
max_in_flight = 8 # concurrent requests actually running
|
||||
max_in_flight = 2 # concurrent requests actually running
|
||||
max_queue_depth = 8 # waiting before rejection
|
||||
max_wait_secs = 30 # how long a queued request waits
|
||||
max_wait_secs = 30 # how long a queued request waits for a slot
|
||||
kv_max_wait_secs = 300 # how long it waits for enough KV budget
|
||||
max_per_principal = 2 # fair share, so one caller cannot monopolise
|
||||
```
|
||||
|
||||
`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
|
||||
The first waits for a slot to turn over; the second waits for a whole
|
||||
long-context sequence to finish, which on a 27B can be minutes. A
|
||||
request that needs KV budget nobody has yet is not a request to reject
|
||||
in thirty seconds.
|
||||
|
||||
Over-capacity requests are rejected promptly with `429`/`503` and a
|
||||
`Retry-After` rather than being allowed to queue invisibly. A fast
|
||||
rejection a client can retry is far better than a request that hangs
|
||||
@@ -102,8 +135,19 @@ until something times out.
|
||||
|
||||
`max_in_flight` above 1 enables batched decode on architectures that
|
||||
support it, so concurrent requests share a decode step instead of
|
||||
serialising. Raise it on big cards, leave it at 1 on small ones — the
|
||||
KV cache for several concurrent long contexts is itself substantial.
|
||||
serialising.
|
||||
|
||||
Higher is not better, because **the model's KV budget is divided by it
|
||||
at load**. A host advertising a 131k context with eight slots needs
|
||||
eight times the KV of one slot, which it does not have — the practical
|
||||
result is that each slot gets a fraction of the advertised window and a
|
||||
single long session runs out of room while seven idle slots hold
|
||||
reservations. Size it against the concurrency you actually observe: on
|
||||
one production host, seven days of request spans showed 89.6% of busy
|
||||
time was a single request in flight and 93.6% was two or fewer, so two
|
||||
slots cover the real load and give each a usable context. Anything above
|
||||
the limit queues on admission with a `Retry-After` rather than being
|
||||
refused.
|
||||
|
||||
`max_per_principal` is what stops one busy client consuming the whole
|
||||
model.
|
||||
@@ -150,7 +194,8 @@ reservation is doing its job, not that the model is overloaded.
|
||||
neuron refuses work it cannot finish rather than crashing partway:
|
||||
|
||||
- a **static floor** of free VRAM below which prefill is refused
|
||||
(`NEURON_MIN_FREE_VRAM_MB`, default 1500)
|
||||
(`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
|
||||
`[harness.candle.context_limit]`; default 1500)
|
||||
- a **length-aware check** that estimates the KV cache the request will
|
||||
actually need and refuses if it will not fit
|
||||
|
||||
@@ -159,15 +204,206 @@ can pass the static floor and still be impossible, because KV grows with
|
||||
prompt *and* generation length. See [context
|
||||
limits](/docs/operating/context-limits) for how the numbers relate.
|
||||
|
||||
`activation_headroom_mb` reserves VRAM per card for prefill activations
|
||||
on top of the weights and the KV cache, before the context ceiling is
|
||||
computed. Raise it if prefill OOMs on a host whose steady-state
|
||||
occupancy looks fine — activations are transient and do not show up in
|
||||
the resident figure.
|
||||
|
||||
A host that cannot currently serve a model reports that on `/health`,
|
||||
and cortex stops routing to it — an unservable replica is not a routing
|
||||
target.
|
||||
|
||||
## Image models
|
||||
|
||||
```toml
|
||||
[harness.candle.image]
|
||||
te_device = "cpu" # where the text encoder runs
|
||||
te_resident = true # keep it loaded between generations
|
||||
max_dim = 2048 # resolution ceiling per side
|
||||
```
|
||||
|
||||
The text encoder is the placement decision. On CPU the ~8 GB encoder
|
||||
never touches the GPU — prompt encoding is one short forward and the
|
||||
features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
|
||||
all. Move it to `cuda` only on a card with room to spare.
|
||||
|
||||
`te_resident` trades memory for latency: resident costs ~16 GB of system
|
||||
RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
|
||||
request. CPU defaults to resident, CUDA to rebuilding per request, which
|
||||
are the right defaults for where each is scarce.
|
||||
|
||||
`max_dim` rejects oversized requests **before admission**, so an
|
||||
impossible resolution costs nothing. That matters more than it sounds:
|
||||
an OOM on the device poisons the worker context, so refusing early is
|
||||
the difference between one rejected request and a model that needs
|
||||
restarting.
|
||||
|
||||
## Prompt caps
|
||||
|
||||
`NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
|
||||
rejected before any GPU work, so a runaway client costs nothing.
|
||||
|
||||
## Sampling
|
||||
|
||||
Every model publishes its own sampling defaults in
|
||||
`generation_config.json`, and a caller can override them per request.
|
||||
Between those two sits an operator override, for the cases where the
|
||||
model's published defaults are wrong for how you serve it:
|
||||
|
||||
```toml
|
||||
[[default_models]]
|
||||
model_id = "Qwen/Qwen3.8-27B"
|
||||
|
||||
[default_models.sampling]
|
||||
temperature = 0.6
|
||||
presence_penalty = 1.0
|
||||
```
|
||||
|
||||
Precedence is **request > operator > model > built-in**. Anything you
|
||||
leave unset changes nothing, so an empty table behaves exactly as no
|
||||
table at all.
|
||||
|
||||
Set the same values in the gateway's `models.toml` under
|
||||
`[models.sampling]`. A model that is cold-loaded on demand gets its
|
||||
sampling from the catalogue, not from this file, so the two must agree
|
||||
or the same model samples differently depending on how it arrived —
|
||||
`script/check-config-consistency.py` fails the deploy if they diverge.
|
||||
|
||||
### Which knob for which symptom
|
||||
|
||||
| symptom | knob |
|
||||
|---|---|
|
||||
| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
|
||||
| The think block restates the same passage over and over | `presence_penalty` |
|
||||
| Common tokens over-represented across a long answer | `frequency_penalty` |
|
||||
| Short immediate loops — "no, no, no…" | `repeat_penalty` |
|
||||
| You need two runs to be comparable | `seed` |
|
||||
|
||||
**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
|
||||
measured a 20% structural-defect rate on long agentic turns against 0/60
|
||||
at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
|
||||
Qwen recommend for their thinking variants, so this is not fighting the
|
||||
model.
|
||||
|
||||
**`presence_penalty`** and **`frequency_penalty`** are scored over the
|
||||
*whole* generated sequence. They are the ones that matter for reasoning
|
||||
models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
|
||||
remedy for the endless repetition their models fall into during long
|
||||
thinking. Both default to `0.0`, which is off.
|
||||
|
||||
**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
|
||||
`64`) are the older, *windowed* form: the penalty only sees the last
|
||||
`repeat_last_n` tokens. That is fine for chat and close to useless for a
|
||||
think block — a 20,000-token deliberation restating a passage 5,000
|
||||
tokens later is entirely outside a 64-token window. Measured across
|
||||
recorded sessions, **86% of think blocks over 60k characters contained
|
||||
verbatim repeated sentences**, one of them repeating 76% of its own
|
||||
content.
|
||||
|
||||
Widening the window is the obvious move and the wrong one: the penalty
|
||||
is token-level, and source code legitimately repeats tokens constantly
|
||||
(`const`, `function`, brace runs), so a wide window at a meaningful
|
||||
penalty degrades exactly the output you care about. Reach for
|
||||
`presence_penalty` first — what recurs is passages, not tokens.
|
||||
|
||||
**`seed`** pins the sampler's RNG for every request to this model. It
|
||||
exists because parameter search needs a control and the clients that
|
||||
would otherwise pin it often cannot. Leaving it set is a product
|
||||
decision, not just a testing one: every identical prompt then returns an
|
||||
identical answer to every caller, which is predictable and cacheable but
|
||||
removes the variation a user may expect from retrying.
|
||||
|
||||
## Reasoning
|
||||
|
||||
```toml
|
||||
[default_models.sampling]
|
||||
# nothing here — reasoning settings live in their own places, below
|
||||
```
|
||||
|
||||
**`preserve_thinking`** controls whether prior turns keep their think
|
||||
blocks when the conversation is re-rendered. It is the model's own chat
|
||||
template control, not ours:
|
||||
|
||||
```toml
|
||||
[[default_models]]
|
||||
preserve_thinking = false # keep reasoning only for the turn in progress
|
||||
```
|
||||
|
||||
Unset is the default and means the template decides. On Qwen3.8-27B that
|
||||
is `true` — every prior turn's reasoning is replayed. Full replay is
|
||||
what makes prompt growth track *total* completion rather than visible
|
||||
output: one measured session produced 28,672 reasoning tokens on turn 1
|
||||
and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
|
||||
prefill on every turn thereafter.
|
||||
|
||||
Setting it to `false` is a legitimate experiment, not a recommendation.
|
||||
Whether narrowing replay *helps* the model is unmeasured. Set it in both
|
||||
`models.toml` and this file, like sampling, and the value is reported on
|
||||
`/v1/models` and logged at load so a session can be attributed to the
|
||||
arm it ran under.
|
||||
|
||||
**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
|
||||
part of the caller's output budget for the answer:
|
||||
|
||||
```toml
|
||||
[harness.candle.reasoning_budget]
|
||||
answer_reserve_tokens = 4096
|
||||
```
|
||||
|
||||
A reasoning model with no stopping criterion fills whatever budget it is
|
||||
given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
|
||||
given 32,768 it used 32,768 — and both times returned no answer at all.
|
||||
The reserve bounds the think block at `max_tokens - answer_reserve` so
|
||||
the rest survives for the reply.
|
||||
|
||||
Deliberately *not* derived from reasoning effort. Effort is a prompt
|
||||
instruction the model is trained to follow; making it also set a
|
||||
guillotine is how a model came to be told to reason at `xhigh` and then
|
||||
cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
|
||||
reasoning.
|
||||
|
||||
The trade to know: when the reserve fires, the model is interrupted
|
||||
mid-deliberation and its answer phase tends to state an *intention*
|
||||
rather than a conclusion. That is better than returning nothing, but it
|
||||
is not free.
|
||||
|
||||
## The prefix cache
|
||||
|
||||
Snapshots of previously-served prefixes, so a follow-up turn re-uses the
|
||||
KV cache instead of re-prefilling the whole conversation:
|
||||
|
||||
```toml
|
||||
[harness.candle.prefix_cache]
|
||||
enabled = true
|
||||
budget_mb = 2048 # snapshot bytes per loaded model
|
||||
max_entries = 8 # live snapshots per model, regardless of budget
|
||||
```
|
||||
|
||||
**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
|
||||
That is the symptom: the prompt keeps growing, the cached prefix does
|
||||
not, and every turn re-prefills the difference. Observed on a 1 GiB
|
||||
default — `cached_tokens` pinned at 2,016 for ten consecutive turns
|
||||
while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
|
||||
at roughly 37 s each, against ~20 s of actual decode.
|
||||
|
||||
The number is not free, and the direction of the trade is the opposite
|
||||
of what it looks like. **The snapshot budget is reserved before the KV
|
||||
budget, not from what is left over.** On a 2×32 GB host serving a 27B at
|
||||
`max_in_flight = 2`, measured 1:1:
|
||||
|
||||
| `budget_mb` | KV budget | tokens per slot | holds a 53k snapshot |
|
||||
|---|---|---|---|
|
||||
| 1024 | 4713 MB | 75,400 | no |
|
||||
| 2048 | 3689 MB | 59,000 | yes |
|
||||
| 3072 | 2665 MB | 42,600 | yes, but a session cannot reach 53k |
|
||||
|
||||
So raising it too far starves the KV cache and shortens how long a
|
||||
single session can run — the model can no longer reach the context whose
|
||||
snapshot the larger cache was meant to hold. Size it against the deepest
|
||||
session you actually serve, and check `kv_budget_mb` in the journal
|
||||
after a reload.
|
||||
|
||||
## Checking on it
|
||||
|
||||
```sh
|
||||
|
||||
@@ -24,7 +24,7 @@ confined anywhere.
|
||||
|
||||
```toml
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3.6-27B"
|
||||
id = "Qwen/Qwen3.8-27B"
|
||||
harness = "candle"
|
||||
min_devices = 2
|
||||
min_device_vram_mb = 24000
|
||||
|
||||
@@ -32,6 +32,43 @@ daemon now does automatically.
|
||||
| `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
|
||||
| `limit.output` | same | generation reserve opencode leaves below the wall |
|
||||
|
||||
### Tuning the derived limit
|
||||
|
||||
Since #67 the neuron computes `limit` itself (see below), and three
|
||||
`[harness.candle.context_limit]` settings shape that computation:
|
||||
|
||||
```toml
|
||||
[harness.candle.context_limit]
|
||||
target_prefill_latency_secs = 120 # longest acceptable prefill per turn
|
||||
bootstrap_prefill_tok_per_sec = 800 # cold-start estimate until measured
|
||||
output_reserve_tokens = 8192 # what a request gets when it names no cap
|
||||
max_output_tokens = 32768 # the largest output a request may name
|
||||
```
|
||||
|
||||
**`target_prefill_latency_secs`** is the one that usually binds. The
|
||||
throughput ceiling is `target_prefill_latency_secs × measured
|
||||
prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
|
||||
advertises about 168k — above most hard ceilings, in which case it stops
|
||||
mattering. On a slower host, or one whose recent traffic has been small
|
||||
prompts, it is what caps the window.
|
||||
|
||||
**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
|
||||
its own rate, and stops the advertised window collapsing on a
|
||||
freshly-restarted host. Prefill samples below 512 tokens are ignored
|
||||
when computing that rate: a short prompt is dominated by fixed
|
||||
per-request cost and says nothing about throughput — the deploy's own
|
||||
19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
|
||||
So a host that has served only small requests keeps the bootstrap
|
||||
estimate rather than adopting a misleading one, and no warm-up is
|
||||
needed after a restart.
|
||||
|
||||
**`output_reserve_tokens`** is what a request that names no cap
|
||||
receives. **`max_output_tokens`** is the ceiling advertised to clients
|
||||
as `max_output_tokens`, and it is the *ceiling*, not the reserve —
|
||||
publishing the reserve tells every client that reads the advertisement
|
||||
to cap itself there, which on a reasoning model can be less than a
|
||||
single think block.
|
||||
|
||||
## How they must relate
|
||||
|
||||
For a single model on a single neuron, all of these must hold:
|
||||
@@ -60,9 +97,9 @@ Notes:
|
||||
lands, treat the VRAM-safe ceiling in rule 5 as a hard limit you set
|
||||
`NEURON_MAX_PROMPT_TOKENS` below, not something the daemon enforces.
|
||||
|
||||
## VRAM cost of context (Qwen3.6-27B on beast)
|
||||
## VRAM cost of context (Qwen3.8-27B on beast)
|
||||
|
||||
Qwen3.6-27B is a hybrid linear-attention model: of its 64 layers only
|
||||
Qwen3.8-27B is a hybrid linear-attention model: of its 64 layers only
|
||||
every 4th is full-attention (`full_attention_interval = 4` → **16**
|
||||
full-attn layers); the rest are `linear_attention` with constant-size
|
||||
recurrent state. KV cache grows **only** on the 16 full-attn layers
|
||||
@@ -131,7 +168,7 @@ value wins over a hand override of the same variable. Use `local.conf`
|
||||
only for genuinely host-local, transient experiments — and remember a
|
||||
later deploy will re-assert `model.conf`.
|
||||
|
||||
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.6-27B".limit`:
|
||||
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.8-27B".limit`:
|
||||
|
||||
```json
|
||||
{ "context": 131072, "input": 122880, "output": 8192 }
|
||||
|
||||
+5
-5
@@ -7,7 +7,7 @@
|
||||
# returns and what the router can cold-load on demand.
|
||||
#
|
||||
# Field reference:
|
||||
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.6-27B").
|
||||
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.8-27B").
|
||||
# Exact match.
|
||||
# harness - which engine handles inference (currently "candle").
|
||||
# quant - quantisation to serve this model with. Either a
|
||||
@@ -118,7 +118,7 @@
|
||||
# The example pins to a specific neuron name; adjust or remove the
|
||||
# pinned_on entry for your own fleet.
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3.6-27B"
|
||||
id = "Qwen/Qwen3.8-27B"
|
||||
harness = "candle"
|
||||
vram_mb = 54000
|
||||
min_devices = 2
|
||||
@@ -169,13 +169,13 @@ limit.context = 8192
|
||||
limit.output = 2048
|
||||
|
||||
# Helexa registry model — `source` pins this entry to the helexa
|
||||
# scheme so cortex forwards `helexa:Helexa/Qwen3.6-27B-Uncensored` to
|
||||
# scheme so cortex forwards `helexa:Helexa/Qwen3.8-27B-Uncensored` to
|
||||
# neuron's /models/load. Requires the neuron config to declare a
|
||||
# matching [harness.candle.sources.helexa] entry pointing at the
|
||||
# helexa registry endpoint (see neuron.example.toml).
|
||||
#
|
||||
# [[models]]
|
||||
# id = "Helexa/Qwen3.6-27B-Uncensored"
|
||||
# id = "Helexa/Qwen3.8-27B-Uncensored"
|
||||
# harness = "candle"
|
||||
# source = "helexa"
|
||||
# vram_mb = 54000
|
||||
@@ -194,7 +194,7 @@ limit.output = 2048
|
||||
# [aliases]
|
||||
# "helexa/small" = "Qwen/Qwen3-1.7B"
|
||||
# "helexa/balanced" = "Qwen/Qwen3-8B"
|
||||
# "helexa/large" = "Qwen/Qwen3.6-27B"
|
||||
# "helexa/large" = "Qwen/Qwen3.8-27B"
|
||||
|
||||
# Text-to-image (#197): Z-Image-Turbo (Apache 2.0). 6B S3-DiT + Qwen3-4B
|
||||
# text encoder (host CPU by default; see [harness.candle.image]) + VAE.
|
||||
|
||||
+26
-46
@@ -18,11 +18,11 @@
|
||||
# quadbrat = 1x RTX 3060 (12 GB) → small / quantised
|
||||
|
||||
# Qwen3-Next single-GPU validation target (Stage 8c). Same architecture
|
||||
# as Qwen3.6-27B (`model_type = "qwen3_5"`, same layer_types pattern,
|
||||
# same Gated DeltaNet hyperparams) but small enough to fit on any
|
||||
# fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
|
||||
# as the 27B flagship (`model_type = "qwen3_5"`, same layer_types
|
||||
# pattern, same Gated DeltaNet hyperparams) but small enough to fit on
|
||||
# any fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
|
||||
# Tbilisi probe correctly, the qwen3_5 code path is right and TP-2 on
|
||||
# Qwen3.6-27B is purely a sharding/collective problem.
|
||||
# the 27B is purely a sharding/collective problem.
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3.5-0.8B"
|
||||
harness = "candle"
|
||||
@@ -75,49 +75,29 @@ residency_priority = 300
|
||||
cost.input = 0.40
|
||||
cost.output = 1.60
|
||||
|
||||
# Tensor-parallel target. Beast is the only neuron with two big GPUs.
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3.6-27B"
|
||||
harness = "candle"
|
||||
# REQUIRED, not cosmetic (#252). Without it a cortex cold-load builds an
|
||||
# unquantised request: the model loads bf16, leaves ~2 GB free per card
|
||||
# against a ~3.9 GB prefill requirement, reports `loaded`, and then fails
|
||||
# every request. Activation from neuron's default_models has always said
|
||||
# q6k; this is the same fact on the path cortex uses.
|
||||
quant = "q6k"
|
||||
# Measured q6k residency across the pair (17.9 + 22.9 GB), not the 51.7 GB
|
||||
# bf16 checkpoint size. The old 54000 described a model we do not serve.
|
||||
vram_mb = 42000
|
||||
min_devices = 2
|
||||
min_device_vram_mb = 24000
|
||||
pinned_on = ["beast"]
|
||||
# The big-node class. Above the everyday class, so image generation can
|
||||
# never take beast however idle the flagship is; equal to the frontier
|
||||
# tier, so those two swap freely. `pinned_on` here is affinity only --
|
||||
# it no longer implies immunity, so this number is what protects it.
|
||||
residency_priority = 300
|
||||
# Illustrative pricing (USD per 1M tokens) — top tier, the 27B flagship.
|
||||
cost.input = 0.30
|
||||
cost.output = 1.20
|
||||
# DELIBERATE, TEMPORARY operator override (issue #67): unblocks the opencode
|
||||
# helexa-provider implementation by giving /v1/models a limit to consume on
|
||||
# beast. Mirrors the enforced NEURON_MAX_PROMPT_TOKENS=131072 there.
|
||||
# To be REMOVED once neuron self-derives these — its removal is the
|
||||
# acceptance test that the computed values replace this hand-set one.
|
||||
#limit.context = 131072
|
||||
#limit.input = 122880
|
||||
#limit.output = 8192
|
||||
|
||||
# Qwen3.8-27B (#251). Architecturally identical to the 3.6-27B — same
|
||||
# model_type, same layer counts, same vision tower; the two configs
|
||||
# differ only in `transformers_version` — so it serves through the same
|
||||
# qwen3_5 TP path with the same KV arithmetic.
|
||||
# Qwen3.8-27B (#251) — the resident flagship, and beast's `default_models`
|
||||
# pre-warm. Serves through the qwen3_5 TP path.
|
||||
#
|
||||
# Same residency class as the 3.6-27B *deliberately*: equal rank is what
|
||||
# lets the two displace each other in BOTH directions, so whichever is
|
||||
# asked for takes beast and the other can take it back. A higher number
|
||||
# here would let this one claim the node permanently and make the
|
||||
# comparison between them impossible.
|
||||
# It replaced Qwen3.6-27B, which was retired from this catalogue on
|
||||
# 2026-08-27 after the A/B it existed for had run its course. The two were
|
||||
# architecturally identical (same model_type, layer counts and vision
|
||||
# tower; the configs differed only in `transformers_version`) and were
|
||||
# deliberately given equal `residency_priority` so they could displace
|
||||
# each other in both directions.
|
||||
#
|
||||
# That symmetry is exactly what made keeping both expensive. The router's
|
||||
# `helexa/balanced` alias still pointed at the 3.6, so every
|
||||
# authenticated web-chat turn evicted the resident 3.8 and cold-loaded the
|
||||
# 3.6 — a ~70 s TP layer load each way — and the next request for the
|
||||
# flagship swapped it straight back. On 2026-08-27 four of those swaps
|
||||
# overlapped and left beast holding 32 GB on one card with no model
|
||||
# registered and nothing serving.
|
||||
#
|
||||
# The lesson is not "don't make models swappable": mutual displacement is
|
||||
# the right design and #99 wants it. It is that two near-identical models
|
||||
# at equal rank need a *reason* to both be present, and once the
|
||||
# comparison is over one of them is just a way for a stale alias to take
|
||||
# the node down.
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3.8-27B"
|
||||
harness = "candle"
|
||||
|
||||
Reference in New Issue
Block a user