helexa/helexa
1
0
Fork 0
Code Issues 80 Pull Requests Actions Packages Projects Releases Wiki Activity

platform: application-owned system prompts — verify faithful passthrough on every API surface + document #179

New Issue
Closed
opened 2026-07-17 08:55:32 +00:00 by grenade · 2 comments
grenade commented 2026-07-17 08:55:32 +00:00
Owner
Copy Link

Principle (decided 2026-07-17)

System prompts are application-owned, not operator-managed. The helexa.ai ecosystem serves an unenumerable diversity of workloads through OpenAI/Anthropic-compatible APIs; operators cannot and should not curate per-use-case prompts. Applications tailor model behaviour by sending their system prompt through the standard API fields, and the serving chain (edge → router → cortex → neuron) passes it through faithfully — no injection, no rewriting, no defaults.

(An SME that genuinely wants centrally-managed prompts for their own workloads runs their own helexa mesh — we're open source; that's independent of the helexa.ai ecosystem and out of scope here.)

Work

1. Verify the passthrough is actually faithful, per surface × per arch:

  • /v1/chat/completions — messages[0].role == "system" reaches the neuron chat template intact (multiple system messages? document behaviour)
  • /v1/responses — the instructions field maps to the system slot (and input items with role: system)
  • /v1/messages (Anthropic) — top-level system field (string AND content-block array forms) survives translation to the OpenAI envelope
  • neuron chat templating applies the system role correctly for each arch family (qwen3 dense, qwen3.5/next MoE, qwen3.6-VL) — including interaction with /no_think handling
  • streaming and non-streaming parity
  • integration tests pinning each of the above (a system prompt that measurably changes output, e.g. "reply only with the word PONG")

2. Document it as a platform feature:

  • README/docs section: "Tailoring model behaviour" — how to send a system prompt on each API surface, with curl examples; state the passthrough guarantee explicitly (what helexa will never do to your payload)
  • Note the guarantee in the operator-facing docs too: operators don't manage prompts, and cortex/router will never inject content into proxied requests

First consumer of the documented feature: the chat SPA's own prompt (#178). Related epic: #177.

## Principle (decided 2026-07-17) System prompts are **application-owned, not operator-managed**. The helexa.ai ecosystem serves an unenumerable diversity of workloads through OpenAI/Anthropic-compatible APIs; operators cannot and should not curate per-use-case prompts. Applications tailor model behaviour by sending their system prompt through the standard API fields, and the serving chain (edge → router → cortex → neuron) passes it through **faithfully — no injection, no rewriting, no defaults**. (An SME that genuinely wants centrally-managed prompts for their own workloads runs their own helexa mesh — we're open source; that's independent of the helexa.ai ecosystem and out of scope here.) ## Work **1. Verify the passthrough is actually faithful, per surface × per arch:** - [ ] `/v1/chat/completions` — `messages[0].role == "system"` reaches the neuron chat template intact (multiple system messages? document behaviour) - [ ] `/v1/responses` — the `instructions` field maps to the system slot (and `input` items with `role: system`) - [ ] `/v1/messages` (Anthropic) — top-level `system` field (string AND content-block array forms) survives translation to the OpenAI envelope - [ ] neuron chat templating applies the system role correctly for each arch family (qwen3 dense, qwen3.5/next MoE, qwen3.6-VL) — including interaction with `/no_think` handling - [ ] streaming and non-streaming parity - [ ] integration tests pinning each of the above (a system prompt that measurably changes output, e.g. "reply only with the word PONG") **2. Document it as a platform feature:** - [ ] README/docs section: "Tailoring model behaviour" — how to send a system prompt on each API surface, with curl examples; state the passthrough guarantee explicitly (what helexa will never do to your payload) - [ ] Note the guarantee in the operator-facing docs too: operators don't manage prompts, and cortex/router will never inject content into proxied requests First consumer of the documented feature: the chat SPA's own prompt (#178). Related epic: #177.
grenade added the p2-next label 2026-07-17 08:56:02 +00:00
grenade commented 2026-08-02 12:10:53 +00:00
Author
Owner
Copy Link

Status check (2026-08-02): the passthrough works; the deliverables don't exist yet

Tested live against the deployed chain (cortex on hanzalova → benjy), model helexa/balanced (Qwen3-8B, classic qwen3 dense), using "reply with exactly the word PONG" as a prompt that measurably changes output.

Surface Form Result
/v1/chat/completions messages[0].role == "system" PONG ✅
/v1/responses instructions PONG ✅
/v1/messages system as string PONG ✅
/v1/messages system as content-block array PONG ✅
/v1/chat/completions stream: true PONG ✅ (streaming parity holds)
control — no system prompt — does not say PONG; replies normally ✅

The control matters as much as the positives: it shows the chain injects no default of its own, so the observed behaviour comes from the caller's prompt alone.

Multiple system messages (the open question in the checklist): the last one wins. Sending "end with ALPHA" then "end with OMEGA" yields Hello! OMEGA. — ALPHA absent. That's the behaviour to document; worth deciding whether we want to guarantee it or concatenate instead, since OpenAI clients do sometimes send several.

So what's actually outstanding

The functionality is done — this issue is now purely test + document, not build:

  • Integration tests. Only cortex-gateway/tests/anthropic.rs::test_anthropic_with_system_prompt exists (the Anthropic→OpenAI translation). Nothing pins /v1/chat/completions, /v1/responses, streaming parity, or the multi-system-message rule — all of which I verified by hand above and any of which could regress silently.
  • Per-arch coverage. Verified on qwen3 dense only. qwen3_5/qwen3_next MoE and qwen3.6-VL are untested, and the interaction with /no_think handling differs per family.
  • Documentation. No "Tailoring model behaviour" section in the README or doc/ — the only hit for "system prompt" in the repo docs is an incidental mention in doc/plan/prefix-kv-cache.md. The passthrough guarantee (what helexa will never do to your payload) is stated nowhere user-facing, and nothing tells operators that cortex/router never inject into proxied requests.

First consumer #178 is closed, so the feature is in use by the SPA today — undocumented and unpinned by tests.

## Status check (2026-08-02): the passthrough works; the deliverables don't exist yet Tested live against the deployed chain (cortex on hanzalova → benjy), model `helexa/balanced` (Qwen3-8B, classic qwen3 dense), using "reply with exactly the word PONG" as a prompt that measurably changes output. | Surface | Form | Result | |---|---|---| | `/v1/chat/completions` | `messages[0].role == "system"` | **PONG** ✅ | | `/v1/responses` | `instructions` | **PONG** ✅ | | `/v1/messages` | `system` as **string** | **PONG** ✅ | | `/v1/messages` | `system` as **content-block array** | **PONG** ✅ | | `/v1/chat/completions` | `stream: true` | **PONG** ✅ (streaming parity holds) | | control — no system prompt | — | does **not** say PONG; replies normally ✅ | The control matters as much as the positives: it shows the chain injects no default of its own, so the observed behaviour comes from the caller's prompt alone. **Multiple system messages** (the open question in the checklist): the **last one wins**. Sending "end with ALPHA" then "end with OMEGA" yields `Hello! OMEGA.` — ALPHA absent. That's the behaviour to document; worth deciding whether we want to guarantee it or concatenate instead, since OpenAI clients do sometimes send several. ## So what's actually outstanding The **functionality is done** — this issue is now purely *test + document*, not *build*: - [ ] **Integration tests.** Only `cortex-gateway/tests/anthropic.rs::test_anthropic_with_system_prompt` exists (the Anthropic→OpenAI translation). Nothing pins `/v1/chat/completions`, `/v1/responses`, streaming parity, or the multi-system-message rule — all of which I verified by hand above and any of which could regress silently. - [ ] **Per-arch coverage.** Verified on qwen3 dense only. `qwen3_5`/`qwen3_next` MoE and `qwen3.6-VL` are untested, and the interaction with `/no_think` handling differs per family. - [ ] **Documentation.** No "Tailoring model behaviour" section in the README or `doc/` — the only hit for "system prompt" in the repo docs is an incidental mention in `doc/plan/prefix-kv-cache.md`. The passthrough guarantee (what helexa will never do to your payload) is stated nowhere user-facing, and nothing tells operators that cortex/router never inject into proxied requests. First consumer #178 is closed, so the feature is in use by the SPA today — undocumented and unpinned by tests.
grenade commented 2026-08-02 12:25:33 +00:00
Author
Owner
Copy Link

Verification evidence

The behaviour this issue asks for was already correct — cortex forwards a
caller's system prompt untouched and injects nothing of its own on any
surface. What was missing was anything holding it in place, or telling
anyone it exists. PR #224 supplies both.

Live, through cortex on hanzalova (:31313), sending
"You must reply with exactly the word PONG and nothing else." as the
system prompt and "hello there" as the user turn — a deliberately
odd instruction, so an honoured prompt is unmistakable from a default one:

model / arch /v1/chat/completions /v1/responses /v1/messages
Qwen3-8B (qwen3 dense) PONG PONG PONG
Qwen3.6-27B (qwen3_5, VL, TP-2) PONG PONG PONG
Qwen3-Coder-Next / Next-80B (qwen3_next) not live-tested — see below

Streaming and non-streaming both. Controls with no system prompt sent
returned ordinary conversational replies, confirming the PONG is coming
from the caller's instruction and not from something helexa added.

The qwen3_next gap, stated plainly. Neither Next variant is warm:
both are catalogue-feasible only on beast, where the pinned 27B is
resident and serving helexa/large. Live-testing them means displacing
a production model for a multi-minute 80B cold load, which is out of
proportion to what is being checked here. What makes that acceptable
rather than a hole: the system slot is not arch-branched. Rendering goes
through the shared chat_template.rs path, applying the template the
model ships in its own tokenizer_config, so there is no qwen3_next
-specific code for a system prompt to fall down. The three new template
tests exercise that shared path directly. Worth a probe next time a Next
model is warm for other reasons; not worth an eviction now.

Regression cover. Live checks prove today, not tomorrow, so the tests
assert on what cortex forwarded upstream, captured from a mock neuron —
not on the reply. A gateway that quietly dropped or replaced the system
prompt would still return a convincing answer, so an assertion on the
response would not catch the regression this is meant to catch. Six cases
in cortex-gateway/tests/system_prompt.rs, three in neuron's
chat_template.rs. The load-bearing one is the negative: nothing is
injected when the caller sends none. If cortex ever grows a default
prompt, house style or safety preamble, that is the test that fails — all
the positive ones would still pass.

Docs. README gains "Tailoring model behaviour" with per-surface curl
examples and the guarantee stated as what helexa will never do to your
payload; CLAUDE.md records it as an operator contract so a future change
that injects into a proxied body reads as breaking rather than as a
feature.

Two behaviours found while verifying, both documented:

  • Several system messages are forwarded unmerged and in order; the model
    decides precedence, and observably the last one wins. cortex does not
    editorialise.
  • /no_think suppresses reasoning on chat/completions but not on
    Responses, where a small max_output_tokens can be spent entirely on
    the think block — the caller gets "" with status: "incomplete".
    That is a genuine inconsistency between surfaces rather than a system
    prompt problem, so it is filed separately as #223; the README carries
    the caveat until it is fixed.
## Verification evidence The behaviour this issue asks for was already correct — cortex forwards a caller's system prompt untouched and injects nothing of its own on any surface. What was missing was anything holding it in place, or telling anyone it exists. PR #224 supplies both. **Live, through cortex on hanzalova** (`:31313`), sending `"You must reply with exactly the word PONG and nothing else."` as the system prompt and `"hello there"` as the user turn — a deliberately odd instruction, so an honoured prompt is unmistakable from a default one: | model / arch | `/v1/chat/completions` | `/v1/responses` | `/v1/messages` | |---|---|---|---| | Qwen3-8B (`qwen3` dense) | PONG | PONG | PONG | | Qwen3.6-27B (`qwen3_5`, VL, TP-2) | PONG | PONG | PONG | | Qwen3-Coder-Next / Next-80B (`qwen3_next`) | not live-tested — see below | | | Streaming and non-streaming both. Controls with no system prompt sent returned ordinary conversational replies, confirming the PONG is coming from the caller's instruction and not from something helexa added. **The `qwen3_next` gap, stated plainly.** Neither Next variant is warm: both are catalogue-feasible only on beast, where the pinned 27B is resident and serving `helexa/large`. Live-testing them means displacing a production model for a multi-minute 80B cold load, which is out of proportion to what is being checked here. What makes that acceptable rather than a hole: the system slot is not arch-branched. Rendering goes through the shared `chat_template.rs` path, applying the template the model ships in its own `tokenizer_config`, so there is no `qwen3_next` -specific code for a system prompt to fall down. The three new template tests exercise that shared path directly. Worth a probe next time a Next model is warm for other reasons; not worth an eviction now. **Regression cover.** Live checks prove today, not tomorrow, so the tests assert on what cortex *forwarded upstream*, captured from a mock neuron — not on the reply. A gateway that quietly dropped or replaced the system prompt would still return a convincing answer, so an assertion on the response would not catch the regression this is meant to catch. Six cases in `cortex-gateway/tests/system_prompt.rs`, three in neuron's `chat_template.rs`. The load-bearing one is the negative: nothing is injected when the caller sends none. If cortex ever grows a default prompt, house style or safety preamble, that is the test that fails — all the positive ones would still pass. **Docs.** README gains "Tailoring model behaviour" with per-surface curl examples and the guarantee stated as what helexa will never do to your payload; CLAUDE.md records it as an operator contract so a future change that injects into a proxied body reads as breaking rather than as a feature. **Two behaviours found while verifying**, both documented: - Several system messages are forwarded unmerged and in order; the model decides precedence, and observably the last one wins. cortex does not editorialise. - `/no_think` suppresses reasoning on chat/completions but **not** on Responses, where a small `max_output_tokens` can be spent entirely on the think block — the caller gets `""` with `status: "incomplete"`. That is a genuine inconsistency between surfaces rather than a system prompt problem, so it is filed separately as #223; the README carries the caveat until it is fixed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No Label p2-next
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#179
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?