fix(deploy): the nginx sync must be its own job, not a step that gates the fleet

Run 1284 proved the first attempt wrong, and worse than predicted. The
sync step hard-fails until an operator runs script/infra-setup.sh to
install its sudoers grant; a failed step fails its job; and
`deploy-neurons` has `needs: [deploy-cortex]`. So a missing nginx
permission skipped the rollout of all three neuron flavours. The fleet
sat two commits behind on eb783c3 while the run reported nothing worse
than "deploy-cortex: failure" — the neuron jobs were *skipped*, which
reads like "nothing to do".

Moving the step later within deploy-cortex, as the previous commit did,
does not fix this. The job still ends red and the dependents still skip.
Order was never the problem.

`deploy-nginx` now stands alone with no `needs`, so its failure is a
statement about the vhost and nothing else. It still fails hard —
drift between the repo copy and the live file is a real defect and
should be red — but it can no longer take the fleet with it.

#287 names this trap: adding a sync step before its sudoers grant exists
breaks the deploy. Having written that down and then done it anyway, the
narrower rule is now recorded beside the job: a step whose prerequisite
is installed by hand does not belong in a job that other jobs depend on.

Refs #287.

.gitea/workflows/deploy.yml

@@ -526,6 +526,124 @@ jobs:
   # dependency on cortex or the neurons (the sweep loop is version-aware
   # and picks up whatever each neuron reports whenever). So it runs
   # alongside the cortex→neurons chain rather than after it.
+  # The vhost sync gets its own job, with NO `needs`, because it is not
+  # part of deploying the gateway binary and must not be able to stop
+  # one.
+  #
+  # It was briefly a step inside deploy-cortex, and that broke the fleet
+  # on its first run (2026-08-27, run 1284): the step hard-fails until an
+  # operator has run script/infra-setup.sh to install its sudoers grant,
+  # a failed step fails the job, and `deploy-neurons` has
+  # `needs: [deploy-cortex]` — so a missing nginx permission silently
+  # skipped the rollout of all three neuron flavours. The fleet stayed
+  # two commits behind while the run reported only "deploy-cortex:
+  # failure".
+  #
+  # #287 names this ordering trap. The lesson is narrower than "order the
+  # steps": a step whose prerequisite is installed by hand does not
+  # belong in a job that something else depends on, because its failure
+  # is not a statement about that job. Moving it later within
+  # deploy-cortex would not have helped — the job still ends red and the
+  # dependents still skip.
+  #
+  # Failing hard here is still right: drift between the repo and the live
+  # file is a real defect, and this job going red says that and nothing
+  # else.
+  deploy-nginx:
+    runs-on: fedora-43
+    if: >-
+      ${{
+        github.event_name == 'workflow_dispatch'
+        || github.event.workflow_run.conclusion == 'success'
+      }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: SSH init
+        run: |
+          mkdir -p ~/.ssh
+          echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
+          chmod 600 ~/.ssh/id_ed25519
+          ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
+              gitea_ci@hanzalova.internal 'hostname -f'
+
+      # nginx vhost config for the internal site. Version-controlled in
+      # `asset/nginx/` since the beginning, but until now only ever
+      # installed by `script/infra-setup.sh` — a one-time, run-by-hand
+      # script. So the repo copy and the live file could diverge with
+      # nothing to detect it, which is the #252 shape: two sources of
+      # truth for one thing and nothing forcing agreement.
+      #
+      # Writes to sites-available and (re)points sites-enabled at it as
+      # a symlink. On this host sites-enabled was a *regular file*, not a
+      # symlink — two independent copies, of which nginx reads one. An
+      # operator editing "the config" would reasonably edit
+      # sites-available, watch `nginx -t` pass, reload, and see nothing
+      # change. Making the link canonical is part of the sync rather
+      # than a separate chore, so the layout cannot drift back.
+      #
+      # `nginx -t` gates the reload: a config that fails validation is
+      # left un-reloaded with the previous one still serving.
+      - name: Sync internal nginx vhost
+        run: |
+          src="asset/nginx/helexa.internal.conf"
+
+          # The sudoers drop-in that permits this is installed by
+          # script/infra-setup.sh, which is run by hand. Check first, so
+          # a missing permission reports the remedy instead of surfacing
+          # as an opaque rsync failure.
+          if ! ssh gitea_ci@hanzalova.internal 'sudo -n /usr/sbin/nginx -t' >/dev/null 2>&1; then
+            echo "FAIL: gitea_ci lacks the nginx sudoers entries on hanzalova." >&2
+            echo "      Re-run 'script/infra-setup.sh' once from an operator shell" >&2
+            echo "      to install the updated asset/sudoers.d/cortex-host.conf." >&2
+            exit 1
+          fi
+
+          rl="asset/nginx/helexa-ratelimit.conf"
+          local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
+          local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
+          remote_sum=$(ssh gitea_ci@hanzalova.internal \
+            'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
+          remote_rl=$(ssh gitea_ci@hanzalova.internal \
+            'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
+          link_ok=$(ssh gitea_ci@hanzalova.internal \
+            'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
+
+          if [ "${local_sum}" = "${remote_sum}" ] \
+             && [ "${local_rl}" = "${remote_rl}" ] \
+             && [ "${link_ok}" = "yes" ]; then
+            echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
+            exit 0
+          fi
+
+          # Order matters: the vhost references names defined at http
+          # context by helexa-ratelimit.conf (`limit_req_zone`s and the
+          # `helexa_noquery` log format). Ship that first, or a vhost
+          # referencing a not-yet-installed name fails `nginx -t` and the
+          # deploy stops with the two files out of step.
+          echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
+          rsync --archive --compress \
+            --chown root:root --chmod 0644 \
+            --rsync-path 'sudo rsync' \
+            "${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
+
+          echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
+          rsync --archive --compress \
+            --chown root:root --chmod 0644 \
+            --rsync-path 'sudo rsync' \
+            "${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
+
+          ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
+          set -eu
+          sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
+                                /etc/nginx/sites-enabled/helexa.internal.conf
+          if ! sudo /usr/sbin/nginx -t; then
+            echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
+            exit 1
+          fi
+          sudo /usr/bin/systemctl reload nginx
+          echo "nginx reloaded"
+          NGINX
+
   # Roll helexa-router + helexa-upstream onto the federation service host
   # (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
   # two services are independent daemons on one host, so one job handles

asset/nginx/helexa-ratelimit.conf

@@ -24,3 +24,24 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
 limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
 limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
 limit_req_status 429;
+
+# Access-log format for the search locations, which must not record what
+# people searched for.
+#
+# The default `main` format logs `$request` — method, full URI *and
+# query string*. For a search proxy that means every query a user types
+# is written to /var/log/nginx/access.log next to their IP, for
+# authenticated and anonymous visitors alike. SearXNG's own web UI
+# defaults to POST partly to avoid exactly this.
+#
+# `$uri` is the path with the query stripped, so the operational signal
+# survives — who called, when, which endpoint, what status, how big, how
+# long — and the query itself does not. `$request_method` is kept
+# explicit since `$uri` drops it.
+#
+# Rate-limit debugging and abuse detection both work from this: they
+# need counts, statuses and client addresses, never the search terms.
+log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
+                          '"$request_method $uri $server_protocol" '
+                          '$status $body_bytes_sent "$http_referer" '
+                          '"$http_user_agent" $request_time';

asset/nginx/helexa.internal.conf

@@ -73,6 +73,9 @@ server {
     location = /tools/fetch {
         limit_req zone=helexa_tools burst=10 nodelay;
         limit_except GET { deny all; }
+        # The `url=` parameter is as revealing as a search query: it is
+        # the page a user is having read to them. Same treatment.
+        access_log /var/log/nginx/access.log helexa_noquery;
         proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -83,7 +86,40 @@ server {
     # federation service host. GET only; the SPA requests format=json.
     location = /tools/web_search {
         limit_req zone=helexa_tools burst=10 nodelay;
-        limit_except GET { deny all; }
+        # GET and POST: SearXNG accepts both, and a client that POSTs
+        # keeps its query out of any intermediary's request line. Other
+        # methods have no meaning here.
+        limit_except GET POST { deny all; }
+        # A search query is small. Bounding the body stops this becoming
+        # a way to push arbitrary bytes at the upstream.
+        client_max_body_size 8k;
+        # Never log the query — see helexa_noquery in
+        # conf.d/helexa-ratelimit.conf.
+        access_log /var/log/nginx/access.log helexa_noquery;
+        proxy_pass http://gallumbits.kosherinata.internal:8888/search;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_read_timeout 30s;
+    }
+
+    # The same upstream under SearXNG's own path, for clients that speak
+    # the SearXNG API rather than our app's.
+    #
+    # `/tools/web_search` is our name for it, and a generic SearXNG
+    # client cannot guess it: they take a base URL and append `/search`,
+    # because that is what every SearXNG instance serves. Without this
+    # block such a request falls through to `location /` and is answered
+    # by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
+    # client. A 404 would at least be honest; 200-with-HTML sends people
+    # looking for a bug in their own parser.
+    #
+    # Same upstream, same rate-limit zone, same GET-only restriction:
+    # this is an alias, not a second, laxer door.
+    location = /search {
+        limit_req zone=helexa_tools burst=10 nodelay;
+        limit_except GET POST { deny all; }
+        client_max_body_size 8k;
+        access_log /var/log/nginx/access.log helexa_noquery;
         proxy_pass http://gallumbits.kosherinata.internal:8888/search;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

asset/sudoers.d/cortex-host.conf

@@ -13,6 +13,18 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
 gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
 # deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
 gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
+# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
+# sites-enabled a symlink to it, validate, reload. Scoped to the one
+# file and the one service — `nginx -t` is included because a reload
+# must never happen on a config nginx has rejected.
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
+gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
+gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
+
 gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
 gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
 gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service

crates/neuron/src/api.rs

@@ -2,6 +2,7 @@
 
 use crate::activation::ActivationTracker;
 use crate::harness::HarnessRegistry;
+use crate::harness::candle::WireSurface;
 use crate::harness::candle::{CandleHarness, InferenceError};
 use crate::harness::preflight::PreflightError;
 use crate::health::HealthCache;
@@ -362,7 +363,7 @@ async fn chat_completions(
 
     if req.stream.unwrap_or(false) {
         match candle
-            .chat_completion_stream_with(req, chat_config, principal)
+            .chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
             .await
         {
             Ok(rx) => {
@@ -381,7 +382,10 @@ async fn chat_completions(
             Err(e) => inference_error_response(e),
         }
     } else {
-        match candle.chat_completion(req, principal).await {
+        match candle
+            .chat_completion(req, principal, WireSurface::Chat)
+            .await
+        {
             Ok(resp) => Json(resp).into_response(),
             Err(e) => inference_error_response(e),
         }
@@ -434,7 +438,13 @@ async fn responses(
 
     if stream_requested {
         match candle
-            .responses_stream(chat_req, response_id, message_item_id, principal)
+            .responses_stream(
+                chat_req,
+                response_id,
+                message_item_id,
+                principal,
+                WireSurface::Responses,
+            )
             .await
         {
             Ok(rx) => {
@@ -458,7 +468,10 @@ async fn responses(
         // and translate the result. We don't currently re-tokenise
         // to compute usage; the harness returns it via the chat
         // response and we pass it through.
-        match candle.chat_completion(chat_req, principal).await {
+        match candle
+            .chat_completion(chat_req, principal, WireSurface::Responses)
+            .await
+        {
             Ok(chat_resp) => {
                 // Extract the assistant text (chat completions
                 // always emits one choice on the candle path).

crates/neuron/src/harness/candle.rs

@@ -669,6 +669,34 @@ fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::Reque
     }
 }
 
+/// Which API surface a request arrived on.
+///
+/// Carried into the tracing span because neuron translates
+/// `/v1/responses` into a chat request at the door
+/// (`openai_responses::request_to_chat`) and every line after that
+/// point — spans, finish lines, rendered prompts — is emitted by the
+/// shared chat path. Without this the journal reads as though every
+/// client used `/v1/chat/completions`, which is not a missing detail
+/// but an actively wrong one: it says something false about the caller.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum WireSurface {
+    /// `POST /v1/chat/completions`
+    Chat,
+    /// `POST /v1/responses`
+    Responses,
+}
+
+impl WireSurface {
+    /// The value recorded as `wire=` on every span, matching the names
+    /// cortex already logs via `log_inbound` so the two journals join.
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Chat => "openai-chat",
+            Self::Responses => "openai-responses",
+        }
+    }
+}
+
 pub struct LoadedModel {
     pub model_id: String,
     /// What the model published in `generation_config.json` (#272).
@@ -2879,8 +2907,9 @@ impl CandleHarness {
         &self,
         request: ChatCompletionRequest,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<ChatCompletionResponse, InferenceError> {
-        let stream = self.inference_stream(request, principal).await?;
+        let stream = self.inference_stream(request, principal, wire).await?;
         wire_chat::collect_chat_completion(
             stream.events,
             stream.id,
@@ -2907,11 +2936,13 @@ impl CandleHarness {
         &self,
         request: ChatCompletionRequest,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
         self.chat_completion_stream_with(
             request,
             wire_chat::ChatProjectionConfig::default(),
             principal,
+            wire,
         )
         .await
     }
@@ -2925,8 +2956,9 @@ impl CandleHarness {
         request: ChatCompletionRequest,
         mut config: wire_chat::ChatProjectionConfig,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
-        let stream = self.inference_stream(request, principal).await?;
+        let stream = self.inference_stream(request, principal, wire).await?;
         // Fill in the model's reasoning markers if the caller
         // didn't pre-populate them — they're a property of the
         // loaded model (which the HTTP handler doesn't reach into
@@ -2954,9 +2986,10 @@ impl CandleHarness {
         response_id: String,
         message_item_id: String,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<mpsc::Receiver<crate::wire::openai_responses::ResponseStreamFrame>, InferenceError>
     {
-        let stream = self.inference_stream(request, principal).await?;
+        let stream = self.inference_stream(request, principal, wire).await?;
         let meta = crate::wire::openai_responses::ResponseMeta {
             response_id,
             created_at: stream.created,
@@ -2978,6 +3011,7 @@ impl CandleHarness {
         &self,
         request: ChatCompletionRequest,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<InferenceStream, InferenceError> {
         let handle = {
             let models = self.models.read().await;
@@ -3002,7 +3036,7 @@ impl CandleHarness {
             LoadedHandle::Single(m) => m,
             #[cfg(feature = "cuda")]
             LoadedHandle::Tp(m) => {
-                return self.inference_tp_stream(m, request, principal).await;
+                return self.inference_tp_stream(m, request, principal, wire).await;
             }
             LoadedHandle::Image(_) => {
                 return Err(InferenceError::WrongModality {
@@ -3140,7 +3174,12 @@ impl CandleHarness {
         // executor so we capture the span explicitly and re-enter it
         // inside the closure to keep the req_id on every emitted line.
         let req_id = new_req_id();
-        let span = tracing::info_span!("chat_stream", req_id = %req_id, model = %model_id);
+        let span = tracing::info_span!(
+            "chat_stream",
+            req_id = %req_id,
+            model = %model_id,
+            wire = wire.as_str()
+        );
         let prompt_len = prompt_tokens.len();
         let req_start = std::time::Instant::now();
         // Cloned `Arc<LoadedModel>` so the spawned task can mark the
@@ -4714,6 +4753,7 @@ impl CandleHarness {
         tp: Arc<TpLoadedModel>,
         request: ChatCompletionRequest,
         principal: Option<String>,
+        wire: WireSurface,
     ) -> Result<InferenceStream, InferenceError> {
         if tp.poisoned.load(Ordering::Acquire) {
             return Err(self.trigger_recovery(&request.model).await);
@@ -4865,14 +4905,19 @@ impl CandleHarness {
         // of this inference; concurrent requests against the same TP
         // model serialise behind it.
         //
-        // Tagged with the same req_id span as the non-streaming path
-        // so the journal can be reconstructed regardless of which API
-        // surface the client hit.
+        // Tagged with the same req_id span as the non-streaming path,
+        // and with the surface the request actually arrived on. Both
+        // matter: neuron translates `/v1/responses` into a chat request
+        // at the door, so without `wire` every line here reads as
+        // though the caller used `/v1/chat/completions`. The value
+        // matches what cortex logs via `log_inbound`, so the two
+        // journals join on it.
         let req_id = new_req_id();
         let span = tracing::info_span!(
             "tp_chat_stream",
             req_id = %req_id,
-            model = %model_id
+            model = %model_id,
+            wire = wire.as_str()
         );
         let req_start = std::time::Instant::now();
         let (vram_free_mb, vram_total_mb) = tp.query_vram().await;
@@ -7573,6 +7618,26 @@ mod tests {
         assert_eq!(effort_in(&out), None);
     }
 
+    /// The wire surface must serialise as the string cortex logs, so a
+    /// journal search joins the two.
+    ///
+    /// neuron translates `/v1/responses` into a chat request at the
+    /// door, and every line after that is emitted by the shared chat
+    /// path. Before `wire` was recorded the journal showed
+    /// `chat_completion (stream): done` for every request whatever the
+    /// surface — worse than omitting the detail, because it asserts
+    /// something false about the caller. It cost a real investigation
+    /// into why a Responses-configured client "was using chat
+    /// completions". It was not.
+    #[test]
+    fn the_wire_surface_matches_what_cortex_logs() {
+        // These strings are load-bearing: `cortex-gateway`'s
+        // `log_inbound` emits the same two, so `journalctl | grep
+        // openai-responses` spans both services.
+        assert_eq!(WireSurface::Chat.as_str(), "openai-chat");
+        assert_eq!(WireSurface::Responses.as_str(), "openai-responses");
+    }
+
     /// Effort must not select a *rung's* budget — that was the
     /// guillotine (#290): the model was told to reason at `xhigh` and
     /// cut off at `medium`'s 12,288. The reserve below is not that; it

crates/neuron/src/harness/context_limit.rs

@@ -36,6 +36,28 @@ use crate::config::ContextLimitConfig;
 /// few turns.
 const RATE_EMA_ALPHA: f64 = 0.3;
 
+/// Smallest prompt whose prefill time says anything about throughput.
+///
+/// A prefill has a fixed per-request cost — tokenize, launch, sync —
+/// that dominates a short prompt completely. The deploy's own smoke
+/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
+/// host that measures ~1,400. Folding that in does not make the
+/// estimate slightly worse; it replaces it.
+///
+/// That mattered because the fallback it displaces is the whole
+/// cold-start guard: with no sample at all, `derive_limit` uses
+/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
+/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
+/// the host advertised **17,408** — a sixth of what it can serve —
+/// until real traffic happened to warm it (#295).
+///
+/// 512 excludes the smoke probe and the short auxiliary calls agent
+/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
+/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
+/// so fixed overhead is roughly a tenth of it — informative rather than
+/// dominant.
+const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
+
 /// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
 /// `bits`. No-op for degenerate inputs so a probe request or a clock blip
 /// can't poison the average.
@@ -99,7 +121,15 @@ impl ThroughputEma {
 
     /// Fold one prefill measurement (`prompt_tokens` processed in
     /// `elapsed`) into the prefill EMA.
+    ///
+    /// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
+    /// measure fixed per-request cost rather than throughput, and the
+    /// value they displace — the bootstrap estimate — is better than
+    /// they are.
     pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
+        if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
+            return;
+        }
         fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
     }
 
@@ -356,6 +386,60 @@ pub fn derive_limit(
 
 #[cfg(test)]
 mod tests {
+
+    /// The deploy's smoke probe must not set the advertised context.
+    ///
+    /// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
+    /// ~1,400. Before this guard that one sample displaced the
+    /// bootstrap estimate and the host advertised 17,408 instead of
+    /// ~96,000 — until real traffic happened along (#295).
+    #[test]
+    fn a_smoke_probe_does_not_poison_the_prefill_rate() {
+        let ema = ThroughputEma::new();
+        ema.record(19, Duration::from_millis(130));
+        assert_eq!(
+            ema.get(),
+            None,
+            "a 19-token prefill must leave the EMA unset so the bootstrap value stands"
+        );
+    }
+
+    /// The short auxiliary calls agent harnesses make are the same
+    /// shape as the probe and equally uninformative — observed at 143
+    /// and 469 tokens against prompts three orders of magnitude larger.
+    #[test]
+    fn short_auxiliary_prompts_are_ignored_too() {
+        let ema = ThroughputEma::new();
+        for tokens in [143usize, 469] {
+            ema.record(tokens, Duration::from_millis(200));
+        }
+        assert_eq!(ema.get(), None);
+    }
+
+    /// A real prompt still records, and still tracks a shift.
+    #[test]
+    fn representative_prefills_still_measure() {
+        let ema = ThroughputEma::new();
+        ema.record(1400, Duration::from_secs(1));
+        let first = ema.get().expect("a 1400-token sample must record");
+        assert!((first - 1400.0).abs() < 1.0, "got {first}");
+
+        // A slower host is tracked, not ignored.
+        for _ in 0..20 {
+            ema.record(700, Duration::from_secs(1));
+        }
+        let after = ema.get().unwrap();
+        assert!(after < first && after > 600.0, "got {after}");
+    }
+
+    /// Exactly at the threshold counts — the boundary is inclusive so
+    /// the constant means what it says.
+    #[test]
+    fn the_threshold_itself_is_a_valid_sample() {
+        let ema = ThroughputEma::new();
+        ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
+        assert!(ema.get().is_some());
+    }
     use super::*;
 
     /// The ceiling a client is told to plan against must not be the

helexa.ai/content/docs/2-operating/03-neurons.md

@@ -58,6 +58,32 @@ Two things that bite:
 - **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
   very different experience from the same model on NVMe.
 
+### Where weights come from
+
+By default a bare `org/name` id is fetched from HuggingFace. Operators
+running a mirror, or serving models from more than one place, can
+declare sources explicitly:
+
+```toml
+[harness.candle]
+default_source = "huggingface"
+
+[harness.candle.sources.huggingface]
+endpoint = "https://huggingface.co"
+auth_env = "HF_TOKEN"          # env var name, never the token itself
+cache_dir = "/archive/llm"     # optional; per-source cache
+```
+
+`auth_env` names an environment variable rather than holding a token, so
+credentials stay out of the config file — worth keeping to even when the
+file is not obviously shared, because config is exactly what gets copied
+between hosts.
+
+Give a source its own `cache_dir` when two sources can serve the same
+`org/name`. The cache tree is keyed on the model id alone, so sharing a
+directory between sources that disagree about what `org/name` means will
+collide on disk.
+
 ## Default models
 
 Models listed as defaults load when the daemon activates, so the host
@@ -89,12 +115,19 @@ Each loaded model has bounded admission control:
 
 ```toml
 [harness.candle.admission]
-max_in_flight = 8        # concurrent requests actually running
+max_in_flight = 2        # concurrent requests actually running
 max_queue_depth = 8      # waiting before rejection
-max_wait_secs = 30       # how long a queued request waits
+max_wait_secs = 30       # how long a queued request waits for a slot
+kv_max_wait_secs = 300   # how long it waits for enough KV budget
 max_per_principal = 2    # fair share, so one caller cannot monopolise
 ```
 
+`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
+The first waits for a slot to turn over; the second waits for a whole
+long-context sequence to finish, which on a 27B can be minutes. A
+request that needs KV budget nobody has yet is not a request to reject
+in thirty seconds.
+
 Over-capacity requests are rejected promptly with `429`/`503` and a
 `Retry-After` rather than being allowed to queue invisibly. A fast
 rejection a client can retry is far better than a request that hangs
@@ -102,8 +135,19 @@ until something times out.
 
 `max_in_flight` above 1 enables batched decode on architectures that
 support it, so concurrent requests share a decode step instead of
-serialising. Raise it on big cards, leave it at 1 on small ones — the
-KV cache for several concurrent long contexts is itself substantial.
+serialising.
+
+Higher is not better, because **the model's KV budget is divided by it
+at load**. A host advertising a 131k context with eight slots needs
+eight times the KV of one slot, which it does not have — the practical
+result is that each slot gets a fraction of the advertised window and a
+single long session runs out of room while seven idle slots hold
+reservations. Size it against the concurrency you actually observe: on
+one production host, seven days of request spans showed 89.6% of busy
+time was a single request in flight and 93.6% was two or fewer, so two
+slots cover the real load and give each a usable context. Anything above
+the limit queues on admission with a `Retry-After` rather than being
+refused.
 
 `max_per_principal` is what stops one busy client consuming the whole
 model.
@@ -150,7 +194,8 @@ reservation is doing its job, not that the model is overloaded.
 neuron refuses work it cannot finish rather than crashing partway:
 
 - a **static floor** of free VRAM below which prefill is refused
-  (`NEURON_MIN_FREE_VRAM_MB`, default 1500)
+  (`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
+  `[harness.candle.context_limit]`; default 1500)
 - a **length-aware check** that estimates the KV cache the request will
   actually need and refuses if it will not fit
 
@@ -159,15 +204,206 @@ can pass the static floor and still be impossible, because KV grows with
 prompt *and* generation length. See [context
 limits](/docs/operating/context-limits) for how the numbers relate.
 
+`activation_headroom_mb` reserves VRAM per card for prefill activations
+on top of the weights and the KV cache, before the context ceiling is
+computed. Raise it if prefill OOMs on a host whose steady-state
+occupancy looks fine — activations are transient and do not show up in
+the resident figure.
+
 A host that cannot currently serve a model reports that on `/health`,
 and cortex stops routing to it — an unservable replica is not a routing
 target.
 
+## Image models
+
+```toml
+[harness.candle.image]
+te_device = "cpu"     # where the text encoder runs
+te_resident = true    # keep it loaded between generations
+max_dim = 2048        # resolution ceiling per side
+```
+
+The text encoder is the placement decision. On CPU the ~8 GB encoder
+never touches the GPU — prompt encoding is one short forward and the
+features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
+all. Move it to `cuda` only on a card with room to spare.
+
+`te_resident` trades memory for latency: resident costs ~16 GB of system
+RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
+request. CPU defaults to resident, CUDA to rebuilding per request, which
+are the right defaults for where each is scarce.
+
+`max_dim` rejects oversized requests **before admission**, so an
+impossible resolution costs nothing. That matters more than it sounds:
+an OOM on the device poisons the worker context, so refusing early is
+the difference between one rejected request and a model that needs
+restarting.
+
 ## Prompt caps
 
 `NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
 rejected before any GPU work, so a runaway client costs nothing.
 
+## Sampling
+
+Every model publishes its own sampling defaults in
+`generation_config.json`, and a caller can override them per request.
+Between those two sits an operator override, for the cases where the
+model's published defaults are wrong for how you serve it:
+
+```toml
+[[default_models]]
+model_id = "Qwen/Qwen3.8-27B"
+
+[default_models.sampling]
+temperature = 0.6
+presence_penalty = 1.0
+```
+
+Precedence is **request > operator > model > built-in**. Anything you
+leave unset changes nothing, so an empty table behaves exactly as no
+table at all.
+
+Set the same values in the gateway's `models.toml` under
+`[models.sampling]`. A model that is cold-loaded on demand gets its
+sampling from the catalogue, not from this file, so the two must agree
+or the same model samples differently depending on how it arrived —
+`script/check-config-consistency.py` fails the deploy if they diverge.
+
+### Which knob for which symptom
+
+| symptom | knob |
+|---|---|
+| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
+| The think block restates the same passage over and over | `presence_penalty` |
+| Common tokens over-represented across a long answer | `frequency_penalty` |
+| Short immediate loops — "no, no, no…" | `repeat_penalty` |
+| You need two runs to be comparable | `seed` |
+
+**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
+measured a 20% structural-defect rate on long agentic turns against 0/60
+at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
+Qwen recommend for their thinking variants, so this is not fighting the
+model.
+
+**`presence_penalty`** and **`frequency_penalty`** are scored over the
+*whole* generated sequence. They are the ones that matter for reasoning
+models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
+remedy for the endless repetition their models fall into during long
+thinking. Both default to `0.0`, which is off.
+
+**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
+`64`) are the older, *windowed* form: the penalty only sees the last
+`repeat_last_n` tokens. That is fine for chat and close to useless for a
+think block — a 20,000-token deliberation restating a passage 5,000
+tokens later is entirely outside a 64-token window. Measured across
+recorded sessions, **86% of think blocks over 60k characters contained
+verbatim repeated sentences**, one of them repeating 76% of its own
+content.
+
+Widening the window is the obvious move and the wrong one: the penalty
+is token-level, and source code legitimately repeats tokens constantly
+(`const`, `function`, brace runs), so a wide window at a meaningful
+penalty degrades exactly the output you care about. Reach for
+`presence_penalty` first — what recurs is passages, not tokens.
+
+**`seed`** pins the sampler's RNG for every request to this model. It
+exists because parameter search needs a control and the clients that
+would otherwise pin it often cannot. Leaving it set is a product
+decision, not just a testing one: every identical prompt then returns an
+identical answer to every caller, which is predictable and cacheable but
+removes the variation a user may expect from retrying.
+
+## Reasoning
+
+```toml
+[default_models.sampling]
+# nothing here — reasoning settings live in their own places, below
+```
+
+**`preserve_thinking`** controls whether prior turns keep their think
+blocks when the conversation is re-rendered. It is the model's own chat
+template control, not ours:
+
+```toml
+[[default_models]]
+preserve_thinking = false   # keep reasoning only for the turn in progress
+```
+
+Unset is the default and means the template decides. On Qwen3.8-27B that
+is `true` — every prior turn's reasoning is replayed. Full replay is
+what makes prompt growth track *total* completion rather than visible
+output: one measured session produced 28,672 reasoning tokens on turn 1
+and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
+prefill on every turn thereafter.
+
+Setting it to `false` is a legitimate experiment, not a recommendation.
+Whether narrowing replay *helps* the model is unmeasured. Set it in both
+`models.toml` and this file, like sampling, and the value is reported on
+`/v1/models` and logged at load so a session can be attributed to the
+arm it ran under.
+
+**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
+part of the caller's output budget for the answer:
+
+```toml
+[harness.candle.reasoning_budget]
+answer_reserve_tokens = 4096
+```
+
+A reasoning model with no stopping criterion fills whatever budget it is
+given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
+given 32,768 it used 32,768 — and both times returned no answer at all.
+The reserve bounds the think block at `max_tokens - answer_reserve` so
+the rest survives for the reply.
+
+Deliberately *not* derived from reasoning effort. Effort is a prompt
+instruction the model is trained to follow; making it also set a
+guillotine is how a model came to be told to reason at `xhigh` and then
+cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
+reasoning.
+
+The trade to know: when the reserve fires, the model is interrupted
+mid-deliberation and its answer phase tends to state an *intention*
+rather than a conclusion. That is better than returning nothing, but it
+is not free.
+
+## The prefix cache
+
+Snapshots of previously-served prefixes, so a follow-up turn re-uses the
+KV cache instead of re-prefilling the whole conversation:
+
+```toml
+[harness.candle.prefix_cache]
+enabled = true
+budget_mb = 2048    # snapshot bytes per loaded model
+max_entries = 8     # live snapshots per model, regardless of budget
+```
+
+**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
+That is the symptom: the prompt keeps growing, the cached prefix does
+not, and every turn re-prefills the difference. Observed on a 1 GiB
+default — `cached_tokens` pinned at 2,016 for ten consecutive turns
+while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
+at roughly 37 s each, against ~20 s of actual decode.
+
+The number is not free, and the direction of the trade is the opposite
+of what it looks like. **The snapshot budget is reserved before the KV
+budget, not from what is left over.** On a 2×32 GB host serving a 27B at
+`max_in_flight = 2`, measured 1:1:
+
+| `budget_mb` | KV budget | tokens per slot | holds a 53k snapshot |
+|---|---|---|---|
+| 1024 | 4713 MB | 75,400 | no |
+| 2048 | 3689 MB | 59,000 | yes |
+| 3072 | 2665 MB | 42,600 | yes, but a session cannot reach 53k |
+
+So raising it too far starves the KV cache and shortens how long a
+single session can run — the model can no longer reach the context whose
+snapshot the larger cache was meant to hold. Size it against the deepest
+session you actually serve, and check `kv_budget_mb` in the journal
+after a reload.
+
 ## Checking on it
 
 ```sh

helexa.ai/content/docs/2-operating/05-context-limits.md

@@ -32,6 +32,43 @@ daemon now does automatically.
 | `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
 | `limit.output` | same | generation reserve opencode leaves below the wall |
 
+### Tuning the derived limit
+
+Since #67 the neuron computes `limit` itself (see below), and three
+`[harness.candle.context_limit]` settings shape that computation:
+
+```toml
+[harness.candle.context_limit]
+target_prefill_latency_secs = 120     # longest acceptable prefill per turn
+bootstrap_prefill_tok_per_sec = 800   # cold-start estimate until measured
+output_reserve_tokens = 8192          # what a request gets when it names no cap
+max_output_tokens = 32768             # the largest output a request may name
+```
+
+**`target_prefill_latency_secs`** is the one that usually binds. The
+throughput ceiling is `target_prefill_latency_secs × measured
+prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
+advertises about 168k — above most hard ceilings, in which case it stops
+mattering. On a slower host, or one whose recent traffic has been small
+prompts, it is what caps the window.
+
+**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
+its own rate, and stops the advertised window collapsing on a
+freshly-restarted host. Prefill samples below 512 tokens are ignored
+when computing that rate: a short prompt is dominated by fixed
+per-request cost and says nothing about throughput — the deploy's own
+19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
+So a host that has served only small requests keeps the bootstrap
+estimate rather than adopting a misleading one, and no warm-up is
+needed after a restart.
+
+**`output_reserve_tokens`** is what a request that names no cap
+receives. **`max_output_tokens`** is the ceiling advertised to clients
+as `max_output_tokens`, and it is the *ceiling*, not the reserve —
+publishing the reserve tells every client that reads the advertisement
+to cap itself there, which on a reasoning model can be less than a
+single think block.
+
 ## How they must relate
 
 For a single model on a single neuron, all of these must hold: