# P1 learnings — briefing for the P2 session

Written 2026-06-12 at the close of the P1 sprint (#20, #19, #21, #24
closed; #22 landed minus competitor columns — see the pinned tracking
issue #27). Everything below was learned the hard way during P1 and is
directly useful for P2 (#11 prefix KV caching, #23 chunked prefill,
#1 cold-load, #15 numerical validation).

> Relocated from `doc/plan/p1-learnings.md` (gitignored) into
> `doc/learnings/` so it lives in source control alongside
> [`p2.md`](p2.md). Content is the original P1 briefing, verbatim.

## Working agreements that are now live

- **merge-on-green is standing policy** for roadmap PRs: open the PR,
  background-poll `commits/<sha>/status`, merge with branch-delete the
  moment it's green. On red: investigate, never merge.
- **Branch → PR per roadmap issue**, `Closes #N` in the commit message
  (Gitea auto-closes on merge). Tick the checkbox on #27 afterwards.
- **Every merge to main self-validates on the fleet**: deploy.yml waits
  for `/health` activation `ready` (per-host timeouts: beast 900 s,
  others 300 s), fails on any `activation.failed` entry with the
  per-model error, then asks the loaded model to say a specific word
  ("LLM probe"). A P2 engine change that breaks load or inference
  turns the deploy run red — trust it, watch it after engine merges.
- **Deploy gating is manifest-equality**: a host restarts only when
  `rpm -q` differs from the newest packages.json entry. Never
  reintroduce unprivileged `dnf check-update` — it both hung (quadbrat)
  and silently lied (benjy/beast) in the same run.

## CI behaviour you must plan around

- **`crates/cortex-core/` touches trigger a FULL fleet rebuild and
  neuron restarts** (it's in both change-detection regexes in
  build-prerelease.yml). Each neuron restart costs a cold-load
  (~3m40s on beast, measured by the deploy validation). For P2 engine
  work this is unavoidable (neuron changes anyway); for incidental
  core type tweaks, batch them into the engine PR rather than landing
  them separately.
- Gateway-only and docs-only pushes skip neuron builds/restarts
  entirely. The skip logic diffs against the *last published RPM's*
  sha, so nothing is ever silently missed.
- **`GITHUB_ENV` does not propagate between steps** on these runners
  (observed: probe step set RUSTC_WRAPPER, build ran unwrapped).
  Job-level `env:` and same-step `export` work; cross-JOB outputs
  (`needs.X.outputs.Y`) work. Set env where you use it.
- **sccache**: present in the CUDA runner image, wired with escalation
  (retry → server restart → final attempt uncached) in lint/test/build
  jobs. A sick cache costs minutes, never the run. Same-ref pushes
  coalesce (`cancel-in-progress: true` on build-prerelease) — pushing
  twice in quick succession wastes the first build, so batch commits.
- **`deploy-dev` workflow** (Actions → deploy-dev → pick host): builds
  ONE flavour and scp's the raw binary onto the host — the fast
  iteration path for engine work (~build + restart, no RPM/publish).
  The sudoers rule requires the exact command form in the workflow;
  change both together. The next regular deploy reconciles the host.
- Reading Gitea job logs via API: runner "expression evaluated"
  noise and the rendered-script dump both match naive greps — filter
  with `grep -v 'expression\|rewritten'` and remember the masker
  replaces any string equal to a secret value with `***` (including,
  comically, the word "sccache").

## Engine facts P2 will need (learned from live traffic)

- **neuron streams exactly one SSE chunk per generated visible
  token.** The gateway token metrics (#21) and bench harness (#22)
  rely on this invariant when no usage frame is present. If P2 work
  changes chunking, fix those consumers or land #31 first.
- **No usage frame on streams** (`stream_options.include_usage`
  ignored — #31, p2-labelled). Non-streaming responses DO carry usage.
  Gateway streaming token counters stay silent until #31 lands.
- **Reasoning deltas are off-wire by default** (dropped unless the
  request opts into thinking). Consequences: thinking models burn
  `max_tokens` invisibly, and short `max_tokens` can yield ZERO
  visible chunks (`finish_reason: length`, empty stream). Benchmarks
  and probes append Qwen's `/no_think` soft switch — it renders an
  empty think block and works on all fleet models. The deploy probe
  uses `max_tokens: 512` headroom instead.
- **27B baseline (doc/benchmarks.md, 2026-06-12)**: decode ~35 tok/s
  FLAT from 128→4k context (Gated DeltaNet's O(1) recurrent state);
  TTFT 1.66 s @128 / 7.07 s @4k. The 4k TTFT is the #23 before-number.
  Re-run `script/bench.py` (same flags as the doc) after every engine
  change — it takes ~5 min for the fleet and the table is
  append-friendly via `--json`.

## Design notes for #11 (prefix KV caching) specifically

- The per-request cache clear lives in `harness/candle.rs` (~line
  1393, `clear_kv_cache()` before every inference) — that call site is
  the entry point: "stop deleting it" is the issue's framing.
- **Hybrid-architecture caveat**: for the qwen3_5/3.6 family, a
  "prefix cache" is NOT just attention KV. Three out of four decoder
  layers are GatedDeltaNet carrying `conv_state` + `recurrent_state`
  (see `harness/arch/qwen3_5/linear_attn.rs` doc-comment), and the
  vision/M-RoPE position counters also persist (`tp_qwen3_5.rs`
  clear_kv_cache resets rope counters). A reusable prefix snapshot at
  token N = attention KV **plus** GDN states **plus** position
  counters, all consistent at exactly N. Recurrent state makes partial
  prefix reuse impossible mid-stride — match must be exact-prefix at
  the snapshot boundary.
- **All cache state must live inside the device worker** (the slab,
  per CLAUDE.md "Per-device worker thread"): tensors never escape the
  worker thread; new state types mean new `Job` variants in
  `device_worker/jobs.rs` + handlers in `dispatch.rs`. The recovery
  path (#17/#20) unloads and reloads models — cached prefixes must be
  invalidated on unload (the snapshot pattern used by the `recovering`
  map in candle.rs is a reference for state that must outlive registry
  slots — but for caches, dropping on unload is correct and simpler).
- The eviction story (#11 body: LRU bounded by a per-device VRAM
  budget) can reuse the worker's existing VRAM-query plumbing
  (`device_vram_mb` routes through the worker since Phase 1).
- Measurement: `cortex_time_to_first_token_seconds{model,node}` is
  live in Prometheus, and `script/bench.py` gives the before/after
  table. For agent-shaped workloads (the win case), benchmark a
  multi-turn conversation: send turn N, then turn N+1 = same prefix +
  new question, and compare TTFT-with-cache vs today's full re-prefill.

## Misc operational facts

- Fleet: beast 2×5090 (27B Q6K TP=2), benjy 4090 (8B), quadbrat 3060
  (1.7B); gateway on hanzalova:31313; all reachable by ssh as the
  operator (passwordless sudo) — used for sudoers sync, live probes,
  and killing wedged processes during P1.
- The tea CLI token (`~/.config/tea/config.yml`) authenticates against
  git.lair.cafe for everything the gitea-mcp tools don't cover
  (milestones, labels, pin, merge).
- Anthropic streaming is now production-real: Claude Code can be
  pointed at the gateway. Real-traffic feedback from that is useful
  input for P2 priorities (especially tool-use-heavy streams).
