# P2 learnings — briefing for the P3 session

Written 2026-06-13 at the close of the P2 sprint. The whole P2 queue
landed in one session: #11 (prefix KV caching), #23 (chunked
delta-rule prefill), #1 (TP cold-load), #15 (numerical validation) —
plus #42 filed (CI flake). Numbers and PRs are on the issues and the
tracker (#27); this doc is the hard-won *how*, for #18, #25, #7, #26,
#4 and anything after.

Read [`p1.md`](p1.md) first — its working agreements (merge-on-green,
branch→PR per issue, fleet self-validation, the `deploy-dev` fast
path) all held through P2 and are not repeated here except where P2
changed or sharpened them.

## The single most important lesson: CI green ≠ correct

P2 shipped four correctness bugs that **passed every CI gate** and
only surfaced on the live fleet. Each now has a regression test that
forbids the wrong thing. The pattern is the lesson: the engine math
is exercised by CPU unit tests on tiny random tensors, and random
data is *benign* — it doesn't reproduce the conditions real model
weights and real prompts create.

1. **Prefix snapshot taken post-generation never re-matched** (#11).
   The cached token sequence included the model's `<think>` reasoning
   tokens, which the client strips when it echoes the assistant turn
   back — so the next prompt was never a token-prefix of the cached
   sequence. Caught only by watching `reused=0` in the journal on a
   real multi-turn conversation against the 27B. Fix: snapshot at the
   *prefill boundary* (prompt-only), not after generation (PR #36).
2. **Full-prompt snapshots broke on BPE retokenization** (#11). A
   prompt ends `…<|im_start|>assistant\n`; when the next turn
   re-tokenizes that text followed by the reply, the trailing `\n`
   merges with the reply's first characters into *different* token
   ids — so the last 1–2 tokens of the cached sequence diverge and the
   exact-prefix match (forced by GDN recurrent state — no partial
   rewind) never fires. Fix: snapshot one past the **last special
   token** (`<|im_start|>`); special tokens are hard tokenizer
   segmentation points, so that prefix is provably stable across
   renders (PR #37).
3. **The 0.8B "passed" both of the above by luck** — its replies
   started with the atomic `<think>` special token, which both keeps
   the reasoning marker in the echoed text *and* blocks the BPE merge.
   **Validate any model-text-dependent behaviour on more than one
   model.** A single model can mask a class of bug entirely.
4. **The chunked delta-rule's tempting optimization was numerically
   wrong in f32** (#23). `(I − T)⁻¹` for the strictly-lower-triangular
   `T` *can* be written as the nilpotent squaring product
   `Π(I + T^(2^j))` (6 matmuls vs 63 sequential row updates), and it
   **passed parity on random data**. On real prompts with repetitive
   text → correlated keys, raw powers of `T` grow combinatorially
   (path counts ≈ C(62,31) ≈ 4.6e17) before nilpotency collapses
   them, destroying f32 precision → NaN logits → `"!!!"` replies.
   The HF reference uses forward substitution for a reason; port it
   faithfully. The regression test
   (`chunked_ut_transform_survives_correlated_keys`) builds
   near-identical keys with β≈1 and diverges to ~8e30 under the
   squaring form.

**Takeaway for P3:** when you write an engine-math unit test, ask
"what would real weights/prompts do that random tensors don't?" —
correlated keys, special-token boundaries, reasoning markers,
repetitive text. Build at least one adversarial fixture per new path.
The live two-turn / fixed-prompt probe on the fleet caught all four;
budget time for it after every engine merge, not just at the end.

## Validating on the fleet (what worked)

- **Live multi-turn / fixed-prompt probe over ssh** is the fastest
  truth: `curl <host>:13131/v1/chat/completions` and read
  `journalctl -u neuron` (fleet runs `RUST_LOG=debug`) for the
  request-path lines — `prefix cache: hit reused_tokens=…`,
  `prefill complete … reused=… elapsed_ms=…`, the new per-phase
  load timings. Greedy (`temperature: 0`) so outputs are comparable.
- **A/B a change behind an env kill-switch** without redeploying:
  drop `Environment=NEURON_GDN_CHUNKED=0` (or whatever the switch is)
  into `/etc/systemd/system/neuron.service.d/<name>.conf`,
  `daemon-reload`, `restart`, measure; remove the drop-in, restart,
  measure again. This gave the clean 8156 ms → 3694 ms chunked-vs-
  recurrent prefill comparison on the *same binary, same prompt*. Add
  such a switch to any perf-sensitive path you land — it's the only
  way to get an honest before/after without a second deploy.
- **`deploy-dev` is the iteration loop for engine work**: it builds
  one flavour and scp's the raw binary, ~build + restart, no
  RPM/publish (~10 min on beast vs ~25 for a full pipeline). Dispatch
  it from the feature branch via the API
  (`POST actions/workflows/deploy-dev.yml/dispatches`,
  `{"ref":"<branch>","inputs":{"target":"beast"}}`). The next regular
  deploy reconciles the host back to the packaged binary. Used it to
  catch the `"!!!"` NaN bug before merging, then again to verify the
  fix.
- **`nvcc` is on beast** (`/usr/local/cuda-13.0/bin/nvcc`, not on
  PATH) — so beast can build the `cuda` feature locally for an
  apples-to-apples bf16 numerical comparison, and has the full Rust
  toolchain (`~/.cargo/bin`). Useful for the #15 harness and any
  cuda-gated test you can't run on the dev box.

## CI behaviour P2 added to the P1 list

- **The "CUDA type-check" job is your only pre-merge check of
  `#[cfg(feature = "cuda")]` code.** Local clippy/test on the dev box
  compile no-cuda only (no nvcc), so cuda-gated mistakes compile
  clean locally and fail in CI. P2 hit two:
  - a `std::sync::MutexGuard` (the prefix-cache registry) held across
    an `.await` inside a `let`-chain made the TP request futures
    non-`Send` → `tokio::spawn` rejected them. **Bind the guard's
    result before any await; never hold a registry lock across a
    suspension point.**
  - a `match` arm returning `candle_core::Error` vs `anyhow::Error`
    — fine until the cuda arm widened the type set.
  Push the branch early and watch that one job after touching
  cuda-gated code; don't wait for the full green.
- **Change-detection skip is per-package, and a flavour build failure
  splits the fleet.** build-prerelease diffs each package against the
  last *published* RPM's sha and skips unchanged ones — good. But when
  one flavour's *build* fails (see #42), only that package is skipped
  from publish; the other two publish and deploy, leaving the fleet on
  mixed versions. **Watch all three `Build neuron-<flavour>` +
  `Package …` jobs, not just the combined commit status** (which can
  read `success`/`pending` while a flavour quietly failed).
- **Retrigger a skipped/failed flavour with an empty commit.** Gitea
  on this version has no run-rerun API endpoint
  (`actions_run_write rerun` returns 404). `git commit --allow-empty`
  + push re-runs the pipeline; change-detection sees the still-stale
  flavour (its last *published* sha is older) and rebuilds it. P2
  needed this twice for #42.
- **The RPM `git<sha>` stamp is the per-package input-change sha, NOT
  main HEAD.** Don't confirm a rollout by grepping `rpm -qa` for the
  merge commit's short sha — it won't match. Confirm via an **uptime
  reset** on `/health` plus a behaviour only the new build has (a new
  log line, a new metric). The fleet-validation memory has this.
- **`cancel-in-progress` can split a host mid-deploy.** A same-ref
  push that lands between a host's `dnf upgrade` and its `restart`
  cancels the restart, leaving the host with the *new* RPM installed
  but the *old* binary running. The next successful deploy heals it
  (the rpm-vs-packages.json compare sees the next version). Filed as
  part of #42; avoid pushing again while a deploy is mid-flight.

## Engine facts P2 established (for #18, #25, and beyond)

- **The device-worker discipline scales cleanly to new state.** Adding
  prefix snapshots was: new `Job` variants
  (`SnapshotKv`/`RestoreKv`/`DropKvSnapshot`) + handlers in
  `dispatch.rs` + a `HashMap` beside the model slab, dropped with the
  model in `DropArch`. The async side holds only an opaque id + the
  token sequence + a byte size — tensors never escape the worker.
  Reuse this shape for any new per-model device state (#25's drafter
  KV, for instance).
- **TP mirrors single-GPU through one pool-minted id.** For TP the
  leader's snapshot lives in its device worker (`Job::Tp*`) and each
  subprocess rank stores its shard's snapshot in-process via new
  `WorkerRequest` variants in `tp/rpc.rs`, all keyed by the *same*
  id the pool mints and broadcasts. Step fan-out is synchronous, so
  all ranks sit at the same token boundary — that's what makes a
  cross-rank snapshot consistent. Partial-failure rule: any rank fails
  restore → clear all ranks + full prefill; any rank fails snapshot →
  drop the id everywhere. The TP shard state has the *same shape* as
  single-GPU, so the `arch/qwen3_5/snapshot.rs` types are shared
  verbatim.
- **GDN state copy semantics are not uniform.** The CUDA delta-rule
  kernels mutate the recurrent-state buffer **in place**
  (`&mut state_bh`; `flatten`/`contiguous` on a contiguous tensor is a
  view), so GDN `conv_state`/`recurrent_state` snapshots must
  **deep-copy** (`Tensor::copy`) in both directions. Attention
  `ConcatKvCache` k/v can share storage (its `append` cats into fresh
  allocations and never mutates stored tensors). Get this wrong and
  the snapshot silently tracks the live cache.
- **The reusable-prefix snapshot boundary is "one past the last
  `<|im_start|>`"** — not the full prompt (BPE retokenization), not
  post-generation (reasoning/tool-call tokens stripped on echo). The
  request path prefills to that boundary, snapshots, then finishes the
  ~2-token `assistant\n` tail. This generalises: any future
  cross-request state reuse must key on a tokenizer-stable boundary,
  and special tokens are the only provably-stable ones.
- **Chunked prefill composes with the prefix cache.** A restored
  conversation's divergent suffix still prefills chunked (the chunked
  path takes a non-zero `initial_state`, validated by a parity test).
  Decode and short (<64-token) prompts keep the recurrent path —
  decode was deliberately untouched. #18 (single-GPU vision prefill
  chunking) and #25 (speculative decoding) both build on these paths.
- **Cold-load is now disk-read-bound.** Parallel ISQ (#1) cut the 27B
  TP cold-load 221 s → 86 s by fanning candle's per-block k-quant math
  (`k_quants::GgmlType::from_float`, public API — no candle fork)
  across the rayon pool, **byte-identical** to `QTensor::quantize`.
  The residual 79 s layer-loop is now ~1.2 s/layer dominated by
  reading the 54 GB bf16 safetensors (~700 MB/s with both ranks
  reading), not quantization. The next lever (issue #1 item 3) is a
  post-ISQ disk cache; raise a fresh issue only if 86 s still hurts.
  Phase timing is now instrumented (`layer loop complete elapsed_ms`,
  `lm_head loaded elapsed_ms`) — read it before optimizing further.

## Numerical validation (#15) — reusable rig for every arch

- **Capture reference fixtures in f32, not bf16.** f32-vs-f32 pins the
  *math*; the implementation matched HF exactly (text logits
  `max_abs 0.000`, vision end-to-end `cosine 1.000000`). Cross-dtype
  comparison is dominated by bf16 rounding chaos through a deep tower
  (global cosine ~0.997, *worst patch* ~0.92, worst-patch index
  unstable across runs) — that's production-dtype noise, not bug, and
  it drowns the signal. The script defaults to f32 for this reason.
- **The 0.8B validates the same arch modules as the 27B** — they
  differ only in hyperparameters — so the automated test runs against
  the 0.8B (an f32 27B forward needs ~108 GB; beast has ~91 GB free).
  A bf16 27B text fixture is checked in for the manual procedure.
- **Mutation sensitivity is the test that the test works.** Re-running
  with `NEURON_VISION_LEGACY_POS=1` (the deliberately-wrong sequential
  pos-embed lookup) collapses tower cosine 0.999998 → 0.753 and fails
  loudly. Every numerical harness needs a known-bad toggle to prove it
  isn't asserting on noise.
- **The harness found a real production bug.** The pos-embed bilinear
  blend was rounding interpolation weights to bf16 *before* blending;
  the reference keeps them f32 and casts once at the end. Fixed in the
  #15 PR. Numerical validation pays for itself.
- **Reproducing HF on beast needs two shims** (documented in
  `crates/neuron/tests/fixtures/numerical/README.md` and the dump
  script): (a) transformers 5.9 ↔ kernels 0.15 import breakage —
  monkeypatch `LayerRepository.__init__` / `FuncRepository.__init__`
  to inject a `revision`, with `USE_HUB_KERNELS=NO`; (b) neuron's
  local HF snapshot lacks `preprocessor_config.json` (hf-hub only
  fetched what neuron needed) — load the *processor* from the repo id
  with `HF_HUB_CACHE` pointed at a scratch dir. The test self-skips
  without `NEURON_REF_MODEL_PATH`, so CI compiles it without weights.
- **Extend this for any new arch.** `script/dump_reference.py` +
  `tests/numerical_reference.rs` are the template; capture
  text + vision fixtures, calibrate thresholds against the observed
  f32 agreement, add a mutation toggle.

## State of the board for P3

- **P1 + P2 are fully closed** with published before/after numbers on
  every issue and on #27. The milestone-7→8 perf story is in hand:
  prefix cache (warm 27B turn ~10 s → ~2.3 s), chunked prefill (cold
  ~5k prefill 8.2 s → 3.7 s), cold-load (221 s → 86 s), and a
  fidelity harness proving f32-exact parity with HF.
- **#26 (tagged release + public writeup) is now unblocked** — it was
  gated on the #22/#23 numbers, which all exist. It's the natural next
  pick: the engine work it would showcase is done.
- **#25 (speculative decoding) is sequenced after #23** (done) and
  reuses the device-worker + TP-rank state patterns above for the
  drafter. **#18 (single-GPU vision prefill chunking)** is the
  single-card-tier parity counterpart to #23 and the chunked-prefill
  machinery already exists.
- **#42 (ampere `ptxas` SIGSEGV in candle-flash-attn sm_86 kernels)**
  is open runner-infra: intermittent, sticky within a run, clears
  across runs; cost two retriggers in the P2 session. Mitigation ideas
  (serialize the flash-attn kernel compile, cap concurrent flavour
  builds, check runner memory headroom) are on the issue. Until it's
  fixed, watch the three flavour jobs and retrigger with an empty
  commit when one segfaults.
- The fleet, gateway, tea-CLI auth, and ssh/sudo access are unchanged
  from P1 — see [`p1.md`](p1.md)'s "Misc operational facts".
