rob thijssen 4f05a87449 feat(neuron): self-derived context-limit core — physics + policy (#67 phase 1) ...

Refs #67. The correct limit{context,input,output} for a deployment is a
computed function of model architecture + live free VRAM + a
coherence/throughput trade-off, not an operator-declared static fact that
goes stale on model swap. This lands the arch-agnostic derivation core;
later phases capture per-model physics at load, measure throughput, and
advertise/enforce the computed limit.

- crates/neuron/src/harness/context_limit.rs (new):
  - kv_bytes_per_token(): shared per-card KV cost (counts only
    full-attention layers; sharded by TP world size). The TP load paths'
    inline math folds onto this in phase 2.
  - ContextProfile: per-model physics snapshot (max_position_embeddings,
    kv_bytes_per_token_per_card, world_size).
  - derive_limit(): context = min(max_pos, vram_ceiling,
    throughput_ceiling) clamped by an optional backstop; input = context −
    output; rounded to 1024. 6 unit tests.
- config.rs: [harness.candle.context_limit] block (mirrors prefix_cache):
  target_prefill_latency_secs, bootstrap_prefill_tok_per_sec,
  activation_headroom_mb, min_free_floor_mb, output_reserve_tokens.
- neuron.example.toml: documented the new block.

No runtime behaviour change yet. fmt/clippy/test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-17 13:00:52 +03:00

4.9 KiB

Raw Blame History

View Raw