rob thijssen 7984d27553 feat(#73): capacity-aware dispatch with region affinity + failover ...

The router's data path. Wires the topology poller (#72) and the shared
streaming proxy (#71) into real request routing.

- `dispatch.rs`: `select_cortexes(model)` ranks reachable cortexes that
  can serve the model, best-first — loaded/warm before cold-loadable,
  region match before not, more healthy nodes before fewer, name for
  determinism. `dispatch()` extracts `model`, picks candidates, and
  forwards via `helexa_stream::forward_streaming` (bearer + bytes
  verbatim, SSE streamed back). Cortex's #63 rejections (429/400/…) pass
  through untouched; transport failures fail over to the next candidate;
  a genuine HTTP response — any status — is returned as-is, never retried
  away.
- Router-originated rejections use the #63 envelope: 404 model_not_found
  (no operator serves it), 503 service_unavailable + Retry-After (known
  but all unreachable / all candidates failed to connect), 400
  missing_model_field. `error.rs` is the router's envelope→axum adapter
  (mirrors cortex-gateway's).
- `handlers.rs`: `/v1/chat/completions`, `/v1/completions`,
  `/v1/responses`, `/v1/messages` dispatch to the same path on a chosen
  cortex. The router holds zero entitlement logic — routes on capacity,
  not budget.
- Config: optional `region` on the router and per-cortex for geo affinity.

Tests (`dispatch.rs`): routes to a serving cortex + forwards the bearer;
cortex 429 passes through and is NOT retried; transport failure fails
over to a live cortex; unknown→404, known-but-unreachable→503,
missing-model→400; ranking order (warm/region/headroom). 7 new, existing
skeleton/topology suites unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh

2026-06-21 19:40:07 +03:00

832 B

Raw Blame History

View Raw