rob thijssen 24e20dcb5c feat(catalogue,gateway): model aliases (helexa/small, helexa/balanced, helexa/large) ...

Operators can now define tier aliases in models.toml:

  [aliases]
  "helexa/small" = "Qwen/Qwen3-1.7B"
  "helexa/balanced" = "Qwen/Qwen3-8B"
  "helexa/large" = "Qwen/Qwen3.6-27B"

A client request for `model: "helexa/small"` is resolved to the concrete
model id at routing time. The gateway also rewrites the proxied body's
`model` field to the concrete id so neuron sees a name that matches its
loaded handle (otherwise the harness rejects the request).

Motivated by the finger-in-the-wind benchmark: same "what's the capital
of Georgia" probe runs in 2.5s on the 1.7B vs 6.7s on the 27B with
identical correctness. Aliases let clients pick a latency tier without
hardcoding model ids, and let operators swap targets without changing
client code.

Changes:
  * cortex-core: `ModelCatalogue` gains `aliases: HashMap<String, String>`
    + `resolve_alias(&str) -> &str`. Unit tests cover the basic
    resolution + TOML round-trip.
  * cortex-gateway:
    * `RouteDecision` gains `resolved_model_id: String`. `router::resolve`
      consumes aliases at entry and threads the concrete id through.
    * Handlers (chat_completions, completions, anthropic_messages
      streaming + non-streaming) rewrite the body's `model` field with
      `rewrite_model_in_body` before proxying, using the resolved id
      for metrics labels, LRU touch, and the body itself.
    * `/v1/models` (Pass 4) emits each alias as its own entry mirroring
      the target's `loaded` flag, feasible_on, and locations — clients
      browsing the endpoint see both names and can pick either.
  * `models.toml` declares the three tier aliases; `models.example.toml`
    documents the section as opt-in.
  * Integration tests verify: end-to-end alias→concrete request flow,
    alias surfacing in /v1/models, and no-op fall-through for
    non-alias model ids.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 16:10:41 +03:00

2.6 KiB

Raw Blame History

View Raw