rob thijssen
63f578cb15
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m38s
CI / Clippy (push) Successful in 2m18s
CI / Test (push) Successful in 4m54s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The router's /v1/models is now the deduped union of every reachable cortex's catalogue, so an opencode client doing discovery against the router resolves the whole federation without knowing about operators or cortexes (resolves #61's "Router/discovery contract"). To preserve per-model limit/cost, the topology poller now retains each cortex's full `cortex_core::node::CortexModelEntry` (was distilled to a {loaded, feasible} bool). `entry_feasible()` replaces the dropped field; dispatch (#73) and `cortexes_serving` use it — no routing behaviour change. `catalogue.rs::aggregate_models`: - Dedupe by model id; a model served by >=1 reachable cortex appears once. - Merge availability: `loaded` OR across operators; only feasible (loaded-or-cold-loadable) entries surface — a catalogue-only model no neuron can host is hidden. - Re-tier to operator names: `feasible_on` becomes the cortexes that can serve it and `locations` the operators it's loaded on (node = cortex name), so the federation view doesn't leak each operator's neuron names or per-device VRAM. - Conflict resolution: `limit` → tightest (smallest context, so a client never overflows the most-constrained operator); `cost` → cheapest (the federation "from" price). Richer range/region policy couples to #68, noted as follow-up. Tests: 4 unit (dedupe+merge, unreachable excluded, infeasible hidden, tightest-limit+cheapest-cost) + 1 end-to-end (two mock cortexes overlapping on a model → GET /v1/models over HTTP asserts the merged union). dispatch/topology suites updated for the entry-storage change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
90 lines
3.1 KiB
Rust
90 lines
3.1 KiB
Rust
use crate::state::RouterState;
|
|
use crate::{catalogue, dispatch};
|
|
use axum::body::Bytes;
|
|
use axum::http::HeaderMap;
|
|
use axum::response::Response;
|
|
use axum::{Json, Router, extract::State, routing::get, routing::post};
|
|
use serde_json::{Value, json};
|
|
use std::sync::Arc;
|
|
|
|
/// Routes served by the router. Inference paths are capacity-aware-dispatched
|
|
/// (#73) to a downstream cortex; `/health` and a stub `/v1/models` are local.
|
|
pub fn api_routes() -> Router<Arc<RouterState>> {
|
|
Router::new()
|
|
.route("/v1/chat/completions", post(chat_completions))
|
|
.route("/v1/completions", post(completions))
|
|
.route("/v1/responses", post(responses))
|
|
.route("/v1/messages", post(messages))
|
|
.route("/v1/models", get(list_models))
|
|
.route("/health", get(health))
|
|
.route("/", get(health))
|
|
}
|
|
|
|
// ── Inference paths — forwarded verbatim to a chosen cortex ──────────
|
|
//
|
|
// Each handler dispatches to the same path on a capacity-bearing cortex.
|
|
// The body is parsed only to read `model`; the bearer and bytes are
|
|
// forwarded unchanged, and the SSE response streams back verbatim.
|
|
|
|
async fn chat_completions(
|
|
State(state): State<Arc<RouterState>>,
|
|
headers: HeaderMap,
|
|
body: Bytes,
|
|
) -> Response {
|
|
dispatch::dispatch(&state, "/v1/chat/completions", headers, body).await
|
|
}
|
|
|
|
async fn completions(
|
|
State(state): State<Arc<RouterState>>,
|
|
headers: HeaderMap,
|
|
body: Bytes,
|
|
) -> Response {
|
|
dispatch::dispatch(&state, "/v1/completions", headers, body).await
|
|
}
|
|
|
|
async fn responses(
|
|
State(state): State<Arc<RouterState>>,
|
|
headers: HeaderMap,
|
|
body: Bytes,
|
|
) -> Response {
|
|
dispatch::dispatch(&state, "/v1/responses", headers, body).await
|
|
}
|
|
|
|
async fn messages(
|
|
State(state): State<Arc<RouterState>>,
|
|
headers: HeaderMap,
|
|
body: Bytes,
|
|
) -> Response {
|
|
dispatch::dispatch(&state, "/v1/messages", headers, body).await
|
|
}
|
|
|
|
/// `GET /health` — router liveness plus a summary of downstream cortex
|
|
/// reachability from the topology poller (#72). `status` reflects the
|
|
/// router process itself (always `ok` if it answers); downstream health is
|
|
/// the informational `cortexes` block, so a fully-degraded fleet doesn't
|
|
/// make the router look dead to its own liveness probe.
|
|
async fn health(State(state): State<Arc<RouterState>>) -> Json<Value> {
|
|
let topo = state.topology.read().await;
|
|
let reachable = topo.values().filter(|t| t.reachable).count();
|
|
Json(json!({
|
|
"status": "ok",
|
|
"cortexes": {
|
|
"configured": state.cortexes.len(),
|
|
"reachable": reachable,
|
|
}
|
|
}))
|
|
}
|
|
|
|
/// `GET /v1/models` — the federation catalogue (#75): the deduped union of
|
|
/// every reachable cortex's `/v1/models`, so a client doing discovery
|
|
/// against the router resolves the whole federation without knowing about
|
|
/// operators or cortexes.
|
|
async fn list_models(State(state): State<Arc<RouterState>>) -> Json<Value> {
|
|
let topo = state.topology.read().await;
|
|
let data: Vec<Value> = catalogue::aggregate_models(&topo)
|
|
.iter()
|
|
.map(|e| json!(e))
|
|
.collect();
|
|
Json(json!({ "object": "list", "data": data }))
|
|
}
|