Files
helexa/crates/cortex-gateway/tests/upstream_chain.rs
rob thijssen a9d7382be8
All checks were successful
CI / Format (push) Successful in 40s
CI / CUDA type-check (push) Successful in 1m39s
CI / Clippy (push) Successful in 2m56s
CI / Test (push) Successful in 6m36s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
feat(B3): cortex upstream entitlement client (#57) + chained provider
cortex can now validate locally-unrecognised bearer keys against the
helexa-upstream authority and reserve/settle their budget there — mesh
accounts work for real inference. The EntitlementProvider trait is the
seam, so cortex's enforcement (auth.rs, metering.rs) is otherwise
unchanged.

- entitlements_upstream.rs: UpstreamEntitlementProvider over reqwest →
  B2's /authz/v1 (resolve/reserve/settle/release/snapshot), presenting the
  operator client bearer. Maps the wire contract back to the trait: granted
  → Reservation, rejected → BudgetError, 401 → InvalidKey. Fail-closed —
  unreachable resolve → AuthError::Unavailable (503, never 401);
  unreachable reserve → retryable BudgetError::RateLimited (refuse, never
  serve un-authorized). settle/release are best-effort (the upstream
  sweeper reaps a lost one).
- entitlements_chain.rs: ChainedEntitlementProvider tries local first
  (operator + infra keys, no network), falls through to upstream for
  unknown keys, and dispatches reserve/settle/release/snapshot to whichever
  backend resolved each account (local treats unknown principals as
  uncapped, so it can't be the blind default).
- cortex-core: AuthError::Unavailable{retry_after_secs}; [upstream] config
  (enabled/url/bearer/timeout). auth.rs maps Unavailable → 503 +
  Retry-After distinctly from InvalidKey → 401, regardless of require_auth.
- state.rs wires the chain when [upstream].enabled, else stays purely local.

Tests (upstream_chain.rs, 4): local key resolves without touching upstream;
unknown key falls through to a mock upstream; unknown-everywhere → 401
InvalidKey; upstream-unreachable → Unavailable (503-mapped), with local keys
still resolving. Existing gateway suites updated for the new config field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 10:41:48 +03:00

106 lines
3.8 KiB
Rust

//! B3: the chained entitlement provider (local → upstream) and fail-closed
//! semantics, exercised against a mock helexa-upstream `/authz/v1`.
use axum::{Json, Router, routing::post};
use cortex_core::config::{ApiKeyConfig, EntitlementsConfig, UpstreamClientConfig};
use cortex_core::entitlements::{AuthError, EntitlementProvider};
use cortex_gateway::entitlements_chain::ChainedEntitlementProvider;
use cortex_gateway::entitlements_local::LocalEntitlementProvider;
use cortex_gateway::entitlements_upstream::UpstreamEntitlementProvider;
use serde_json::{Value, json};
use tokio::net::TcpListener;
/// Mock upstream: `mesh-key` resolves to a mesh account; anything else 401.
/// reserve always grants reservation 1.
async fn spawn_mock_upstream() -> String {
async fn resolve(Json(body): Json<Value>) -> axum::response::Response {
use axum::response::IntoResponse;
if body["api_key"] == "mesh-key" {
Json(json!({"principal": {"account_id": "mesh-acct", "key_id": "mesh-key-1"}}))
.into_response()
} else {
(
axum::http::StatusCode::UNAUTHORIZED,
Json(json!({"error": {"code": "invalid_api_key"}})),
)
.into_response()
}
}
async fn reserve() -> Json<Value> {
Json(json!({ "reservation_id": 1 }))
}
let app = Router::new()
.route("/authz/v1/resolve", post(resolve))
.route("/authz/v1/reserve", post(reserve));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
fn local_with_key() -> LocalEntitlementProvider {
let cfg = EntitlementsConfig {
require_auth: false,
keys: vec![ApiKeyConfig {
key: "local-key".into(),
account_id: "op".into(),
key_id: None,
hard_cap: None,
window: Default::default(),
}],
};
LocalEntitlementProvider::from_config(&cfg)
}
fn chain(local: LocalEntitlementProvider, url: &str) -> ChainedEntitlementProvider {
let upstream = UpstreamEntitlementProvider::new(&UpstreamClientConfig {
enabled: true,
url: url.to_string(),
bearer: "client-secret".into(),
timeout_secs: 5,
});
ChainedEntitlementProvider::new(local, upstream)
}
#[tokio::test]
async fn local_key_resolves_locally() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("local-key").await.expect("local resolves");
assert_eq!(p.account_id, "op");
}
#[tokio::test]
async fn unknown_key_falls_through_to_upstream() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("mesh-key").await.expect("upstream resolves");
assert_eq!(p.account_id, "mesh-acct");
assert_eq!(p.key_id, "mesh-key-1");
}
#[tokio::test]
async fn unknown_everywhere_is_invalid_key() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
match c.resolve("nope").await {
Err(AuthError::InvalidKey) => {}
other => panic!("expected InvalidKey, got {other:?}"),
}
}
#[tokio::test]
async fn upstream_unreachable_fails_closed_as_unavailable() {
// No mock — point at a dead port. A locally-unknown key must surface
// Unavailable (→ 503), never InvalidKey (→ 401).
let c = chain(local_with_key(), "http://127.0.0.1:1");
match c.resolve("some-mesh-key").await {
Err(AuthError::Unavailable { retry_after_secs }) => assert!(retry_after_secs > 0),
other => panic!("expected Unavailable, got {other:?}"),
}
// A local key still resolves without touching upstream.
assert_eq!(c.resolve("local-key").await.unwrap().account_id, "op");
}