Files
helexa/crates/cortex-gateway/tests/upstream_chain.rs
rob thijssen f4117224fc
All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m53s
CI / Test (push) Successful in 6m41s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
feat(B6): served-usage ledger + reconciliation (#58)
Operators are now metered for the tokens they serve on behalf of mesh
accounts, and the upstream rolls that up for compensation.

cortex-gateway:
- served_usage.rs: an in-process per-(account,key,UTC-day) served-token
  counter, incremented in metering::usage_sink alongside spend/settle for
  every authenticated request. A flush task (spawned in run() when
  [upstream].enabled, mirroring poller/evictor) POSTs ABSOLUTE cumulative
  counters to upstream on [upstream].served_usage_report_interval_secs.
- The no-limit infra key is the operator's local key with hard_cap=None
  (already supported) — it's metered for served-usage but never budget-
  refused and never hits upstream.

helexa-upstream:
- POST /authz/v1/served-usage: upserts rows keyed by (operator_id from the
  client bearer, account, key, period) with
  GREATEST(existing, incoming) — monotonic + idempotent, so re-sends,
  races, and a restarted cortex's lower counter never regress the total.
- reconcile.rs + `helexa-upstream reconcile` CLI: rolls up unreconciled
  served_usage per operator/period (SUM::bigint), stamps reconciled_at,
  prints the totals. Payout mechanism out of scope.

Validated against a throwaway Postgres: monotonic upsert (100→250, a 50
re-send stays 250, same-value idempotent) and reconcile rollup +
stamp-once; cortex counter unit test for per-principal accumulation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
2026-06-23 11:45:57 +03:00

107 lines
3.9 KiB
Rust

//! B3: the chained entitlement provider (local → upstream) and fail-closed
//! semantics, exercised against a mock helexa-upstream `/authz/v1`.
use axum::{Json, Router, routing::post};
use cortex_core::config::{ApiKeyConfig, EntitlementsConfig, UpstreamClientConfig};
use cortex_core::entitlements::{AuthError, EntitlementProvider};
use cortex_gateway::entitlements_chain::ChainedEntitlementProvider;
use cortex_gateway::entitlements_local::LocalEntitlementProvider;
use cortex_gateway::entitlements_upstream::UpstreamEntitlementProvider;
use serde_json::{Value, json};
use tokio::net::TcpListener;
/// Mock upstream: `mesh-key` resolves to a mesh account; anything else 401.
/// reserve always grants reservation 1.
async fn spawn_mock_upstream() -> String {
async fn resolve(Json(body): Json<Value>) -> axum::response::Response {
use axum::response::IntoResponse;
if body["api_key"] == "mesh-key" {
Json(json!({"principal": {"account_id": "mesh-acct", "key_id": "mesh-key-1"}}))
.into_response()
} else {
(
axum::http::StatusCode::UNAUTHORIZED,
Json(json!({"error": {"code": "invalid_api_key"}})),
)
.into_response()
}
}
async fn reserve() -> Json<Value> {
Json(json!({ "reservation_id": 1 }))
}
let app = Router::new()
.route("/authz/v1/resolve", post(resolve))
.route("/authz/v1/reserve", post(reserve));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}")
}
fn local_with_key() -> LocalEntitlementProvider {
let cfg = EntitlementsConfig {
require_auth: false,
keys: vec![ApiKeyConfig {
key: "local-key".into(),
account_id: "op".into(),
key_id: None,
hard_cap: None,
window: Default::default(),
}],
};
LocalEntitlementProvider::from_config(&cfg)
}
fn chain(local: LocalEntitlementProvider, url: &str) -> ChainedEntitlementProvider {
let upstream = UpstreamEntitlementProvider::new(&UpstreamClientConfig {
enabled: true,
url: url.to_string(),
bearer: "client-secret".into(),
timeout_secs: 5,
served_usage_report_interval_secs: 60,
});
ChainedEntitlementProvider::new(local, upstream)
}
#[tokio::test]
async fn local_key_resolves_locally() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("local-key").await.expect("local resolves");
assert_eq!(p.account_id, "op");
}
#[tokio::test]
async fn unknown_key_falls_through_to_upstream() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
let p = c.resolve("mesh-key").await.expect("upstream resolves");
assert_eq!(p.account_id, "mesh-acct");
assert_eq!(p.key_id, "mesh-key-1");
}
#[tokio::test]
async fn unknown_everywhere_is_invalid_key() {
let url = spawn_mock_upstream().await;
let c = chain(local_with_key(), &url);
match c.resolve("nope").await {
Err(AuthError::InvalidKey) => {}
other => panic!("expected InvalidKey, got {other:?}"),
}
}
#[tokio::test]
async fn upstream_unreachable_fails_closed_as_unavailable() {
// No mock — point at a dead port. A locally-unknown key must surface
// Unavailable (→ 503), never InvalidKey (→ 401).
let c = chain(local_with_key(), "http://127.0.0.1:1");
match c.resolve("some-mesh-key").await {
Err(AuthError::Unavailable { retry_after_secs }) => assert!(retry_after_secs > 0),
other => panic!("expected Unavailable, got {other:?}"),
}
// A local key still resolves without touching upstream.
assert_eq!(c.resolve("local-key").await.unwrap().account_id, "op");
}