All checks were successful
CI / Format (push) Successful in 35s
CI / CUDA type-check (push) Successful in 1m37s
CI / Clippy (push) Successful in 2m53s
CI / Test (push) Successful in 6m41s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Operators are now metered for the tokens they serve on behalf of mesh accounts, and the upstream rolls that up for compensation. cortex-gateway: - served_usage.rs: an in-process per-(account,key,UTC-day) served-token counter, incremented in metering::usage_sink alongside spend/settle for every authenticated request. A flush task (spawned in run() when [upstream].enabled, mirroring poller/evictor) POSTs ABSOLUTE cumulative counters to upstream on [upstream].served_usage_report_interval_secs. - The no-limit infra key is the operator's local key with hard_cap=None (already supported) — it's metered for served-usage but never budget- refused and never hits upstream. helexa-upstream: - POST /authz/v1/served-usage: upserts rows keyed by (operator_id from the client bearer, account, key, period) with GREATEST(existing, incoming) — monotonic + idempotent, so re-sends, races, and a restarted cortex's lower counter never regress the total. - reconcile.rs + `helexa-upstream reconcile` CLI: rolls up unreconciled served_usage per operator/period (SUM::bigint), stamps reconciled_at, prints the totals. Payout mechanism out of scope. Validated against a throwaway Postgres: monotonic upsert (100→250, a 50 re-send stays 250, same-value idempotent) and reconcile rollup + stamp-once; cortex counter unit test for per-principal accumulation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6o3ddqmYNh9kzdwq6eowh
107 lines
3.9 KiB
Rust
107 lines
3.9 KiB
Rust
//! B3: the chained entitlement provider (local → upstream) and fail-closed
|
|
//! semantics, exercised against a mock helexa-upstream `/authz/v1`.
|
|
|
|
use axum::{Json, Router, routing::post};
|
|
use cortex_core::config::{ApiKeyConfig, EntitlementsConfig, UpstreamClientConfig};
|
|
use cortex_core::entitlements::{AuthError, EntitlementProvider};
|
|
use cortex_gateway::entitlements_chain::ChainedEntitlementProvider;
|
|
use cortex_gateway::entitlements_local::LocalEntitlementProvider;
|
|
use cortex_gateway::entitlements_upstream::UpstreamEntitlementProvider;
|
|
use serde_json::{Value, json};
|
|
use tokio::net::TcpListener;
|
|
|
|
/// Mock upstream: `mesh-key` resolves to a mesh account; anything else 401.
|
|
/// reserve always grants reservation 1.
|
|
async fn spawn_mock_upstream() -> String {
|
|
async fn resolve(Json(body): Json<Value>) -> axum::response::Response {
|
|
use axum::response::IntoResponse;
|
|
if body["api_key"] == "mesh-key" {
|
|
Json(json!({"principal": {"account_id": "mesh-acct", "key_id": "mesh-key-1"}}))
|
|
.into_response()
|
|
} else {
|
|
(
|
|
axum::http::StatusCode::UNAUTHORIZED,
|
|
Json(json!({"error": {"code": "invalid_api_key"}})),
|
|
)
|
|
.into_response()
|
|
}
|
|
}
|
|
async fn reserve() -> Json<Value> {
|
|
Json(json!({ "reservation_id": 1 }))
|
|
}
|
|
let app = Router::new()
|
|
.route("/authz/v1/resolve", post(resolve))
|
|
.route("/authz/v1/reserve", post(reserve));
|
|
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
|
let addr = listener.local_addr().unwrap();
|
|
tokio::spawn(async move {
|
|
axum::serve(listener, app).await.unwrap();
|
|
});
|
|
format!("http://{addr}")
|
|
}
|
|
|
|
fn local_with_key() -> LocalEntitlementProvider {
|
|
let cfg = EntitlementsConfig {
|
|
require_auth: false,
|
|
keys: vec![ApiKeyConfig {
|
|
key: "local-key".into(),
|
|
account_id: "op".into(),
|
|
key_id: None,
|
|
hard_cap: None,
|
|
window: Default::default(),
|
|
}],
|
|
};
|
|
LocalEntitlementProvider::from_config(&cfg)
|
|
}
|
|
|
|
fn chain(local: LocalEntitlementProvider, url: &str) -> ChainedEntitlementProvider {
|
|
let upstream = UpstreamEntitlementProvider::new(&UpstreamClientConfig {
|
|
enabled: true,
|
|
url: url.to_string(),
|
|
bearer: "client-secret".into(),
|
|
timeout_secs: 5,
|
|
served_usage_report_interval_secs: 60,
|
|
});
|
|
ChainedEntitlementProvider::new(local, upstream)
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn local_key_resolves_locally() {
|
|
let url = spawn_mock_upstream().await;
|
|
let c = chain(local_with_key(), &url);
|
|
let p = c.resolve("local-key").await.expect("local resolves");
|
|
assert_eq!(p.account_id, "op");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn unknown_key_falls_through_to_upstream() {
|
|
let url = spawn_mock_upstream().await;
|
|
let c = chain(local_with_key(), &url);
|
|
let p = c.resolve("mesh-key").await.expect("upstream resolves");
|
|
assert_eq!(p.account_id, "mesh-acct");
|
|
assert_eq!(p.key_id, "mesh-key-1");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn unknown_everywhere_is_invalid_key() {
|
|
let url = spawn_mock_upstream().await;
|
|
let c = chain(local_with_key(), &url);
|
|
match c.resolve("nope").await {
|
|
Err(AuthError::InvalidKey) => {}
|
|
other => panic!("expected InvalidKey, got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn upstream_unreachable_fails_closed_as_unavailable() {
|
|
// No mock — point at a dead port. A locally-unknown key must surface
|
|
// Unavailable (→ 503), never InvalidKey (→ 401).
|
|
let c = chain(local_with_key(), "http://127.0.0.1:1");
|
|
match c.resolve("some-mesh-key").await {
|
|
Err(AuthError::Unavailable { retry_after_secs }) => assert!(retry_after_secs > 0),
|
|
other => panic!("expected Unavailable, got {other:?}"),
|
|
}
|
|
// A local key still resolves without touching upstream.
|
|
assert_eq!(c.resolve("local-key").await.unwrap().account_id, "op");
|
|
}
|