feat(cortex): rank models by residency priority instead of pinning them
Some checks failed
CI / Clippy (push) Blocked by required conditions
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 12s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m58s
CI / Format (push) Successful in 9s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Some checks failed
CI / Clippy (push) Blocked by required conditions
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 12s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m58s
CI / Format (push) Successful in 9s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
`pinned_on` was doing two unrelated jobs under one name. It restricted *where* a model could be placed (`is_feasible_on`) and it made that model immune to eviction *there* (`is_pinned`). Because one field carried both meanings, neither could be asked for on its own, and the immunity it granted was absolute: a pinned model was protected from everything, an unpinned one from nothing. That is not enough to describe a fleet. A real catalogue needs to say that the image generator may take the mid tier's node when someone asks for an image, must never take the flagship's, and that the frontier coder model may. Those three statements are consistent with each other and inexpressible as a boolean, because two of them concern the same protected model and disagree about who is doing the taking. Eviction was also blind to that question. `evict_lru_on_node` received only the node, chose the oldest unpinned resident, and never learned what it was making room for -- so "may X displace Y" had nowhere to be asked even if the catalogue could express it. This splits the two concepts. `pinned_on` keeps only its affinity meaning. A new `residency_priority` decides who yields to whom, and the evictor now takes the incoming model so the comparison is between a pair rather than a property of one. The router's evictable estimate uses the same predicate, so a node can no longer rank as able to make room and then decline to make any. Three details are load-bearing: Displacement requires a strictly higher rank, so equal-ranked models never evict each other. Without that, two models of the same class thrash a node, each evicting the other on alternate requests. Priority governs only whether a displacement is permitted, never whether one is needed. Free-fit still outranks evict-fit, so a model never displaces anything while a node with room exists. A profile carrying `pinned_on` and no explicit priority defaults high rather than to the ordinary case. A catalogue written when `pinned_on` implied immunity must not silently start letting its flagship be evicted -- the failure would be invisible until a cold-load took the top tier off its node. Models absent from the catalogue rank at the default rather than being unevictable, so an unlisted resident cannot wedge a node permanently. Operator-visible: models.toml is gitignored, so fleet priorities are a deploy-time config change, not part of this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0165r11RzqkMqWWXfJE8tAVU
This commit is contained in:
@@ -241,7 +241,8 @@ quant = "Q4_K_M"
|
||||
vram_mb = 19000
|
||||
min_devices = 2
|
||||
min_device_vram_mb = 10000
|
||||
pinned_on = ["beast"] # optional: never evict from these neurons
|
||||
pinned_on = ["beast"] # optional: affinity — run only on these
|
||||
residency_priority = 300 # optional: who may displace whom under VRAM pressure
|
||||
```
|
||||
|
||||
### neuron.toml (per-host)
|
||||
|
||||
@@ -620,7 +620,8 @@ quant = "Q4_K_M"
|
||||
vram_mb = 19000
|
||||
min_devices = 2
|
||||
min_device_vram_mb = 10000
|
||||
pinned_on = ["beast"] # optional: never evict from these neurons
|
||||
pinned_on = ["beast"] # optional: affinity — run only on these
|
||||
residency_priority = 300 # optional: who may displace whom under VRAM pressure
|
||||
|
||||
[[models]]
|
||||
id = "Qwen/Qwen3-VL-8B"
|
||||
|
||||
@@ -22,9 +22,28 @@ pub struct ModelProfile {
|
||||
/// Minimum VRAM per device in MB.
|
||||
#[serde(default)]
|
||||
pub min_device_vram_mb: Option<u64>,
|
||||
/// Neurons where this model should never be evicted.
|
||||
/// Neurons this model is allowed to run on. Empty = anywhere its
|
||||
/// device constraints are satisfied.
|
||||
///
|
||||
/// This is an *affinity* constraint — where the model may be placed.
|
||||
/// It says nothing about whether the model may be evicted once
|
||||
/// resident; that is [`ModelProfile::residency_priority`]. The two
|
||||
/// were a single field once, which made "run only here" and "never
|
||||
/// evict here" impossible to ask for separately.
|
||||
#[serde(default)]
|
||||
pub pinned_on: Vec<String>,
|
||||
/// How strongly this model holds its place when a node runs out of
|
||||
/// VRAM. A model may displace a resident one only if it ranks
|
||||
/// strictly higher, so equal-ranked models never evict each other.
|
||||
///
|
||||
/// Unset means [`DEFAULT_RESIDENCY_PRIORITY`], except for profiles
|
||||
/// carrying `pinned_on`, which default to
|
||||
/// [`PINNED_RESIDENCY_PRIORITY`] — before these were separate
|
||||
/// fields, `pinned_on` implied immunity from eviction, and a
|
||||
/// catalogue written against that meaning must not silently start
|
||||
/// allowing its flagship to be evicted.
|
||||
#[serde(default)]
|
||||
pub residency_priority: Option<u32>,
|
||||
/// Source scheme this profile's weights come from. When set, the
|
||||
/// router prefixes `id` with `scheme:` before forwarding the load
|
||||
/// request to neuron, ensuring the daemon fetches from the right
|
||||
@@ -57,6 +76,18 @@ fn default_min_devices() -> u32 {
|
||||
1
|
||||
}
|
||||
|
||||
/// Residency priority for a model that declares none. Deliberately not
|
||||
/// zero: an operator needs room to rank something *below* the ordinary
|
||||
/// case (a scratch or experimental model that should yield to anything)
|
||||
/// without editing every other entry.
|
||||
pub const DEFAULT_RESIDENCY_PRIORITY: u32 = 100;
|
||||
|
||||
/// Residency priority assumed for a profile that carries `pinned_on` but
|
||||
/// no explicit priority. High enough that nothing with a default
|
||||
/// priority can evict it, preserving the immunity `pinned_on` used to
|
||||
/// grant on its own.
|
||||
pub const PINNED_RESIDENCY_PRIORITY: u32 = 1000;
|
||||
|
||||
/// The full model catalogue.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct ModelCatalogue {
|
||||
@@ -95,11 +126,39 @@ impl ModelCatalogue {
|
||||
}
|
||||
}
|
||||
|
||||
/// Check if a model is pinned on a given neuron.
|
||||
pub fn is_pinned(&self, model_id: &str, neuron_name: &str) -> bool {
|
||||
self.models
|
||||
.iter()
|
||||
.any(|p| p.id == model_id && p.pinned_on.contains(&neuron_name.to_string()))
|
||||
/// How strongly `model_id` holds its place. Models absent from the
|
||||
/// catalogue rank at the default — a model can be resident on a
|
||||
/// neuron without a profile (loaded directly, or left over from an
|
||||
/// earlier catalogue), and treating those as unevictable would let
|
||||
/// an unlisted model wedge a node permanently.
|
||||
pub fn residency_priority(&self, model_id: &str) -> u32 {
|
||||
self.get(model_id)
|
||||
.map(|p| {
|
||||
p.residency_priority.unwrap_or({
|
||||
if p.pinned_on.is_empty() {
|
||||
DEFAULT_RESIDENCY_PRIORITY
|
||||
} else {
|
||||
PINNED_RESIDENCY_PRIORITY
|
||||
}
|
||||
})
|
||||
})
|
||||
.unwrap_or(DEFAULT_RESIDENCY_PRIORITY)
|
||||
}
|
||||
|
||||
/// May `incoming` take VRAM from `resident` when a node cannot hold
|
||||
/// both?
|
||||
///
|
||||
/// Strictly-greater, so equal-ranked models never displace one
|
||||
/// another. That matters more than it looks: it stops two models of
|
||||
/// the same class thrashing a node, each evicting the other on
|
||||
/// alternate requests, which is the failure mode a boolean
|
||||
/// pinned/unpinned split cannot express at all.
|
||||
///
|
||||
/// This governs only *whether* a displacement is permitted, never
|
||||
/// whether one is needed. A node with room for both evicts nothing,
|
||||
/// however the two rank.
|
||||
pub fn may_displace(&self, incoming_id: &str, resident_id: &str) -> bool {
|
||||
self.residency_priority(incoming_id) > self.residency_priority(resident_id)
|
||||
}
|
||||
|
||||
/// Find a profile by model id.
|
||||
@@ -167,6 +226,7 @@ mod tests {
|
||||
min_devices: 2,
|
||||
min_device_vram_mb: Some(24_000),
|
||||
pinned_on: vec![],
|
||||
residency_priority: None,
|
||||
source: None,
|
||||
limit: None,
|
||||
cost: None,
|
||||
@@ -212,6 +272,137 @@ mod tests {
|
||||
assert!(p.is_feasible_on("anywhere", &devices));
|
||||
}
|
||||
|
||||
/// A catalogue shaped like a real fleet: a flagship confined to one
|
||||
/// big node, a frontier model that may take that node from it, an
|
||||
/// image generator that may take the mid tier's node but never the
|
||||
/// flagship's, and the mid tier itself.
|
||||
fn tiered_catalogue() -> ModelCatalogue {
|
||||
toml::from_str(
|
||||
r#"
|
||||
[[models]]
|
||||
id = "flagship"
|
||||
harness = "candle"
|
||||
pinned_on = ["big-node"]
|
||||
residency_priority = 300
|
||||
|
||||
[[models]]
|
||||
id = "frontier"
|
||||
harness = "candle"
|
||||
residency_priority = 400
|
||||
|
||||
[[models]]
|
||||
id = "image"
|
||||
harness = "candle"
|
||||
residency_priority = 200
|
||||
|
||||
[[models]]
|
||||
id = "mid"
|
||||
harness = "candle"
|
||||
residency_priority = 100
|
||||
"#,
|
||||
)
|
||||
.expect("parse tiered catalogue")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_generation_displaces_the_mid_tier() {
|
||||
assert!(tiered_catalogue().may_displace("image", "mid"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_generation_never_displaces_the_flagship() {
|
||||
// The image generator's device constraints alone would let it
|
||||
// land on the flagship's node, so this is the case priority
|
||||
// exists to prevent -- not a hypothetical one.
|
||||
assert!(!tiered_catalogue().may_displace("image", "flagship"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_frontier_tier_displaces_the_flagship() {
|
||||
// The requirement a boolean pin cannot express: the flagship is
|
||||
// protected from one model and not from another.
|
||||
assert!(tiered_catalogue().may_displace("frontier", "flagship"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn equal_rank_never_displaces_either_way() {
|
||||
let cat = tiered_catalogue();
|
||||
assert!(!cat.may_displace("mid", "mid"));
|
||||
assert!(!cat.may_displace("flagship", "flagship"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_lower_tier_cannot_displace_a_higher_one() {
|
||||
assert!(!tiered_catalogue().may_displace("mid", "image"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pinned_on_alone_still_protects_a_catalogue_written_before_priorities() {
|
||||
// `pinned_on` used to mean "never evict here". A catalogue that
|
||||
// predates the split says nothing about priority, and must not
|
||||
// silently start allowing its flagship to be evicted.
|
||||
let cat: ModelCatalogue = toml::from_str(
|
||||
r#"
|
||||
[[models]]
|
||||
id = "flagship"
|
||||
harness = "candle"
|
||||
pinned_on = ["big-node"]
|
||||
|
||||
[[models]]
|
||||
id = "ordinary"
|
||||
harness = "candle"
|
||||
"#,
|
||||
)
|
||||
.expect("parse legacy catalogue");
|
||||
assert!(!cat.may_displace("ordinary", "flagship"));
|
||||
assert!(cat.may_displace("flagship", "ordinary"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unlisted_resident_is_displaceable_by_a_ranked_model() {
|
||||
// A model can be resident without a profile. Treating it as
|
||||
// unevictable would let an unlisted model wedge a node forever.
|
||||
let cat = tiered_catalogue();
|
||||
assert_eq!(
|
||||
cat.residency_priority("never-heard-of-it"),
|
||||
DEFAULT_RESIDENCY_PRIORITY
|
||||
);
|
||||
assert!(cat.may_displace("image", "never-heard-of-it"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn affinity_and_immunity_are_independently_expressible() {
|
||||
// The whole point of the split: confine a model to a node
|
||||
// without protecting it there, and protect one without
|
||||
// confining it anywhere.
|
||||
let cat: ModelCatalogue = toml::from_str(
|
||||
r#"
|
||||
[[models]]
|
||||
id = "confined-but-evictable"
|
||||
harness = "candle"
|
||||
pinned_on = ["big-node"]
|
||||
residency_priority = 50
|
||||
|
||||
[[models]]
|
||||
id = "roaming-but-protected"
|
||||
harness = "candle"
|
||||
residency_priority = 900
|
||||
"#,
|
||||
)
|
||||
.expect("parse catalogue");
|
||||
let devices = [device(0, 32_000)];
|
||||
|
||||
let confined = cat.get("confined-but-evictable").unwrap();
|
||||
assert!(confined.is_feasible_on("big-node", &devices));
|
||||
assert!(!confined.is_feasible_on("other-node", &devices));
|
||||
|
||||
let roaming = cat.get("roaming-but-protected").unwrap();
|
||||
assert!(roaming.is_feasible_on("other-node", &devices));
|
||||
|
||||
assert!(cat.may_displace("roaming-but-protected", "confined-but-evictable"));
|
||||
assert!(!cat.may_displace("confined-but-evictable", "roaming-but-protected"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_alias_returns_target_when_alias_present() {
|
||||
let mut cat = ModelCatalogue::default();
|
||||
|
||||
@@ -17,11 +17,23 @@ pub async fn eviction_loop(fleet: Arc<CortexState>) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Evict the least-recently-used model on a given node.
|
||||
/// Returns the model ID that was evicted, or None if nothing could be evicted.
|
||||
/// Evict the least-recently-used model on a given node that `incoming`
|
||||
/// is permitted to displace.
|
||||
///
|
||||
/// `incoming` is the model the caller is trying to make room for. It is
|
||||
/// load-bearing, not diagnostic: whether a displacement is allowed is a
|
||||
/// question about the *pair*, so an evictor that only knows the victim
|
||||
/// cannot answer it. Passing `None` means "no particular model" and
|
||||
/// permits displacing anything at or below the default priority, which
|
||||
/// is the right reading for maintenance-driven eviction rather than a
|
||||
/// cold-load making room for itself.
|
||||
///
|
||||
/// Returns the model ID that was evicted, or None when nothing on the
|
||||
/// node may be displaced.
|
||||
pub async fn evict_lru_on_node(
|
||||
fleet: &CortexState,
|
||||
node_name: &str,
|
||||
incoming: Option<&str>,
|
||||
) -> anyhow::Result<Option<String>> {
|
||||
let (neuron_endpoint, candidate) = {
|
||||
let nodes = fleet.nodes.read().await;
|
||||
@@ -29,13 +41,18 @@ pub async fn evict_lru_on_node(
|
||||
anyhow::bail!("node '{node_name}' not found");
|
||||
};
|
||||
|
||||
// Find the loaded model with the oldest last_accessed,
|
||||
// excluding models pinned on this neuron (from catalogue).
|
||||
// Oldest first, among the models this incoming model outranks.
|
||||
let candidate = node
|
||||
.models
|
||||
.values()
|
||||
.filter(|m| m.status == ModelStatus::Loaded)
|
||||
.filter(|m| !fleet.catalogue.is_pinned(&m.id, node_name))
|
||||
.filter(|m| match incoming {
|
||||
Some(inc) => fleet.catalogue.may_displace(inc, &m.id),
|
||||
None => {
|
||||
fleet.catalogue.residency_priority(&m.id)
|
||||
<= cortex_core::catalogue::DEFAULT_RESIDENCY_PRIORITY
|
||||
}
|
||||
})
|
||||
.min_by_key(|m| m.last_accessed)
|
||||
.map(|m| m.id.clone());
|
||||
|
||||
@@ -43,7 +60,11 @@ pub async fn evict_lru_on_node(
|
||||
};
|
||||
|
||||
let Some(model_id) = candidate else {
|
||||
tracing::info!(node = node_name, "no evictable models found");
|
||||
tracing::info!(
|
||||
node = node_name,
|
||||
incoming = incoming.unwrap_or("(none)"),
|
||||
"no displaceable models found"
|
||||
);
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
|
||||
@@ -254,7 +254,8 @@ pub async fn resolve(
|
||||
// free enough was ranked below one that can).
|
||||
if !fits_free {
|
||||
for _ in 0..3 {
|
||||
match crate::evictor::evict_lru_on_node(fleet, &node_name).await {
|
||||
match crate::evictor::evict_lru_on_node(fleet, &node_name, Some(&profile.id)).await
|
||||
{
|
||||
Ok(Some(evicted)) => {
|
||||
tracing::info!(
|
||||
model = %profile.id,
|
||||
@@ -298,10 +299,15 @@ async fn node_fits_free(fleet: &Arc<CortexState>, node_name: &str, profile: &Mod
|
||||
/// profile. Preference order (#203):
|
||||
/// 1. A neuron from `profile.pinned_on` that is healthy + feasible.
|
||||
/// 2. One whose live free VRAM already fits the profile.
|
||||
/// 3. One whose *evictable* (loaded, not catalogue-pinned) models
|
||||
/// plus free VRAM could fit it after a cold-swap eviction.
|
||||
/// 3. One whose *displaceable* models — loaded, and outranked by this
|
||||
/// profile's residency priority — plus free VRAM could fit it
|
||||
/// after a cold-swap eviction.
|
||||
/// 4. Any healthy + feasible neuron, stable by name.
|
||||
///
|
||||
/// Free-fit outranks evict-fit, so a model never displaces anything
|
||||
/// while a node with room exists. Priority decides who may be
|
||||
/// displaced, never whether a displacement is needed.
|
||||
///
|
||||
/// Returns `(name, endpoint, fits_free)` — `fits_free = false` tells
|
||||
/// the caller a cold-swap eviction is needed before loading.
|
||||
async fn pick_feasible_neuron(
|
||||
@@ -329,14 +335,16 @@ async fn pick_feasible_neuron(
|
||||
.unwrap_or(0);
|
||||
let need = profile.vram_mb.unwrap_or(0);
|
||||
let fits_free = max_free >= need;
|
||||
// Evictable estimate: free + the VRAM of loaded models that the
|
||||
// catalogue does not pin to this node.
|
||||
// Evictable estimate: free + the VRAM of loaded models this
|
||||
// profile is permitted to displace. The predicate must be the
|
||||
// same one the evictor applies, or a node ranks as able to make
|
||||
// room and then declines to make any.
|
||||
let evictable: u64 = node
|
||||
.models
|
||||
.values()
|
||||
.filter(|m| {
|
||||
matches!(m.status, cortex_core::node::ModelStatus::Loaded)
|
||||
&& !fleet.catalogue.is_pinned(&m.id, &node.name)
|
||||
&& fleet.catalogue.may_displace(&profile.id, &m.id)
|
||||
})
|
||||
// neuron reports vram_used_mb: null today; fall back to the
|
||||
// catalogue's declared footprint so the evictable estimate
|
||||
@@ -632,6 +640,7 @@ mod tests {
|
||||
min_devices: 1,
|
||||
min_device_vram_mb: None,
|
||||
pinned_on: vec![],
|
||||
residency_priority: None,
|
||||
source: source.map(String::from),
|
||||
limit: None,
|
||||
cost: None,
|
||||
|
||||
@@ -56,6 +56,74 @@ async fn spawn_eviction_mock() -> (String, Arc<tokio::sync::Mutex<Vec<String>>>)
|
||||
(base_url, unloaded)
|
||||
}
|
||||
|
||||
/// A fleet whose catalogue ranks models, so the displacement rules are
|
||||
/// exercised rather than defaulted. Writes the catalogue to a temp file
|
||||
/// because `CortexState` loads it from a path.
|
||||
fn make_fleet_with_catalogue(
|
||||
endpoint: &str,
|
||||
catalogue_toml: &str,
|
||||
tag: &str,
|
||||
) -> (Arc<CortexState>, std::path::PathBuf) {
|
||||
let path = std::env::temp_dir().join(format!("cortex-evict-catalogue-{tag}.toml"));
|
||||
std::fs::write(&path, catalogue_toml).expect("write test catalogue");
|
||||
let config = GatewayConfig {
|
||||
gateway: GatewaySettings {
|
||||
listen: "127.0.0.1:0".into(),
|
||||
metrics_listen: "127.0.0.1:0".into(),
|
||||
},
|
||||
eviction: EvictionSettings {
|
||||
strategy: EvictionStrategy::Lru,
|
||||
defrag_after_cycles: 0,
|
||||
},
|
||||
neurons: vec![NeuronEndpoint {
|
||||
name: "gpu-node".into(),
|
||||
endpoint: endpoint.to_string(),
|
||||
}],
|
||||
models_config: path.to_string_lossy().into_owned(),
|
||||
entitlements: Default::default(),
|
||||
upstream: Default::default(),
|
||||
};
|
||||
(Arc::new(CortexState::from_config(&config)), path)
|
||||
}
|
||||
|
||||
fn loaded(id: &str, age_secs: i64) -> ModelEntry {
|
||||
ModelEntry {
|
||||
id: id.into(),
|
||||
status: ModelStatus::Loaded,
|
||||
last_accessed: Some(Utc::now() - chrono::Duration::seconds(age_secs)),
|
||||
vram_estimate_mb: Some(8000),
|
||||
capabilities: Vec::new(),
|
||||
tool_call: false,
|
||||
reasoning: false,
|
||||
limit: None,
|
||||
servable: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The fleet policy under test: image generation outranks the mid tier
|
||||
/// but not the flagship; the frontier tier outranks the flagship.
|
||||
const TIERED: &str = r#"
|
||||
[[models]]
|
||||
id = "flagship"
|
||||
harness = "candle"
|
||||
residency_priority = 300
|
||||
|
||||
[[models]]
|
||||
id = "frontier"
|
||||
harness = "candle"
|
||||
residency_priority = 400
|
||||
|
||||
[[models]]
|
||||
id = "image"
|
||||
harness = "candle"
|
||||
residency_priority = 200
|
||||
|
||||
[[models]]
|
||||
id = "mid"
|
||||
harness = "candle"
|
||||
residency_priority = 100
|
||||
"#;
|
||||
|
||||
fn make_fleet(endpoint: &str, defrag_after: u32) -> Arc<CortexState> {
|
||||
let config = GatewayConfig {
|
||||
gateway: GatewaySettings {
|
||||
@@ -116,7 +184,7 @@ async fn test_evict_lru_model() {
|
||||
);
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node")
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", None)
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
@@ -149,7 +217,7 @@ async fn test_eviction_nothing_to_evict() {
|
||||
nodes.get_mut("gpu-node").unwrap().healthy = true;
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node")
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", None)
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
@@ -184,7 +252,7 @@ async fn test_eviction_increments_lifecycle_cycles() {
|
||||
);
|
||||
}
|
||||
|
||||
cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node")
|
||||
cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", None)
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
@@ -231,3 +299,103 @@ async fn test_last_accessed_updated_on_request() {
|
||||
.is_some()
|
||||
);
|
||||
}
|
||||
|
||||
/// Image generation must take the mid tier's node when it needs it —
|
||||
/// this is existing fleet behaviour and the change must preserve it.
|
||||
#[tokio::test]
|
||||
async fn image_generation_evicts_the_mid_tier() {
|
||||
let (mock_url, unloaded) = spawn_eviction_mock().await;
|
||||
let (fleet, path) = make_fleet_with_catalogue(&mock_url, TIERED, "image-takes-mid");
|
||||
|
||||
{
|
||||
let mut nodes = fleet.nodes.write().await;
|
||||
let node = nodes.get_mut("gpu-node").unwrap();
|
||||
node.healthy = true;
|
||||
node.models.insert("mid".into(), loaded("mid", 60));
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", Some("image"))
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
assert_eq!(evicted, Some("mid".to_string()));
|
||||
assert_eq!(unloaded.lock().await.as_slice(), ["mid"]);
|
||||
std::fs::remove_file(path).ok();
|
||||
}
|
||||
|
||||
/// Image generation must never take the flagship's node. Its device
|
||||
/// constraints alone would let it land there, so nothing but priority
|
||||
/// stops this.
|
||||
#[tokio::test]
|
||||
async fn image_generation_cannot_evict_the_flagship() {
|
||||
let (mock_url, unloaded) = spawn_eviction_mock().await;
|
||||
let (fleet, path) = make_fleet_with_catalogue(&mock_url, TIERED, "image-spares-flagship");
|
||||
|
||||
{
|
||||
let mut nodes = fleet.nodes.write().await;
|
||||
let node = nodes.get_mut("gpu-node").unwrap();
|
||||
node.healthy = true;
|
||||
node.models
|
||||
.insert("flagship".into(), loaded("flagship", 9999));
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", Some("image"))
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
assert_eq!(
|
||||
evicted, None,
|
||||
"the flagship outranks image generation, however stale it is"
|
||||
);
|
||||
assert!(unloaded.lock().await.is_empty());
|
||||
std::fs::remove_file(path).ok();
|
||||
}
|
||||
|
||||
/// The frontier tier may cold-swap the flagship off its node.
|
||||
#[tokio::test]
|
||||
async fn the_frontier_tier_evicts_the_flagship() {
|
||||
let (mock_url, unloaded) = spawn_eviction_mock().await;
|
||||
let (fleet, path) = make_fleet_with_catalogue(&mock_url, TIERED, "frontier-takes-flagship");
|
||||
|
||||
{
|
||||
let mut nodes = fleet.nodes.write().await;
|
||||
let node = nodes.get_mut("gpu-node").unwrap();
|
||||
node.healthy = true;
|
||||
node.models
|
||||
.insert("flagship".into(), loaded("flagship", 10));
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", Some("frontier"))
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
assert_eq!(evicted, Some("flagship".to_string()));
|
||||
assert_eq!(unloaded.lock().await.as_slice(), ["flagship"]);
|
||||
std::fs::remove_file(path).ok();
|
||||
}
|
||||
|
||||
/// LRU still decides *which* victim, but only among the models the
|
||||
/// incoming one outranks. Here the flagship is by far the stalest, so a
|
||||
/// purely age-ordered evictor would take it.
|
||||
#[tokio::test]
|
||||
async fn lru_picks_the_oldest_displaceable_model_not_the_oldest_model() {
|
||||
let (mock_url, unloaded) = spawn_eviction_mock().await;
|
||||
let (fleet, path) = make_fleet_with_catalogue(&mock_url, TIERED, "lru-within-rank");
|
||||
|
||||
{
|
||||
let mut nodes = fleet.nodes.write().await;
|
||||
let node = nodes.get_mut("gpu-node").unwrap();
|
||||
node.healthy = true;
|
||||
node.models
|
||||
.insert("flagship".into(), loaded("flagship", 9999));
|
||||
node.models.insert("mid".into(), loaded("mid", 60));
|
||||
}
|
||||
|
||||
let evicted = cortex_gateway::evictor::evict_lru_on_node(&fleet, "gpu-node", Some("image"))
|
||||
.await
|
||||
.expect("eviction should succeed");
|
||||
|
||||
assert_eq!(evicted, Some("mid".to_string()));
|
||||
assert_eq!(unloaded.lock().await.as_slice(), ["mid"]);
|
||||
std::fs::remove_file(path).ok();
|
||||
}
|
||||
|
||||
@@ -19,8 +19,30 @@
|
||||
# min_device_vram_mb - each device must meet this VRAM floor for the
|
||||
# neuron to be considered "feasible".
|
||||
# pinned_on - optional whitelist of neuron names. Non-empty
|
||||
# narrows feasibility to just those neurons and
|
||||
# protects the model from LRU eviction there.
|
||||
# narrows feasibility to just those neurons.
|
||||
# Affinity only: it says where the model may run,
|
||||
# not whether it may be evicted once there.
|
||||
# residency_priority - how strongly the model holds its place when a
|
||||
# node runs out of VRAM. A model may displace a
|
||||
# resident one only if it ranks STRICTLY higher,
|
||||
# so equal-ranked models never evict each other.
|
||||
# Default 100; a profile with pinned_on and no
|
||||
# explicit priority defaults to 1000, preserving
|
||||
# the eviction immunity pinned_on used to imply
|
||||
# on its own.
|
||||
#
|
||||
# This is what lets a fleet say "the image model
|
||||
# may take the mid tier's node, but never the
|
||||
# flagship's, while the frontier tier may take
|
||||
# the flagship's" — three rules that a single
|
||||
# pinned/unpinned flag cannot express, because
|
||||
# it can only protect a model from everything or
|
||||
# from nothing.
|
||||
#
|
||||
# Priority decides who MAY be displaced, never
|
||||
# whether a displacement is NEEDED: a node with
|
||||
# room for both evicts nothing, however the two
|
||||
# rank.
|
||||
# source - optional source scheme ("huggingface", "helexa",
|
||||
# operator mirror tag). When set, cortex forwards
|
||||
# the load to neuron as `scheme:id` so the daemon
|
||||
@@ -49,6 +71,9 @@ vram_mb = 54000
|
||||
min_devices = 2
|
||||
min_device_vram_mb = 24000
|
||||
pinned_on = ["your-multi-gpu-neuron"]
|
||||
# Ranked above the everyday models so a smaller one cannot cold-swap the
|
||||
# flagship out, but below anything you want to be able to displace it.
|
||||
residency_priority = 300
|
||||
# Token budget: context wall, compaction trigger (input headroom), max output.
|
||||
limit.context = 32768
|
||||
limit.input = 28672
|
||||
|
||||
Reference in New Issue
Block a user