rob thijssen
526b662c5e
All checks were successful
CI / Format (push) Successful in 44s
CI / CUDA type-check (push) Successful in 1m31s
CI / Clippy (push) Successful in 2m13s
CI / Test (push) Successful in 5m9s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Defense-in-depth for the agent0 NoFeasibleNeuron storm (root cause fixed in neuron). Two cortex resilience gaps this incident exposed: 1. Brittle health flip: the poller marked a node unhealthy on a SINGLE missed /models poll, instantly yanking the node and all its models from routing. A busy neuron briefly slow to answer shouldn't be declared dead. Now debounced: NodeState.consecutive_poll_failures must reach POLL_FAILURE_THRESHOLD (3) before the node flips unhealthy (~20s at the 10s poll interval); any successful poll resets it. A never-healthy node stays unhealthy (the counter only protects an already-healthy node from blips). 2. Transient surfaced as permanent: when a catalogued model's only feasible neuron is momentarily unhealthy, the router returned 404 NoFeasibleNeuron — which litellm/clients treat as non-retryable, so agent0 hard-failed. pick_feasible_neuron now distinguishes "a feasible node exists but is unhealthy right now" → new RouteError::FeasibleNodeUnhealthy (503 + Retry-After: 3, retryable) from "no node could ever satisfy the topology" → 404 NoFeasibleNeuron (permanent). Mirrors the beast case exactly: healthy 1-GPU nodes + an unhealthy 2-GPU node → retry, don't fail. Tests: poller test updated to assert debounce (1 miss keeps healthy, 3 flip); new feasibility_routing tests cover transient-503 vs permanent-404. Local fmt/clippy/test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
15 KiB
15 KiB