Retry executor discovery V2 failures with backoff (#62679)

## Why

Discovery could retry transport failures on successive lookups while
caching server errors until invalidation. That could repeat requests
against an unhealthy executor or leave a session without an inventory
after the server recovered.

## What changed

Cache failed discovery loads per context and return the cached error
during cooldown. The first lookup after cooldown retries, with delays
doubling from 1 second up to 30 seconds on consecutive failures.
Successful inventories are reused, canceled loads add no cooldown, and
explicit refresh clears the cooldown for the next lookup without
interrupting in-flight readers.

Compare discovery snapshots to detect inventory changes so recovered
plugins and MCP capabilities can become available without an explicit
refresh.

## Testing

Cover backoff timing, shared results for concurrent callers,
cancellation, and refresh bypassing cooldown. Exercise recovery after
failed prewarming and verify that MCP inventory updates after recovery.

GitOrigin-RevId: 6de34bb8957a0b24ee874e2447157bd3af562430
This commit is contained in:
Anthony Tafoya
2026-11-20 06:05:04 +00:00
committed by copyberry
parent ea8ccc1d20
commit 2fb728da7d
7 changed files with 529 additions and 264 deletions
@@ -1,4 +1,4 @@
//! Core owns a thread discovery cache shared across turns. Explicit refresh dirties every entry.
//! Core owns a thread discovery cache shared across turns. Explicit refresh evicts every entry.
//! In-flight readers may keep their inventories; invalidation applies to the next lookup.
//! Prewarming bounds discovery before readiness without publishing MCP state or failing startup.
@@ -166,7 +166,6 @@ impl Session {
.zip(&snapshots)
.any(|(previous, current)| !previous.same_discovery(current))
});
let inventory_changed = cache.take_recovered_discovery() || inventory_changed;
(Some(snapshots), inventory_changed)
}
}