Commit Graph
2 Commits
Author SHA1 Message Date
Alexi Christakis ddabe594e6 Bound relay connection attempts and handle pongs during blocked writes (#51502)
## Why

A stalled rendezvous WebSocket upgrade can prevent reconnects, while short-lived connections reset reconnect backoff too early. Blocked relay writes also prevent received pongs from clearing their heartbeat deadline.

## What changed

- Bound rendezvous WebSocket connection attempts with `DEFAULT_REMOTE_EXEC_SERVER_CONNECT_TIMEOUT`.
- Retain exponential reconnect backoff across short-lived connections, resetting it only after a connection lasts at least 120 seconds.
- Process pongs during blocked data writes while preserving the in-flight send and its independent write timeout. Prefer queued pongs over an expired heartbeat deadline.
- Abort the relay writer when its owning connection task is dropped.

## Testing

Add coverage for pongs arriving during blocked writes, missing and queued pongs, independent write timeouts, and writer cancellation. Extend reconnect tests to cover stalled WebSocket upgrades without re-registration and backoff reset after a stable session.

GitOrigin-RevId: 0efa350c2b4ada9b89e23df8909063fbb69f33dc
2026-10-06 23:07:25 +00:00
Anton Panasenko ef8cfe5e96 Retry registry authentication outages and jitter executor reconnects (#50465)
## Why

Remote executors need to recover from confirmed authentication-service outages and spread reconnect attempts after a shared outage. Registration retries must remain limited to confirmed failures before a write, since replaying an ambiguous request can replace a newer registration.

## What changed

- Retry registry responses with HTTP 502 and `authentication_service_unavailable`, alongside the existing HTTP 503 `registration_conflict` retries.
- Accept structured registry errors in `detail`, keeping the canonical `error` authoritative when both are present.
- Randomize reconnect delays between half and all of the current exponential backoff, retaining the 30-second cap.

## Testing

Add coverage for reconnect jitter bounds, authentication-outage retry cancellation, error-envelope precedence, and preservation of Noise identity and initialized sessions across retries. Make the HTTP-header helper cleanup test wait for the helper PID before cancellation and verify process termination when the provider is dropped.

GitOrigin-RevId: 35fdb57732d17b0bbfeca8a4dc557a0988e29a31
2026-10-02 23:54:32 +00:00