mirror of
https://github.com/openai/codex.git
synced 2026-10-31 19:28:49 +00:00
Add trace-safe diagnostics for Noise relay request progress (#58025)
## Why WebSocket heartbeat progress only establishes relay reachability, not executor receipt or completion. A blocked write or full application queue can also leave a Pong unread, so diagnosing stalled requests requires separate transport and application progress signals. ## What changed - Trace request dequeue and completed writes under the request's trace context, and emit an executor receipt span immediately after decoding, before queueing or execution completes. - Record RPC enqueue wait time and queue occupancy. - Add bounded heartbeat events and a transport closure summary with fixed exit categories, queue depths, message counters, and read/write timing. Distinguish write timeouts from missing Pongs and report application delivery backpressure separately. - Keep request methods, IDs, payloads, raw errors, and peer-provided text out of the new trace-safe transport diagnostics. ## Testing Add coverage for blocked data and keepalive writes, missing Pongs, saturated application queues, read errors, and responsive relays without application responses. Verify trace correlation, immediate receipt span export, bounded Pong telemetry, and exclusion of payloads and raw error text. GitOrigin-RevId: b58b2f747c4050dcde8efb408c35ecadd3561b5f
This commit is contained in:
committed by
copyberry
parent
a29c7b97e8
commit
a3697361df
@@ -28,6 +28,7 @@ use tokio::sync::mpsc;
|
||||
use tokio::sync::oneshot;
|
||||
use tokio::sync::watch;
|
||||
use tokio::task::JoinHandle;
|
||||
use tokio::time::Instant;
|
||||
use tokio::time::timeout;
|
||||
|
||||
use crate::client_telemetry::record_client_request;
|
||||
@@ -740,6 +741,7 @@ impl RpcClient {
|
||||
response_rx: oneshot::Receiver<RpcCompletion>,
|
||||
call_timeout: RpcCallTimeout,
|
||||
) -> Result<Value, RpcCallError> {
|
||||
let enqueue_started = Instant::now();
|
||||
if self
|
||||
.write_tx
|
||||
.send(JSONRPCMessage::Request(JSONRPCRequest {
|
||||
@@ -759,6 +761,9 @@ impl RpcClient {
|
||||
tracing::Level::INFO,
|
||||
event.name = "codex.exec_server.request_enqueued",
|
||||
rpc.method = method,
|
||||
enqueue_wait_ms = enqueue_started.elapsed().as_millis() as u64,
|
||||
outgoing_queue_depth = self.write_tx.max_capacity() - self.write_tx.capacity(),
|
||||
outgoing_queue_capacity = self.write_tx.max_capacity(),
|
||||
);
|
||||
// Do not race in-flight requests directly against the transport-close
|
||||
// watch value. The connection reader receives JSON-RPC messages and
|
||||
|
||||
Reference in New Issue
Block a user