Reduce input-to-display latency #1

New Issue

grenade · 2026-05-07T07:51:13Z

grenade commented

2026-05-07 07:51:13 +00:00

Keystrokes and mouse interactions have variable, inconsistent latency between input and the result appearing on screen. The system is functional but sluggish.

Current bottlenecks

Full framebuffer blit on every update — the proxy sends the entire 640×480×4 = 1.2MB RGBA framebuffer over WebSocket after each FramebufferDirty event, even when only a small region changed. The e-RIC decoder already knows which rectangles were updated.
Proxy chain depth — browser → WS → nginx reverse proxy → ericrfb-proxy → OmniView TCP → KVM → target → video capture → e-RIC → proxy decode → RGBA → WS → canvas. Each hop adds latency.
Blocking session pump — the OmniView session runs on a spawn_blocking thread. Input events are drained via try_recv between process_one() calls, so input can only be sent when the decoder yields between server messages.

Proposed improvements

Send dirty rectangles only (high impact)

The handle_fb_update() loop already iterates per-rectangle. Instead of converting the full framebuffer to RGBA after all rects are processed, send each dirty rect as an individual TAG_BLIT message with only the changed pixels. For a typical incremental update (small cursor movement, text appearing), this reduces the WS payload from 1.2MB to a few KB.

Decouple input from decode (medium impact)

Split the blocking pump into two threads: one for reading OmniView messages (decode), one for writing input events. Currently input is interleaved with decoding via try_recv, which means a slow decode (large Tight rectangle) blocks input delivery.

Request incremental updates immediately (low-hanging fruit)

After sending the initial full-frame request, send incremental FramebufferUpdateRequest as soon as the previous update is processed, rather than waiting for the blit to be sent over WebSocket. This keeps the OmniView pipeline full.

Frontend: use `requestAnimationFrame` for blits (minor)

Batch incoming WS blit messages and apply them in a single requestAnimationFrame callback to avoid layout thrashing when multiple small rects arrive in quick succession.

Keystrokes and mouse interactions have variable, inconsistent latency between input and the result appearing on screen. The system is functional but sluggish. ## Current bottlenecks 1. **Full framebuffer blit on every update** — the proxy sends the entire 640×480×4 = 1.2MB RGBA framebuffer over WebSocket after each `FramebufferDirty` event, even when only a small region changed. The e-RIC decoder already knows which rectangles were updated. 2. **Proxy chain depth** — browser → WS → nginx reverse proxy → ericrfb-proxy → OmniView TCP → KVM → target → video capture → e-RIC → proxy decode → RGBA → WS → canvas. Each hop adds latency. 3. **Blocking session pump** — the OmniView session runs on a `spawn_blocking` thread. Input events are drained via `try_recv` between `process_one()` calls, so input can only be sent when the decoder yields between server messages. ## Proposed improvements ### Send dirty rectangles only (high impact) The `handle_fb_update()` loop already iterates per-rectangle. Instead of converting the full framebuffer to RGBA after all rects are processed, send each dirty rect as an individual `TAG_BLIT` message with only the changed pixels. For a typical incremental update (small cursor movement, text appearing), this reduces the WS payload from 1.2MB to a few KB. ### Decouple input from decode (medium impact) Split the blocking pump into two threads: one for reading OmniView messages (decode), one for writing input events. Currently input is interleaved with decoding via `try_recv`, which means a slow decode (large Tight rectangle) blocks input delivery. ### Request incremental updates immediately (low-hanging fruit) After sending the initial full-frame request, send incremental `FramebufferUpdateRequest` as soon as the previous update is processed, rather than waiting for the blit to be sent over WebSocket. This keeps the OmniView pipeline full. ### Frontend: use `requestAnimationFrame` for blits (minor) Batch incoming WS blit messages and apply them in a single `requestAnimationFrame` callback to avoid layout thrashing when multiple small rects arrive in quick succession.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: grenade/blekin#1

Reduce input-to-display latency #1