Pace the intents backfill only when it reads someone else's endpoint #35

Closed
opened 2026-10-03 20:32:47 +00:00 by grenade · 0 comments
Owner

Why

The intents backfill sleeps BACKFILL_PACE (1 s) between blocks. That was set when every read went to FastNEAR's free RPC (about 2 req/s sustained) or its archive. The live endpoint is now our own node (https://near.internal), and since #27's enabling commit start_height is the oldest block that node holds. So the whole backfill is served by a host we run, and the sleep only slows it down: about 22,000 blocks at 1/s is around 6 hours. The tip side stalls for each descending batch meanwhile, so live pushes (LIVE_WITHIN_BLOCKS) are missed during the backfill.

What

  • Pace by who serves the read, not by direction. A height at or above the live endpoint's earliest_block_height goes to rpc_urls; anything older goes to archive_urls. Sleep only when the endpoints that will answer are not ours.
  • "Ours" is decided from the URL host:
    • a .internal name (split-horizon, mesh-only by house convention)
    • loopback, or a private/link-local IP
    • anything else is someone else's and keeps the 1 s pace
  • The test covers the classifier with each shape, including that a public name that merely contains "internal" is not ours.
## Why The intents backfill sleeps `BACKFILL_PACE` (1 s) between blocks. That was set when every read went to FastNEAR's free RPC (about 2 req/s sustained) or its archive. The live endpoint is now our own node (`https://near.internal`), and since #27's enabling commit `start_height` is the oldest block that node holds. So the whole backfill is served by a host we run, and the sleep only slows it down: about 22,000 blocks at 1/s is around 6 hours. The tip side stalls for each descending batch meanwhile, so live pushes (`LIVE_WITHIN_BLOCKS`) are missed during the backfill. ## What - **Pace by who serves the read, not by direction.** A height at or above the live endpoint's `earliest_block_height` goes to `rpc_urls`; anything older goes to `archive_urls`. Sleep only when the endpoints that will answer are not ours. - **"Ours"** is decided from the URL host: - a `.internal` name (split-horizon, mesh-only by house convention) - loopback, or a private/link-local IP - anything else is someone else's and keeps the 1 s pace - The test covers the classifier with each shape, including that a public name that merely contains "internal" is not ours.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/observer#35