• Joined on 2026-02-17
grenade created branch fix/retire-qwen36-27b in helexa/helexa 2026-08-27 19:34:49 +00:00
grenade pushed to main at helexa/helexa 2026-08-27 10:24:07 +00:00
2d1a4ebb2f Merge branch 'fix/nginx-sync-must-not-block-cortex-deploy'
d87327c3a2 fix(deploy): the nginx sync must be its own job, not a step that gates the fleet
a17b3acb70 fix(deploy): the nginx sync must not hold the cortex binary hostage
Compare 3 commits »
grenade pushed to fix/nginx-sync-must-not-block-cortex-deploy at helexa/helexa 2026-08-27 10:02:16 +00:00
d87327c3a2 fix(deploy): the nginx sync must be its own job, not a step that gates the fleet
grenade commented on issue helexa/helexa#267 2026-08-27 09:38:36 +00:00
Streams go silent for the whole prefill, so clients drop them as dead — and an abandoned request keeps the pool lock

Still open on main as of c6c6fc8a. Re-checked while sweeping issues, with the trim sites enumerated rather than counted — the original figure of "twelve" was close but the breakdown is what…

grenade pushed to fix/nginx-sync-must-not-block-cortex-deploy at helexa/helexa 2026-08-27 09:36:54 +00:00
a17b3acb70 fix(deploy): the nginx sync must not hold the cortex binary hostage
grenade created branch fix/nginx-sync-must-not-block-cortex-deploy in helexa/helexa 2026-08-27 09:36:53 +00:00
grenade commented on issue helexa/helexa#265 2026-08-27 09:35:49 +00:00
a wedged NCCL collective hangs neuron indefinitely — exit non-zero and let systemd restart it

Implemented in c9f7383d (+ 8b0ed913 for the tests), merged. All four requirements are in the code:

  • abort() runs on its own named thread (tp-watchdog-abort), so a blocking abort cannot…
grenade closed issue helexa/helexa#256 2026-08-27 09:35:05 +00:00
Graceful shutdown never completes under load — queued admission waiters aren't cancelled, so every restart SIGKILLs
grenade commented on issue helexa/helexa#256 2026-08-27 09:35:02 +00:00
Graceful shutdown never completes under load — queued admission waiters aren't cancelled, so every restart SIGKILLs

Fixed in 3d28e016, merged, and now validated against nine days of production restarts rather than just tests.

The fix

Two waits had to become abandonable, because closing either alone…

grenade opened issue helexa/helexa#303 2026-08-27 09:34:48 +00:00
The TP watchdog can never fire before systemd gives up, so a wedged collective during shutdown always ends in SIGABRT
grenade pushed to main at helexa/helexa 2026-08-27 09:33:53 +00:00
c6c6fc8a50 Merge branch 'feat/nginx-search-alias-and-sync'
432f8009ce feat(nginx): accept POST on search, and keep queries out of the logs
98ba45cbb4 feat(nginx): serve SearXNG's own path, and deploy the vhost from CI
Compare 3 commits »
grenade closed issue helexa/helexa#253 2026-08-27 09:29:46 +00:00
OpenAI-native tool callers 422 on turn two: "content": null on assistant tool-call turns is rejected
grenade commented on issue helexa/helexa#253 2026-08-27 09:29:43 +00:00
OpenAI-native tool callers 422 on turn two: "content": null on assistant tool-call turns is rejected

Fixed in 3c1decec, on main since 2026-08-15. The issue was left open by oversight — closing it now on a re-read of what the commit trail actually landed.

MessageContent::Null models the…

grenade closed issue helexa/helexa#270 2026-08-27 09:27:46 +00:00
Prefix-cache reuse looks low on long agentic contexts — 2068 tokens reused against a 42K prompt
grenade commented on issue helexa/helexa#270 2026-08-27 09:27:42 +00:00
Prefix-cache reuse looks low on long agentic contexts — 2068 tokens reused against a 42K prompt

Diagnosed and fixed. This was the first sighting of what #292 later pinned down: the cache was working correctly and had nowhere to put the snapshot.

prefix_cache.budget_mb was at its 1024…

grenade closed issue helexa/helexa#295 2026-08-27 09:26:51 +00:00
A restart collapses the advertised context window to a sixth, until real traffic warms the prefill EMA
grenade commented on issue helexa/helexa#295 2026-08-27 09:26:49 +00:00
A restart collapses the advertised context window to a sixth, until real traffic warms the prefill EMA

Options 2–4 carried forward to #302, so the EMA-semantics question does not get lost behind a closed issue. Closing this one on the cold-start collapse, which is the defect it was filed for.

grenade opened issue helexa/helexa#302 2026-08-27 09:26:44 +00:00
The prefill EMA measures total prompt tokens over uncached time, and is named as a throughput rate
grenade commented on issue helexa/helexa#295 2026-08-27 09:26:22 +00:00
A restart collapses the advertised context window to a sixth, until real traffic warms the prefill EMA

Fixed in 176ce473, merged.

Option 1 from the list. MIN_PREFILL_SAMPLE_TOKENS = 512: record() drops any sample whose prompt is smaller, so a request too short to measure throughput cannot…

grenade pushed to main at helexa/helexa 2026-08-27 09:26:08 +00:00
192030dff5 Merge branch 'fix/wire-surface-in-neuron-spans'
24b8de7f82 Merge branch 'fix/295-ignore-tiny-prefill-samples'
522682282c fix(observability): record which API surface a request arrived on
176ce47349 fix(context): ignore prefill samples too small to measure throughput
Compare 4 commits »