• Joined on 2026-02-17
grenade pushed to main at helexa/helexa 2026-08-15 20:10:30 +00:00
fec5b33195 Merge branch 'fix/failure-observability'
5ce4d8481e fix(observability): failures say why, in the journal and on the wire
Compare 2 commits »
grenade pushed to main at grenade/moments 2026-08-15 17:31:49 +00:00
12e9d4097f feat: count private-repo work in the activity summary
grenade closed issue grenade/moments#7 2026-08-15 17:31:48 +00:00
activity summary should count private-repo work, not omit it
grenade opened issue grenade/moments#7 2026-08-15 17:31:34 +00:00
activity summary should count private-repo work, not omit it
grenade pushed to main at grenade/moments 2026-08-15 16:41:44 +00:00
815bfa7deb fix: reconcile repo visibility instead of trusting the ingest-time flag
grenade closed issue grenade/moments#6 2026-08-15 16:41:44 +00:00
repo visibility changes are never noticed: private repos keep serving their history
grenade opened issue grenade/moments#6 2026-08-15 16:41:28 +00:00
repo visibility changes are never noticed: private repos keep serving their history
grenade opened issue helexa/helexa#258 2026-08-15 16:32:18 +00:00
A failing stream tells its client nothing — the body just stops, and cortex records a 200
grenade pushed to fix/failure-observability at helexa/helexa 2026-08-15 16:29:08 +00:00
5ce4d8481e fix(observability): failures say why, in the journal and on the wire
grenade created branch fix/failure-observability in helexa/helexa 2026-08-15 16:29:08 +00:00
grenade commented on issue helexa/helexa#257 2026-08-15 15:54:11 +00:00
VRAM backstop rejects before admission, killing long agentic sessions — admission counts requests, capacity is KV bytes

Auto-recovery from the fault completed on its own — status: loaded, serving again, no intervention. Worth noting because it means the device fault is survivable, not fatal to the node.

The…

grenade commented on issue helexa/helexa#257 2026-08-15 15:50:46 +00:00
VRAM backstop rejects before admission, killing long agentic sessions — admission counts requests, capacity is KV bytes

Device fault under the workload this change admits — do not treat #257 as validated

Roughly an hour after adebbd42 deployed, on beast, during a sustained long-context dsh session:

15:41…
grenade pushed to main at helexa/helexa 2026-08-15 15:37:55 +00:00
c62cfc6a50 fix(servability): a model with KV budget is servable, however saturated
grenade pushed to main at helexa/helexa 2026-08-15 13:46:56 +00:00
adebbd429a Merge branch 'fix/vram-aware-admission'
8f7edd7d00 fix(admission): borrow the context-limit config in the TP non-streaming path
4fd23c5322 fix(admission): reserve the prefix cache's VRAM out of the KV budget
eca4251054 fix(admission): reserve KV budget at admission instead of rejecting
Compare 4 commits »
grenade pushed to fix/vram-aware-admission at helexa/helexa 2026-08-15 13:24:34 +00:00
8f7edd7d00 fix(admission): borrow the context-limit config in the TP non-streaming path
grenade commented on issue helexa/helexa#257 2026-08-15 13:05:43 +00:00
VRAM backstop rejects before admission, killing long agentic sessions — admission counts requests, capacity is KV bytes

A caveat on the fix, found by watching beast at idle after the session that prompted this issue. Recording it because it is what live validation has to check.

**Driver-visible free VRAM drifts…

grenade pushed to fix/vram-aware-admission at helexa/helexa 2026-08-15 13:01:29 +00:00
4fd23c5322 fix(admission): reserve the prefix cache's VRAM out of the KV budget
grenade commented on issue helexa/helexa#255 2026-08-15 12:53:47 +00:00
Concurrent cold-loads spawn duplicate TP worker sets that OOM each other — self-sustaining under load

The deploy pipeline caught this too — run 1132 (main, 9e1a0fa) reports deploy-neurons (blackwell, beast.hanzalova.internal, 900, 131072): failure while every other target succeeded…

grenade pushed to fix/vram-aware-admission at helexa/helexa 2026-08-15 12:52:45 +00:00
eca4251054 fix(admission): reserve KV budget at admission instead of rejecting
grenade created branch fix/vram-aware-admission in helexa/helexa 2026-08-15 12:52:45 +00:00