Prefill panics above ~128k tokens: routing ids become garbage (qwen4exp, Vulkan)

Summary

rustingstrata bench completes a 131072-token prefill (and a 131072-token prompt at --ctx 262144), but a single prefill of 192k or 256k tokens panics inside the expert routing/cache path. The same prompt lengths work fine at smaller contexts, so the trigger is the length of one prefill, not the KV size.

Environment

rustingstrata 1.2.1, main @ 7264e57 (2026-11-10), cargo build --release
Arch Linux, kernel 7.2.11-arch1-1
AMD Radeon RX 9060 XT 16 GB (gfx1200, RADV, Mesa 26.2.6-arch1.1)
Ryzen 7 5700X, 62 GB RAM
Model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, IQ3_XXS (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf, 47.0 GB + 28.8 GB PLE shard; 176.94B params, qwen4exp)
Run with --device 1 (single GPU), --backend vulkan, --kv q8, --vram-margin 512

Reproduction

rustingstrata bench \
  --model Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --gpu --backend vulkan --device 1 \
  --pp 262144 --tg 64 --reps 1 --ctx 262144 --kv q8 --vram-margin 512

Observed

ctx	prompt	result
32768	32768	prefill 259.4 t/s, decode 22.5 t/s
65536	65536	prefill 221.0 t/s, decode 21.9 t/s
131072	131072	prefill 169.3 t/s, decode 20.0 t/s
262144	131072	prefill 170.3 t/s, decode 18.2 t/s
196608	196608	panic
262144	262144	panic

The model loads fine at 256k: plan: weights+state+scratch 8924 MB (KV 3265 MB for 262208 tokens, prefill 1432 MB), expert slots 6764 MB, resident experts 4062/24576.

Panics

Baseline build, 262144-token prompt:

thread 'main' panicked at src/expert_cache.rs:202:24:
index out of bounds: the len is 24576 but the index is 18446744073263861996

(self.counts[il * self.n_expert + e as usize], e: i32 — the index is a negative id cast to usize.)

With a local guard in record() (usize::try_from(e), skip if e >= n_expert || il >= layers.len()), the same 262144-token prompt now fails later, at the consumer:

thread 'main' panicked at src/gpu_model/prefill.rs:375:41:
index out of bounds: the len is 513 but the index is 356189108

(the values are -1/garbage, not a valid 0..511 expert id; off[e as usize + 1]).

196608-token prompt at --ctx 196608: same prefill.rs:375, index 18446744073592964713 (a negative value).

--chunk 4096 does not change the outcome for the 262144-token prompt.

Where the ids come from

prefill.rs:370 reads them straight from the GPU scratch buffer:

let ids = g.get::<i32>(&pf.ints, 0, t * k)?;

The buffer is allocated as gpu.zeros::<i32>(3 * chunk * k) (prefill.rs:66), and prefill_with feeds the chunked path with tokens.chunks(chunk), so t <= chunk should hold. The fact that 131072 works but 196608 does not (with the same chunking) suggests something positional/size-dependent in the routing path rather than the chunk size, but I could not reduce it further.

Expected

Either complete the prefill, or fail with a clean anyhow error. A panic kills the server process, so serve with --ctx 262144 is unusable for any request that carries more than ~128k tokens of prompt.

Workaround

--ctx 131072, or keep single prefill batches <= 131072 tokens and rely on prompt caching for follow-up turns.

Happy to run any instrumented build or enable rustingstrata's diagnostic options; the failure is deterministic.

So, I tried to Reproduce your error, but I can't I think this is AMD specific, so can you pull from main(I added more logs to understand what is the problem) and run it with

STRATA_NAN_TRACE=131000 rustingstrata bench \
  --model Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --gpu --backend vulkan --device 1 \
  --pp 196608 --ctx 196608  --tg 64 --reps 1 --kv q8 --vram-margin 512

Thanks again!

More data on d7f739a, same machine (RX 9060 XT 16 GB, gfx1200, RADV Mesa 26.2.6, kernel 7.2.11).
All runs: same IQ3_XXS model, --pp 196608 --ctx 196608 --kv q8 --vram-margin 512 --device 1.

Your exact command (default chunk 4096, cooperative matrices on):
the NAN trace prints finite values at pos 131072, layers 0-2, then the GPU hangs:

radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
Error: The logical device has been lost.

Kernel:

amdgpu 0000:2d:00.0: ring gfx_0.0.0 timeout, signaled seq=1121420, emitted seq=1121423
amdgpu 0000:2d:00.0: Process rustingstrata pid 578833 thread rustingstrata pid 578833
amdgpu 0000:2d:00.0: [drm] device wedged, but no recovery needed

STRATA_VK_NO_COOP=1 (same command): no hang, but a clean error at pos 53248, layer 3:

Error: prefill: router gave expert id -1559426103 at layer 3, chunk of 4096 tokens from position 53248;
the hidden state is non-finite (try STRATA_NAN_TRACE=1), or lower --chunk / --ctx

STRATA_VK_NO_COOP=1 + STRATA_NAN_TRACE=40960: hangs at pos 40960 after layer 10, and no
non-finite value was printed before the hang (the probes change timing).
--chunk 1024 (cooperative matrices on, trace from 131000): the full 196608-token prefill completes:

pp 196608: prefill 52.6 (52.6-52.6) tok/s | tg 64: decode 10.2 (10.2-10.2) tok/s
expert hit rate 80.3% (6049 misses), cache swaps 4053

The trace has 0 non-finite lines, including pos 131072, 176128 and 195584.

So this is not a hard position limit: 131072 is crossed fine with 1024-token chunks. The failure depends
on the interaction of chunk size and prompt length (chunk 4096 + more than ~131072 tokens), and the NAN
trace shows the numbers stay finite until the GPU hangs.

The kernel side is a ring timeout, so a dispatch that becomes too long/invalid with 4096-token chunks is
plausible. Is there a per-chunk buffer or index that scales with chunk (3*chunk*k, or a state offset), or
a dispatch whose size grows with chunk and can exceed a driver limit?

--chunk 1024 is a working workaround for now, but it costs about 3x prefill throughput
(52.6 vs 169 t/s at 128k).

Okay, my one more try, the fix is on main, please pull it and rerun your chunk-4096 command with cooperative matrices on. If it still hangs, rerun with STRATA_VK_ATTN_BUDGET=16777216 and send the log. A run with STRATA_VK_PROF=1 --pp 100000 would also help.
Thank you so much for your time, since I have no AMD gpu, I can't reproduce bug and your help is really very helpful, thank you again

Pulled main again and rebuilt: HEAD = 9dd6340 ("Merge pull request #28 from yugt/sm61-clear-error", plus 6fd27b3 "eviction fix", 4f054a5 "vulkan support for DeepSeekV4.1 1017"). Same machine, RX 9060 XT 16 GB, gfx1200, RADV Mesa 26.2.6, kernel 7.2.11, IQ3_XXS.

Your chunk-4096 command (coopmat on, STRATA_NAN_TRACE=131000, --pp 196608 --ctx 196608 --kv q8 --vram-margin 512 --device 1) - still hangs, identically to d7f739a:

nan-trace pos 131072 layer 0 ... layer 2   (all finite)
radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
Error: The logical device has been lost.
ring gfx_0.0.0 timeout, signaled seq=5752819, emitted seq=5752822, process rustingstrata

Note: the trace stops after layer 2 and the hang is in layer 3 at pos 131072, i.e. it is reproducible at a fixed place, not a random race.

FWIW the diff d7f739a..9dd6340 touches only Parked/PartCopy in gpu_model.rs (parking of a saved conversation state); prefill.rs and the shaders are unchanged, so this outcome was expected.

STRATA_VK_ATTN_BUDGET does not exist in 9dd6340 - grep -r over the whole tree finds nothing, and the env var is silently ignored. The VK vars that do exist are: STRATA_VK_NO_COOP, NO_COND, NO_TQ, DP_OLD, SMEM_LIMIT, STATS, PROF, F. Did that change stay in a branch, or under another name? I can rerun as soon as it is on main.
STRATA_VK_PROF=1 --pp 100000 --ctx 196608 --kv q8 (default chunk 4096; this length is below the failing point and completes):

pp 100000: prefill 179.2 t/s | tg 64: decode 7.7 t/s
prof attn_prefill_256   18718.3 ms  56.8%    300 calls   62394 us
prof gemmi_dp            7800.2 ms  23.7%  57540 calls     136 us
prof gemm_f              1983.1 ms   6.0%  25052 calls      79 us
prof deltanet             984.5 ms   3.0%    996 calls     989 us
prof qsa_score            758.1 ms   2.3%  18794 calls      40 us
prof quant_q8             443.8 ms   1.3%  40593 calls      11 us
prof gemm_fs              366.4 ms   1.1%  24594 calls      15 us
prof rms_norm             356.7 ms   1.1%  12038 calls      30 us
prof hc_write             333.9 ms   1.0%   2680 calls     125 us
prof hc_mix               304.1 ms   0.9%   2681 calls     113 us
prof qsa_select           264.4 ms   0.8%  18794 calls      14 us

Caveat: the profiled total is ~33 s against ~558 s of prefill, because dispatches recorded inside compute graphs are not timestamped. attn_prefill_256 is nevertheless the longest single dispatch (~62 ms, grid [t/64, n_head], each workgroup loops over the whole KV), and its duration grows linearly with step, so it is my prime suspect for the ring timeout at step > 131072 with t = 4096.

Chunk-size sweep at exactly the failing length (--pp 196608 --ctx 196608 --kv q8, coopmat on):

--chunk 4096: hangs at pos 131072
--chunk 3072: full run - prefill 131.9 t/s, decode 18.5 t/s, 0 non-finite lines
--chunk 2048: full run - prefill 69.2 t/s, decode 13.3 t/s
--chunk 1024: full run - prefill 52.6 t/s, decode 10.2 t/s

So 3072 is a much better workaround than 1024 (~2.5x faster prefill) and 196608 tokens load, prefill and decode correctly with it. What is special about chunk 4096 at >131072 tokens? Does any dispatch become invalid above some size (grid dimension, push-constant count, or a descriptor/buffer range that assumes at most 32 chunks)?

One more data point, and it narrows the cause considerably: the failure position does not depend on the quantization.

Two different quants, same command shape (--ctx 262144 --kv q8 --chunk 3072 --vram-margin 512 --device 1, --pp 262144), RX 9060 XT 16 GB / gfx1200 / RADV Mesa 26.2.6:

IQ3_XXS (47.0 + 28.8 GB): plan 8584 MB weights+state+scratch, KV 3265 MB, slots 7099 MB, 4263/24576 resident
  Error: prefill: router gave expert id 1848116770 at layer 31, chunk of 3072 tokens from position 202752

Q2_0   (37.6 + 28.8 GB): plan 8103 MB weights+state+scratch, KV 3265 MB, slots 7590 MB, 5757/24576 resident
  Error: prefill: router gave expert id -2120408218 at layer 7, chunk of 3072 tokens from position 202752

Same failing position (202752), different layers (31 vs 7), different sign of the garbage id. 66 chunks of 3072 were processed cleanly in both runs; the 67th is the first bad one. VRAM was never the issue in either run - both plans fit with margin 512 and the error is the non-finite hidden state, not an allocation failure.

So the wall is not quantization-specific and not layer-specific: it is a position/length-dependent path whose limit depends on the chunk size. For comparison, with the default chunk 4096 the same model dies much earlier, at pos 131072, and as a GPU ring timeout rather than a clean error:

chunk	ctx	result
4096	196608	hang at pos 131072 (ring gfx_0.0.0 timeout, device lost)
3072	196608	passes, prefill 131.9 t/s, decode 18.5 t/s
3072	262144	clean error at pos 202752, both IQ3_XXS and Q2_0
2048	196608	passes, prefill 69.2 t/s, decode 13.3 t/s
1024	196608	passes, prefill 52.6 t/s, decode 10.2 t/s

Both walls look like an index/counter that overflows at a chunk-size-dependent point (131072 = 32 x 4096, 202752 = 66 x 3072), so whatever scales with chunk (or with the number of chunks fed so far) is the thing to look at - a state offset, a chunk counter, or a buffer sized from chunk x something.

I am not running further configs for now (each failed run costs ~25 minutes), but I can test --chunk 1024 --ctx 262144 or any build you want to try. For daily use I am staying at --ctx 196608 --chunk 3072 (196608 tokens verified end to end), since 262144 does not complete yet with any chunk tried so far.

vulkan: bound prefill attention work per submit (#25)

Thanks again for this report again! Sorry for the runaround: STRATA_VK_ATTN_BUDGET was never merged — it lived in a local branch, which is why grep found nothing in 9dd6340. It is in main now as 8554f84.

What changed:

A chunk's prefill attention is now split into separate submits once t * (pos + t) passes a budget (STRATA_VK_ATTN_BUDGET, default 268435456 query-key pairs; 0 disables slicing). This applies to both the cooperative-matrix and no-coop paths.
While doing that I found a separate bug in the STRATA_VK_NO_COOP=1 path: its online softmax started the accumulator at zero and rescaled it by exp(0 - first_score). For a strongly negative first key that is 0 * inf = NaN — which matches the "router gave expert id ... hidden state is non-finite" you saw at pos 53248 with STRATA_VK_NO_COOP=1. Fixed too.

Could you pull main and rerun, cooperative matrices on:

--pp 196608 --ctx 196608 --chunk 4096 (the run that hung at 131072)
if it still hangs, the same with STRATA_VK_ATTN_BUDGET=16777216
--pp 262144 --ctx 262144 --chunk 3072 (the clean error at 202752)
optionally STRATA_VK_NO_COOP=1 --pp 196608 --chunk 4096 to confirm the non-finite error at 53248 is gone

The case I'm least sure the budget fixes is (3): that one was cooperative matrices on and failed in a later layer, not in the first attention layer. If it persists, please add STRATA_NAN_TRACE=200000 for that run and send the last finite layer / first non-finite layer. And if (2) still hangs, it is not simple dispatch length — your profile showed ~62 ms per attention dispatch at 100k, far under the 10 s job timeout — so the STRATA_VK_PROF=1 --pp 100000 output plus the ring timeout lines again would help. Thanks!

Pulled main, HEAD = a821cdd (contains 8425eb9 "vulkan: bound prefill attention work per submit", plus 9a4310f and a821cdd itself). Rebuilt, same machine (RX 9060 XT 16 GB, gfx1200, RADV Mesa 26.2.6, kernel 7.2.11). Coopmat results first, then the NO_COOP test.

IQ3_XXS, --pp 196608 --ctx 196608 --chunk 4096 --kv q8 (the run that hung at 131072): full pass.

pp 196608: prefill 128.1 t/s | tg 64: decode 13.5 t/s
expert hit rate 76.9% (7087 misses), cache swaps 1044
worker stalls 0

Skipped STRATA_VK_ATTN_BUDGET=16777216 (IQ3_XXS, same command as 1): the default (2^28) already fixed it, so no need. See point 4 for a BUDGET=0 A/B anyway.
Q2_0, --pp 262144 --ctx 262144 --chunk 3072 --kv q8 (the clean error at 202752), with STRATA_NAN_TRACE=200000: full pass.

pp 262144: prefill 80.0 t/s | tg 64: decode 12.5 t/s
expert hit rate 83.5% (5072 misses), cache swaps 1261

The trace ran to pos 261120, layer 47, all values finite. So the case you were least sure about is fixed too: 262144 prefill and decode end to end on Q2_0.

STRATA_VK_NO_COOP=1, IQ3_XXS, --pp 196608 --ctx 196608 --chunk 4096 --kv q8: still broken, and now in a different way. Three runs:

a) before the fix (9dd6340), IQ3_XXS, same command, for reference: clean error at pos 53248, layer 3, "router gave expert id -1559426103 ... the hidden state is non-finite".

b) a821cdd, IQ3_XXS, default budget: GPU hang instead of an error.

radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
Error: The logical device has been lost.
ring gfx_0.0.0 timeout, signaled seq=17747718, emitted seq=17747721, process rustingstrata

It died early, within about a minute of starting the first chunks.

c) a821cdd, IQ3_XXS, same command with STRATA_NAN_TRACE=1: the trace prints 483 lines and the last complete line is pos 45056 layer 2, so it hangs in layer 3 at pos 45056. Zero "non-finite" lines. Layer 3 is the first full-attention layer of this model (attention at 3, 7, 11, ... every 4th layer, DeltaNet in between), and layer 3 printed healthy finite values at pos 4096 through 40960, for example:

nan-trace pos 40960 layer 3: q 9.70e0 k 4.02e0 v 4.72e0 attn 1.18e0 | mixer 4.54e-1 router 9.91e0 weights 3.74e-1 ...

d) a821cdd, IQ3_XXS, same command with STRATA_VK_ATTN_BUDGET=0 (slicing off): a clean error again, but elsewhere and at a different length.

Error: prefill: router gave expert id 292719620 at layer 43, chunk of 4096 tokens from position 49152; lower --chunk (for example --chunk 1024) or --ctx, and try STRATA_NAN_TRACE=1 for a non-finite hidden state

Summary for the tiled path: the specific "0 * inf" NaN at 53248 is gone (no non-finite value is printed now), but the path still fails around 45 to 53k, and the failure mode depends on the budget setting: hang in layer 3 with slicing on (default), out-of-range expert id at layer 43 with slicing off. Coopmat, which is the default on this GPU, is unaffected.

Practical note: on this RX 9060 XT the default budget is enough, 262144 tokens prefill at 80 t/s and decode at 12.5 t/s, so a full 256k context now works. RDNA4 users should keep cooperative matrices on (do not set STRATA_VK_NO_COOP) until the tiled path is fixed.

There's one remaining issue, but it's not from the original report: the STRATA_VK_NO_COOP=1 (tiled attention) path still breaks -it freezes in layer 3 at 45056 with tiling enabled, and there's an error with a garbage ID in layer 43 at 49152 with BUDGET=0. The original repro didn't use this flag, but in my map, coopmat is enabled by default and works. So this is a separate bug from a different path.

Navigation Menu

Repository navigation

Prefill panics above ~128k tokens: routing ids become garbage (qwen4exp, Vulkan) #25

Summary

Environment

Reproduction

Observed

Panics

Where the ids come from

Expected

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Prefill panics above ~128k tokens: routing ids become garbage (qwen4exp, Vulkan) #25

Description

Summary

Environment

Reproduction

Observed

Panics

Where the ids come from

Expected

Workaround

Activity

Vasyl-Trefilov commented on Nov 11, 2026

ksergk commented on Nov 11, 2026

Vasyl-Trefilov commented on Nov 11, 2026

ksergk commented on Nov 12, 2026

ksergk commented on Nov 12, 2026

Vasyl-Trefilov commented on Nov 12, 2026

ksergk commented on Nov 13, 2026

ksergk commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions