RX 9070 XT (Vulkan/RADV), Flash-Next IQ2_XS: decode 0.8 tok/s, thousands of "GPU waits for the CPU expert worker that timed out"

Thanks for sharing RustingStrata. On an RX 9070 XT it builds, loads Qwen3.8-Flash-Next and answers correctly, but decode is very slow. The bench reports thousands of timed-out GPU waits on the CPU expert worker.

Setup

RustingStrata 37ba980, cargo build --release (rustc 1.99.1, Arch)
AMD Radeon RX 9070 XT 16 GB (gfx1201), Mesa/RADV 26.2.5, kernel 7.2.11, Arch Linux
Ryzen 7 7700 (8C/16T), 32 GB RAM, Samsung 990 EVO NVMe
Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf (ISTA-DASLab, sha256 matches HF), on the NVMe
The card also drives the KDE desktop (~1 GB VRAM)
Run under a systemd sandbox (DynamicUser, no home, loopback only); no other GPU job running

strata bench --gpu --backend vulkan --pp 1024 --tg 128 --reps 2

WARNING: radv is not a conformant Vulkan implementation, testing use only.
plan: weights+state+scratch 3852 MB (KV 96 MB for 4096 tokens, prefill 401 MB), expert slots 11677 MB, resident experts 8487/24576
pp   1024: prefill 194.6 (78.8-310.4) tok/s | tg 128: decode 0.8 (0.8-0.9) tok/s
expert hit rate 93.0% (8648 misses), cache swaps 4752
GPU waits for the CPU expert worker that timed out and resumed: 5436

strata generate --gpu --backend vulkan -n 64, short prompt, run twice (cold page cache, then warm):

cold: prompt 10 tok 1.55 tok/s | gen 39 tok 0.51 tok/s
      CPU expert compute, share of gen time: 5.6%
      expert hit rate 48.9% (12016 misses), cache swaps 874
      GPU waits for the CPU expert worker that timed out and resumed: 1463
warm: prompt 10 tok 15.43 tok/s | gen 39 tok 8.42 tok/s
      CPU expert compute, share of gen time: 3.3%
      expert hit rate 99.7% (80 misses), cache swaps 0
      GPU waits for the CPU expert worker that timed out and resumed: 78

The warm run hits 99.7 % of experts, and CPU expert compute is only 3.3 % of generation time. Yet it still records 78 timed-out waits for 39 tokens and decodes at 8.4 tok/s. The waits time out even when the CPU has almost nothing to do, so this looks like a GPU↔CPU synchronisation issue on RADV rather than a lack of compute.

For reference, on the same machine and file, the original Strata (HIP) decodes at 30 to 60 tok/s, depending on the measurement.

strata calibrate was not run. I can run it, or any build or env flag you want tested, and send the output.

Thanks for the detailed report, the cold/warm numbers made this easy to narrow down. All benchmarks so far were done on an NVIDIA GPU (RTX 3060), so problems on AMD were expected, and they are not fixed yet.

The CPU expert path needs a small "mailbox" buffer that the GPU polls while the CPU worker computes the experts that are not in VRAM. On AMD the GPU caches that buffer unless it comes from a device-coherent memory type (VK_AMD_device_coherent_memory). RADV hides that extension on RDNA4 (RX 9000) unless the driconf option radv_device_coherent_memory is set. Without it the GPU can keep reading a stale copy of the mailbox, so a MoE combine misses the worker's answer and spins until it gives up, even when the worker finished long ago. That fits what you see: timeouts in the warm run with 99.7% hits and only 3% CPU time, and far more of them in the bench, where many experts run on the CPU.

Could you rerun with the variable set, on your current build?

radv_device_coherent_memory=true strata bench --gpu --backend vulkan --pp 1024 --tg 128 --reps 2

(Under systemd that is Environment=radv_device_coherent_memory=true in the unit.)

If that fixes it, the "timed out and resumed" line should be gone or close to zero. In the next commit strata sets the variable itself before it opens Vulkan (unless you set it), and prints a warning at load if an AMD GPU still has no device-coherent memory.

If it does not help, please send the output of vulkaninfo | grep -i coherent and the bench output with the variable set, and I will look further.
Thank you again for helping and contributing!

That was it. Thanks for the quick and precise diagnosis.

vulkaninfo confirms it: without the variable only the iGPU (Raphael) lists VK_AMD_device_coherent_memory; with radv_device_coherent_memory=true the RX 9070 XT lists it too.

Same build (37ba980), same file, same sandbox, only Environment=radv_device_coherent_memory=true added:

pp   1024: prefill 450.5 (232.4-668.6) tok/s | tg 128: decode 64.2 (60.7-67.7) tok/s
expert hit rate 94.4% (6855 misses), cache swaps 25

	before	with the variable
bench prefill pp1024	194.6 tok/s	450.5 tok/s
bench decode tg128	0.8 tok/s	64.2 tok/s
"timed out and resumed" (bench)	5436	line absent
`generate -n 64`, 1st run	0.51 tok/s	22.72 tok/s
`generate -n 64`, 2nd run	8.42 tok/s	22.37 tok/s

The "timed out" line no longer appears in any of the three runs. The bench also took 21 s instead of 5.5 min.

Setting the variable from strata itself sounds right to me. Happy to test that commit when it lands.

vulkan: set radv_device_coherent_memory before opening Vulkan.

Great, thanks for confirming! That fix is already in: strata now sets radv_device_coherent_memory=true itself before it opens Vulkan (unless you set the variable yourself). It's on main now, together with the multi-GPU split and the other recent fixes:

git checkout main && git pull && cargo build --release

Could you rerun the same bench and generate commands without the Environment= line, to confirm the "timed out and resumed" line stays absent and the numbers match your run with the variable?

Also, when you have time, the split runs (--device 0,1 and --device 1,0) on this build now print per-device timing ("device N: X ms/token forward, Y ms/token copy to the next device"). Those lines would tell us where the ~8.5 ms/token split overhead goes. So it can be even faster, thank you so much for help!

I will create some .md file where I will write all people that helped with Strata, how do you want me to write you, just github or something else?

@charmoky

Just GitHub is fine: @charmoky. Thanks!

Confirmed on eaf2bb1, without the Environment= line:

ATTENTION: default value of option radv_device_coherent_memory overridden by environment.
vulkan: AMD Radeon RX 9070 XT (RADV GFX1201) (radv Mesa 26.2.5-arch1.1), device-coherent memory yes, cooperative matrix f16 yes int8 no, conditional rendering yes
pp   1024: prefill 378.7 (102.6-654.8) tok/s | tg 128: decode 59.4 (55.4-63.4) tok/s
expert hit rate 93.8% (7639 misses), cache swaps 722

No "timed out and resumed" line in the bench or either generate run.

	`37ba980` + variable	`eaf2bb1`, no variable
bench prefill pp1024	450.5 tok/s	378.7 tok/s
bench decode tg128	64.2 tok/s	59.4 tok/s
bench cache swaps	25	722
`generate -n 64`, 1st / 2nd run	22.7 / 22.4 tok/s	18.7 / 23.9 tok/s

The lower bench numbers come with many more cache swaps (722 vs 25) and a wide prefill range. I ran it once per build, so I can't tell build changes from run-to-run variance.

On the split runs: I haven't done any, and the ~8.5 ms/token figure isn't from me. Device 1 here is the Ryzen 7700's iGPU (2 CUs), so a split on this machine wouldn't tell you much.

One more data point, in case it helps: a 24-turn agent-style replay with IQ2_XS on the same machine. It starts with a 43.8K-token prompt, then adds 130–1 190 new tokens per turn with a fixed history. I ran it A/B/A/B: original Strata 0.2.9 (HIP, --mmap-experts, with the env we use in production: GPU_PINNED_MIN_XFER_SIZE=1048576, STRATA_ROUTE_TAIL_SKIP=4, STRATA_RESIDENT_PIN=0) against RustingStrata eaf2bb1 (serve --ctx 65536, no env). Same file, same turns, 64K context for both, page cache dropped before each start, request wall time with one generated token.

	Strata 0.2.9 (HIP)	RustingStrata (Vulkan)
first 43.8K-token prompt	207 s / 103 s	267 s / 267 s
24 turns, read time total	274 s / 246 s	70 s / 69 s
median read speed per turn	57 / 53 tok/s	156 / 157 tok/s
read from the NVMe, 24 turns	62 / 51 GB	7.0 / 7.1 GB

RustingStrata was the same to the second across both passes, and has no "timed out" line in either. After the first few turns it reads 0.04–0.2 GB per turn, where Strata re-reads 0.4–5 GB of experts. Strata is faster on the long first prompt. One difference in conditions: Strata ran without a memory limit, RustingStrata under MemoryMax=26G (32 GB machine).

Thanks for confirming, and for the agent replay, that is the most useful data point we have had so far.

Sorry about the split question, the ~8.5 ms/token was from another tester on issue #1 (two AMD cards). With a 2-CU iGPU as device 1 a split makes no sense, please ignore it. I just reply to people in reddit and github all day and slowly loosing my mind, so it was just a misstake.

On the lower bench numbers: the 722 cache swaps point to the expert placement. RustingStrata saves an expert profile after a run and starts the next run from it, and the multi-GPU commit between your two builds changed how that profile is saved. So the first run on eaf2bb1 likely started without a usable profile. If you run the same bench twice in a row, the second run should be back near 25 swaps and the old numbers. If it is not, please send both outputs.

The agent replay is great to see: about 3x faster on the follow-up turns and 9x less read from the NVMe than Strata 0.2.9, under a 26 GB memory limit.

The long first prompt is where we are behind (267 s vs 103-207 s), and that is the next thing to fix. A commit that is coming soon lifts a prefill chunk cap for this model (Flash-Next uses hyper-connections, and its prefill was limited to 1024-token chunks; each chunk reloads the experts that are not in VRAM). On my RTX 3060 that took a 4.8K-token prompt from 132 to 158 tok/s; on a 43.8K-token prompt the gain should be bigger. I will ping you here when it is on main. Running strata calibrate --model <file> --gpu --backend vulkan once after updating lets it pick the best chunk size for your card.

I added you to THANKS.md in the repo root. Thanks again!

Thanks for adding me, and no worries about the split question.

Same bench four times in a row on eaf2bb1 (IQ2_XS, no env). Before these, the profile had last been saved by a serve --ctx 262144 --kv q8 session:

run	prefill pp1024	decode tg128	expert hit rate	cache swaps
1	338.5 (97.4-579.6)	60.3 (55.9-64.7)	93.7%	813
2	442.2 (205.5-679.0)	47.5 (36.0-59.1)	85.4%	120
3	401.2 (146.6-655.8)	62.7 (58.8-66.5)	94.1%	163
4	428.7 (171.4-686.0)	47.2 (35.2-59.2)	86.0%	100

You were right about the swaps: 813 on the first run, 100–163 after. Decode alternates, though: about 62 tok/s with a ~94% hit rate on runs 1 and 3, about 47 tok/s with ~86% on runs 2 and 4. It looks as if each run's saved profile places the experts worse for the next one, then better again. No "timed out" line in any of them.

Two more data points from the same machine, in case they help:

262K context works. serve --ctx 262144 --kv q8 (KV 3264 MB, expert slots 8002 MB): a real OpenCode agent session completed a small fix-and-test task, with correct tool calls throughout. A 228,857-token prompt with a fact placed at 10% returned the fact correctly; the prefill took 16 min 07 s (~237 tok/s). A 297K-token prompt got a clean HTTP 400 "fills the context".
When the model fits in RAM, Strata 0.2.9 is well ahead on prefill. Same 24-turn replay, A/B/A/B, on the Coder IQ1_M (Strata with --resident-experts, both at 262K):

	Strata 0.2.9 (HIP)	RustingStrata (Vulkan)
first 43.8K-token prompt	19.8 s / 19.9 s	290 s / 289 s
24 turns, read time total	35 s / 36 s	85 s / 85 s
median read speed per turn	251 / 233 tok/s	126 / 128 tok/s

On IQ3_XXS (RustingStrata only, 64K, same replay): first prompt 548 s / 554 s, 24 turns 137 s / 177 s, 17.8 / 19.5 GB read; no "timed out" line.

The prefill chunk fix you mention should help the first prompt in all three cases. Happy to rerun the replays and strata calibrate when it lands.

Thanks, this is very useful.

Alternating decode (94% / 86%): found and fixed. The saved profile only remembered the last reply, so each run's text decided the next run's placement. It now keeps a long-memory profile (about 8,000 tokens). That fix is on main and ships with the next release.

Prefill chunk fix: it is already in v1.2.0 (your eaf2bb1 predates it). It only affects Flash-Next, not the Coder.

Coder IQ1_M first prompt (290 s vs 19.8 s): this one surprises me. On my RTX 3060 (12 GB, Vulkan), the 80B IQ1_S with only 38% of the experts in VRAM prefills 16K tokens at 780-860 tok/s, and the chunk size makes little difference. Your 43.8K prompt ran at about 150 tok/s, so something on your machine is holding it back. Could you run this twice, once inside your MemoryMax=26G unit and once outside it, and post the plan: and pp lines plus how much was read from the NVMe?

strata bench --model <Coder IQ1_M> --gpu --backend vulkan --mtp 0 --pp 16384 --tg 4 --reps 2 --chunk 4096

My guess is the memory limit: the page cache counts toward MemoryMax, so if the model does not fit under it, every prefill chunk reads the missed experts from the NVMe again.

expert cache: save a long-memory routing profile

The memory limit is not it: same numbers inside and outside MemoryMax=26G. The slow first prompt comes from serve: with the same model and about the same expert slots, bench prefills 43K tokens at 635 tok/s and serve at 199 tok/s.

All on 862cbc7 (main), Coder IQ1_M, Vulkan, page cache dropped before each run.

Your command (bench --mtp 0 --pp 16384 --tg 4 --reps 2 --chunk 4096), A/B/A/B:

run	MemoryMax	`pp 16384`	hit rate / swaps	NVMe read
1	26G	700.7 (435.3-966.1)	94.8% / 2582	29.6 GB
2	none	799.9 (511.2-1088.7)	94.1% / 0	27.2 GB
3	26G	797.7 (510.4-1085.1)	94.1% / 0	27.1 GB
4	none	796.4 (509.4-1083.4)	94.1% / 0	27.2 GB

plan: weights+state+scratch 5911 MB (KV 384 MB for 16388 tokens, prefill 1447 MB), expert slots 9555 MB, resident experts 4896/12288. Run 1 is the profile carry-over again (first run after the build). Memory peak 23.6–24.7 GB in all four.

The 43.8K-token replay prompt, bench vs serve (26G):

	plan	first 43.8K prompt	24 turns, read total
`bench --pp 43008 --chunk 4096`	slots 8849 MB, resident 4534	635.8 tok/s (~68 s)	—
`bench --pp 43008` (default chunk)	same	634.9 tok/s	—
`serve --ctx 65536 --kv q8`	slots 9003 MB, resident 4613	199 tok/s, 220 s	73 s
`serve --ctx 262144 --kv q8` ×2	slots 6230 MB, resident 3192	178 tok/s, 246 s / 246 s	118 s / 114 s

The serve run reads 30 GB from the NVMe for the first prompt, against 27.6 GB for the bench, so it is not re-reading much more. Its slowness looks like compute or scheduling.

One regression against eaf2bb1 at 262K: the follow-up turns are slower. Turns 2–3 (169 and 381 new tokens) take 12–16 s instead of 4–6 s, and the 24 turns take 114–118 s instead of 85 s. At 64K they are back to 4–5 s (73 s total).

Hi! I already fount the problem and currently on it, I will write you and push to main in like 10-20 with fix

@charmoky

@charmoky the fix is on main, can you run benchmark again pls

Same numbers on 95964da. I think the new kernel does not run here: the 9070 XT reports cooperative matrices, and dp_tile is only set when !coop. The prefill attention: tiled line is not printed either:

vulkan: AMD Radeon RX 9070 XT (RADV GFX1201) (radv Mesa 26.2.5-arch1.1), device-coherent memory yes, cooperative matrix f16 yes int8 no, conditional rendering yes

Coder IQ1_M, Vulkan, MemoryMax=26G, page cache dropped before each run:

	`862cbc7`	`95964da`
`bench --pp 43008 --chunk 4096`	635.8 tok/s	621.8 tok/s
`serve --ctx 65536 --kv q8`, first 43.8K prompt	199 tok/s, 220 s	200.9 tok/s, 218 s
same, 24 turns read total	73 s	73 s
`serve --ctx 262144 --kv q8`, first 43.8K prompt	178 tok/s, 246 s	178.3 tok/s, 246 s
same, turns 2–3 (169 / 381 new tokens)	12–16 s	8.8 / 10.3 s
same, 24 turns read total	114–118 s	106 s

So serve still prefills the first long prompt about 3× slower than bench on this card.

chore: raise analyst/developer step budgets; track M0-T4 in review

[#2] Vulkan shared-memory preflight (M0-T4) (#3)

docs: close M0-T4 and record the preflight validation

@charmoky

Hi @charmoky, sorry for so long delay and my next try to fix it is on the main, can you please one more time run

Drop the page cache, then run a single cold pass with the same flags as before:
rustingstrata bench --gpu --backend vulkan --pp 43008 --chunk 4096 --reps 1
Repeat step 1 with STRATA_PF_ADAPT=1, and again with STRATA_PF_ADAPT=4.
For one of the runs, compare the f16 default with --kv q8.
For each run, please paste the pp ...: prefill line, the expert hit rate ... cache swaps ... line and the vulkan: ... line from startup.

Thanks again for testing and waiting!

On ee9e235, STRATA_PF_ADAPT makes no real difference to bench: 568–588 tok/s across all four runs.

Coder IQ1_M, rustingstrata bench --model … --gpu --backend vulkan --mtp 0 --pp 43008 --tg 4 --reps 1 --chunk 4096, MemoryMax=26G, page cache dropped before each run:

run	`pp 43008: prefill`	expert hit rate / cache swaps	expert slots, resident	NVMe read
default (f16 KV)	588.2 tok/s	89.3% (205 misses) / 1199	8136 MB, 4169/12288	28.1 GB
`STRATA_PF_ADAPT=1`	567.5 tok/s	90.6% (180 misses) / 1990	8183 MB, 4193/12288	30.2 GB
`STRATA_PF_ADAPT=4`	583.5 tok/s	89.7% (198 misses) / 1169	8136 MB, 4169/12288	29.0 GB
default, `--kv q8`	577.8 tok/s	90.2% (189 misses) / 917	8605 MB, 4409/12288	28.0 GB

Startup, the same in all four:

vulkan: AMD Radeon RX 9070 XT (RADV GFX1201) (radv Mesa 26.2.5-arch1.1), device-coherent memory yes, cooperative matrix f16 yes int8 no, conditional rendering yes, maxComputeSharedMemorySize 65536

For reference, 95964da got 621.8 tok/s on the same command, with 8849 MB of expert slots against 8136 MB now.

Correction to what I said earlier: serve is not slower than bench. The prompt is. bench prefills prose(), a single ~60-token paragraph repeated. Fed that same text, serve --ctx 65536 --kv q8 prefills 45.4K tokens at 601.4 tok/s (93.6 s), cold, on ee9e235. The slow first prompt in my replay is a real OpenCode coding session, which routes far more experts.

The same replay through serve --ctx 65536 --kv q8 with each STRATA_PF_ADAPT setting (26G, cold):

`STRATA_PF_ADAPT`	first 43.8K prompt	turns 2–3 (169 / 381 new tokens)	24 turns, read total	NVMe read after the first prompt
0	228.9 s, 191.8 tok/s	5.2 / 5.2 s	79 s	4.5 GB
1	270.5 s, 162.2 tok/s	3.3 / 4.0 s	85 s	1.2 GB
4	253.4 s, 173.2 tok/s	3.7 / 4.3 s	85 s	1.2 GB

STRATA_PF_ADAPT makes the first prompt 10–18% slower. After it, the follow-up turns read about 0.07 GB each instead of about 1 GB, but the 24 turns are not faster overall.

--chunk does not help either. On the same replay, 8192 gives 225.6 s (194.5 tok/s) and 16384 gives 261.5 s (167.8 tok/s), against 228.9 s at the default 4096. The larger prefill scratch takes expert slots away, 8320 → 7463 → 4799 MB.

bench might be more representative with a less repetitive prompt, or with a --prompt-file option.

Navigation Menu

Repository navigation

RX 9070 XT (Vulkan/RADV), Flash-Next IQ2_XS: decode 0.8 tok/s, thousands of "GPU waits for the CPU expert worker that timed out" #3

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RX 9070 XT (Vulkan/RADV), Flash-Next IQ2_XS: decode 0.8 tok/s, thousands of "GPU waits for the CPU expert worker that timed out" #3

Description

Activity

Vasyl-Trefilov commented on Nov 5, 2026

charmoky commented on Nov 5, 2026

Vasyl-Trefilov commented on Nov 5, 2026

Vasyl-Trefilov commented on Nov 5, 2026

charmoky commented on Nov 5, 2026

Vasyl-Trefilov commented on Nov 5, 2026

charmoky commented on Nov 6, 2026

Vasyl-Trefilov commented on Nov 6, 2026

charmoky commented on Nov 6, 2026

Vasyl-Trefilov commented on Nov 6, 2026

Vasyl-Trefilov commented on Nov 6, 2026

charmoky commented on Nov 6, 2026

Vasyl-Trefilov commented on Nov 9, 2026

charmoky commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions