RX 6700 XT / RADV: Vulkan works, slow 30K Qwen3.6 prefill #10

Activity

1. Vasyl-Trefilov

   added

   BenchmarkFeature need approved benchmarks before mergeFeature need approved benchmarks before merge

   AMD

   on Nov 6, 2026
2. Vasyl-Trefilov commented on Nov 6, 2026

   

   Vasyl-Trefilov

   on Nov 6, 2026

   Last edited by Vasyl-Trefilov

   Member

   More actions

   @hermeticvm Hi, ty for detailed report! The fix is on main, you can pull and try again, it should give better performance now. Sorry for this problem, there are a lot of problems with AMD GPUs
3. hermeticvm commented on Nov 7, 2026

   

   hermeticvm

   on Nov 7, 2026

   Author

   More actions

   Thanks for the quick fix! I retested commit 95964dabb92a9a050a469ae74622e3a53dc86c15. The tiled prefill change helps substantially on the original 30K workload. I also tried 117K and found a GPU reset with the original chunk size; a smaller chunk completed correctly, but slowly.

   Same machine/model as the original report: RX 6700 XT 12 GiB, RADV Mesa 26.2.5-arch1.1, kernel 7.2.8-9-omarchy, Ryzen 5 5600X. The locally quantized Qwen3.6-35B-A3B Heretic IQ4_XS GGUF (with MTP head) has SHA-256 9985dd184a5c28f1014bce4a293a798ad9ed693d9bfa1ec6189782c2768efa72; it is not a public download. This is a release build with target-cpu=native.

   Controlled 30K retest

   Same new binary, the original 30,444-token prompt, --ctx 32768 --chunk 1024 --kv q8 --vram-margin 1536 --threads 6 --mtp 0, temperature 0. Each run started a fresh server with its own copy of the same expert profile; cached_tokens=0. The old path was selected with STRATA_VK_DP_OLD=1; the new path logged vulkan: prefill attention: tiled. Order was tiled, old, old, tiled.

   Attention path	Prefill (tokens/s)	End-to-end (s)
   Old, run 1	134.02	230.38
   Old, run 2	133.69	230.98
   Tiled, run 1	244.41	127.81
   Tiled, run 2	244.54	127.70

   The four response bodies were byte-identical. After stripping Markdown fences they passed six merge-interval cases; the fences themselves violated the prompt's no-Markdown instruction. This is a same-build comparison, not a controlled comparison against my older v1.2.0 binary.

   117K cold prefill and prefix reuse

   With --ctx 131072 --kv q8 --vram-margin 1536 --threads 6 --mtp 0, an uncached 117,697-token chat request failed at chunk 1024 after 15m17s with HTTP 500. The server logged:

   radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
   

   The kernel journal reported ring gfx_0.0.0 timeout, identified the strata process, then reported Ring gfx_0.0.0 reset succeeded and [drm] device wedged, but no recovery needed. Sampled peak VRAM was 10,808 / 12,272 MiB; the 500 MiB free-memory guard did not trigger. I have not isolated the cause of the ring timeout.

   Following the AMD documentation, I tried one fresh-server retry with --chunk 256 and the same seeded expert profile. It completed:

   	Production llama.cpp/ROCm	RustingStrata, chunk 256
   Cold request, wall	250.44 s	2,181.78 s
   Cold prefill	482.89 tokens/s	54.08 tokens/s
   Follow-up, wall	1.63 s	7.95 s
   Follow-up cached tokens	117,693	117,690

   Both cold and follow-up responses retrieved the correct marker. There were no kernel reset/timeout entries during the retry. Production uses a different server, expert placement and speculation configuration (DSpark + n-gram), so this is a practical comparison, not an isolated backend or chunk-size benchmark. Only one request per arm was run at this length.

   The 117K prompt can be reproduced with the generator below, followed by the JSON request. The model tokenizer counted 117,685 user-text tokens and RustingStrata reported 117,697 with chat formatting. This is intentionally synthetic reference material, not an agent-quality benchmark.

   import json, urllib.request
   
   filler = ' '.join(
       f"Result {i}: The {['northern','southern','eastern','western'][i % 4]} district reported "
       f"{i * 37 % 1000} new households, a {i % 9 + 1}.{i % 7} percent change, while council member "
       f"{['Ahn','Berg','Costa','Diallo','Eze'][i % 5]} proposed plan {i}."
       for i in range(110)
   )
   prompt = ('The following is reference material; keep the verification code for the question at the end.\n'
             + filler * 15 + '\n\nThe verification code is N7K4-Q2M9.\n\n' + filler * 16
             + '\n\nWhat is the verification code? Give the code first, then explain in about 150 words '
               'how you would preserve and retrieve an important fact in a long technical conversation.')
   messages = [{'role': 'user', 'content': prompt}]
   
   def request(messages, limit, cache_prompt):
       body = {'model': 'qwen3.6', 'messages': messages, 'max_tokens': limit,
               'temperature': 0, 'top_p': 0.95, 'top_k': 20, 'min_p': 0, 'seed': 42,
               'cache_prompt': cache_prompt, 'chat_template_kwargs': {'enable_thinking': False}}
       req = urllib.request.Request('http://127.0.0.1:8085/v1/chat/completions',
                                    json.dumps(body).encode(), {'Content-Type': 'application/json'})
       with urllib.request.urlopen(req, timeout=3600) as response:
           result = json.load(response)
       print(json.dumps(result), flush=True)
       with urllib.request.urlopen('http://127.0.0.1:8085/api/stats', timeout=30) as response:
           print(json.dumps(json.load(response)['recent'][-1]), flush=True)
       return result
   
   first = request(messages, 300, False)
   answer = first['choices'][0]['message']['content']
   request(messages + [{'role': 'assistant', 'content': answer},
                       {'role': 'user', 'content': 'Repeat the verification code from the reference material exactly, with no other text.'}], 64, True)

   Other observations

   On 18 short requests per Rust arm (three repeats each of edit/fresh-code/thinking-chat at temperatures 0 and 0.6), enabling MTP improved median decode speed by approximately 6–15% versus MTP off. One API behavior worth flagging: all 12 thinking-enabled Rust chat responses began with text such as Here's a thinking process: in message.content, without a reasoning_content field. The six corresponding production llama.cpp responses separated reasoning_content and final content. The two servers have different thinking settings, so I haven't isolated whether this is template behavior, model output or response parsing. Happy to file a separate issue if useful.
4. Vasyl-Trefilov commented on Nov 7, 2026

   

   Vasyl-Trefilov

   on Nov 7, 2026

   Member

   More actions

   Hi, thanks for report and suggestion. Prefill on AMD is real pain, I will soon get AMD GPU and will fix everything, but for now it's difficult as hell, I will try to do something about this and will write you when it's done, thank you again!
5. Vasyl-Trefilov

   added 3 commits that reference this issue on Nov 8, 2026

   docs: start M1-T2 (issue #10)

   67e1506

   [#10] Warp reduce without __reduce_*_sync (M1-T2) (#11)

   ...

   fe3eb1a

   docs: close M1-T2 (issue #10, PR #11, fe3eb1a)

   70dd826