Repository navigation
[Idea]: Support multiple GPUs and support qwen3.8-flash-next #1
Description
What problem would this solve?
I have an x299-based PC (I9 CPU, 128gb DDR4 RAM) with two RTX 3060s (12gb VRAM each). I have the pi harness using a free AI model from openrouter.ai and I told it to look at the strata inference engine and forks of it, and to pick the best one for my hardware, and it picked RustingStrata. The free AI model says that RustingStrata doesn't support more than one GPU, and it doesn't support qwen3.8-flash-next.
Have you considered having RustingStrata support using multiple GPUs? Are you planning on supporting qwen3.8-flash-next like is done by the main strata build?
Proposed improvement
With multiple GPUs, I would expect that as many experts would be cached in VRAM as possible, with RustingStrata keeping track of which GPU to use for which cached expert. This should speed up things because it wouldn't have to go to the slow RAM as often.
Alternatives and tradeoffs
No response
Activity
Vasyl-Trefilov commentedon Oct 31, 2026
Thanks for the detailed request!
Multi-GPU: right, RustingStrata uses one GPU per process at the moment (--device N picks which one). With two 3060s you'd have 24 GB, which is enough to keep almost all of Qwen3.6-35B-A3B Q2_K_XL in VRAM, so I agree it would help. The likely first step is splitting the layers between the two cards (hidden state copied between them), with leftover experts still on the CPU. I only have one 3060, so I can't test it myself yet. Would you be up for testing a branch on your two cards if I get to it?
Until then you can run it on one card (--device 0 or --device 1) and get the same numbers as in the README, since that's my setup too. With 128 GB of RAM you can also run Qwen3-Next-80B-A3B, with most of its experts on the CPU. You can also start two serve instances, one per GPU.
About "qwen3.8-flash-next", I maybe will add it as next model, why I have not done it yet? Bc I also used original Strata with "qwen3.8-flash-next", but my pc was dieing bc it's still not good enough for this model and I was really upset, bc I would love to use for example Qwen3.8:27b in Strata, but it just never existed, so I though that I will create my own Strata for all Qwen models expect for "qwen3.8-flash-next".
But I will add it anyway
runningman84 commentedon Nov 4, 2026
I have dual amd GPUs 7900xt + 9070xt, if you need a tester I can test it…
Vasyl-Trefilov commentedon Nov 4, 2026
Qwen3.8-Flash-Next is supported now (on main). For multi-GPU I'm starting on a layer split across cards (--device 0,1): each card holds its share of layers plus as many experts as fit, the rest stay on the CPU. @runningman84 thanks, a 7900 XT + 9070 XT pair is a great Vulkan test. I'll post here when a branch is ready to try. Thanks again for help!!!
Vasyl-Trefilov commentedon Nov 4, 2026
@runningman84 multi-GPU support is ready for testing on the
multigpubranch, and your 7900 XT + 9070 XT would be a great first test.It is a layer split: each GPU holds a contiguous range of layers (in proportion to its free VRAM), its own KV cache and its own expert slots. Experts that do not fit stay in RAM and run on the CPU as before. The hidden state moves between cards through host memory once per token (decode) or once per chunk (prefill), so no peer-to-peer is needed and mixed cards work.
On one card opened twice, the split gives bit-identical logits to the single-device path on Vulkan and CUDA (Qwen3.6-35B-A3B, Qwen3-4B, Qwen3.8-Flash-Next). I have only one GPU, so two real cards are untested. To save you round trips, please run the whole loop below once and paste the full output of each run, even if one fails.
Build:
Run each card alone, then both:
radv_device_coherent_memory=trueis needed on RDNA4 (the 9070 XT) for the CPU expert worker. Each run starts with a line per card such asvulkan device 1: AMD Radeon RX 9070 XT (RADV GFX1201). If an integrated GPU shows up as device 0 or 1, use the indexes of the two discrete cards instead.Known limits:
runningman84 commentedon Nov 5, 2026
here are the results:
it looks like 2 cards are slower than my 7900xt alone. The 9070xt seems to perform worse than the 7900xt.
Vasyl-Trefilov commentedon Nov 5, 2026
Very sad, but this is just first try, Thanks! The split is correct: all three runs give the same text. The problem is speed: 43 tok/s split vs 68 on the 7900 XT alone, even with every expert in VRAM. On my side a split over one GPU opened twice costs almost nothing, so I need numbers from your machine to see where the time goes.
I pushed a new multigpu. It also has everything from the main development branch: the AMD coherent-memory option is now set automatically (you no longer need radv_device_coherent_memory=true), the Q4_0/Q5_0 tensors in the unsloth quants are supported, and there is sparse attention for long contexts. A split run now also prints how long each card takes per token.
Please pull, rebuild, and run:
and send the full output, especially the lines at the end:
"device 0: ... ms/token forward, ... ms/token copy to the next device" and "gen total ... ms/token".
1,0 puts the 7900 XT first, which shows whether the 9070 XT's share of the layers is the slow part.
runningman84 commentedon Nov 5, 2026
here are the latest results:
Vasyl-Trefilov commentedon Nov 5, 2026
Thank you for fast response! This is exactly the data I needed. The copy between the cards is cheap (0.3 ms/token), so the hand-off isn't the problem. What stands out is that each card runs its layers about 45% slower in the split than alone: in the 1,0 run the 7900 XT takes 11.98 ms for 27 layers (0.44 ms/layer), while your warm single-card run on it (68 tok/s) works out to about 0.31 ms/layer for all 48.
My guess is power management. In a split each card idles roughly half of every token while the other one works, and the driver drops its clocks during those ~10 ms gaps. Could you try the 1,0 run with both cards forced to high clocks?
If you have amdgpu_top, the shader clock during a normal split run would also tell us a lot.
Two notes:
ike287 commentedon Nov 5, 2026
two-GPU result for the
multigpubranch — this issue is exactly my use case, so I tested it. Everything below ran on commitaadec27.Hardware
Model
Qwen3.8-Flash-NextGGUF, IQ4_XS (98 GB, 3 shards, bartowski). Load + inference both correct across a--device 0,1split; layers 0–24 / 24–48 with the memory plan split ~21.5 GB expert slots per card.Note for other testers: only a subset of quants is loadable. Q4_0/Q4_1/Q5_0/Q5_1 are rejected at load — several popular Unsloth UD files (UD-Q4_K_XL, UD-IQ4_XS, UD-Q3_K_XL) and bartowski Q4_K_M fail because of Q5_1/Q5_0 tensors. IQ4_XS from bartowski works.
Results (
--mtp 0, 3 reps, split vs single card)--device 0,1--device 0(prefill/decode in tok/s; split numbers at 33,000 ctx, single-card at 8,192)
With both cards, 16.8k of 24.6k experts fit resident and the hit rate was 99.5% (8.2k swaps) — CPU expert compute dropped from 84% to ~25% of gen time vs an early disk-bound run. Splitting doubles decode and prefill vs one card as expected.
Two practical notes
primarycache=metadata, CPU-side experts are never resident in page cache, so the engine re-reads ~1 TB from NVMe per load: ~7 min load, 7.9 tok/s decode, GPUs at 0%. After staging the GGUF into a tmpfs (98 GB copies in ~80 s): 13 s load and the numbers above. Worth a docs note for ZFS users.Great work — the layer-split "just worked," and the memory-plan/expert-residency logging is very good. For context, on this same box the single-GPU strata fork and llama.cpp both land around 50–60 tok/s decode on comparable MoE models, so this engine's expert-cache design is pulling real weight.
runningman84 commentedon Nov 5, 2026
the power management makes a difference
RustingStrata utilizes both gpus only about 50% it looks similar to split layer with llama-cpp:
Normally I use qwen3.8-27b:q6 with 200k context. This utilizes both gpus and I get about 60 tokens per second in split tensor.
My original idea was is there some other tools like Strata or something else which can utilize both cards also in MoE models in a way that the speed gets doubled as with 27b. Right now it looks like I can use qwen next but two cards don't really help...
Vasyl-Trefilov commentedon Nov 5, 2026
Thanks, that settles the clocks question: with power_dpm_force_performance_level=high the 1,0 split goes from 44.15 to 57.05 tok/s, and the 7900 XT's 27 layers go from 11.98 to 9.09 ms per token. The cards were dropping their clocks while waiting on each other. I'll add this to the AMD notes. Strata can't set it itself because it needs root.
About ~50% use per card: that's how a layer split works. One request goes through the layers in order, so one card always waits while the other computes. llama.cpp's layer split behaves the same way.
The 2x you get on Qwen3.8-27B with tensor split doesn't carry over to a MoE like Flash-Next. A dense 27B at Q6 reads about 22 GB of weights per token, so each card reading half is a big win. Flash-Next uses only about 3B parameters per token, so one layer takes about 0.3 ms. Splitting each layer across both cards would need two copies between the cards per layer. Your logs show about 0.3 ms per copy, so that's about 29 ms of copies per token, while a whole token on the 7900 XT alone takes about 14.7 ms. Without direct card-to-card copies (which Vulkan doesn't give us on consumer AMD cards), a single request on this model won't decode faster on two cards than on the 7900 XT alone.
What we can still make faster, and what I'd like to try next:
I only have one GPU here, so I can check that results are correct but can't measure two-card speed. When each step lands I'll post the exact commands. If you can run them on your 9070 XT + 7900 XT, that would help a lot.
Until then: for short and medium sessions the 7900 XT alone (--device 1) is fastest, at 68 tok/s in your warm run. The split pays off once one card no longer holds the experts a session needs. Split, about 17k of the 24.5k experts stay in VRAM, against about 8.9k on the 7900 XT alone. ike287's run above shows the same thing more strongly: their 98 GB model doesn't fit one 3090, and the split doubled both decode and prefill.
btw, what for you want to use local LLM and I want to add you in file that will contain all contributors, to show my respect and thanks, so do you want to have only your github there or something else?
runningman84 commentedon Nov 5, 2026
I am happy to test additional releases for you, feel free to add my GitHub username to the contributors.
My usecase is just coding some smaller projects with pi agent and general research with Hermes.
BTW. will your project also support dense models or will it just be a MoE runner?
What about running higher quants and huge contexts? With my setup of two cards and 64gb ddr5 ram strata told me that I should not use bigger quants or contexts. Is this something your project might improve in the 2 gpu setup?
Vasyl-Trefilov commentedon Nov 5, 2026
Thanks a lot, your runs found real problems (the AMD idle downclock among them), so the help is very welcome. I added you to THANKS.md in the repo root.
Dense models: already supported, not only MoE. Qwen3.5-2B, Qwen3-4B and Qwen3.8-27B run today, and the layer split over two cards works for dense models too. The difference to MoE: for a dense model every weight is used for every token, so all weights must sit in VRAM (spread over both cards with --device 0,1). Only MoE experts can live in system RAM.
Higher quants and long contexts: the message you saw is Strata's VRAM check. The model, the KV cache and the prefill scratch must fit on the GPU with some margin left. What helps now:
On MoE models a bigger quant mostly costs speed, not "fit": the experts that do not fit in VRAM stay in RAM, and your 64 GB holds a lot of them.
If you post the exact command and the full message (it prints how many MB were left), I can tell you whether it is a real limit or something I should fix.
Speed on two cards: the split runs the cards one after the other, so for one request it gives you more memory, not 2x speed. Ideas we will try (no promise yet): pipelining long prompts across both cards, keeping the waiting card busy so it does not downclock, and serving two requests at once. A llama.cpp-style tensor split for dense 27B is not planned soon: without a direct GPU-to-GPU link every layer would need two copies through the host, and those cost more than they save at batch size 1. For dense 27B at 200K, llama.cpp's tensor split is likely still the faster choice on your setup for now.
Vasyl-Trefilov commentedon Nov 5, 2026
@ike287 Q4_1 and Q5_1 are supported now, including in the experts, on CUDA, Vulkan and the CPU expert path (it's on main as of the next push). That covers the Unsloth Flash-Next UD-Q4_K_XL files, which use Q5_1 for ffn_down_exps.
I checked it on a Qwen3-30B-A3B requantized with Q4_1 gate/up experts, Q5_1 down experts and Q4_1 attn_v. Greedy top-1 agreement with llama.cpp over 20 prompts x 32 tokens was 98.9% on CUDA and 98.75% on Vulkan, which is the same range as the other types. The misses were near-ties.
I haven't run the UD-Q4_K_XL Flash-Next file itself, so if you try it, I'd like to hear the decode speed and whether the output looks right.
luckydevil13 commentedon Nov 6, 2026
@Vasyl-Trefilov I have 5060 ti + 7900 gre , is it possible to test as vulkan 0,1 backend? is it supported?
Vasyl-Trefilov commentedon Nov 6, 2026
@luckydevil13 Hi! Yes, v1.2.0 supports it:
strata bench --model <gguf> --gpu --backend vulkan --device 0,1Both cards need the same backend, so CUDA on one and Vulkan on the other isn't possible yet. The startup log (vulkan device 0: ...) shows which card is which. Layers split by free VRAM and the cards take turns, so it gives you more memory, not double speed. If the 7900 GRE is slow, run
echo high > /sys/class/drm/cardN/device/power_dpm_force_performance_levelas root. Nobody has tried NVIDIA + AMD in one split yet, so please if you can post the output, plus a single-card run of each.
And I will try to make possible to run on different backends soon, thanks for nice question!
luckydevil13 commentedon Nov 7, 2026
@Vasyl-Trefilov thanks, looks works (both gpus loaded), if need more tests let me know.
[root@BIG RustingStrata]# ./target/release/strata bench --model $M --gpu --backend vulkan --device 0,1
docs: add the living V100 roadmap (5 milestones, 36 tasks)