You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi @luckydevil13 ! There are few problems and reasons why no or not yet.
First: goal of this project is "Strata for all Qwen models"
Second: current best deepseek V4.1 has 552b Total params, 8B active for prefill and 16B for answer and this is make deveplloping and testing this model way harder then small Qwen models
Third: Qwen3.8 Flash Next is better or equal to Deepseek, even tho QFN is 3 times smaller!
So what is the point of deepseek?
I think about adding 1-2 big models, I still think which one, but I guess it will be not deepseek, if you got any other suggestions, I will be happy to hear it!
@luckydevil13 Hi, I will take a look in this project and think about deepseek again and write you in 30-50min back.
Also about your idea, ofc you can try it with Sol6.1, just create PR and I will review it after you finish, to make sure that everything is correct.
Thanks for idea again!
@luckydevil13, I took a look at it and also searched for some other bigger models that can fit in your ram and can actually make sense and I don't think that anything will make sense, maybe only GLM-5.3-Flash. If you like this model, I can try implement it
@luckydevil13 I hope you are happy, bc I am adding DeepSeekV4.1 1017 in Strata, I will tag you when it's ready, it can take few days, since it has different architecture from Qwen models
@luckydevil13 also GLM5.3 Flash support will be released today, with your 128GB ram Q2-Q3 GLM5.3 Flash will be way better then DeepSeekV4.1 1017. For now it perform 3-4 times faster then on llama.cpp, but maybe this is bc I need to read weights from SSD, bc I don't have so much RAM as you do. But you can expect like 8-15t/s which is pretty nice for 330B model
Huge thanks,
But seems not works for Vulkan backend . catch CUDA :(
[root@BIG RustingStrata]# ./target/release/rustingstrata generate --model $M --gpu --backend vulkan --device 0,1 --prompt "The capital of France is"
cuda device 0: NVIDIA GeForce RTX 5060 Ti, sm_120
cuda shared memory: opt-in 101376 B per block, attn_decode_gqa*: 2112 B static, dynamic max 73728 B, carveout default
cuda shared memory: opt-in 101376 B per block, attn_prefill_*: 0 B static, dynamic max 72704 B, carveout default
cuda shared memory: opt-in 101376 B per block, gemvn_*: 0 B static, dynamic max 101376 B, carveout default
Paris. The quick brown fox jumps over the lazy dog^C
[root@BIG RustingStrata]# ./target/release/rustingstrata -V
rustingstrata 1.2.1
Activity
Vasyl-Trefilov commentedon Nov 7, 2026
Hi @luckydevil13 ! There are few problems and reasons why no or not yet.
First: goal of this project is "Strata for all Qwen models"
Second: current best deepseek V4.1 has 552b Total params, 8B active for prefill and 16B for answer and this is make deveplloping and testing this model way harder then small Qwen models
Third: Qwen3.8 Flash Next is better or equal to Deepseek, even tho QFN is 3 times smaller!
So what is the point of deepseek?
I think about adding 1-2 big models, I still think which one, but I guess it will be not deepseek, if you got any other suggestions, I will be happy to hear it!
Vasyl-Trefilov commentedon Nov 7, 2026
and btw I will add you in THANKS.md, since you helped with benchmarks and also Ideas!
luckydevil13 commentedon Nov 8, 2026
@Vasyl-Trefilov take a look at https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF , they say With Strata, every build runs on a single GPU with 12 GB of VRAM. See Usage.
I like lightweght of this project.
BTW I have 128Gb ram ) It's hard to add ds4 support? Could I try to do it with Sol 6.1? how did you think?
Vasyl-Trefilov commentedon Nov 8, 2026
@luckydevil13 Hi, I will take a look in this project and think about deepseek again and write you in 30-50min back.
Also about your idea, ofc you can try it with Sol6.1, just create PR and I will review it after you finish, to make sure that everything is correct.
Thanks for idea again!
Vasyl-Trefilov commentedon Nov 8, 2026
@luckydevil13, I took a look at it and also searched for some other bigger models that can fit in your ram and can actually make sense and I don't think that anything will make sense, maybe only GLM-5.3-Flash. If you like this model, I can try implement it
Vasyl-Trefilov commentedon Nov 8, 2026
Also MiniMax M3.1 Flash looks interesting
Vasyl-Trefilov commentedon Nov 8, 2026
But I think we need just wait from Qwen4 models that will come out very soon
docs: start M1-T3 (issue #12)
[#12] ldmatrix fallback for attention prefill (M1-T3) (#13)
Vasyl-Trefilov commentedon Nov 8, 2026
@luckydevil13 I hope you are happy, bc I am adding DeepSeekV4.1 1017 in Strata, I will tag you when it's ready, it can take few days, since it has different architecture from Qwen models
Vasyl-Trefilov commentedon Nov 9, 2026
@luckydevil13 hey, it's implemented, I got Reddit post about it https://www.reddit.com/r/LocalLLaMA/comments/1xr2v59/comment/19scehy72txfl/?screen_view_count=3 I tested Deepseek only on one quant, but it must work on others too, so you can try it out, I will be very happy to get some feedback from you
Vasyl-Trefilov commentedon Nov 10, 2026
@luckydevil13 also GLM5.3 Flash support will be released today, with your 128GB ram Q2-Q3 GLM5.3 Flash will be way better then DeepSeekV4.1 1017. For now it perform 3-4 times faster then on llama.cpp, but maybe this is bc I need to read weights from SSD, bc I don't have so much RAM as you do. But you can expect like 8-15t/s which is pretty nice for 330B model
luckydevil13 commentedon Nov 11, 2026
Huge thanks,
But seems not works for Vulkan backend . catch CUDA :(
Vasyl-Trefilov commentedon Nov 11, 2026
@luckydevil13 yes, Vulkan was not supported, but now it is, so pls pull from main and try again
Vasyl-Trefilov commentedon Nov 11, 2026
@luckydevil13 Also, what do you think about GLM5.3 Flash as I said above, this should looking interesting for your hardware