You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
hm, I took a look and compared it with Qwen3.6:35b-a3b and it looks interesting, bc gemma is same smart, but has less params.
Good Idea! I will think about it a bit more and write you back in few hours, bc supporting new architecture is not so easy, but if you want and I am interested in it, maybe this will be possible. I will tag you in the evening. but I think yes, but I will think
@beyondzvincent yes, I like your Idea very much, I will add at least "gemma4 26b a4b", it's not so hard as I though, bc it's close to what I already have, so maybe in 1-2 hours I will be done and you can try it out
For reference, using PolyStrata on an RTX 4070 12 GB with 32K context and MTP enabled, I get up to ~140 tok/s (tg) and ~2700 tok/s (pp). Even accounting for the 4070's higher memory bandwidth and MTP, switching from Q5 to QAT Q4 + enabling MTP/expert caching should give your 3060 a noticeable boost.
Activity
Vasyl-Trefilov commentedon Nov 10, 2026
hm, I took a look and compared it with Qwen3.6:35b-a3b and it looks interesting, bc gemma is same smart, but has less params.
Good Idea! I will think about it a bit more and write you back in few hours, bc supporting new architecture is not so easy, but if you want and I am interested in it, maybe this will be possible. I will tag you in the evening. but I think yes, but I will think
Vasyl-Trefilov commentedon Nov 10, 2026
@beyondzvincent yes, I like your Idea very much, I will add at least "gemma4 26b a4b", it's not so hard as I though, bc it's close to what I already have, so maybe in 1-2 hours I will be done and you can try it out
Vasyl-Trefilov commentedon Nov 10, 2026
@beyondzvincent It's implemented, but my RustingStrata is only a bit faster, Q5 gemma is 24–28 tok/s and llama.cpp is 22.1 tok/s on my hardware
Anc813 commentedon Nov 13, 2026
@Vasyl-Trefilov
You should try the QAT version (UD-Q4_K_XL) instead of Q5 — since it's native Q4, quality is on par with Q5, while the file size drops to ~14.2 GB (meaning much less offloading to system RAM on a 12 GB card):
https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF
For reference, using PolyStrata on an RTX 4070 12 GB with 32K context and MTP enabled, I get up to ~140 tok/s (tg) and ~2700 tok/s (pp). Even accounting for the 4070's higher memory bandwidth and MTP, switching from Q5 to QAT Q4 + enabling MTP/expert caching should give your 3060 a noticeable boost.
Vasyl-Trefilov commentedon Nov 13, 2026
@Anc813 ty, I will try and tag you when I will make some benchmarks