So, I am not ready to trust Chinese models yet, and have thus been keeping things western.
I have Gemma4:26b running super fast on my GPU (2000+ tokens/s pre-fill, and 80+ tokens/s evaluation output. I generally use it for web assisted research and problem solving.
Then I have a secondary model (Laguna S 2.1 118B) running on my CPU on my server. Normally this would be extremely frustrating, but with 8 channels of DDR4, I get a total memory bandwidth pretty close to the likes of an Nvidia Spark or a Strix 395+.
CPU’s can’t keep up with the matrix math of even a low end GPU, so pre-fill can be a little slow, but eval output actually isn’t bad on the CPU, with some 13-14 tokens/s
I generally use the heavier Laguna logic model to do things like code review or general purpose reasoning where Gemma4 can’t keep up, and I have spare time to run it in the background.
I’d be curious to hear what others may be using.
I am looking for better models for my use case, but right now I feel stuck.
I am finding that the modern 30B-class models - while excellent for their size still don’t perform anywhere near as well as a good 120B-class model like Laguna S 2.1
My main gripe is that there is very little in-between the 30B-class and the 120B-class. The 30B is kind of small, using only ~27GB of my 64GB GPU RAM with 8bit quantization.120B-class is too large to fit in my 64GB of VRAM even at 4 bit quantization, and thus I can’t benefit from GPU acceleration.
I would really love a modern MoE model in the 55B-75B range which I could run at 4-6bit quantization and make the most out of my VRAM with, but there just isnt much there right now.
I did briefly run the old llama 3.3B 70B model, but it is actually out-performed logic wise by the newer generation of 30B-class models. I just can’t help but wonder how good this new generation of models could be if they could have weights in that size class.
I’m hoping someone comes up with a model like that in the not too distant future.