# So, what local inference models are we using?

> Source: <https://forum.level1techs.com/t/so-what-local-inference-models-are-we-using/254516#post_1>
> Published: 2026-08-27 16:09:55+00:00

So, I am not ready to trust Chinese models yet, and have thus been keeping things western.

I have Gemma4:26b running super fast on my GPU (2000+ tokens/s pre-fill, and 80+ tokens/s evaluation output. I generally use it for web assisted research and problem solving.

Then I have a secondary model (Laguna S 2.1 118B) running on my CPU on my server. Normally this would be extremely frustrating, but with 8 channels of DDR4, I get a total memory bandwidth pretty close to the likes of an Nvidia Spark or a Strix 395+.

CPU’s can’t keep up with the matrix math of even a low end GPU, so pre-fill can be a little slow, but eval output actually isn’t bad on the CPU, with some 13-14 tokens/s

I generally use the heavier Laguna logic model to do things like code review or general purpose reasoning where Gemma4 can’t keep up, and I have spare time to run it in the background.

I’d be curious to hear what others may be using.

I am looking for better models for my use case, but right now I feel stuck.

I am finding that the modern 30B-class models - while *excellent* for their size still don’t perform anywhere near as well as a good 120B-class model like Laguna S 2.1

My main gripe is that there is very little in-between the 30B-class and the 120B-class. The 30B is kind of small, using only ~27GB of my 64GB GPU RAM with 8bit quantization.120B-class is too large to fit in my 64GB of VRAM even at 4 bit quantization, and thus I can’t benefit from GPU acceleration.

I would really love a modern MoE model in the 55B-75B range which I could run at 4-6bit quantization and make the most out of my VRAM with, but there just isnt much there right now.

I did briefly run the old llama 3.3B 70B model, but it is actually out-performed logic wise by the newer generation of 30B-class models. I just can’t help but wonder how good this new generation of models could be if they could have weights in that size class.

I’m hoping someone comes up with a model like that in the not too distant future.