Small LFM Models and Quantization

A developer got the 230M-parameter LiquidAI LFM2.5-230M-Q5_K_M.gguf model working with a local MCP server after switching from 4-bit to 5-bit quantization, following an earlier setup that ran the 350M-parameter LFM2.5-350M-QAD-Q4_0.gguf model. The 5-bit 230M model is faster and smaller than the 4-bit 350M model, reaching 500-1000 tokens per second, and the developer noted small models are more sensitive to quantization and that the 230M model could not handle basic syntax at 4 bits. The developer recommends LiquidAI 2.5 LFM models for simple tool-calling agents on low-end hardware.

I have a local MCP server that mostly just calls a handful of functions. I had managed to get it working with a 350 million parameter model LFM2.5-350M-QAD-Q4 0.gguf , but could not go any lower. However I spent some time today to improve the code and now the 230M model LFM2.5-230M-Q5 K M.gguf works. The main thing I needed to do was change from 4-bit quantization to 5-bit quantization . It is a known issue that small models are more sensitive to quantization; and the 230M model seemed like it couldn't handle basic syntax at 4-bits. The 5-bit version of the 230M model is faster and smaller than the 4-bit version of the 350M model. I can even get 500-1000 tokens per second on it, which is nearly instant. For simple tool-calling agents the LiquidAI 2.5 LFM models are a good choice. These are models that can be run on low-end hardware or really any hardware . And don't give up on the smallest model—the 230M model seems to be useful but apparently needs 5 bits per weight.