{"slug": "small-lfm-models-and-quantization", "title": "Small LFM Models and Quantization", "summary": "A developer got the 230M-parameter LiquidAI LFM2.5-230M-Q5_K_M.gguf model working with a local MCP server after switching from 4-bit to 5-bit quantization, following an earlier setup that ran the 350M-parameter LFM2.5-350M-QAD-Q4_0.gguf model. The 5-bit 230M model is faster and smaller than the 4-bit 350M model, reaching 500-1000 tokens per second, and the developer noted small models are more sensitive to quantization and that the 230M model could not handle basic syntax at 4 bits. The developer recommends LiquidAI 2.5 LFM models for simple tool-calling agents on low-end hardware.", "body_md": "I have a local **MCP server** that mostly just calls a handful of functions. I had managed to get it working with a **350 million** parameter model (`LFM2.5-350M-QAD-Q4_0.gguf`), but could not go any lower. However I spent some time today to improve the code and now the 230M model (`LFM2.5-230M-Q5_K_M.gguf`) works.\n\nThe main thing I needed to do was change from **4-bit** quantization to **5-bit quantization**. It is a known issue that small models are more sensitive to quantization; and the 230M model seemed like it couldn't handle basic syntax at 4-bits.\n\nThe 5-bit version of the 230M model is **faster and smaller** than the 4-bit version of the 350M model. I can even get 500-1000 tokens per second on it, which is nearly instant.\n\nFor simple **tool-calling agents** the **LiquidAI 2.5 LFM models** are a good choice. These are models that can be run on low-end hardware (or really any hardware). And don't give up on the smallest model—the 230M model seems to be useful but apparently needs 5 bits per weight.", "url": "https://wpnews.pro/news/small-lfm-models-and-quantization", "canonical_source": "https://www.dotnetperls.com/2026_9_10_small-lfm-models-quantization", "published_at": "2026-09-10 07:00:00+00:00", "updated_at": "2026-09-10 22:18:18.483243+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["LiquidAI", "LFM2.5-230M-Q5_K_M.gguf", "LFM2.5-350M-QAD-Q4_0.gguf", "MCP server"], "alternates": {"html": "https://wpnews.pro/news/small-lfm-models-and-quantization", "markdown": "https://wpnews.pro/news/small-lfm-models-and-quantization.md", "text": "https://wpnews.pro/news/small-lfm-models-and-quantization.txt", "jsonld": "https://wpnews.pro/news/small-lfm-models-and-quantization.jsonld"}}