Last night I spent some time trying to make my MCP server work with a smaller local model, one that is just 1.2 billion parameters and 696 MB on disk. This is LiquidAI's LFM 2.5 1.2B, with file name LFM2.5-1.2B-Instruct-QAD-Q4_0.gguf
.
Most small models I test fail too badly and I just delete them right away. However this model seemed to have some potential to work for my agentic system. It required some changes in the MCP server.
• I had to add more examples and details about how to respond on certain user queries.
• I had to remove a confusing tool (which it did not seem to understand, and would randomly call just to annoy me).
So basically I have a system with just 1 tool that can be called in many different ways. The small model thus does not get confused about what it is supposed to do. It still gets confused by certain things: it has trouble keeping track of who said what, and what certain words mean in a phrase. However it runs at over 300 tokens per second, uses minimal RAM, and will make it so I basically will never need to update my GPU to run local agentic tasks (it even will run speedily on CPU alone).