I have had good results with small language models (such as 1.2B parameters) for interacting with my MCP server. I decided to try to see how small I could go—if 1.2B worked, would 350M work, would 230M? Each size decrease means more efficiency.
I have found LiquidAI's small models to be quite effective at calling tools. I could use the 1.2B parameter model with minimal changes to the MCP server code. With 350M, I had to redesign a feature and use a different approach, but it eventually worked as well.
Some notes:
• While 2.6B (`LFM2.5-2.6B-QAD-Q4_0.gguf`
) and 1.2B (`LFM2.5-1.2B-Instruct-QAD-Q4_0.gguf`
) work well with minimal modifications, 350M (`LFM2.5-350M-QAD-Q4_0.gguf`
) required fairly extensive code changes and rewrites.
• 230M (LFM2.5-230M-QAD-Q4_0.gguf
) does not appear viable for my use—it is simply too small.
• The performance improvements with small models are astonishing—a 1.2B model can do 300 tokens a second, while a 350M model can do 500 tokens a second.
How can one build a program that works well with a 350M model? Basically I recommend designing it from scratch to work with the model—test each feature, and only add features that the model can handle. A 350M parameter model has limitations; it is possible to build up a program that works with it though.