Small Language Models (350M Parameters)

LiquidAI's 350M-parameter small language model required extensive code rewrites but achieved 500 tokens per second, compared to 300 tokens per second for the 1.2B model, according to a developer's tests with MCP servers. The 230M model proved too small for the use case, while the 1.2B and 2.6B models worked with minimal modifications.

I have had good results with small language models such as 1.2B parameters for interacting with my MCP server. I decided to try to see how small I could go—if 1.2B worked, would 350M work, would 230M? Each size decrease means more efficiency . I have found LiquidAI's small models to be quite effective at calling tools. I could use the 1.2B parameter model with minimal changes to the MCP server code. With 350M , I had to redesign a feature and use a different approach, but it eventually worked as well. Some notes: • While 2.6B LFM2.5-2.6B-QAD-Q4 0.gguf and 1.2B LFM2.5-1.2B-Instruct-QAD-Q4 0.gguf work well with minimal modifications, 350M LFM2.5-350M-QAD-Q4 0.gguf required fairly extensive code changes and rewrites. • 230M LFM2.5-230M-QAD-Q4 0.gguf does not appear viable for my use—it is simply too small. • The performance improvements with small models are astonishing—a 1.2B model can do 300 tokens a second, while a 350M model can do 500 tokens a second. How can one build a program that works well with a 350M model? Basically I recommend designing it from scratch to work with the model—test each feature, and only add features that the model can handle. A 350M parameter model has limitations; it is possible to build up a program that works with it though.