Now that I have a custom MCP server running locally, I need an LLM (running in llama-cpp) to control it. I have found most any model will work; the main challenge is finding the fastest, smallest, and most accurate one. This brings us to the age-old question of choosing a local LLM: is quantization level or parameter count more important?
If I allocate a budget of 300 MB for the model, I can use either a model like LFM 2.5 350M (which is 219.3 MB) or Bonsai 1.7B (which is 248.3 MB). Both models work and can call tools correctly. So which is better? I guess this question comes down to testing and preferences. The LFM model is faster; I can get over 500 tokens per second. Meanwhile Bonsai 1.7B is closer to 250 tokens per second. I feel like Bonsai 1.7B tends to be more reliable—occasionally the 350M model can make some sort of mistake that slows me down (like an incorrect tool argument or something). So at least for myself, I feel like models with more parameters that are quantized down are a better option, due to increased reliability; but other use cases may favor the lower-parameter models (they are faster).