I have mostly concluded my efforts in building (and optimizing) a local MCP server for llama-cpp
. Making things fast is really important when designing agentic AI systems. Can you imagine if a chat bot took an hour to return a response? No one would use it—so being fast is important.
Some optimizations I made:
• Used Stdio instead of HTTP to reduce overhead in MCP tool calls.
• Switched to a faster model—with LFM2.5-2.6B-Q4_K_M.gguf
I can get 146 tokens per second and still have fairly correct tool calls.
• Various Rust optimizations (reducing allocations, caching structs for re-use) improved performance for some tasks from 1 millisecond to 15 microseconds (I mostly used Microsoft Copilot).
• Removed unnecessary words in tool call descriptions (removed "the" words) for about 7% prompt size reduction.
• Merged similar tool calls (insert_file
, insert_image
) for additional prompt size minimization.
My AI agent can handle most requests in 1-3 seconds (including the actual MCP tool calls). It remains to be seen how useful my AI agent system is, but at least it has some potential to be helpful (a slow system would never be helpful).