Code mode can help smaller LLM models

Code mode, which prompts LLMs to generate executable code instead of structured tool-call signatures, can help smaller models perform tool calling more reliably without additional training, according to a developer analysis. Anthropic uses the approach to reduce token spend, and Cloudflare introduced the concept when models struggled with tool-call formatting. The method requires ephemeral sandbox execution for safety, with providers like the author's spinning up in under 200ms.

It's been clear that LLMs alone are not as powerful as "LLMs with tools". How does LLM call tools? It can generate a signature of the tool call, somewhat like a function call search web "query" or bash "ls -l" etc. This usually calls a function written to either call the google API, exa, or execute a bash command locally. This is kind of fine as long as tool calls are properly formatted — some slightly less powerful models often make mistakes in generating those signatures. The training was so tedious that lot of models were released without the "tool calling" ability. We had to include training data involving special tokens, for example - |tool call| . How to fix it? We learned that the same less powerful models are still very good at generating code, since they are already trained on them. If we ask the LLM to generate say a python/typescript code which eventually calls the search API, bash etc then that has higher chances of success - especially when multiple tool calls are involved. One Issue Now we will need to execute code all the time, with this comes the risk of executing malformed code still happens with lower probibility — to execute each code we spin-up an ephemeral linux machine & execute the code there. There is not much overhead as most of the ephemeral sandbox providers spin-up under 200ms including us & you can just assume it will not be a noticeable delay. How is it used so far? Anthropic uses it to reduce token spends by a lot. See Programmatic Tool Calling https://platform.claude.com/cookbook/tool-use-programmatic-tool-calling-ptc . The code mode concept was probably introduced by Cloudfare https://blog.cloudflare.com/code-mode/ first when models weren't as good in writing tool calls. Today, both arguments make sense - double benefit. How does it relate to smaller models Microsoft recently said https://microsoft.ai/news/hill-climbing-mai-models-for-github-copilot-and-excel/ to use specially trained models for specific tasks — now they haven't said anything about code mode but my hunch is that code mode will certainly help if you don't want additional training to make the model learn tool calling. Along with helping you reduce the token count like always.