Tool calling for small on-device LLMs that don't support tool calling

A developer built prompt-based tool calling for small on-device LLMs running in the browser via transformers.js or WebLLM, which return text only and lack a tool-calling API, after LangChain's tool-calling agents refused local models with the error "bind_tools() must be implemented." The approach renders OpenAI-style tool definitions into a system prompt and asks models to reply in <tool_call> JSON blocks, then parses the output with a list of regex patterns covering tag drift, Mistral's [TOOL_CALLS] arrays, Llama's <function=name> syntax, fenced JSON, nested stringified arguments, <think> reasoning blocks and truncated replies. The parser pairs those patterns with a strict filter so the agent accepts the formats small models actually emit without executing tools that were never offered.

You attach a few tools to an AI agent, pick a small model that runs on your own machine, and press run. With a cloud model this is routine. With a model running in the browser, the agent refuses to start. In AWFlow you can build agents: an LLM node with tools attached search the page, call a sub-workflow, query a vector store . With OpenAI or Anthropic you pass tool schemas and get structured tool calls back. With a model running in the browser transformers.js or WebLLM , you don’t. Those runtimes give you text in and text out. LangChain’s tool-calling agents check for bindTools and refuse local models outright: Error: bind tools must be implemented So I implemented it with prompt-based tool calling. Prompting the model is the easy part. The interesting part is parsing what it actually sends back , because every model family answers in the format it was trained on, whatever you asked for. How the agent itself is wired the graph, the loop, the pause is covered in LangChain and LangGraph in the browser https://awflow.io/blog/langchain-and-langgraph-in-the-browser-how-an-ai-agent-builds-your-workflow . This post zooms into one piece of it: tool calls on models that have no tool-calling API. Describe the tools in the prompt The tools are converted to OpenAI-style definitions name, description, JSON Schema parameters and rendered into a system-prompt section, one JSON object per line: Tools You can call the following tools. Each is described as JSON name, description, JSON Schema parameters : {"name":"search page","description":"Find text on the current page","parameters":{...}} {"name":"save row","description":"Append a row to the sheet","parameters":{...}} To call a tool, reply with exactly this, one block per call, and nothing else: <tool call {"name": "<tool name ", "arguments": {<arguments matching the parameters }} </tool call Only call the tools listed above. You will receive each result in a <tool response block. When you have what you need, answer the user in plain text, without a tool call. Earlier tool calls in the conversation are re-rendered in that same <tool call format, and results go back wrapped in <tool response . That way the transcript keeps showing the model the format you want. I picked <tool call tags because many open models Qwen-style chat templates, for one were trained on something close to it. But “close” is the key word. Accept what models actually write Here’s what small models really return when asked for that format: <tool call {"name": "search page", "arguments": {"query": "price"}}</tool call as asked <function call {"name": "search page", ...}</function call tag drift TOOL CALLS {"name": "search page", "arguments": {...}} Mistral style <function=search page {"query": "price"}</function Llama style json {"name": "search page", "parameters": {"query": "price"}} fenced JSON {"function": {"name": "search page", "arguments": "{\"query\":\"price\"}"}} nested + stringified args …plus <think …</think reasoning blocks in front, and replies cut off at the token limit before the closing tag. Rejecting all of that gives you an agent that “never calls tools”. Accepting all of it naively gives you an agent that runs things nobody offered it. So the parser is a list of patterns plus a strict filter: js const CALL FORMATS = // <tool call …</tool call / <function call / <tools ; closing tag optional truncation { pattern: /< ?:tool call|function call|tools \s \s\S ? \s ?:<\/ ?:tool call|function call|tools |$ /g }, // TOOL CALLS … — array runs to the end { pattern: /\ TOOL CALLS\ \s \ \s\S \ /g }, // <function=name {…}</function — the name lives outside the JSON { pattern: /<function= \w.- + \s \s\S ? \s ?:<\/function |$ /g, nameGroup: 1 }, // json {…} { pattern: / ?:json|tool call ?\s { \s\S ? \ } \s /g }, ; Each match goes through one normaliser that accepts the usual spellings: - name: name or function.name - arguments: arguments , parameters , args or input , as an object or a JSON-encoded string - a single call or an array of calls function toRawCall value: unknown, forcedName?: string { if isPlainObject value return undefined; const fn = isPlainObject value.function ? value.function : value; const name = forcedName ?? fn.name; if typeof name == 'string' || name return undefined; // With <function=name {…}, the object IS the arguments. let args = forcedName ? value : fn.arguments ?? fn.parameters ?? fn.args ?? fn.input ?? {} ; if typeof args === 'string' args = tryParseJson args ; if isPlainObject args return undefined; return { name, args }; } Keep the leniency safe A lenient parser is only useful if it can’t be talked into running things. Here is the path a reply takes, step by step. Reasoning is stripped first <think …</think blocks are removed including an unclosed one before parsing, so a model that thinks about calling a tool doesn’t actually call it. Shapes are matched, not model names Each pattern runs over the reply. A closing tag is optional, because replies often stop at the token limit before it. Keys are normalised The name and arguments are read in whichever spelling the model used, and string arguments are parsed as JSON. Only offered tools count A block is accepted only if every call in it names a tool that was actually bound. A model that invents delete all rows gets nothing. Nothing is silently dropped If a block looks like a call but is malformed or names an unknown tool, it stays in the prose. The user sees exactly what the model said instead of an empty reply. Last resort: if nothing matched and the whole reply is JSON that parses as calls, accept that too. Some small models just answer {"name": …} with no wrapper. Here is what that means for a few real-looking replies: | Model reply | |---| | <tool call search page | | <function=search page | | call: delete all rows | | <think search?</think | | {"name": "search pa | Tool-call parser pure function | Model reply | Result | |---|---| | <tool call search page | call: search page | | <function=search page | call: search page | | call: delete all rows | kept as text | | <think search?</think | no call | | {"name": "search pa | kept as text | Only real calls to offered tools become calls.An unknown tool or malformed JSON stays in the prose; reasoning never triggers a call. Text version 1. Model reply: <tool call search page · <function=search page · call: delete all rows · <think search?</think · {"name": "search pa 2. Tool-call parser 3. Model reply | Result: <tool call search page | call: search page · <function=search page | call: search page · call: delete all rows | kept as text · <think search?</think | no call · {"name": "search pa | kept as text Why model-agnostic matters My first version special-cased the model I was testing with. It worked great until the next model. A user can add chat models from the Hugging Face Hub to the extension any with a browser-runnable ONNX build , so the parser can’t know the family. Matching on shapes instead of model names means any model that writes one of these formats works, with no per-family lookup table. Even the prompt itself is checked by a test to make sure it never names a model family. It’s also a pure function : string in, { content, toolCalls } out. So every accepted format, malformed call and edge case has a unit test, with no model in the test run. Does it actually work? With 1–4B models, for agents with a few well-described tools: yes, well enough for things like “find the price on this page and save it to the sheet”. Where it breaks down: - many tools with overlapping descriptions the model picks the wrong one - long multi-step plans it loses the thread after a few calls - deeply nested argument schemas For those, the same agent node can switch to a cloud model or a bigger local one via Ollama Ollama with workflow automation https://awflow.io/blog/ollama-workflow-automation-local-llm . Same workflow, different brain. If you are choosing an on-device model in the first place, how to choose, install and compare a local AI model https://awflow.io/blog/how-to-choose-install-and-compare-a-local-ai-model walks through it. If you’re doing this yourself: If you’ve seen tool-call formats I haven’t covered, I’d like to see them in the comments. The agents described here are part of AWFlow, a free browser extension for workflows and AI agents that runs in your browser.