Making Local AI Tool Calls More Reliable
While testing our local-first AI assistant, I found an intermittent problem: the model sometimes answered with plain text instead of calling the tool needed to read or update local data.
The original system used tool_choice="auto"
. This normally worked, but it allowed the model to skip a required tool call. A prompt can guide a model, but it cannot guarantee that the model will always follow the tool protocol.
I fixed this by adding a safe recovery step. The first request still uses automatic tool selection, so normal conversation works as before. If an explicit local-data request returns no tool call, the assistant retries once with tool_choice="required"
. The retry happens only before any tool has run, which prevents duplicate database writes.
The assistant also checks whether the recovered tool belongs to the correct read or write group. During streaming, it buffers the first response so an incorrect, ungrounded answer is not shown before recovery completes.
I tested the change with automated regression tests and the running local Gemma model. The live recovery flow was:
auto -> required -> auto
The main lesson was simple: prompts describe expected behavior, but reliable agent systems also need program-level checks around model decisions.