Weak models write tool calls as prose. We started executing them

FIIT.AI added a runtime step that extracts prose-shaped tool calls from weak models' message bodies and converts them into real tool_calls, after its didNothing guard failed to fire on a turn where an agent wrote functions.fiitx_affiliate_ops({"operation":"target_invite","commit":true}) inside a fenced code block instead of the tool_calls channel, resulting in zero tool calls and no messages sent. The original guard keyed on zero tool calls AND an empty body, so the non-empty body let the turn close normally and the UI render "done"; a nudge-and-retry fix also failed four days later on a different agent, where the model repeated the same prose call across three "continue" presses with zero executions. The new step runs extracted calls through the ordinary path with the same preflight, approval gates, and execution, restricted to tools declared that turn, skipping placeholder arguments, taking only the first occurrence, and still requiring human approval for anything that sends, posts, pays, or deletes.

← FIIT.AI https://fiit.ai/ Weak models write tool calls as prose. We started executing them. A user told one of our agents to send a batch of outreach messages. The agent replied with this: functions.fiitx affiliate ops {"operation":"target invite","commit":true} — inside a fenced code block, in the body of the message. Not in the tool calls channel. The runtime only executes tool calls , so the turn made zero tool calls. Nothing was sent. The interesting part is what our own guard did. We had a check for exactly this class of dishonesty — an agent that finishes a turn without doing anything. It fired on didNothing : zero tool calls and an empty body. Here the body was not empty. It was full of text that looked like a successful execution. So the guard stayed quiet, the turn closed normally, and the UI rendered “done”. The user saw a code block that looked like a receipt, believed the invites had gone out, and only found out on the next turn when they asked why nobody had replied. The guard was testing the wrong thing zero calls AND empty body encodes an assumption: that a model which did nothing will also say nothing. That assumption is wrong for the models where it matters most. A model good enough to produce well-formed arguments but not reliable enough to emit them through the function-calling channel will happily narrate the call instead — and the narration is the most confident-looking text in the whole transcript. The fix for detection was to stop guessing at shape and start keying on identity. For each turn we already build the list of tools we declared to the model. We now keep that list and look for a declared tool name in the body followed immediately by { or " . Mentioning a tool by name in a sentence does not trip it. A normal function call in a code sample does not trip it either, because the names never match. Retrying was not enough The first version just pushed back: a nudge telling the model it had not actually called anything, try again. That works on capable models. It does not work on the ones that produce this failure in the first place. We hit it again four days later on a different agent. The nudge fired, the model got the message, and on the next turn it wrote the same call into the body again. The user pressed “continue” three times and got three identical blocks of text and zero executions. At that point the argument for materialising the text became hard to ignore. The model had already done the hard part. The arguments were complete and correct. It had picked the right tool. The only thing wrong was the transport. Refusing to honour it was costing the user a working turn to protect a purity that bought nothing. So we added a step before the assistant message is pushed onto the transcript: extract prose-shaped calls, convert them into real tool calls on that assistant message, and let them run through the ordinary path — same preflight, same approval gates, same execution. Nothing bypasses anything. The assistant message ends up carrying the calls it should have emitted, which also keeps the following role: "tool" messages structurally valid. What keeps this from being reckless Materialising text into side effects deserves suspicion. The constraints that make it safe: - Only tools declared this turn. Same identity check as the detector. The model cannot invent a tool by writing its name. - A plan is not a call. If the arguments contain placeholders — /path/to/… , <your-token , “replace this with…” — it is the model describing what it would do, not asking for it to happen. Those are left to the nudge. - First occurrence only, deduplicated. When a model writes three calls in sequence, the later ones usually depend on the output of the earlier ones, which does not exist yet. - Approval still applies. Anything that sends, posts, pays or deletes stops for a human exactly as it would have through the normal channel. Materialising changes the transport, not the policy. That placeholder rule has its own scar. An early version of the regex treated the bare word your as a placeholder marker, and a poster-generation call whose prompt contained the phrase “YOUR AI WORKFORCE” got classified as a plan and silently dropped. The pattern now requires the connected forms — your api key , your-token — because a space-separated “your” is just English. Three shapes so far Once we started looking, the same failure showed up wearing different clothes. 1. The function-call shape. functions.NAME {...} in a fenced block. The original. 2. The shell shape. Three commands in a bash block, with prose saying “task started, running silently”. Zero calls, three turns in a row. Our extractor only knew the NAME { form, so it saw nothing. 3. The XML shape. <function name="…" wrapping a JSON blob, or a bare json block near the top of the reply. The shell case also exposed a second bug in the honesty check. It had an exemption for replies that end in a question — added earlier so that a legitimate clarifying question would not be flagged as a placeholder answer. The model ended with “want me to monitor progress in real time?” and collected the exemption. Another exemption keyed on the words “fail” and “error”, on the theory that a genuine failure report mentions them; the reply contained “if any platform fails” and collected that one too. Both exemptions now lose to an explicit claim of execution. If the text asserts that work is underway — running, started, downloading, in the background, please wait — there is no exemption. Weak assertions “I will now…” keep theirs. When it still refuses If the guards run out and the model has still not made a real call, we append to its reply rather than replacing it: Nothing was executed in this turn. Append, never replace. An earlier version of a different guard replaced bodies it judged to be placeholders, and it occasionally ate a real answer. The rule now is that the model’s text is never deleted — a note is added underneath it, and the note is the only thing we control. What we would tell someone building this - Do not define “did nothing” as “said nothing”. The expensive failures are the ones with a confident paragraph attached. - Key your detectors on identity, not shape. The tool names you declared this turn are ground truth you already have. Shape heuristics chase an infinite tail. - Distinguish a plan from a call before you act on either. The difference is placeholders in the arguments, and it is worth getting the regex exactly right. - Treat a claim of execution as a load-bearing assertion. “Running in the background” with zero calls is not a formatting quirk; it is the failure. - Model tier predicts this. We have seen it from mid-tier models, and most often when a stronger pinned model was unavailable and the router fell back. A fallback that silently downgrades capability will manufacture this class of bug. Pin to something with credit on it. Both behaviours have regression tests whose fixtures are the verbatim transcripts from the days they bit us. That is the only kind of fixture we trust for this — a synthetic example tests the regex you already wrote. We are FIIT.AI https://fiit.ai/ . We build AI employees that run on your own machine and inside the systems a company already uses — the desktop client is called Fiitx. The failures above are from our own production harness.