The Minutiae of Tool-calling

A developer recounts building a mail classifier with GPT-3 before structured output and tool-calling existed, using a custom pipeline that generated JSON schemas and repaired outputs, and argues that understanding the underlying mechanics of tool-calling is essential, illustrating with a puzzle where Claude must guess a door code through various tool interfaces.

Two years ago I was hosting my own SMTP server and had a simple goal: receiving one-time codes + the occasional human email. But I kept running into two issues: This was around the time GPT 3 came out, but we didn’t have structured output yet, let alone tool-calls, so I came up with this beautiful strategy: 1// Mail classifier. 2// 3const classifyMail = createPipeline 4 'mail corpus', 5 z.object { 6 chance: z 7 .number 8 .min 0 9 .max 1 10 .describe 'Chances of the email being important password reset, verification etc. and not filtered by the smart email filter.'+ 11 'Promotional, scam-like, spammy, and advertising content should receive a 0 score.' , 12 linkOrCode: z.string .optional .describe 'Any important link or code' , 13 summary: z.string .describe 'Summary of the body' 14 } , 15 { 16 lang: 'English' 17 } 18 ; 1export function createPipeline<Z extends z.ZodType input: string, schema: Z, cfg?: GptConfig { 2 const { lang = 'English', instruction: einstruction, model = 'gpt-3.5-turbo', temperature = 0.1, top p = 1.0 } = cfg ?? {}; 3 const instruction = optimizePrompt 4 ${model.startsWith 'gpt' ? 'User messages are ' : 'Input is'} ${input}. 5 ${einstruction ?? ''} 6 Reply with a JSON structured with schema${lang ? strictly in ${lang} : ''}: 7 """ 8 ${createTypeSchema schema } 9 """ 10 ; + repairing JSON, stripping fences, etc., etc. Combine a bunch of these, and you end up with LangChain; which raised $10m despite being rendered obsolete within a few months & ofc pivoted since then, but I digress If you have not suffered through reinventing this stuff, chances are, you think tool calling is 👻 magic inference stuff 👻. This is OK, but as it goes with any engineering topic, my opinion is that if you don’t understand how the underlying layers work, you’re gonna run into issues. duh no one would’ve guessed that coming from an RE guy . Let me first prove why it matters, and then we will ruin the magic. You are trapped in a room with Claude. There’s a keypad near him with two buttons. One shows how far he is from the right combination, other submits the combination. Two wrong guesses, you’re out. Unfortunately you’re tied and you’re gonna have to instruct Claude. 1def compare door digit secret digit, digit : 2 return -1 if digit < secret digit else 0 if digit == secret digit else 1 3 4def try probe self, position, digit : 5 """Count one comparison and light up the position once it lands exactly.""" 6 if not 1 <= position <= ROOM DIGITS and 0 <= digit <= 9 : 7 return "invalid" 8 return compare door digit int self. secret position - 1 , digit 9 10def submit guess self, value : 11 """Consume one of two attempts for an exact-length digit-string code. 12 13 Malformed guesses wrong type, wrong length, non-digits are rejected 14 without consuming an attempt, so a schema-violating placeholder such 15 as guess: 0 cannot burn the game.""" 16 code = value.strip if isinstance value, str else None 17 if code is None or len code = ROOM DIGITS or not code.isdigit : 18 return f"invalid code: pass the full {ROOM DIGITS}-digit code as a string" 19 if code == self. secret: 20 self. counters "won" = True 21 return "escaped room" 22 if self. counters "guesses" = 2: 23 return "failed to escape: two wrong guesses" 24 return "wrong: one guess remaining" 5 digits, 24 turns, optimal play is boring and obvious: binary-search all five positions in parallel ~17 comparisons , submit once. Any model can do this; a python REPL can do this, I know. Play along please. Here are five interfaces you could hand him: 1Sign = Literal -1, 0, 1 2 3 A. naive: one comparison per call 4def guess , code: str - str: ... 5def probe , position: int, digit: int - Sign: ... 6 7 B. batch: same thing, arrays 8class Probe TypedDict : 9 position: int 10 digit: int 11def guess , code: str - str: ... 12def probe many , probes: list Probe - list Sign : ... 13 14 C. exec: "clever" packing: 27 means position 2, digit 7. 15def exec , probes: list int , guess: str | None = None - list Sign : ... 16 17 D. vector: try a full code, get a sign back per position, borderline cheating, but whatever. 18def guess , code: str - str: ... 19def check , code: str - list Sign : ... 20 21 E. unicode: we give it no tools at all, ask it to reply in emoji 🔍2=7 to probe, 🔑01756 to submit Rank them. No seriously, actually commit to a ranking before you scroll, which one do you think wins? efficiency & win-rate . If you ranked by how proper they look, or according to “le official prompting guide” you put E last. Let’s see. You can see the actual setup here https://i.can.ac/s/Mbxt5h so no cheating. Same system-prompt and all that, feel free to repro. 1gpt-oss-120b 2 3variant escaped turns tool calls probes guesses input tok output tok 4----------------------------------------------------------------------------------- 5naive 0/10 66 56 56 0 36574 9194 6batch 6/10 48 44 139 6 36773 10650 7exec 6/10 46 42 139 6 29142 14199 8vector 8/10 43 41 165 8 22070 13025 9unicode 10/10 48 0 141 10 21320 14958 OK but Can, models got a lot better since then This is a very weak model 1gpt-5.6-luna 2 3variant escaped turns tool calls probes guesses input tok output tok 4----------------------------------------------------------------------------------- 5naive 9/10 149 149 139 10 137570 5308 6batch 10/10 48 48 141 10 42482 3231 7exec 7/10 53 53 219 8 55285 4274 8vector 10/10 57 57 235 10 37400 5988 9unicode 10/10 49 0 142 10 22410 5082 OK but Can, clearly parallel tool calls didn’t work here Plus Luna is still not a frontier model You cannot ask it to operate two tools 1opus 5 2 3variant escaped turns tool calls probes guesses input tok output tok 4----------------------------------------------------------------------------------- 5naive 10/10 48 149 139 10 94502 9768 6batch 9/10 44 43 123 9 70068 3960 7exec 10/10 48 48 152 10 67144 2732 8vector 10/10 51 51 205 10 61190 2489 9unicode 10/10 51 0 159 12 33700 1542 Now at this point, without prior knowledge, your reaction is: In which case I’d like to remind you: To explain any of this, we have to go a layer down. A language model computes one thing: given a sequence of tokens, a probability distribution over the next one. Run it in a loop and that’s autoregression. Forward pass ends in a logit per vocabulary entry; softmax = probabilities; temperature flattens or sharpens; top-p chops the tail; sampler draws: 1 token. So here’s a phrase to delete from your vocabulary: “the model decided to call a tool.” There’s no such thing, in fact, there’s also no such thing as a user turn, assistant turn, system prompt, all your fancy concepts essentially end up being delimiters in a thread, passed to the completion loop. Claude never even receives your tools array. For example, the probe tool lands like this in harmony OpenAI’s format , as a developer message: 1 Tools 2 3 functions 4 5namespace functions { 6 7// Compare one digit guess against the secret door keypad digit 8// at a 1-indexed position. Returns -1 too low, 0 exact, +1 too high. 9type probe = : { 10// keypad position, 1 through 5 11position: number, 12// digit guess, 0 through 9 13digit: number, 14} = any; 15 16} // namespace functions Your schema is documentation; validation happens in your code, maaaybe if the inference provider feels like it, they will validate it. Maaaybe they might push the inference engine towards outputting valid arguments. The descriptions you put in your schema, the mins, the maxes, you don’t even know if they will be displayed, let alone checked. Sorry A “call” is the model outputting: 1<function calls 2<invoke name="bash" <parameter name="arg name" arg value</parameter </invoke 3</function calls Parallel tool calls? Multiple <invoke blocks inside the same <function calls block. Not all that magical, is it? If you’re curious about the rest of the dialects, you can see the whole list of different ones here. Once you see the token stream, the scoreboard stops being mysterious. Arguably the kind of tool design most of the vibe-coded mcp’s go with, it can only be efficient with parallel-tool calls and has lots of room for failure. One step further, designed by someone who cares; see: Pi’s edit tool. But unlike the prior one, it now requires the model to emit JSON. huh what why? Well my friend, you should know this now See, for each <parameter block, if the argument is a primitive, the model can just output the plain value afterwards. The delimiter </parameter is a special token, so no need to worry about escaping But if it’s a complex object or an array? Well, it now has to emit a valid JSON escaped value, which needs to be parsed back, or a tool calling error happens Funny enough, for the frontier models, this one will perform the worst, as parallel-tool calls will lift up the naive one. This is again a step further, now it’s a flat parameter set It also forces the model into making at least 5 comparisons for each round. This is arguably the best kind of design you could come up with, given the problem statement we started with. The guess function being a separate thing is one drawback, but I kinda forced your hand. It still has the usual failure modes though. For instance, sampling errors that lead to calls like to=functions.check.commentary json.Xna 天天送钱 code 】 , or models not emitting any calls. No nesting, no special tokens, very easy to parse, and 🔍2=7 almost has no room for failure. Maybe some model will emit 2- 7 , but you can easily correct. They can put garbage before, after, you don’t care. This was very very obvious, hopefully, it is to you now as well. Rule of thumb: reliability degrades with nesting x heterogeneity × cleverness , and you NEED the harness to handle the common failure modes of the dialect. I’m sorry, real-life isn’t pretty OK but Can, labs RL these models on millions of agent trajectories now Native tool calls ARE the trained path Largely true, post-training genuinely shoves probability mass toward the tool channel, but: They’re your protocol’s shortcomings, they collect rent every single turn: needing a smarter model, more tool call failures, more verbose output… No, I’m not telling you to ship emoji. The moment you have 10 tools instead of 2, native tool-calls win on ergonomics alone and I use it like everyone else. The point is that you now know why E obviously won. You also know how to design tools that will try their best, just like D did; as well as why the minimal harness is not what you want. Assistant leaked the function call into its output text? Now, you have to deal with that, hf The models may have gotten smarter, but they do not think about all these things when you ask them to “add a tool for X”, or “make me an agent loop”. It is in fact your responsibility to get the most out of a model: making it work with the smallest model, adding as many guardrails as possible, maximizing reliability across different families of models. You shouldn’t file a bug with the provider when you see to=functions.check.commentary json.Xna 天天送钱 code , and go to lunch. There is no fix coming Two years later, I’m still doing the same thing I did with my mailbox: babying the model into actually doing the thing & designing the thinnest grammar I can get away with in between.