SMITH: S chema-grounded M ulti-task I terative T ool H oning
1Appier AI Research 2National Taiwan University
Abstract #
Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can actually invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.9 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. When invoked by a frozen 350M student, tools written by our 4B match those produced by a writer an order of magnitude larger. The same recipe also lifts Qwen3-8B and Granite-3.3-8B without modification.
Why tool creation and tool use need to be trained together #
Tool-augmented LLMs are only as capable as the tools someone already wrote for them: a calculator, a search API, a Python sandbox. When the right tool doesn't exist, the agent is stuck. Recent work (LATM, CRAFT, TroVE, KTCE) lets a model synthesize new tools on the fly, but almost always with a powerful model writing the tool and a separate, weaker model using it. The writer never finds out whether its interface was actually easy to call.
-
Large frozen LLM writes a tool at inference time
-
A different, weaker model tries to invoke it
-
Ambiguous schema → wrong call → wrong answer
-
No gradient: the writer never learns its schema failed
-
Same policy writes the tool (code + JSON schema)
-
Same policy invokes it later from the schema alone
-
Reward is computed from whether that use succeeded
-
Gradient flows straight back to the tool writer
This creates two concrete training problems the paper has to solve. Reward decomposition: a tool can fail because its code is wrong, its schema is wrong, the two disagree with each other, or the tool is technically correct but poorly designed. Each needs a different corrective signal. Circular evaluation: scoring tool quality needs a judge, but a model judging its own live weights is unreliable, and a frozen external judge never improves alongside the policy.
SMITH: Schema-grounded Multi-task Iterative Tool Honing #
SMITH is a multi-task RL framework, trained with DAPO (a clip-higher variant of GRPO), that mixes two rollout types into every batch: build and use. Both are optimized inside the same policy, so gradients from tool creation and tool consumption update the same weights every step.
Task 1Build: write the tool
The policy sees N = 4 question–answer pairs and must infer the general procedure behind them, then express it as an OpenAI-compatible ``` (Python function, JSON schema)
pair. The tool is then run against **K = 16** held-out questions
drawn from a *harder* difficulty band than the examples it was built from; the model never
sees the ground-truth answers at generation time. A tool that only pattern-matches the easy
induction examples scores near zero; only a genuinely reusable abstraction survives.
Walk through an example
4 in-context examples (easy band)
**Q:** How many 1-bits in the binary form of 42?**A:** 3**Q:** How many 1-bits in the binary form of 255?**A:** 8- … 2 more
Tool the policy writes
``` php
def solve(question: str) -> str:
n = int(re.search(r"\d+", question).group())
return str(bin(n).count("1"))
{
"name": "solve",
"description": "Counts the 1-bits in the
binary form of the integer named in
the question.",
"parameters": {
"type": "object",
"properties": {
"question": { "type": "string" }
},
"required": ["question"]
}
}
13 / 16 held-out (harder) questions correct → admitted to the tool pool
Task 2Use: call the tool
The policy receives a target question and a tool pool entry: one matching domain tool plus two distractor tools from unrelated categories, forcing it to identify the right schema. It has up to T = 5 turns to call the tool and answer; an efficiency penalty discourages burning through the turn budget. If no pool tool exists yet for the category, the model must build one first from the same in-context examples.
Try it: pick the right tool
Target question
“How many 1-bits are in the binary form of 92,401?”
Pool entry — every schema is named solve(question)
; only the description tells them apart
Three reward axes, kept separate on purpose
Rather than collapsing everything into one score, SMITH keeps format, execution, and judge feedback as independent axes fed to DAPO, so each failure mode gets its own gradient instead of being averaged away.
Format reward
Checks the response contains exactly one Python block and one JSON block whose function name and parameters actually match. A malformed pair terminates the rollout with zero reward on every axis.
$$r^{\mathrm{fmt}} \in {0,\ 0.5}$$
Evaluation reward
The tool is handed to an evaluator model, which must call it to answer each held-out question. Only answers that came through a real tool call count. A model can't shortcut this by reasoning the answer out in text.
$$r^{\mathrm{eval}} = \frac{1}{|\mathcal{T}|}\sum_{j=1}^{|\mathcal{T}|} \mathbf{1}!\left[\pi^{\mathrm{eval}}(q_j \mid \mathcal{C}, \mathcal{S}) \approx a_j\right]$$
Judge reward
An LLM judge scores code correctness, schema quality, and overall quality, kept as a separate axis from execution reward, not folded in. A syntax error is penalized directly; a schema/code mismatch halves the score.
$$r^{\mathrm{judge}} = \begin{cases}-0.5 & \text{syntax error}\ 0.5, s_{\mathrm{overall}} & \text{schema}\neq\text{code}\ s_{\mathrm{overall}} & \text{otherwise}\end{cases}$$
Breaking the circularity of self-judging. The evaluator (\pi^{\mathrm{eval}}) and judge (\pi^{\mathrm{judge}}) both start from the same base checkpoint as the policy, then are periodically re-synced to the latest policy weights. This lets the evaluator improve alongside the policy without the instability of scoring against live, still-updating weights.
Use-task correctness, with an efficiency penalty
Let (c \in {0,1}) mark whether the final answer matches ground truth. The reward is scaled by an efficiency multiplier (\eta(\rho)) that decays as the turn fraction (\rho = \min(n/T, 1)) grows, with a floor (\eta_{\min}=0.3) so the policy is never indifferent to correctness even at the turn limit ((\eta_{\mathrm{mid}} = 0.7)):
$$ r^{\mathrm{correct}} = 2c,\eta(\rho), \qquad \eta(\rho) = \begin{cases} 1 - 2(1-\eta_{\mathrm{mid}})\rho & \rho \le 0.5 \[2pt] \max!\bigl(\eta_{\min},\ \eta_{\mathrm{mid}},(1-2(\rho-0.5))^2\bigr) & \rho > 0.5 \end{cases} $$
Every training batch is split evenly, (\mathcal{B} = \mathcal{B}{\mathrm{build}} \sqcup \mathcal{B}{\mathrm{use}}) with (|\mathcal{B}{\mathrm{build}}| = |\mathcal{B}{\mathrm{use}}| = B/2), and DAPO accumulates both losses in a single backward pass: (\mathcal{L} = \mathcal{L}{\mathrm{DAPO}}(\mathcal{B}{\mathrm{build}}) + \mathcal{L}{\mathrm{DAPO}} (\mathcal{B}{\mathrm{use}})). Build and use gradients therefore update the same parameters (\theta) every step. Any tool with (r^{\mathrm{eval}} > 0) is pushed into a shared Tool Pool for reuse by future use-task rollouts. This is the "iterative honing" in SMITH's name.
Experimental setup #
SMITH is trained on 13 procedural task categories from Reasoning-Gym, spanning arithmetic, algorithms, algebra, games, and logical reasoning, chosen because their answers are exact and automatically verifiable and each exposes a difficulty curriculum. Tools are induced on easy examples but graded on the hardest band of the same task family, deliberately separating tool-writing quality from instance difficulty.
The primary backbone is Qwen3-4B-Instruct, fine-tuned with LoRA (r = 64, α = 128) using DAPO for 60 gradient steps, build/use rollouts mixed 1:1. The recipe is also applied unmodified to Qwen3-8B and Granite-3.3-8B to test generality across model families. Baselines share the Qwen3-4B-Instruct backbone wherever possible: inference-time tool writers LATM, CRAFT, TroVE, and KTCE; distillation baselines ReTool (from Qwen-32B traces) and LATM (distilled from GPT-4.1); and a deliberate scaling probe, LATM on Qwen3-30B-A3B. Transfer is measured on TabMWP-Hard (a strengthened tabular-reasoning benchmark) and GQA (visual question answering), neither seen during training, and on BFCL v4, an externally specified function-calling benchmark.
Results #
Main results on Reasoning-Gym
SMITH is the only method that leads on genuinely held-out tasks while also using the fewest tokens. Against distillation, a 4B model trained with SMITH's RL objective generalizes more reliably than 4B models distilled from far larger oracles: ReTool leads on seen tasks (92.2) but drops nearly 30 points on unseen ones, a sign of overfitting to the demonstrator's distribution rather than learning a transferable build-and-use policy. Against more elaborate inference-time scaffolds (CRAFT, TroVE, KTCE) and a bigger tool-writer (Qwen3-30B-A3B), SMITH still wins on unseen tasks, using roughly 32× fewer output tokens than standard chain-of-thought.
| Method | Seen Avg | Unseen RG | I/O tokens | ||||| |---|---|---|---|---|---|---|---|---| | Logic | Game | Algebra | Arith | Algo | Avg | ||| | Standard CoT | 58.0 | 49.9 | 60.3 | 56.8 | 62.7 | 48.6 | 55.7 | 173 / 3,206 | | LATM* [Cai et al.] | 77.6 | 53.9 | 55.5 | 38.6 | 53.0 | 90.2 | 58.3 | 607 / 174 | | LATM* – Qwen3-30B-A3B | 74.0 | 68.7 | 64.2 | 97.3 | 56.5 | 84.0 | 74.1 | 659 / 405 | | CRAFT [Yuan et al.] | 74.1±0.7 | 27.4±1.3 | 89.5±0.0 | 94.2±1.2 | 76.6±1.1 | 95.0±0.0 | 76.5±0.4 | 1,226 / 418 | | TroVE [Wang et al.] | 52.6±0.4 | 60.7±2.4 | 10.6±1.3 | 51.2±3.5 | 59.8±0.8 | 97.0±0.0 | 55.9±0.6 | 347 / 575 | | KTCE [Ma et al.] | 61.0±1.5 | 60.2±1.5 | 79.8±1.6 | 70.6±0.3 | 45.6±0.4 | 69.3±1.8 | 65.1±0.2 | 319 / 404 | | ReTool (distill Qwen-32B) | 92.2±0.8 | 50.3±2.3 | 55.0±0.8 | 48.7±0.6 | 79.8±2.7 | 82.4±0.1 | 63.2±0.4 | 1,707 / 633 | | LATM (distill GPT-4.1) | 81.7±4.4 | 37.6±15.2 | 58.1±12.2 | 91.3±7.7 | 51.6±10.4 | 93.2±5.9 | 65.8±4.1 | 638 / 207 | SMITH | 85.2±2.7 | 74.2±0.6 | 63.7±1.1 | 97.9±2.6 | 70.6±2.1 | 93.0±0.4 | 79.9±2.2 | 664 / 100 |
Tools transfer across model scale
Do SMITH's tools encode a genuinely reusable solution, or a private convention only the writer understands? We test both directions: pairing the RL-trained 4B writer with a much smaller consumer, and with a much larger one.
Pairing tools from the RL-trained 4B writer with a frozen LFM2.5-350M consumer lifts its held-out accuracy from 11.6 to 42.9, matching a tool-writer eighty times larger (Qwen3-30B-A3B, untrained, at 41.5).
RG (Unseen) accuracy of LFM2.5-350M as the tool-writing source changes; the writer itself is never fine-tuned on LFM2.5's outputs.
The reverse direction also holds. Pairing SMITH's 4B-written tools with a much stronger Qwen3-30B-A3B consumer beats that same 30B model writing tools for itself (LATM*) on every task group, most sharply on TabMWP-Hard (0.7 → 38.8), lifting the task-weighted overall score from 70.2 to 76.6. A bigger tool user doesn't make its own self-written tool preferable. SMITH's 4B writer remains the better source of tools either way.
Out-of-domain generalization
Neither TabMWP-Hard (tabular reasoning) nor GQA (visual question answering) appears anywhere in SMITH's training data. SMITH leads TabMWP-Hard outright and is the only 4B, non-distilled method to top either column. The GQA gap to GPT-4.1-distilled LATM is the acknowledged cost of not distilling visual primitives from a stronger oracle.
| Task | CoT | LATM | LATM* (30B) | CRAFT | TroVE | KTCE | ReTool | LATM (distill) | SMITH |
|---|---|---|---|---|---|---|---|---|---|
| TabMWP-Hard | 7.2 | 19.7 | 0.7 | 30.0 | 36.4 | 27.2 | 3.0 | 7.1 | 40.4 |
| GQA | 11.5 | 35.0 | 29.8 | 21.9 | 21.4 | 0.0 | 26.1 | 56.0 | 42.6 |
Scaling across backbones
The same RL recipe, applied unmodified, improves every backbone tested, including Granite-3.3-8B, which starts from a much weaker base. A Self-Judge variant (Qwen3-8B judging its own rollouts instead of an external 30B-A3B judge) pushes held-out RG even higher (85.9) but trades off OOD GQA, suggesting a smaller self-judge is weaker but less biased on in-distribution data.
| Method | RG (Seen) | RG (Unseen) | TabMWP | GQA |
|---|---|---|---|---|
| Qwen3-8B (baseline) | 72.6 | 72.2 | 42.4 | 17.3 |
| SMITH: Qwen3-8B | 79.4 | 81.7 | 56.7 | 28.7 |
| SMITH: Self-Judge | 74.7 | 85.9 | 54.5 | 16.3 |
| Granite-3.3-8B (baseline) | 31.2 | 22.0 | 3.9 | 7.8 |
| SMITH: Granite-3.3-8B | 39.1 | 28.5 | 4.5 | 11.7 |
Generalization to external tool-calling
SMITH is never trained on BFCL v4's schemas, multi-turn traces, or judges, so a gain here isolates a learned tool-use prior rather than benchmark-specific fitting. It lifts BFCL overall accuracy on both Qwen backbones, most sharply on Qwen3-8B.
| Qwen3-4B-Instruct | Qwen3-8B | Granite-3.3-8B | ||||
|---|---|---|---|---|---|---|
| Metric | Base | SMITH | Base | SMITH | Base | SMITH |
| BFCL v4 | 45.1 | 48.6 | 43.3 | 55.8 | 36.3 | 38.7 |
Ablation: what's actually driving the gain? #
We isolate whether joint training, rather than simply having a specialist builder and a specialist user, is the active ingredient. Pairing a separately-trained 30B-A3B builder with a separately-trained 4B tool-use specialist (Decoupled Create/Use) does not beat a single model trained on tool creation alone (58.9 vs. 68.8 RG Unseen). Only the full objective, which keeps execution reward and judge reward as disentangled axes and closes the loop in one policy, reaches the best aggregate score.
| Method | πeval | Tool use? | RG (Seen) | RG (Unseen) | TabMWP | GQA |
|---|---|---|---|---|---|---|
| Qwen3-4B-Instruct (base) | n/a | n/a | 61.9 | 47.0 | 19.7 | 20.9 |
| Tool Create | 30B-A3B | No | 77.4 | 59.4 | 15.6 | 37.0 |
| Tool Create | 4B | No | 73.9 | 68.8 | 17.7 | 35.8 |
| Decoupled Create/Use | 30B-A3B | Yes | 76.4 | 58.9 | 13.9 | 35.0 |
| SMITH: No Sync | 4B | Yes | 80.3 | 66.9 | 25.8 | 24.6 |
| SMITH: No LLM Judge | self | Yes | 82.6 | 67.8 | 18.3 | 42.6 |
| SMITH: K = 1 | self | Yes | 78.6 | 73.9 | 46.8 | 32.3 |
| SMITH: Full | self | Yes | 86.6 | 78.3 | 40.4 | 42.6 |
Tool creation alone yields a strong in-distribution bump but underperforms on OOD GQA; coupling build and use without disentangled rewards lifts in-distribution accuracy but hurts held-out transfer. Only keeping process quality (format, judge) and outcome correctness (execution, use) as separate reward axes gets both.
Conclusion #
SMITH jointly trains a single language model to create and use reusable tools, closing the feedback loop between tool writer and tool user so the policy is optimized directly on its own execution outcomes. Trained on 13 Reasoning-Gym tasks, the 4B model attains the highest held-out accuracy among all evaluated methods, leads TabMWP-Hard, and writes tools that transfer to a 350M student never seen during training, matching tools from a writer an order of magnitude larger. The same recipe lifts Qwen3-8B and Granite-3.3-8B without modification: coupling creation and use inside one trained policy is a scalable path to generalization, without a larger frozen teacher, a more elaborate scaffold, or out-of-domain supervision.
BibTeX #
@article{tam2026smith,
title = {Joint Optimization of Tool Creation and Use for Large Language Model Agents},
author = {Tam, Zhi Rui and Lin, Chieh-Yen and Chen, Yun-Nung and Sun, Shao-Hua and Lee, Hung-yi},
year = {2026},
journal = {arXiv preprint},
url = {https://tool-use-smith.github.io}
}