# Joint Optimization of Tool Creation and Use for Large Language Model Agents > Source: > Published: 2026-08-26 01:36:48+00:00 # Joint Optimization of Tool Creation and Use for Large Language Model Agents SMITH: **S** chema-grounded **M** ulti-task **I** terative **T** ool **H** oning 1Appier AI Research 2National Taiwan University ## Abstract Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can actually invoke. We propose **SMITH** (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a *build* task (write a tool from a few examples) or a *use* task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches **79.9** macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches **40.4** on TabMWP-Hard and **42.6** on out-of-domain GQA (**+7.6** over the best same-backbone inference-time baseline), without any visual or tabular training data. When invoked by a frozen 350M student, tools written by our 4B match those produced by a writer an order of magnitude larger. The same recipe also lifts Qwen3-8B and Granite-3.3-8B without modification. ## Why tool creation and tool use need to be trained together Tool-augmented LLMs are only as capable as the tools someone already wrote for them: a calculator, a search API, a Python sandbox. When the right tool doesn't exist, the agent is stuck. Recent work (LATM, CRAFT, TroVE, KTCE) lets a model synthesize new tools on the fly, but almost always with a powerful model *writing* the tool and a separate, weaker model *using* it. The writer never finds out whether its interface was actually easy to call. - Large frozen LLM writes a tool at inference time - A different, weaker model tries to invoke it - Ambiguous schema → wrong call → wrong answer - No gradient: the writer never learns its schema failed - Same policy writes the tool (code + JSON schema) - Same policy invokes it later from the schema alone - Reward is computed from whether that use succeeded - Gradient flows straight back to the tool writer This creates two concrete training problems the paper has to solve. **Reward decomposition:** a tool can fail because its *code* is wrong, its *schema* is wrong, the two disagree with each other, or the tool is technically correct but poorly designed. Each needs a different corrective signal. **Circular evaluation:** scoring tool quality needs a judge, but a model judging its own live weights is unreliable, and a frozen external judge never improves alongside the policy. ## SMITH: Schema-grounded Multi-task Iterative Tool Honing SMITH is a multi-task RL framework, trained with [DAPO](#) (a clip-higher variant of GRPO), that mixes two rollout types into every batch: **build** and **use**. Both are optimized inside the *same* policy, so gradients from tool creation and tool consumption update the same weights every step. ### Task 1Build: write the tool The policy sees **N = 4** question–answer pairs and must infer the general procedure behind them, then express it as an OpenAI-compatible ``` (Python function, JSON schema) ``` pair. The tool is then run against **K = 16** held-out questions drawn from a *harder* difficulty band than the examples it was built from; the model never sees the ground-truth answers at generation time. A tool that only pattern-matches the easy induction examples scores near zero; only a genuinely reusable abstraction survives. Walk through an example 4 in-context examples (easy band) **Q:** How many 1-bits in the binary form of 42?**A:** 3**Q:** How many 1-bits in the binary form of 255?**A:** 8- … 2 more Tool the policy writes ``` php def solve(question: str) -> str: n = int(re.search(r"\d+", question).group()) return str(bin(n).count("1")) { "name": "solve", "description": "Counts the 1-bits in the binary form of the integer named in the question.", "parameters": { "type": "object", "properties": { "question": { "type": "string" } }, "required": ["question"] } } ``` **13 / 16** held-out (harder) questions correct → admitted to the tool pool ### Task 2Use: call the tool The policy receives a target question and a tool pool entry: **one** matching domain tool plus **two** distractor tools from unrelated categories, forcing it to identify the right schema. It has up to **T = 5** turns to call the tool and answer; an efficiency penalty discourages burning through the turn budget. If no pool tool exists yet for the category, the model must build one first from the same in-context examples. Try it: pick the right tool Target question “How many 1-bits are in the binary form of 92,401?” Pool entry — every schema is named `solve(question)` ; only the description tells them apart ### Three reward axes, kept separate on purpose Rather than collapsing everything into one score, SMITH keeps **format**, **execution**, and **judge** feedback as independent axes fed to DAPO, so each failure mode gets its own gradient instead of being averaged away. #### Format reward Checks the response contains exactly one Python block and one JSON block whose function name and parameters actually match. A malformed pair terminates the rollout with zero reward on every axis. $$r^{\mathrm{fmt}} \in \{0,\ 0.5\}$$ #### Evaluation reward The tool is handed to an evaluator model, which must call it to answer each held-out question. Only answers that came through a real tool call count. A model can't shortcut this by reasoning the answer out in text. $$r^{\mathrm{eval}} = \frac{1}{|\mathcal{T}|}\sum_{j=1}^{|\mathcal{T}|} \mathbf{1}\!\left[\pi^{\mathrm{eval}}(q_j \mid \mathcal{C}, \mathcal{S}) \approx a_j\right]$$ #### Judge reward An LLM judge scores code correctness, schema quality, and overall quality, kept as a *separate* axis from execution reward, not folded in. A syntax error is penalized directly; a schema/code mismatch halves the score. $$r^{\mathrm{judge}} = \begin{cases}-0.5 & \text{syntax error}\\ 0.5\, s_{\mathrm{overall}} & \text{schema}\neq\text{code}\\ s_{\mathrm{overall}} & \text{otherwise}\end{cases}$$ **Breaking the circularity of self-judging.** The evaluator \(\pi^{\mathrm{eval}}\) and judge \(\pi^{\mathrm{judge}}\) both start from the same base checkpoint as the policy, then are periodically re-synced to the latest policy weights. This lets the evaluator improve alongside the policy without the instability of scoring against live, still-updating weights. ### Use-task correctness, with an efficiency penalty Let \(c \in \{0,1\}\) mark whether the final answer matches ground truth. The reward is scaled by an efficiency multiplier \(\eta(\rho)\) that decays as the turn fraction \(\rho = \min(n/T, 1)\) grows, with a floor \(\eta_{\min}=0.3\) so the policy is never indifferent to correctness even at the turn limit (\(\eta_{\mathrm{mid}} = 0.7\)): $$ r^{\mathrm{correct}} = 2c\,\eta(\rho), \qquad \eta(\rho) = \begin{cases} 1 - 2(1-\eta_{\mathrm{mid}})\rho & \rho \le 0.5 \\[2pt] \max\!\bigl(\eta_{\min},\ \eta_{\mathrm{mid}}\,(1-2(\rho-0.5))^2\bigr) & \rho > 0.5 \end{cases} $$ Every training batch is split evenly, \(\mathcal{B} = \mathcal{B}_{\mathrm{build}} \sqcup \mathcal{B}_{\mathrm{use}}\) with \(|\mathcal{B}_{\mathrm{build}}| = |\mathcal{B}_{\mathrm{use}}| = B/2\), and DAPO accumulates both losses in a single backward pass: \(\mathcal{L} = \mathcal{L}_{\mathrm{DAPO}}(\mathcal{B}_{\mathrm{build}}) + \mathcal{L}_{\mathrm{DAPO}} (\mathcal{B}_{\mathrm{use}})\). Build and use gradients therefore update the same parameters \(\theta\) every step. Any tool with \(r^{\mathrm{eval}} > 0\) is pushed into a shared **Tool Pool** for reuse by future use-task rollouts. This is the "iterative honing" in SMITH's name. ## Experimental setup SMITH is trained on **13 procedural task categories** from [Reasoning-Gym](https://github.com/open-thought/reasoning-gym), spanning arithmetic, algorithms, algebra, games, and logical reasoning, chosen because their answers are exact and automatically verifiable and each exposes a difficulty curriculum. Tools are induced on *easy* examples but graded on the *hardest* band of the same task family, deliberately separating tool-writing quality from instance difficulty. The primary backbone is **Qwen3-4B-Instruct**, fine-tuned with LoRA (r = 64, α = 128) using DAPO for 60 gradient steps, build/use rollouts mixed 1:1. The recipe is also applied unmodified to **Qwen3-8B** and **Granite-3.3-8B** to test generality across model families. Baselines share the Qwen3-4B-Instruct backbone wherever possible: inference-time tool writers **LATM**, **CRAFT**, **TroVE**, and **KTCE**; distillation baselines **ReTool** (from Qwen-32B traces) and **LATM** (distilled from GPT-4.1); and a deliberate scaling probe, **LATM on Qwen3-30B-A3B**. Transfer is measured on **TabMWP-Hard** (a strengthened tabular-reasoning benchmark) and **GQA** (visual question answering), neither seen during training, and on **BFCL v4**, an externally specified function-calling benchmark. ## Results ### Main results on Reasoning-Gym SMITH is the only method that leads on genuinely held-out tasks while also using the fewest tokens. Against distillation, a 4B model trained with SMITH's RL objective generalizes more reliably than 4B models distilled from far larger oracles: ReTool leads on seen tasks (92.2) but drops nearly 30 points on unseen ones, a sign of overfitting to the demonstrator's distribution rather than learning a transferable build-and-use policy. Against more elaborate inference-time scaffolds (CRAFT, TroVE, KTCE) and a bigger tool-writer (Qwen3-30B-A3B), SMITH still wins on unseen tasks, using roughly **32× fewer output tokens** than standard chain-of-thought. | Method | Seen Avg | Unseen RG | I/O tokens | ||||| |---|---|---|---|---|---|---|---|---| | Logic | Game | Algebra | Arith | Algo | Avg | ||| | Standard CoT | 58.0 | 49.9 | 60.3 | 56.8 | 62.7 | 48.6 | 55.7 | 173 / 3,206 | | LATM* [Cai et al.] | 77.6 | 53.9 | 55.5 | 38.6 | 53.0 | 90.2 | 58.3 | 607 / 174 | | LATM* – Qwen3-30B-A3B | 74.0 | 68.7 | 64.2 | 97.3 | 56.5 | 84.0 | 74.1 | 659 / 405 | | CRAFT [Yuan et al.] | 74.1±0.7 | 27.4±1.3 | 89.5±0.0 | 94.2±1.2 | 76.6±1.1 | 95.0±0.0 | 76.5±0.4 | 1,226 / 418 | | TroVE [Wang et al.] | 52.6±0.4 | 60.7±2.4 | 10.6±1.3 | 51.2±3.5 | 59.8±0.8 | 97.0±0.0 | 55.9±0.6 | 347 / 575 | | KTCE [Ma et al.] | 61.0±1.5 | 60.2±1.5 | 79.8±1.6 | 70.6±0.3 | 45.6±0.4 | 69.3±1.8 | 65.1±0.2 | 319 / 404 | | ReTool (distill Qwen-32B) | 92.2±0.8 | 50.3±2.3 | 55.0±0.8 | 48.7±0.6 | 79.8±2.7 | 82.4±0.1 | 63.2±0.4 | 1,707 / 633 | | LATM (distill GPT-4.1) | 81.7±4.4 | 37.6±15.2 | 58.1±12.2 | 91.3±7.7 | 51.6±10.4 | 93.2±5.9 | 65.8±4.1 | 638 / 207 | SMITH | 85.2±2.7 | 74.2±0.6 | 63.7±1.1 | 97.9±2.6 | 70.6±2.1 | 93.0±0.4 | 79.9±2.2 | 664 / 100 | ### Tools transfer across model scale Do SMITH's tools encode a genuinely reusable solution, or a private convention only the writer understands? We test both directions: pairing the RL-trained 4B writer with a much smaller consumer, and with a much larger one. Pairing tools from the RL-trained 4B writer with a frozen **LFM2.5-350M** consumer lifts its held-out accuracy from 11.6 to **42.9**, matching a tool-writer *eighty times larger* (Qwen3-30B-A3B, untrained, at 41.5). RG (Unseen) accuracy of LFM2.5-350M as the tool-writing source changes; the writer itself is never fine-tuned on LFM2.5's outputs. The reverse direction also holds. Pairing SMITH's 4B-written tools with a much stronger **Qwen3-30B-A3B** consumer beats that same 30B model writing tools for itself (LATM*) on every task group, most sharply on TabMWP-Hard (0.7 → 38.8), lifting the task-weighted overall score from 70.2 to 76.6. A bigger tool user doesn't make its own self-written tool preferable. SMITH's 4B writer remains the better source of tools either way. ### Out-of-domain generalization Neither TabMWP-Hard (tabular reasoning) nor GQA (visual question answering) appears anywhere in SMITH's training data. SMITH leads TabMWP-Hard outright and is the only 4B, non-distilled method to top either column. The GQA gap to GPT-4.1-distilled LATM is the acknowledged cost of not distilling visual primitives from a stronger oracle. | Task | CoT | LATM | LATM* (30B) | CRAFT | TroVE | KTCE | ReTool | LATM (distill) | SMITH | |---|---|---|---|---|---|---|---|---|---| | TabMWP-Hard | 7.2 | 19.7 | 0.7 | 30.0 | 36.4 | 27.2 | 3.0 | 7.1 | 40.4 | | GQA | 11.5 | 35.0 | 29.8 | 21.9 | 21.4 | 0.0 | 26.1 | 56.0 | 42.6 | ### Scaling across backbones The same RL recipe, applied unmodified, improves every backbone tested, including **Granite-3.3-8B**, which starts from a much weaker base. A **Self-Judge** variant (Qwen3-8B judging its own rollouts instead of an external 30B-A3B judge) pushes held-out RG even higher (85.9) but trades off OOD GQA, suggesting a smaller self-judge is weaker but less biased on in-distribution data. | Method | RG (Seen) | RG (Unseen) | TabMWP | GQA | |---|---|---|---|---| | Qwen3-8B (baseline) | 72.6 | 72.2 | 42.4 | 17.3 | SMITH: Qwen3-8B | 79.4 | 81.7 | 56.7 | 28.7 | SMITH: Self-Judge | 74.7 | 85.9 | 54.5 | 16.3 | | Granite-3.3-8B (baseline) | 31.2 | 22.0 | 3.9 | 7.8 | SMITH: Granite-3.3-8B | 39.1 | 28.5 | 4.5 | 11.7 | ### Generalization to external tool-calling SMITH is never trained on [BFCL v4](#)'s schemas, multi-turn traces, or judges, so a gain here isolates a learned tool-use prior rather than benchmark-specific fitting. It lifts BFCL overall accuracy on both Qwen backbones, most sharply on Qwen3-8B. | Qwen3-4B-Instruct | Qwen3-8B | Granite-3.3-8B | |||| |---|---|---|---|---|---|---| | Metric | Base | SMITH | Base | SMITH | Base | SMITH | | BFCL v4 | 45.1 | 48.6 | 43.3 | 55.8 | 36.3 | 38.7 | ## Ablation: what's actually driving the gain? We isolate whether joint training, rather than simply having a specialist builder and a specialist user, is the active ingredient. Pairing a separately-trained 30B-A3B builder with a separately-trained 4B tool-use specialist (**Decoupled Create/Use**) does *not* beat a single model trained on tool creation alone (58.9 vs. 68.8 RG Unseen). Only the full objective, which keeps execution reward and judge reward as disentangled axes and closes the loop in one policy, reaches the best aggregate score. | Method | πeval | Tool use? | RG (Seen) | RG (Unseen) | TabMWP | GQA | |---|---|---|---|---|---|---| | Qwen3-4B-Instruct (base) | n/a | n/a | 61.9 | 47.0 | 19.7 | 20.9 | | Tool Create | 30B-A3B | No | 77.4 | 59.4 | 15.6 | 37.0 | | Tool Create | 4B | No | 73.9 | 68.8 | 17.7 | 35.8 | | Decoupled Create/Use | 30B-A3B | Yes | 76.4 | 58.9 | 13.9 | 35.0 | | SMITH: No Sync | 4B | Yes | 80.3 | 66.9 | 25.8 | 24.6 | | SMITH: No LLM Judge | self | Yes | 82.6 | 67.8 | 18.3 | 42.6 | | SMITH: K = 1 | self | Yes | 78.6 | 73.9 | 46.8 | 32.3 | SMITH: Full | self | Yes | 86.6 | 78.3 | 40.4 | 42.6 | Tool creation alone yields a strong in-distribution bump but underperforms on OOD GQA; coupling build and use without disentangled rewards lifts in-distribution accuracy but hurts held-out transfer. Only keeping process quality (format, judge) and outcome correctness (execution, use) as separate reward axes gets both. ## Conclusion SMITH jointly trains a single language model to *create* and *use* reusable tools, closing the feedback loop between tool writer and tool user so the policy is optimized directly on its own execution outcomes. Trained on 13 Reasoning-Gym tasks, the 4B model attains the highest held-out accuracy among all evaluated methods, leads TabMWP-Hard, and writes tools that transfer to a 350M student never seen during training, matching tools from a writer an order of magnitude larger. The same recipe lifts Qwen3-8B and Granite-3.3-8B without modification: coupling creation and use inside one trained policy is a scalable path to generalization, without a larger frozen teacher, a more elaborate scaffold, or out-of-domain supervision. ## BibTeX ``` @article{tam2026smith, title = {Joint Optimization of Tool Creation and Use for Large Language Model Agents}, author = {Tam, Zhi Rui and Lin, Chieh-Yen and Chen, Yun-Nung and Sun, Shao-Hua and Lee, Hung-yi}, year = {2026}, journal = {arXiv preprint}, url = {https://tool-use-smith.github.io} } ```