{"slug": "joint-optimization-of-tool-creation-and-use-for-large-language-model-agents", "title": "Joint Optimization of Tool Creation and Use for Large Language Model Agents", "summary": "Appier AI Research and National Taiwan University introduced SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use within a single policy, achieving 79.9 macro-average accuracy on held-out tasks with a 4B Qwen3 model across 13 procedural reasoning tasks. The framework also reached 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), and tools written by the 4B model matched those from a writer an order of magnitude larger when invoked by a frozen 350M student.", "body_md": "# Joint Optimization of Tool Creation and Use for Large Language Model Agents\n\nSMITH: **S** chema-grounded **M** ulti-task **I** terative **T** ool **H** oning\n\n1Appier AI Research 2National Taiwan University\n\n## Abstract\n\nTool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation\nsystems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool\ndecoupled from the one that uses it, with no signal that the schemas it produces are schemas it can\nactually invoke. We propose **SMITH** (Schema-grounded Multi-task Iterative Tool Honing), a\nreinforcement learning framework that jointly trains tool creation and tool use inside a single policy.\nEach rollout is either a *build* task (write a tool from a few examples) or a *use* task\n(invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and\noutcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained\nwith SMITH on 13 procedural reasoning tasks with exact verifiers reaches **79.9**\nmacro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an\nuntrained 30B-A3B tool-writer. It also reaches **40.4** on TabMWP-Hard and\n**42.6** on out-of-domain GQA (**+7.6** over the best same-backbone\ninference-time baseline), without any visual or tabular training data. When invoked by a frozen 350M\nstudent, tools written by our 4B match those produced by a writer an order of magnitude larger. The\nsame recipe also lifts Qwen3-8B and Granite-3.3-8B without modification.\n\n## Why tool creation and tool use need to be trained together\n\nTool-augmented LLMs are only as capable as the tools someone already wrote for them: a calculator,\na search API, a Python sandbox. When the right tool doesn't exist, the agent is stuck. Recent work\n(LATM, CRAFT, TroVE, KTCE) lets a model synthesize new tools on the fly, but almost always with a\npowerful model *writing* the tool and a separate, weaker model *using* it. The writer\nnever finds out whether its interface was actually easy to call.\n\n- Large frozen LLM writes a tool at inference time\n- A different, weaker model tries to invoke it\n- Ambiguous schema → wrong call → wrong answer\n- No gradient: the writer never learns its schema failed\n\n- Same policy writes the tool (code + JSON schema)\n- Same policy invokes it later from the schema alone\n- Reward is computed from whether that use succeeded\n- Gradient flows straight back to the tool writer\n\nThis creates two concrete training problems the paper has to solve. **Reward decomposition:**\na tool can fail because its *code* is wrong, its *schema* is wrong, the two disagree with\neach other, or the tool is technically correct but poorly designed. Each needs a different corrective\nsignal. **Circular evaluation:** scoring tool quality needs a judge, but a model judging\nits own live weights is unreliable, and a frozen external judge never improves alongside the policy.\n\n## SMITH: Schema-grounded Multi-task Iterative Tool Honing\n\nSMITH is a multi-task RL framework, trained with [DAPO](#) (a clip-higher\nvariant of GRPO), that mixes two rollout types into every batch: **build** and\n**use**. Both are optimized inside the *same* policy, so gradients from tool\ncreation and tool consumption update the same weights every step.\n\n### Task 1Build: write the tool\n\nThe policy sees **N = 4** question–answer pairs and must infer the general\nprocedure behind them, then express it as an OpenAI-compatible ```\n(Python function, JSON\nschema)\n```\n\npair. The tool is then run against **K = 16** held-out questions\ndrawn from a *harder* difficulty band than the examples it was built from; the model never\nsees the ground-truth answers at generation time. A tool that only pattern-matches the easy\ninduction examples scores near zero; only a genuinely reusable abstraction survives.\n\nWalk through an example\n\n4 in-context examples (easy band)\n\n**Q:** How many 1-bits in the binary form of 42?**A:** 3**Q:** How many 1-bits in the binary form of 255?**A:** 8- … 2 more\n\nTool the policy writes\n\n``` php\ndef solve(question: str) -> str:\n    n = int(re.search(r\"\\d+\", question).group())\n    return str(bin(n).count(\"1\"))\n{\n  \"name\": \"solve\",\n  \"description\": \"Counts the 1-bits in the\n    binary form of the integer named in\n    the question.\",\n  \"parameters\": {\n    \"type\": \"object\",\n    \"properties\": {\n      \"question\": { \"type\": \"string\" }\n    },\n    \"required\": [\"question\"]\n  }\n}\n```\n\n**13 / 16** held-out (harder) questions correct → admitted to the tool pool\n\n### Task 2Use: call the tool\n\nThe policy receives a target question and a tool pool entry: **one** matching\ndomain tool plus **two** distractor tools from unrelated categories, forcing it to\nidentify the right schema. It has up to **T = 5** turns to call the tool and answer;\nan efficiency penalty discourages burning through the turn budget. If no pool tool exists yet for\nthe category, the model must build one first from the same in-context examples.\n\nTry it: pick the right tool\n\nTarget question\n\n“How many 1-bits are in the binary form of 92,401?”\n\nPool entry — every schema is named `solve(question)`\n\n; only the description tells them apart\n\n### Three reward axes, kept separate on purpose\n\nRather than collapsing everything into one score, SMITH keeps **format**,\n**execution**, and **judge** feedback as independent axes fed to DAPO, so\neach failure mode gets its own gradient instead of being averaged away.\n\n#### Format reward\n\nChecks the response contains exactly one Python block and one JSON block whose function name and parameters actually match. A malformed pair terminates the rollout with zero reward on every axis.\n\n$$r^{\\mathrm{fmt}} \\in \\{0,\\ 0.5\\}$$\n\n#### Evaluation reward\n\nThe tool is handed to an evaluator model, which must call it to answer each held-out question. Only answers that came through a real tool call count. A model can't shortcut this by reasoning the answer out in text.\n\n$$r^{\\mathrm{eval}} = \\frac{1}{|\\mathcal{T}|}\\sum_{j=1}^{|\\mathcal{T}|} \\mathbf{1}\\!\\left[\\pi^{\\mathrm{eval}}(q_j \\mid \\mathcal{C}, \\mathcal{S}) \\approx a_j\\right]$$\n\n#### Judge reward\n\nAn LLM judge scores code correctness, schema quality, and overall quality, kept as a\n*separate* axis from execution reward, not folded in. A syntax error is penalized directly;\na schema/code mismatch halves the score.\n\n$$r^{\\mathrm{judge}} = \\begin{cases}-0.5 & \\text{syntax error}\\\\ 0.5\\, s_{\\mathrm{overall}} & \\text{schema}\\neq\\text{code}\\\\ s_{\\mathrm{overall}} & \\text{otherwise}\\end{cases}$$\n\n**Breaking the circularity of self-judging.** The evaluator \\(\\pi^{\\mathrm{eval}}\\)\nand judge \\(\\pi^{\\mathrm{judge}}\\) both start from the same base checkpoint as the policy, then are\nperiodically re-synced to the latest policy weights. This lets the evaluator improve alongside the\npolicy without the instability of scoring against live, still-updating weights.\n\n### Use-task correctness, with an efficiency penalty\n\nLet \\(c \\in \\{0,1\\}\\) mark whether the final answer matches ground truth. The reward is scaled by an efficiency multiplier \\(\\eta(\\rho)\\) that decays as the turn fraction \\(\\rho = \\min(n/T, 1)\\) grows, with a floor \\(\\eta_{\\min}=0.3\\) so the policy is never indifferent to correctness even at the turn limit (\\(\\eta_{\\mathrm{mid}} = 0.7\\)):\n\n$$ r^{\\mathrm{correct}} = 2c\\,\\eta(\\rho), \\qquad \\eta(\\rho) = \\begin{cases} 1 - 2(1-\\eta_{\\mathrm{mid}})\\rho & \\rho \\le 0.5 \\\\[2pt] \\max\\!\\bigl(\\eta_{\\min},\\ \\eta_{\\mathrm{mid}}\\,(1-2(\\rho-0.5))^2\\bigr) & \\rho > 0.5 \\end{cases} $$\n\nEvery training batch is split evenly, \\(\\mathcal{B} = \\mathcal{B}_{\\mathrm{build}} \\sqcup\n\\mathcal{B}_{\\mathrm{use}}\\) with \\(|\\mathcal{B}_{\\mathrm{build}}| = |\\mathcal{B}_{\\mathrm{use}}| =\nB/2\\), and DAPO accumulates both losses in a single backward pass:\n\\(\\mathcal{L} = \\mathcal{L}_{\\mathrm{DAPO}}(\\mathcal{B}_{\\mathrm{build}}) + \\mathcal{L}_{\\mathrm{DAPO}}\n(\\mathcal{B}_{\\mathrm{use}})\\). Build and use gradients therefore update the same parameters\n\\(\\theta\\) every step. Any tool with \\(r^{\\mathrm{eval}} > 0\\) is pushed into a shared\n**Tool Pool** for reuse by future use-task rollouts. This is the \"iterative honing\" in\nSMITH's name.\n\n## Experimental setup\n\nSMITH is trained on **13 procedural task categories** from\n[Reasoning-Gym](https://github.com/open-thought/reasoning-gym), spanning arithmetic, algorithms, algebra,\ngames, and logical reasoning, chosen because their answers are exact and automatically verifiable and\neach exposes a difficulty curriculum. Tools are induced on *easy* examples but graded on the\n*hardest* band of the same task family, deliberately separating tool-writing quality from\ninstance difficulty.\n\nThe primary backbone is **Qwen3-4B-Instruct**, fine-tuned with LoRA (r = 64, α = 128)\nusing DAPO for 60 gradient steps, build/use rollouts mixed 1:1. The recipe is also applied unmodified\nto **Qwen3-8B** and **Granite-3.3-8B** to test generality across model\nfamilies. Baselines share the Qwen3-4B-Instruct backbone wherever possible: inference-time tool\nwriters **LATM**, **CRAFT**, **TroVE**, and **KTCE**;\ndistillation baselines **ReTool** (from Qwen-32B traces) and **LATM**\n(distilled from GPT-4.1); and a deliberate scaling probe, **LATM on Qwen3-30B-A3B**.\nTransfer is measured on **TabMWP-Hard** (a strengthened tabular-reasoning benchmark) and\n**GQA** (visual question answering), neither seen during training, and on\n**BFCL v4**, an externally specified function-calling benchmark.\n\n## Results\n\n### Main results on Reasoning-Gym\n\nSMITH is the only method that leads on genuinely held-out tasks while also using the fewest tokens.\nAgainst distillation, a 4B model trained with SMITH's RL objective generalizes more reliably than 4B\nmodels distilled from far larger oracles: ReTool leads on seen tasks (92.2) but drops nearly 30 points\non unseen ones, a sign of overfitting to the demonstrator's distribution rather than learning a\ntransferable build-and-use policy. Against more elaborate inference-time scaffolds (CRAFT, TroVE,\nKTCE) and a bigger tool-writer (Qwen3-30B-A3B), SMITH still wins on unseen tasks, using roughly\n**32× fewer output tokens** than standard chain-of-thought.\n\n| Method | Seen Avg |\nUnseen RG | I/O tokens | |||||\n|---|---|---|---|---|---|---|---|---|\n| Logic | Game | Algebra | Arith | Algo | Avg | |||\n| Standard CoT | 58.0 | 49.9 | 60.3 | 56.8 | 62.7 | 48.6 | 55.7 | 173 / 3,206 |\n| LATM* [Cai et al.] | 77.6 | 53.9 | 55.5 | 38.6 | 53.0 | 90.2 | 58.3 | 607 / 174 |\n| LATM* – Qwen3-30B-A3B | 74.0 | 68.7 | 64.2 | 97.3 | 56.5 | 84.0 | 74.1 | 659 / 405 |\n| CRAFT [Yuan et al.] | 74.1±0.7 | 27.4±1.3 | 89.5±0.0 | 94.2±1.2 | 76.6±1.1 | 95.0±0.0 | 76.5±0.4 | 1,226 / 418 |\n| TroVE [Wang et al.] | 52.6±0.4 | 60.7±2.4 | 10.6±1.3 | 51.2±3.5 | 59.8±0.8 | 97.0±0.0 | 55.9±0.6 | 347 / 575 |\n| KTCE [Ma et al.] | 61.0±1.5 | 60.2±1.5 | 79.8±1.6 | 70.6±0.3 | 45.6±0.4 | 69.3±1.8 | 65.1±0.2 | 319 / 404 |\n| ReTool (distill Qwen-32B) | 92.2±0.8 | 50.3±2.3 | 55.0±0.8 | 48.7±0.6 | 79.8±2.7 | 82.4±0.1 | 63.2±0.4 | 1,707 / 633 |\n| LATM (distill GPT-4.1) | 81.7±4.4 | 37.6±15.2 | 58.1±12.2 | 91.3±7.7 | 51.6±10.4 | 93.2±5.9 | 65.8±4.1 | 638 / 207 |\nSMITH | 85.2±2.7 | 74.2±0.6 | 63.7±1.1 | 97.9±2.6 | 70.6±2.1 | 93.0±0.4 | 79.9±2.2 | 664 / 100 |\n\n### Tools transfer across model scale\n\nDo SMITH's tools encode a genuinely reusable solution, or a private convention only the writer understands? We test both directions: pairing the RL-trained 4B writer with a much smaller consumer, and with a much larger one.\n\nPairing tools from the RL-trained 4B writer with a frozen **LFM2.5-350M**\nconsumer lifts its held-out accuracy from 11.6 to **42.9**, matching a tool-writer\n*eighty times larger* (Qwen3-30B-A3B, untrained, at 41.5).\n\nRG (Unseen) accuracy of LFM2.5-350M as the tool-writing source changes; the writer itself is never fine-tuned on LFM2.5's outputs.\n\nThe reverse direction also holds. Pairing SMITH's 4B-written tools with a much stronger\n**Qwen3-30B-A3B** consumer beats that same 30B model writing tools for itself (LATM*) on\nevery task group, most sharply on TabMWP-Hard (0.7 → 38.8), lifting the task-weighted overall score\nfrom 70.2 to 76.6. A bigger tool user doesn't make its own self-written tool preferable. SMITH's 4B\nwriter remains the better source of tools either way.\n\n### Out-of-domain generalization\n\nNeither TabMWP-Hard (tabular reasoning) nor GQA (visual question answering) appears anywhere in SMITH's training data. SMITH leads TabMWP-Hard outright and is the only 4B, non-distilled method to top either column. The GQA gap to GPT-4.1-distilled LATM is the acknowledged cost of not distilling visual primitives from a stronger oracle.\n\n| Task | CoT | LATM | LATM* (30B) | CRAFT | TroVE | KTCE | ReTool | LATM (distill) | SMITH |\n|---|---|---|---|---|---|---|---|---|---|\n| TabMWP-Hard | 7.2 | 19.7 | 0.7 | 30.0 | 36.4 | 27.2 | 3.0 | 7.1 | 40.4 |\n| GQA | 11.5 | 35.0 | 29.8 | 21.9 | 21.4 | 0.0 | 26.1 | 56.0 | 42.6 |\n\n### Scaling across backbones\n\nThe same RL recipe, applied unmodified, improves every backbone tested, including\n**Granite-3.3-8B**, which starts from a much weaker base. A **Self-Judge**\nvariant (Qwen3-8B judging its own rollouts instead of an external 30B-A3B judge) pushes held-out RG\neven higher (85.9) but trades off OOD GQA, suggesting a smaller self-judge is weaker but less biased\non in-distribution data.\n\n| Method | RG (Seen) | RG (Unseen) | TabMWP | GQA |\n|---|---|---|---|---|\n| Qwen3-8B (baseline) | 72.6 | 72.2 | 42.4 | 17.3 |\nSMITH: Qwen3-8B | 79.4 | 81.7 | 56.7 | 28.7 |\nSMITH: Self-Judge | 74.7 | 85.9 | 54.5 | 16.3 |\n| Granite-3.3-8B (baseline) | 31.2 | 22.0 | 3.9 | 7.8 |\nSMITH: Granite-3.3-8B | 39.1 | 28.5 | 4.5 | 11.7 |\n\n### Generalization to external tool-calling\n\nSMITH is never trained on [BFCL v4](#)'s schemas, multi-turn traces, or\njudges, so a gain here isolates a learned tool-use prior rather than benchmark-specific fitting. It\nlifts BFCL overall accuracy on both Qwen backbones, most sharply on Qwen3-8B.\n\n| Qwen3-4B-Instruct | Qwen3-8B | Granite-3.3-8B | ||||\n|---|---|---|---|---|---|---|\n| Metric | Base | SMITH | Base | SMITH | Base | SMITH |\n| BFCL v4 | 45.1 | 48.6 | 43.3 | 55.8 | 36.3 | 38.7 |\n\n## Ablation: what's actually driving the gain?\n\nWe isolate whether joint training, rather than simply having a specialist builder and a specialist\nuser, is the active ingredient. Pairing a separately-trained 30B-A3B builder with a separately-trained\n4B tool-use specialist (**Decoupled Create/Use**) does *not* beat a single model\ntrained on tool creation alone (58.9 vs. 68.8 RG Unseen). Only the full objective, which keeps\nexecution reward and judge reward as disentangled axes and closes the loop in one policy, reaches the\nbest aggregate score.\n\n| Method | πeval | Tool use? | RG (Seen) | RG (Unseen) | TabMWP | GQA |\n|---|---|---|---|---|---|---|\n| Qwen3-4B-Instruct (base) | n/a | n/a | 61.9 | 47.0 | 19.7 | 20.9 |\n| Tool Create | 30B-A3B | No | 77.4 | 59.4 | 15.6 | 37.0 |\n| Tool Create | 4B | No | 73.9 | 68.8 | 17.7 | 35.8 |\n| Decoupled Create/Use | 30B-A3B | Yes | 76.4 | 58.9 | 13.9 | 35.0 |\n| SMITH: No Sync | 4B | Yes | 80.3 | 66.9 | 25.8 | 24.6 |\n| SMITH: No LLM Judge | self | Yes | 82.6 | 67.8 | 18.3 | 42.6 |\n| SMITH: K = 1 | self | Yes | 78.6 | 73.9 | 46.8 | 32.3 |\nSMITH: Full | self | Yes | 86.6 | 78.3 | 40.4 | 42.6 |\n\nTool creation alone yields a strong in-distribution bump but underperforms on OOD GQA; coupling build and use without disentangled rewards lifts in-distribution accuracy but hurts held-out transfer. Only keeping process quality (format, judge) and outcome correctness (execution, use) as separate reward axes gets both.\n\n## Conclusion\n\nSMITH jointly trains a single language model to *create* and *use* reusable tools,\nclosing the feedback loop between tool writer and tool user so the policy is optimized directly on\nits own execution outcomes. Trained on 13 Reasoning-Gym tasks, the 4B model attains the highest\nheld-out accuracy among all evaluated methods, leads TabMWP-Hard, and writes tools that transfer to a\n350M student never seen during training, matching tools from a writer an order of magnitude larger.\nThe same recipe lifts Qwen3-8B and Granite-3.3-8B without modification: coupling creation and use\ninside one trained policy is a scalable path to generalization, without a larger frozen teacher, a\nmore elaborate scaffold, or out-of-domain supervision.\n\n## BibTeX\n\n```\n@article{tam2026smith,\n  title   = {Joint Optimization of Tool Creation and Use for Large Language Model Agents},\n  author  = {Tam, Zhi Rui and Lin, Chieh-Yen and Chen, Yun-Nung and Sun, Shao-Hua and Lee, Hung-yi},\n  year    = {2026},\n  journal = {arXiv preprint},\n  url     = {https://tool-use-smith.github.io}\n}\n```\n\n", "url": "https://wpnews.pro/news/joint-optimization-of-tool-creation-and-use-for-large-language-model-agents", "canonical_source": "https://tool-use-smith.github.io/", "published_at": "2026-08-26 01:36:48+00:00", "updated_at": "2026-08-26 02:14:19.949468+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-agents"], "entities": ["Appier AI Research", "National Taiwan University", "SMITH", "Qwen3", "DAPO", "GRPO", "TabMWP-Hard", "GQA"], "alternates": {"html": "https://wpnews.pro/news/joint-optimization-of-tool-creation-and-use-for-large-language-model-agents", "markdown": "https://wpnews.pro/news/joint-optimization-of-tool-creation-and-use-for-large-language-model-agents.md", "text": "https://wpnews.pro/news/joint-optimization-of-tool-creation-and-use-for-large-language-model-agents.txt", "jsonld": "https://wpnews.pro/news/joint-optimization-of-tool-creation-and-use-for-large-language-model-agents.jsonld"}}