{"slug": "toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms", "title": "ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs", "summary": "Researchers have developed ToolSense, an open-source diagnostic framework that automatically generates three benchmarks to audit whether large language models truly understand their parametric tool knowledge. Testing on ToolBench's 47,000 tools revealed a knowledge-retrieval dissociation where models collapsed by 50-64 percentage points on realistic queries compared to standard benchmarks, with some scoring near-random on factual probes despite strong retrieval performance. The findings expose critical limitations in current tool-retrieval evaluation methods, as embedding-based and parametric approaches may achieve high scores on fully-specified queries without genuine tool comprehension.", "body_md": "arXiv:2606.12451v1 Announce Type: new\nAbstract: Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck. As embedding-based retrieval approaches rely on compact encoders that may under-capture specialized tool semantics, parametric tool retrieval addresses this by encoding each tool as a virtual token appended to the LLM vocabulary, fine-tuned in two stages (memorization then retrieval SFT) to use the LLM as a retriever, achieving strong performance on standard ToolBench retrieval benchmarks. Yet these benchmarks use verbose, fully-specified queries, and their evaluation applies constrained decoding that restricts outputs to valid token paths, neither reveals whether the model actually understands its tools. We introduce \\textbf{ToolSense}, an open-source LLM-powered diagnostic framework that takes any tool catalog as input and automatically generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. Applying ToolSense to ToolBench (~47k tools) and evaluating five parametric model training configurations reveals a knowledge-retrieval dissociation: on RRB queries, several configurations collapse by ~50-64 percentage points compared to fully-specified ToolBench benchmarks, falling below the embedding-model baseline. Additionally, despite strong retrieval performance, some models score near-random on factual probes, suggesting a knowledge-retrieval dissociation. We open-source the ToolSense framework and the ToolBench diagnostic benchmarks at https://github.com/SAP/toolsense.", "url": "https://wpnews.pro/news/toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms", "canonical_source": "https://arxiv.org/abs/2606.12451", "published_at": "2026-06-12 04:00:00+00:00", "updated_at": "2026-06-12 04:51:06.879018+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-agents", "ai-research", "ai-tools"], "entities": ["ToolSense", "ToolBench", "LLM", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms", "markdown": "https://wpnews.pro/news/toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms.md", "text": "https://wpnews.pro/news/toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms.txt", "jsonld": "https://wpnews.pro/news/toolsense-a-diagnostic-framework-for-auditing-parametric-tool-knowledge-in-llms.jsonld"}}