Uncertainty Quantification for LLM Function-Calling

Researchers at Apple and other institutions published the first evaluation of uncertainty quantification methods for LLM function-calling, finding that multi-sample methods like Semantic Entropy offer no clear advantage over simpler single-sample approaches. The study proposes improvements leveraging function-calling output structure, such as clustering based on abstract syntax tree parsing and selecting semantically meaningful tokens for logit-based scores.

content type paper /research/ published July 2026 Uncertainty Quantification for LLM Function-Calling AuthorsZihuiwen Ye† , Lukas Aichberger† , Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal†, Arno Blaas‡, Adam Goliński‡ Uncertainty Quantification for LLM Function-Calling AuthorsZihuiwen Ye† , Lukas Aichberger† , Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal†, Arno Blaas‡, Adam Goliński‡ Large Language Models LLMs are increasingly deployed to autonomously solve real-world tasks. A key ingredient for this is the LLM Function-Calling paradigm, a widely used approach for equipping LLMs with tool-use capabilities. However, an LLM calling functions incorrectly can have severe implications, especially when their effects are irreversible, e.g., transferring money or deleting data. Hence, it is of paramount importance to consider the LLM’s confidence that a function call solves the task correctly prior to executing it. Uncertainty Quantification UQ methods can be used to quantify this confidence and prevent potentially incorrect function calls. In this work, we present what is, to our knowledge, the first evaluation of UQ methods for LLM Function-Calling FC . While multi-sample UQ methods, such as Semantic Entropy, show strong performance for natural language Q&A tasks, we find that in the FC setting, it offers no clear advantage over simple single-sample UQ methods. Additionally, we find that the particularities of FC outputs can be leveraged to improve the performance of existing UQ methods in this setting. Specifically, multi-sample UQ methods benefit from clustering FC outputs based on their abstract syntax tree parsing, while single-sample UQ methods can be improved by selecting only semantically meaningful tokens when calculating logit-based uncertainty scores. Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results June 27, 2025 research area Methods and Algorithms /research/?domain=Methods%20and%20Algorithms , research area Speech and Natural Language Processing /research/?domain=Speech%20and%20Natural%20Language%20Processing conference ACL /research/?event=ACL Uncertainty Quantification UQ in Language Models LMs is key to improving their safety and reliability. Evaluations often use metrics like AUROC to assess how well UQ methods e.g., negative sequence probabilities correlate with task correctness functions e.g., ROUGE-L . We show that mutual biases—when both UQ methods and correctness functions are biased by the same factors—systematically distort evaluation. First, we formally prove that… Efficient and Effective Uncertainty Quantification in LLMs November 21, 2024 research area Speech and Natural Language Processing /research/?domain=Speech%20and%20Natural%20Language%20Processing Workshop at NeurIPS /research/?event=NeurIPS%20Workshop This paper was accepted at the Safe Generative AI Workshop SGAIW 2024 at NeurIPS 2024. Uncertainty quantification UQ is crucial for ensuring the safe deployment of large language model, particularly in high-stakes applications where hallucinations can be harmful. However, existing UQ methods often demand substantial computational resources, e.g., multi-sample methods such as Semantic Entropy Kuhn et al., 2023 usually require 5-10 inference…