content type paperpublished July 2026 Uncertainty Quantification for LLM Function-Calling
AuthorsZihuiwen Ye†, Lukas Aichberger†, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal†, Arno Blaas‡, Adam Goliński‡
Uncertainty Quantification for LLM Function-Calling
AuthorsZihuiwen Ye†, Lukas Aichberger†, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal†, Arno Blaas‡, Adam Goliński‡
Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks. A key ingredient for this is the LLM Function-Calling paradigm, a widely used approach for equipping LLMs with tool-use capabilities. However, an LLM calling functions incorrectly can have severe implications, especially when their effects are irreversible, e.g., transferring money or deleting data. Hence, it is of paramount importance to consider the LLM’s confidence that a function call solves the task correctly prior to executing it. Uncertainty Quantification (UQ) methods can be used to quantify this confidence and prevent potentially incorrect function calls. In this work, we present what is, to our knowledge, the first evaluation of UQ methods for LLM Function-Calling (FC). While multi-sample UQ methods, such as Semantic Entropy, show strong performance for natural language Q&A tasks, we find that in the FC setting, it offers no clear advantage over simple single-sample UQ methods. Additionally, we find that the particularities of FC outputs can be leveraged to improve the performance of existing UQ methods in this setting. Specifically, multi-sample UQ methods benefit from clustering FC outputs based on their abstract syntax tree parsing, while single-sample UQ methods can be improved by selecting only semantically meaningful tokens when calculating logit-based uncertainty scores.
Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results
June 27, 2025research area Methods and Algorithms, research area Speech and Natural Language Processingconference ACL
Uncertainty Quantification (UQ) in Language Models (LMs) is key to improving their safety and reliability. Evaluations often use metrics like AUROC to assess how well UQ methods (e.g., negative sequence probabilities) correlate with task correctness functions (e.g., ROUGE-L). We show that mutual biases—when both UQ methods and correctness functions are biased by the same factors—systematically distort evaluation. First, we formally prove that…
Efficient and Effective Uncertainty Quantification in LLMs
November 21, 2024research area Speech and Natural Language ProcessingWorkshop at NeurIPS This paper was accepted at the Safe Generative AI Workshop (SGAIW) 2024 at NeurIPS 2024.
Uncertainty quantification (UQ) is crucial for ensuring the safe deployment of large language model, particularly in high-stakes applications where hallucinations can be harmful. However, existing UQ methods often demand substantial computational resources, e.g., multi-sample methods such as Semantic Entropy (Kuhn et al., 2023) usually require 5-10 inference…