Recently, I have been exploring uncertainty quantification in language models and, as part of a broader interest in AI literacy, what happens when it’s exposed to users in meaningful ways. Having attended this year the 10th edition of the Toronto ML Summit (and, relevant to the following, a NHL game in Montréal), I can attest this topic still seems to be relegated to backwater conversations at conferences targeting industry rather than research.
I set out to build a simple agent and to test its limits while interacting with it as if this was a friendly conversation with an expert. For this purpose, I narrowed down the domain knowledge to something I myself still have a lot to learn about. Shall I be able to spot false information, or shall I be fooled by its ability at confabulating plausible answers?
I used Wikipedia as the basis for what constitutes ground truth, and provided the agent with external tools to grab specific, possibly real-time information such as weather conditions. I also decided not to give it access to web search directly, to see what happens when it doesn’t know that it doesn’t know.
Keep reading to know more about slashing, icing, predictions for the upcoming NHL season, and what happened to a Scottish physicist I worked for whose name didn’t go unnoticed at the Toronto Pearson airport.
To set things up, I decided to measure the uncertainty from logprobs (short for log probabilities), a numerical representation of how confident a language model is in the words (tokens) it generates. One can calculate the overall confidence of a model’s answer by averaging the logprobs over the generated paragraph. The highest the average logprob of a sentence, the more naturally the text generation “flows”. On the other hand, if an LLM outputs a low average logprob, it usually means the model was “stretching” its knowledge, which may be an indication of hallucination, or more generally of poor coverage of the domain in the training dataset.
Besides logprobs, I also ask the LLM to self-report its level of confidence, quantified in low/mid/high buckets, which are usually easier to interpret than a percentage. There is not obvious mechanism for the LLM to observe its own state (something often called “introspection”), so self-reporting is prone to hallucination as well. Notoriously, LLMs are particularly good at confabulating credible answers, so much so that self-reporting basically should not be trusted, but can still be used as diagnostic for consistency.
Since the main providers of frontier models do not return logprobs, and also for reproducibility and privacy reasons, I decided to run a llama.cpp server locally to serve a small model called Qwen2.5–7B-Instruct-Q4_K_M .
To provide the ground truth, I let the user decide to pre-fetch a number of Wikipedia pages, which constitute the knowledge base (KB). I also added two tools, one to fetch weather conditions, one to get NHL results, to let the LLM extend its reach when the intrinsic information coming from its training data does not cover the asked question. However, I also added a fact-checking layer that makes use once again of Wikipedia, triggered when the self-reported confidence is low.
I created the UI of P.U.C.K. (“Power plays, Uncertainty, Confidence & Knowledge”) such that the chat and the diagnostics are separated in the two halves of the screen. Questions can be asked by typing in a textbox at the bottom right, or repeated by clicking on a small arrow that appears in each bubble. On the right, a gauge indicator shows the measured level of confidence (a.k.a. b.s.-meter). For each question, the source of retrieved information is indicated: the knowledge base from Wikipedia, the intrinsic knowledge from model training, or from a tool call.
The colour scheme is supposed to remind a hockey rink :-) As for the knowledge base, the user can load a number of pages (the names have to match actual Wikipedia pages), with some pre-defined suggestions. This is particularly important for some topics such as stats and bio of specific players, which arguably only appear generically in the model training dataset. Additional sources can be added anytime during the chat, if the user thinks they might be relevant.
When a tool call happens, a printout of the API call is reported in another tab in the right panel.
This is it. Let’s ask some questions, and let’s see how the average confidence evolves during the conversation.
I started with something simple, to test its ability to retrieve factual information from the given sources:
Q: First up, who holds the all-time NHL points record?
A: The all-time NHL points record is held by Wayne Gretzky. He has the most career points and assists in NHL history.
✔️fact-check: supported. checked against your loaded KB: Ice hockey, National Hockey League, Stanley Cup, Wayne Gretzky, Toronto Maple Leafs, Montreal Canadiens, Bobby Orr.
Hallucination Risk: Low
Up next, a couple of basic questions:
Q: What can cause a 4-on-4 power play part of the game? A: A 4-on-4 power play in ice hockey occurs when both teams are on the ice but one team has been assessed a penalty, reducing their number of skaters to four for a brief period. This situation can happen when a team commits a minor penalty, such as holding, tripping, or slashing, and the penalty has not yet expired.
Q: Can you remind me what slashing means?
A: Slashing in ice hockey is a penalty called when a player makes an illegal, forceful swing with their stick that contacts an opponent. This action is considered dangerous and is against the rules because it can lead to injuries. The referee will stop the play and assess a minor penalty to the player who committed the infraction.
Ok, that was good. It’s also worth noting that logprobs and self-reported confidence scores do not always line up.
Then, I tested something that does not appear in the knowledge base, but is supposed to be common lore:
Q: Okay, off the cheat sheet now. Just hockey. What does “icing” mean?
A: Icing in ice hockey is a violation where a player shoots the puck from behind the center red line across the opposite goal line without it being touched by any player before it crosses the goal line. If the referee deems this to be a deliberate attempt to waste time and not a normal play, the puck is dropped at the center red line, and the team that iced the puck is assessed a minor penalty.
This is an interesting situation: the self-report was very high, but the logprobs were not. While the answer was factually correct, the system was essentially unable to establish its truthfulness, hence the average confidence was raised to “moderate”.
I’ll skip ahead a bit to show an example of when the agent actually gives a honest answer and recommends an external cross-check.
Q: I’m curious about hockey’s origins. What was the attendance at the very first NHL game?
A: The first National Hockey League (NHL) game was played on November 22, 1917, between the Montreal Canadiens and the Montreal Wanderers at the Victoria Skating Rink in Montreal, Quebec. The attendance for this game is not definitively recorded in historical records, but it was likely around 3,000 to 4,000 people.
In this case, the self-report was low, source was model training data, and the fact-checking layer reported:
? fact-check: The excerpt does not provide specific attendance figures for the first NHL game. checked against History of the National Hockey League • freshly fetched, not your loaded KB
Do you think this answer is factually correct, a total hallucination, or somewhere in between? If you’re curious, you can read more about the very first NHL games on this page.
Then, I tried to trick the agent into avoiding external calls, look into its own memory, and expose its own shortcomings. Please note that, at the time of writing (August 2026), the 2025/2026 NHL season was already over and the winner was already well established.
Q: Now for something you couldn’t possibly know — not because it’s obscure, but because it hasn’t happened yet, as far as you’re concerned. Who won the Stanley Cup last season?
A: I cannot provide information on who won the Stanley Cup last season because that information has not yet occurred in my data. The Stanley Cup playoffs and finals have not concluded as of the time you are asking this question.
𐄂 fact-check: contradicted — The Wikipedia excerpt clearly states that the 2022 Stanley Cup playoffs concluded on June 26, 2022, with the Colorado Avalanche winning the Stanley Cup, which contradicts the claim.
Interestingly, this also shows that the model training data stop at 2022 for what concerns NHL and possibly other sports. And yes, the Colorado Avalanche did win the 2022 Stanley Cup.
I then deployed a card I had been keeping up my sleeve for a while. This is a true story, and I was eager to see what the agent was able to confabulate.
Q: I have an anecdote. My ex-boss’ name is Robert Orr. He’s a physicist from Scotland, but lives in Toronto. He told me that once he came back from the US. At the border, the customs officer looked at his passport, glanced at him, and said ‘Welcome back, Bobby!’
A: That’s an intriguing anecdote! While Bobby Orr is indeed a famous figure in the world of ice hockey, it’s less likely that a customs officer would recognize a physicist named Robert Orr by his nickname “Bobby.” However, such an encounter is not impossible, especially if the customs officer has a personal connection to Bobby Orr or is simply impressed by the name. It’s possible that your ex-boss Robert Orr might have a well-known surname or a name that sounds similar to Bobby Orr’s, leading to this interaction. Alternatively, the customs officer might have a personal connection to the Orr family or simply be familiar with the nickname due to its prominence in sports history. Your anecdote adds a personal touch to the well-known figure of Bobby Orr and highlights the unique interactions that can occur at border crossings.
One can’t talk about sports without asking for predictions. To do that, I added a skill to the agent so that it can look up team stats and make use of them to generate an answer. While this is not intended to be a sport-betting platform, it still shows how tool calls happen, and how a language model can make use of the numbers coming back.
Q: Let’s see what you think. What are the Canadiens’ chances of winning the Cup in the next three years?
A: Based on the current form and recent performance, the Montreal Canadiens have shown strong potential but face significant competition in the NHL. Their chances of winning the Stanley Cup in the next three years depend on several factors, including the performance of their roster, the health of key players, and the strength of the other teams in the league. The Canadiens have a talented roster with some high-profile players and solid prospects. However, they have not been consistent in recent seasons and have faced challenges in the playoffs. To significantly increase their chances of winning the Cup, the team needs to improve their consistency and performance in the latter stages of the season and playoffs. Given the current state of the team, my guess is that the Canadiens have a moderate chance of winning the Stanley Cup in the next three years, but it would require a strong performance and some favorable playoff matchups.
Note that, to answer this question, the agent made a tool call to an API to retrieve stats about the Montréal Canadiens.
Q: How about the Maple Leafs? Do you think they finally have a chance this year?
A: Based on the current form and recent performance, the Toronto Maple Leafs have shown significant improvement under their current coaching and management. They have a strong roster with several talented players and a solid young core. However, their success in the playoffs has been inconsistent, and they have faced tough competition from other teams in the league. The Maple Leafs have the potential to be a formidable team, but winning the Stanley Cup requires more than just talent; it also involves consistency, health, and a bit of luck. Given their current state, my guess is that the Maple Leafs have a moderate chance of making a deep playoff run this year, but winning the Stanley Cup would be a significant challenge.
This was just a simple attempt at demonstrating how an “honest agent” is supposed to expose its own limits to users. Ideally, information about confidence scores could be leveraged to invoke the intervention of humans-in-the-loop, or more in general, of external review.
However, I think a broader question remains. Do people become better users of AI platforms after being exposed to their shortcomings? Or do they simply stop trusting them across the board? I don’t think there’s yet a definite answer for that, even though there’s an ever growing body of literature on the subject (Bhatt et al. 2021, Papenmeier et al. 2022, Kim et al., 2024, Zhao et al., 2024). What I noticed is that so far most of these studies have been conducted in specific settings such as German-speaking communities (Koch et al. (MAILS), 2024) or videogame players (Reyes et al., 2025), rather than “in the wild”. There’s nothing wrong with it, as long as we acknowledge that more work has to be done to force the upper hand and convince the main players in the field (e.g. OpenAI and Anthropic) to be more explicit about the limitations of their otherwise apparently almighty AI models.
I can not leave this last one unmentioned. This is arguably the mother of all hallucinations, one that is also particularly ridiculous for anyone living in Toronto:
Q: When did the Leafs last win the Cup?
A: The Toronto Maple Leafs last won the Stanley Cup in 2018. This was their 14th championship, the most recent of which came after an extended championship drought that lasted 57 seasons, the longest in NHL history.
𐄂 fact-check: contradicted — The text states that the Toronto Maple Leafs’ championship drought is the longest in league history at 57 seasons, which contradicts the claim that they last won the Stanley Cup in 2018. checked against your loaded KB: Ice hockey, National Hockey League, Montreal Canadiens, Toronto Maple Leafs, Bobby Orr, Wayne Gretzky, Stanley Cup
Source code: https://github.com/rdisipio/honest-agent Conversation With An Honest Agent was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.