Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable A developer behind langchainrust 0.30.0 has published an approach for making LLM-judge scores comparable in best-of-N agent selection by requesting logprobs and top_logprobs from the judge model instead of relying on the returned integer. The method computes an expected score and confidence from the token distribution, so two candidates that both score 8 can be distinguished by whether the judge assigned 8 a probability of 0.90 or 0.50. The implementation, in lc-evaluation's verifier and judge_reliability modules, requires only a one-line config change, with_top_logprobs(5), and the author cautions that the resulting confidence reflects the model's certainty about its token choice, not a calibrated measure of answer quality. One sentence: the model gives you an integer; a decision needs a measurement. This article gives the fix first three steps , then explains why it works, and ends with the engineering constraints you cannot skip. The code below comes from the real implementation in langchainrust 0.30.0: lc-core/src/language models/chat.rs , lc-evaluation/src/verifier.rs , lc-evaluation/src/judge reliability.rs . This is a common agent scenario: the agent produces several candidates at one step answers, plans, or tool calls , then calls another LLM as a judge and picks one to continue. In agent systems this is called best-of-N / verifier-guided selection . It shows up in: The common factor: the judge is itself an LLM. So the judge returns: Candidate A: 8 Candidate B: 8 Which one do you pick? You can't. The integer 8 only tells you "it chose 8". It does not tell you "how confident it was". The distribution behind that same call looks like this: Candidate A: 8 @ 0.90, 7 @ 0.08, 9 @ 0.02 Candidate B: 8 @ 0.50, 7 @ 0.40, 9 @ 0.10 A is very sure about 8. B only picked 8 by a 0.50-to-0.40 margin over 7 — it had a 40% chance of actually wanting to say 7. Two 8s are not the same 8. You do not need a different model, and you do not need to retrain anything. You only need to take back the information the model already computed but does not return by default: logprobs and top logprobs . Same two candidates, two selection methods: | Candidate | Judge output | Integer only | Verifier expected / confidence | |---|---|---|---| | A | 8 | 8 | 7.94 / 0.90 | | B | 8 | 8 | 7.70 / 0.50 | That is the whole fix. In code it is one extra line: js let config = OpenAIConfig::from env result ?.with top logprobs 5 ; The rest of the article unpacks this: The first reaction is usually: isn't the judge "just directly giving a score"? Why would there be a probability distribution? Because the model has no "scoring module" and no regression head that emits a continuous value. It does exactly what it does when writing a poem or code: predict the next token. The prompt looks like this: Score the answer below on a 0-9 scale. Output exactly one digit. Answer: ... Score: The model's job is to complete the token after Score: . It computes one softmax over the whole vocabulary; 8 has the highest probability, so it emits 8 . So: What you see: 8 What the model did: computed a probability over the vocabulary, 8 ranked first, emitted 8 "Direct scoring" is an analogy to human behavior, not the model's operation. Because of that, there is always a distribution behind the chosen token. The default API response simply does not return it. | Layer | Content | Who provides it | |---|---|---| | Choice | It emitted 8 | The model | | Confidence | Whether 8 had probability 0.9 or 0.5 | The model logprobs | | True quality | How good an 8 actually is | The model does not measure this | The third layer is the easiest to misread: 0.90 does not mean "this answer has a 90% chance of being an 8-quality answer". It means "the model is very sure the next token is 8". The model gives you an opinion and the strength of that opinion , not a calibrated quality measurement. This is not a thermometer. It is a multiple-choice answer. You ask a person, "Rate this answer from 1 to 10." They say "8". You see the answer; you do not see whether they said it instantly or hesitated between 7 and 8. The model is different: its hesitation is digitized and readable. 0.90 is an instant answer; 0.50 is a hesitant one. Step 2 says "take the distribution back". This section explains exactly what you are taking, what the fields are called, and what the response looks like. The model does not generate characters or words. It generates tokens . A token can be: 8 10 points a space the At each step the model does one thing: given the text so far, predict the next token. It computes a probability over the entire next token "8" probability 0.90 next token "7" probability 0.08 next token "9" probability 0.02 next token "points" probability 0.001 ... logprob is If the probability is p , the logprob is ln p . | Probability p | logprob = ln p | |---|---| | 0.90 | -0.105 | | 0.50 | -0.693 | | 0.08 | -2.526 | | 0.02 | -3.912 | | 0.001 | -6.908 | Why logarithms instead of probabilities? Converting back is one step: probability = exp logprob exp -0.105 ≈ 0.90 . In the request: { "logprobs": true } This means: Return the log probability of the token you actually generated . If the model generated 8 , the response includes: { "token": "8", "logprob": -0.105 } Without it you only see 8 and have no idea how sure the model was. { "logprobs": true, "top logprobs": 5 } At each generated position, in addition to the generated token, return the 5 most likely candidate tokens . The response looks like this: { "token": "8", "logprob": -0.105, "top logprobs": { "token": "8", "logprob": -0.105 }, { "token": "7", "logprob": -2.526 }, { "token": "9", "logprob": -3.912 }, { "token": "6", "logprob": -4.605 }, { "token": "1", "logprob": -5.298 } } 8@0.90, 7@0.08, 9@0.02 comes from. One naming quirk: In the code, entry.top logprobs refers to the response array. ln p ≤ 0 . Closer to 0 means more likely. Section 2 gave the result. This section fills in the derivation so you can check it yourself. Taking the argmax throws away both at the same instant. | Candidate | Distribution score @ probability | Expected | Confidence | argmax | |---|---|---|---|---| | A | 8@0.90 mailto:8@0.90 , 7@0.08 mailto:7@0.08 , 9@0.02 mailto:9@0.02 | 7.94 | 0.90 | 8 | | B | 8@0.50 mailto:8@0.50 , 7@0.40 mailto:7@0.40 , 9@0.10 mailto:9@0.10 | 7.70 | 0.50 | 8 | | C | 9@0.50 mailto:9@0.50 , 6@0.40 mailto:6@0.40 , 7@0.10 mailto:7@0.10 | 7.60 | 0.50 | 9 | There are only two formulas: expected score = Σ probability i × score i confidence = max probability / total probability In the example the probabilities already sum to 1, so the denominator is 1. In real calls the top-k probabilities usually sum to less than 1, so you must normalize — Section 7 covers this. Method A: integer only argmax A = 8, B = 8, C = 9 → A and B are indistinguishable; C ranks first Method B: expected score A 7.94 B 7.70 C 7.60 → select A Method C: expected score + confidence A 7.94, 0.90 → better on average and more certain The three methods disagree A selects C; B and C select A . This is not a contrived coincidence: The problem is not "whether the judge is accurate". It is that the information argmax discards is exactly the information you need to judge accuracy. The verifier fixes "scores are not comparable". It does not fix "the judge itself is biased". Even with a distribution, you cannot rely on a single judgment. These are not occasional mistakes. They are repeatable: There is also a statistical trap: raw agreement is inflated . It includes "random chance agreement"; subtract that and the real consistency drops noticeably. Do not treat "85% agreement" as a conclusion — look at kappa first. lc-evaluation/src/judge reliability.rs provides: Order swap php pub fn order swap consistent original: Verdict, swapped: Verdict - Verdict { match original, swapped { Verdict::AWins, Verdict::BWins = Verdict::AWins, Verdict::BWins, Verdict::AWins = Verdict::BWins, = Verdict::Tie, } } In the swapped round, "B wins" means the original A won. A candidate wins only if both rounds point to the same original answer. Cohen's kappa js let kappa = cohen kappa &judge a, &judge b ; How much agreement remains after subtracting chance agreement. Degenerate samples where everything is positive or everything is negative return None instead of pretending to have a conclusion. Same-family detection assert same family "gpt-5", "gpt-4o" ; assert same family "gpt-5", "claude-3" ; Either reject same-family judging or mark it explicitly in the report. Summary report pub struct ReliabilityReport { pub n: usize, pub agreement: f64, pub kappa: Option