{"slug": "are-you-benchmarking-the-model-or-the-harness", "title": "Are You Benchmarking the Model—or the Harness?", "summary": "An engineer investigating AI model behavior in the Kai!AI Arena discovered that apparent model traits were actually artifacts of the evaluation harness. After fixing four bugs, including a probability leak and missing dialogue, the blind-bid rate of DeepSeek V4-Pro dropped from 40% to 6%, and other model differences shrank. The findings highlight the importance of distinguishing model performance from harness effects in AI evaluation.", "body_md": "This article was edited with AI assistance. The cases and data come from real batch runs in the\n\nKai!AI Arena. Every claim is limited to the game rules, model versions, and experimental setup used at the time. This is not a general model ranking.\n\nIf I had published one day earlier, DeepSeek V4-Pro might have acquired a personality trait it never had: **bidding without looking at its dice.**\n\nFirst, a quick explanation of the table. In Liar's Dice, each player has a set of hidden dice. Players take turns claiming that the whole table contains at least *N* dice of a particular face. The next player must either raise the bid or challenge it. Everyone then reveals their dice: if the bid holds, the bidder wins; if it does not, the challenger wins. In *Kai!*, looking at your own dice is an explicit action, so a player may bid before looking. I call that a blind bid.\n\nThe evidence looked solid. In the first batch of AI matches, DeepSeek V4-Pro made nearly 40% of its bids before looking at its dice.\n\nA story almost wrote itself. Perhaps it trusted intuition. Perhaps it was unusually willing to gamble or liked to seize the initiative. Another model usually looked first and calculated before bidding. Put the win rate, dialogue, and action logs side by side, and two distinct “personalities” seemed to emerge.\n\nThen I inspected the context.\n\nEven when the model had not used the probability tool, the system was still inserting a rough probability estimate into its prompt. The candidate actions were also sorted by probability. The model appeared to be bidding under the cup, but the harness was feeding it a strong hint from offstage.\n\nAfter I fixed the leak, V4-Pro's blind-bid rate fell from roughly **40% to 6%**.\n\nWhat looked like a model insight had been an explanation of a software bug. A compelling conclusion vanished from the data.\n\nThen I found three more bugs of the same kind. Across the first roughly 60 matches, more than half of the most visible differences between models shrank after the fixes.\n\nThat changed how I think about model evaluation:\n\n**An arena directly measures a system made of a model and a harness. Before attributing the result to the model, you must show that the measurement system did not quietly think for it, hide part of its input, or rewrite its failures.**\n\nBy *harness*, I mean everything wrapped around the model: prompts, context assembly, tools, action spaces, token budgets, provider routing, output parsing, retries, and fallbacks. The model produces an answer. The harness decides what it sees, what it is allowed to do, and which part of that answer survives into the database.\n\nAt first I thought I had found one implementation mistake. I kept looking and found a second, a third, and a fourth. The troubling part was that none of them stopped the matches. Every game still produced actions, dialogue, and a final score. The dataset looked complete.\n\n| Harness problem | The apparent model trait | What changed after the fix |\n|---|---|---|\n`max_tokens=400` truncated long outputs |\nPoor formatting; often replaced by a fallback bot | Format failures and fallbacks dropped sharply |\n| Rough probabilities were included without a tool call, and actions were sorted by probability | Liked to bid without looking | Blind-bid rate fell from about 40% to 6% |\n| Opponent dialogue was not forwarded | Weak player with little social reasoning | Win rate returned to roughly 40–60% |\n| Subjective judgments were stored only up to the first 100 characters | Rambling reasoning and incoherent records | The original chain of reasoning reappeared when full text was saved |\n\nThe four bugs interfered at four different points.\n\nThe probability leak changed the input. Missing dialogue removed information the model should have had. The token cap truncated its output. The database then changed the evidence I used to interpret that output. The harness shaped both the move and my explanation of the move.\n\nThese distortions are especially dangerous because each one generates a plausible story. A truncated answer becomes weak instruction following. Missing dialogue becomes poor social reasoning. An action list that has already ranked the safe moves becomes decision-making ability. As long as the system still produces a score, it is easy to skip the measurement process and attach a label to the model.\n\nThe uncomfortable conclusion is that the evaluation framework is also playing.\n\nSo when I see a model leaderboard now, I do not begin with who won. I begin with a different question: **What task did each model actually receive?**\n\nModel arenas often point to a shared prompt as evidence of fairness. It is necessary, but it controls only one part of the experiment.\n\nIn the early version of *Kai!*, both seats received the exact same system prompt. Any of the following could still change the task:\n\nA prompt is a string sent to a model. The task also includes its information boundary, tool permissions, compute budget, and failure policy.\n\nThis is why apparently uniform settings can create systematic bias. A concise model may escape truncation while a long-reasoning model spends its entire budget before producing a final answer. A model sensitive to option order will react to the placement of candidate actions. A model that relies heavily on conversational cues will lose more when dialogue is omitted.\n\nDoes a more uniform setup always make an experiment fairer? That depends on what you are trying to measure.\n\nMany arguments about benchmark fairness are really arguments about different questions.\n\nIf I already have a fixed product interface and want to know which model can replace another with the least work, I should hold the prompt, tools, budget, and parser constant. That measures compatibility with a shared product contract.\n\nIf I want to measure the ceiling of each model, I should optimize the prompt, tools, and reasoning settings separately. The result now includes adaptation work, so it no longer represents models running under identical conditions.\n\nIf I care about return on resources, I should fix cost, latency, or token use. That experiment measures output under the same constraint, not absolute capability.\n\nAll three protocols are valid:\n\n| Protocol | What is controlled | What it can answer |\n|---|---|---|\n| Interface fairness | Same prompt, tools, budget, and action space | Which model fits the same product contract best? |\n| Capability ceiling | Model-specific prompt, tool, and reasoning optimization | What can each model do after adaptation? |\n| Resource fairness | Same cost, latency, or token budget | Which model produces more under the same constraint? |\n\nProblems begin when the claim outruns the protocol. A shared-prompt benchmark can tell us which model works better with that prompt. It cannot directly establish each model's capability ceiling. Individually tuned results may show a ceiling, but the evaluator's tuning skill has entered the experiment.\n\nEvery benchmark report should state its fairness constraint first, then limit its claims accordingly.\n\nThe current “bare table” track in *Kai!* uses interface fairness. It answers a product question: if I swap the model behind the same game interface, what kind of opponent does the player get?\n\nOnce that protocol is chosen, the difficult work begins: keeping the harness from appearing in the score as model ability.\n\nI turned that goal into a set of concrete constraints.\n\nFirst, every model acts through the same player interface. It can see only its own dice and public events, and every action is checked by the same deterministic engine. The information boundary lives in the schema rather than in a promise written into the prompt.\n\nSecond, the same dice seed is played twice with the seats swapped. This reduces the effect of first move, seat position, and random rolls. A single win never becomes a model-level conclusion.\n\nThird, I freeze more than the prompt. Candidate order, context serialization, tool responses, sampling parameters, parsers, and code versions are all experimental conditions. Each batch stores a prompt hash and Git commit. If one of them changes, I start a new batch.\n\nFourth, I record what the provider actually did. Sending `max_tokens`\n\nor a reasoning flag does not prove that the server honored it. Each call records completion tokens, reasoning tokens, finish reason, latency, cost, and actual route. Empty responses and timeouts remain in the dataset.\n\nFinally, retries, repairs, and bot takeovers appear beside the score. A production system needs fallbacks, but an evaluation cannot let a fallback quietly impersonate model behavior. The leaderboard reports formatting failures, refusals, and fallback rates, with a separate view for zero-fallback samples.\n\nThese rules sound like engineering hygiene, but they decide whether the conclusion holds. A reliable product harness hides failures so the player can continue. A credible evaluation harness exposes them. The two can share code; they cannot share an unmarked data definition.\n\nEven after all this, a leaderboard has boundaries. Controlling variables can make an answer more reliable. It cannot make the experiment answer a question it never asked.\n\nI divide the *Kai!* metrics into three layers:\n\nEach layer is useful, but its interpretation is limited. A low format-failure rate may come from better instruction following or a more forgiving parser. A high win rate shows strength at this game; it does not automatically become a general reasoning score. Behavioral differences matter to the product experience, but with thin samples they are observations, not permanent personalities.\n\nThe current clean set contains seven model IDs, 11 pairings, 22 seat arms, and 44 matches. That is enough to expose obvious harness failures. It is nowhere near enough for a general model ranking. The memory track also introduces cross-match path dependence, so it is stored separately from the memory-free bare-table track.\n\nThere is only one narrow claim I am willing to make from this dataset:\n\n**Under the recorded versions, routes, budgets, rules, and samples, different models behaved differently as opponents in Kai!.**\n\nOne step beyond that requires another experiment.\n\nThe original story—“DeepSeek likes to bid without looking”—was easy to write. It had numbers, contrast, and the appeal of model personality. It was also false.\n\nThe rate moved from 40% to 6% without a model upgrade or prompt optimization. I merely removed information the harness had been leaking.\n\nThis is the part of model evaluation that leaderboards tend to hide. The model stands under the spotlight while the measurement tool disappears into the dark. Yet the tool still organizes the input, allocates the budget, handles errors, and decides which answer becomes evidence.\n\nA perfectly neutral harness may be impossible. The design of the action space, the budget, and the metric set all involve choices. We can at least make those choices visible: pin versions, retain raw calls, publish failure rates, separate fallback samples, and state where the conclusion stops.\n\nNow, whenever I am about to write “this model is bolder” or “that model cannot read people,” I ask one counterfactual question:\n\n**If I changed only the harness, would the difference survive?**\n\nWithout that check, a model trait is only an observation waiting to be debugged.\n\nSo the next time a model looks brave, cautious, clever, or foolish, resist the personality analysis for a moment.\n\nCheck the table first.``", "url": "https://wpnews.pro/news/are-you-benchmarking-the-model-or-the-harness", "canonical_source": "https://dev.to/haoxiang_li_a709204042e6b/are-you-benchmarking-the-model-or-the-harness-2bke", "published_at": "2026-08-14 05:26:01+00:00", "updated_at": "2026-08-14 06:18:36.079519+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products", "developer-tools"], "entities": ["DeepSeek V4-Pro", "Kai!AI Arena"], "alternates": {"html": "https://wpnews.pro/news/are-you-benchmarking-the-model-or-the-harness", "markdown": "https://wpnews.pro/news/are-you-benchmarking-the-model-or-the-harness.md", "text": "https://wpnews.pro/news/are-you-benchmarking-the-model-or-the-harness.txt", "jsonld": "https://wpnews.pro/news/are-you-benchmarking-the-model-or-the-harness.jsonld"}}