{"slug": "the-model-knew-the-bid-was-true-then-it-challenged-anyway", "title": "The Model Knew the Bid Was True. Then It Challenged Anyway.", "summary": "In Kai, a Liar's Dice game, an engineer found that large language models sometimes challenge a bid they know is true, losing the round. The issue was traced to the action schema, where the 'challenge' action did not explicitly state its meaning as asserting the bid is false. Adding an explicit 'assert' field to the action contract reduced the error rate across multiple models.", "body_md": "The game logs and replay results in this article are real. Model traces originally written in Chinese have been translated into English. The findings apply only to the recorded models, prompts, routes, and Liar's Dice positions.\n\nGPT-5.6 Luna was holding three fives and a wild one.\n\nThe current bid was four fives.\n\nIts own dice already made the bid true. Challenging could only lose. The private trace recognized the situation:\n\n“The current bid of four fives is guaranteed to hold.”\n\nThen the same trace continued:\n\n“Challenging directly has a high chance of winning.”\n\nAnd the model challenged.\n\nThis was not a rare malformed response or a fallback bot taking over. The JSON was valid. The action was legal. The engine accepted it, revealed the dice, and made the challenger lose.\n\nI first filed it under bad reasoning. Then I found more examples with the same shape: the model described a bid as true, sometimes even used the word “guaranteed,” and still chose the action that asserted the opposite.\n\nThe arithmetic was sitting in the trace. The failure happened somewhere between the arithmetic and the button.\n\nIn Kai, my Liar's Dice game, the model does not type arbitrary commands. The harness gives it a small action schema. One option looked like this:\n\n```\n{\"type\":\"challenge\"}\n```\n\nThe rules described its mechanical effect: reveal all dice and settle the round immediately.\n\nThat description was accurate, but incomplete. It said what the engine would do. It did not state what the player meant by choosing it.\n\nIn Liar's Dice, a challenge is a claim:\n\nThe current bid is false.\n\nWithout that sentence, “challenge” could also be read as “stop raising and settle now.” Several traces looked exactly like that interpretation. The model knew the bid was safe, saw further bidding as unnecessary risk, and treated the reveal action as a way to cash out its advantage.\n\nThe engine knew that revealing a true bid punishes the challenger. I knew it. The model could reconstruct it from the full rules. The action contract still made the wrong reading easy.\n\nSo I changed the action to make the assertion explicit:\n\n```\n{\n  \"type\": \"challenge\",\n  \"assert\": \"current_bid_is_false\"\n}\n```\n\nThis was not a new rule or a strategic hint. The field states the meaning the action already had.\n\nI wanted to know whether that small semantic change would actually alter decisions, or whether I had simply found a few strange transcripts and built another story around them.\n\nThe target error is uncommon in full matches. A model first needs to see its dice, face a bid already satisfied by those dice, get another turn, and then consider challenging. Running hundreds of complete games would spend most calls waiting for those positions to appear.\n\nI extracted ten distinct positions from recorded matches where the model's own dice already guaranteed the current bid. In every target position, a challenge was an objective error. No opponent read or risk preference could rescue it.\n\nI also extracted ten control positions where the model's dice were one short of the bid. In those positions, challenging could be reasonable because the unknown opponent dice still determined the result.\n\nEach model received the same position, history, dice, sampling settings, and underlying legal moves. Each position was sampled eight times under three versions:\n\n`{\"type\":\"challenge\"}`\n\n.`\"assert\":\"current_bid_is_false\"`\n\n, both in the system contract and in the current legal-action description.The cleanest comparison is v8 against v9. Those two share the prose rulebook; v9 changes the challenge contract. I kept v7 in the table as the historical baseline, not as part of the narrow action-schema claim.\n\nThe state, rather than each stochastic answer, is the unit that matters here. One position had appeared twice in the original extraction, so I deduplicated positions and gave each distinct state equal weight.\n\nThree model runs completed without transport errors: DeepSeek V4 Flash, DeepSeek V4 Pro, and GPT-5.6 Luna. Two other runs stayed in the artifacts but not in the behavioral comparison: the Haiku route returned HTTP 403 for many calls, and the DeepSeek Chat route returned HTTP 404 for all of them.\n\nHere is the challenge rate averaged equally across the ten distinct states for each model:\n\n| Model | Guaranteed-loss states: v7 | v8 | v9 | Control states: v7 | v8 | v9 |\n|---|---|---|---|---|---|---|\n| DeepSeek V4 Flash | 30.0% | 28.8% | 23.1% | 28.8% | 17.5% | 23.8% |\n| DeepSeek V4 Pro | 11.6% | 11.3% | 6.3% | 13.8% | 26.6% | 21.3% |\n| GPT-5.6 Luna | 38.8% | 25.0% | 2.5% | 17.5% | 21.3% | 16.3% |\nEqual-weight mean |\n26.8% |\n21.7% |\n10.6% |\n20.0% |\n21.8% |\n20.4% |\n\nThe prose rewrite helped a little. The explicit assertion helped much more.\n\nIn the clean v8-to-v9 comparison, guaranteed-loss challenges fell from 21.7% to 10.6%. The control rate stayed almost flat: 21.8% to 20.4%.\n\nThat control matters. If every challenge rate had collapsed, the new schema might merely have made the models afraid to use the action. Instead, the large change was concentrated in positions where challenging contradicted information already visible in the model's own hand.\n\nLuna reacted most strongly. Its target error rate fell from 25.0% to 2.5%, while its control rate moved from 21.3% to 16.3%. The older v7 baseline was worse still, at 38.8%.\n\nIn the original four-fives position, the v8 contract produced a challenge in four of eight samples. With the explicit assertion, it produced none. The actions changed even though some of the private arithmetic remained messy.\n\nDeepSeek V4 Flash moved much less. V4 Pro started with a lower error rate and improved modestly. The same harness repair did not have the same value for every model.\n\nNothing about the model weights changed. I did not give the models extra dice, more tokens, a calculator, or a worked example. I made one action say what it meant.\n\nThat distinction is useful because tool schemas are often treated as plumbing. We compare models behind the same set of function names and assume they received the same task. But a shared ambiguous contract can be easy for one model to infer and costly for another.\n\nThe original action mixed three layers:\n\nMy schema exposed the first two and left the third implicit. The bad traces suggest that at least one model sometimes optimized around the consequence—settle now—while losing track of the intent that determines who wins the settlement.\n\nAdding the assertion brought the intent into the action itself. It turned a vague verb into a falsifiable statement that could be checked against the model's own reasoning.\n\nThis also explains why the prose-only rewrite had a smaller effect. Better-written rules do not guarantee that the decisive meaning will be present at the moment of action selection. The v9 contract repeated that meaning exactly where the model had to commit to it.\n\nThere are several limits to the result.\n\nThe study used ten target states and ten controls from one game. Each state was sampled repeatedly, so the table contains repeated decisions, not hundreds of independent game situations. The positions came from one seat's recorded matches. Only three model routes produced clean results for all three arms.\n\nThe v9 change also touched the action contract as a unit: the system definition and the legal-action representation both gained the explicit assertion. This experiment does not isolate whether the JSON field, the nearby wording, or their consistency produced the effect.\n\nMost importantly, this is not evidence that one model generally understands negation, games, or tools better than another. It shows that these models reacted differently to one ambiguous action contract in these recorded positions.\n\nThat is already enough to change how I test a harness.\n\nAn engine test usually asks whether an action is legal and whether the state transition is correct. For an LLM tool, that is only half the contract. The model also needs to understand what choosing the action claims about the world.\n\nI now want three tests for every important action:\n\nThen I replay those states when the prompt, schema, parser, or model changes. Full matches are still useful, but targeted states expose semantic regressions before they dissolve into a win rate.\n\nI also keep provider failures beside the behavioral results. A route returning 403 or 404 is not a model personality, and a fallback bot is not a quiet version of the model. If a batch cannot answer the action contract reliably, that failure belongs in the report rather than disappearing from the denominator.\n\nThe strangest part of this bug was that the model had already written down the fact I needed. It knew the bid was true. The harness then offered a verb whose meaning was loose enough for that fact to stop controlling the action.\n\nThe model did not receive a reasoning upgrade. The task finally said what I thought it had said all along.\n\nThis article was translated into English with AI assistance.", "url": "https://wpnews.pro/news/the-model-knew-the-bid-was-true-then-it-challenged-anyway", "canonical_source": "https://dev.to/haoxiang_li_a709204042e6b/the-model-knew-the-bid-was-true-then-it-challenged-anyway-2k6f", "published_at": "2026-08-17 06:40:05+00:00", "updated_at": "2026-08-17 07:12:42.390439+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-safety", "ai-research"], "entities": ["GPT-5.6 Luna", "DeepSeek V4 Flash", "DeepSeek V4 Pro", "Kai"], "alternates": {"html": "https://wpnews.pro/news/the-model-knew-the-bid-was-true-then-it-challenged-anyway", "markdown": "https://wpnews.pro/news/the-model-knew-the-bid-was-true-then-it-challenged-anyway.md", "text": "https://wpnews.pro/news/the-model-knew-the-bid-was-true-then-it-challenged-anyway.txt", "jsonld": "https://wpnews.pro/news/the-model-knew-the-bid-was-true-then-it-challenged-anyway.jsonld"}}