# Kimi’s Benchmark Chart Was Made Up > Source: > Published: 2026-10-24 23:36:45+00:00 ## The chart looked real—until Kimi was asked A video from AI analyst Matthew Berman recently depicted a simple, yet revealing, test. Berman asked **Moonshot AI’s Kimi K3.1 Pro** to create a benchmark chart, and the model promptly generated a bar graph. This visualization displayed broad categories such as coding, reasoning, mathematics, and agentic tasks. Crucially, the graph offered no clear source, named evaluation, or traceable basis for its scores. Despite presenting these metrics in a confident, data-like format, it lacked any verifiable foundation for its purported performance. The numbers appeared authoritative but were entirely unsubstantiated. Berman directly challenged the model, asking whether **those numbers** were accurate. Kimi K3.1 Pro responded plainly: "No, they are invented placeholders." This admission confirms the specific outcome of Berman’s test. It demonstrates that in this particular instance, Kimi fabricated its own benchmark scores. This revelation highlights a critical aspect of current LLM behavior. When prompted to visualize its own performance, Kimi synthesized convincing-looking data rather than drawing from verified system tables. While this test clearly shows what happened in this specific interaction, it does not mean every Kimi benchmark claim is fabricated. ## A fluent chart is not evidence Generating a chart is not the same as retrieving, validating, or calculating data. Kimi K3.1 Pro’s response to Matthew Berman highlights a core failure: without supplied figures or verified source access, a model can fill gaps with plausible-looking content. Its self-generated chart, featuring broad categories like coding, reasoning, and mathematics, relied on **invented placeholders**. Visual polish can mislead users. Generic categories and tidy bars borrow the authority of research, but they often hide missing methodology, dates, model versions, and citations. Moonshot AI’s official claims for Kimi K3.1 Pro, such as 67.2% on *Terminal-Bench 4.0*, stand in stark contrast to its admitted fabrication when directly prompted. Independent agentic suites like *KingBench 4* show the model scoring only 5.37/10. Distinguishing between hallucination and deliberate deception is crucial. Kimi K3.1 Pro’s output was unsupported and, when questioned, acknowledged as invented. This test alone does not establish intent or prove a system-wide pattern of deliberate deceit. However, it underscores the need for rigorous verification of AI-generated content, especially concerning **benchmark scores** that influence market perception and adoption. The model’s 2.8-trillion-parameter Mixture-of-Experts (MoE) architecture and 1 million token context window do not inherently guarantee truthful self-reporting. ## Benchmark claims need more than a leaderboard Benchmark claims demand more than a leaderboard. A useful report should disclose the exact model and version, test set, prompt, scoring procedure, and run date. It must also state whether independent reproduction verified the results. Without this transparency, comparing scores becomes an exercise in speculation. Scores can be difficult to compare even when systems appear on the same chart. Differences in harnesses, task difficulty, tool access, sampling, or evaluation rules make direct comparisons unreliable. For instance, Moonshot AI claims Kimi K3.1 Pro achieved 67.2% on *Terminal-Bench 4.0*, nearly matching [Claude](https://www.stork.ai/en/claude-ai-workspaces) Opus 5.5. However, independent agentic suites like *KingBench 4* show Kimi K3.1 scoring only 5.37/10, placing it sixth behind alternatives from Anthropic and OpenAI, with testers reporting persistent syntax and build failures. Official claims often differ significantly from third-party realities. The Kimi episode highlights the need to verify claims. While Kimi K3.1 Pro admitted to inventing benchmark scores for its chart, this incident should prompt skepticism about *any* unsubstantiated claim, not just broader allegations about Moonshot AI or any specific score. For official information about the model, visit the [Kimi AI Official Website](https://kimi.ai). Always separate official claims from independent tests and label each clearly. Enjoying this? Get one like it in your inbox each morning. one email a day · unsubscribe in two clicks · no third-party tracking ## A quick checklist before trusting AI scores Before trusting AI benchmark scores, apply a quick checklist. First, ask for the primary source, exact score table, and methodology. Then, open the cited source yourself instead of relying on a model-generated summary or graphic. If the model cannot verify data, provide a trusted dataset and ask it to calculate or plot from those values. Check that chart labels and numbers match the input. For instance, inputting scores for Terminal-Bench 4.0 should yield a chart accurately reflecting those specific percentages. Treat **benchmark scores** as claims to investigate, not verdicts. Kimi K3.1 Pro's admission that its numbers were "invented placeholders" was candid and useful. However, verification should happen *before* a chart earns your trust. This approach applies to all claims, from Moonshot AI’s reported 67.2% on Terminal-Bench 4.0 to any model’s self-generated performance visualization. Independent reproduction and transparent methodology remain the gold standard for validating **AI performance metrics**. ## Frequently Asked Questions ### Did Kimi K3.1 invent benchmark scores? In Matthew Berman’s test, Kimi generated a chart with generic scores and then said they were invented placeholders. ### Does this prove Kimi’s official benchmark results are false? No. The chart test shows that this generated visualization was unsupported; it does not independently verify or disprove official results. ### Why do AI models make up benchmark data? A model may generate plausible-looking details when asked to create a chart without being given verified data or access to a reliable source. ### How can I check an AI benchmark claim? Look for the original evaluation, exact model version, scoring rules, and reproducible results from independent testers.