I gave 8 LLMs my analytics product's AI job. The cheap ones either invent a reason or shrug.

A developer built InsightTrack Analyst Bench, a 480-case auto-graded benchmark that tests eight language models on the real tasks behind Pulse, the AI analyst in the open-source InsightTrack web analytics platform. Claude Sonnet 5 scored a perfect 100.0 overall at $2.08 for the full run, while cheaper models such as Gemini 3.5 Flash-Lite (81.0, $0.20) and Claude Haiku 4.5 (77.9, $0.63) collapsed on diagnosis, scoring 49.2 and 44.2 respectively by inventing causes or misreading real traffic changes. The benchmark grades tool choice, data reading, diagnosis and SQL, and labels each failure mode so scores translate into actionable model-selection evidence.

This is a submission for the Kaggle Benchmarking Challenge https://dev.to/challenges/kaggle-2026-09-23 I build InsightTrack https://github.com/NishikantaRay/InsightTrack , an open-source web analytics platform. It has an AI analyst called Pulse . You ask "why did traffic to /pricing drop last week?" , and Pulse picks the right tools, reads the numbers, and answers in plain English. Pulse needs a language model behind it. I wanted to choose that model with evidence, not vibes, so I turned Pulse's real job into a benchmark: 480 questions, graded automatically, with answers that come from InsightTrack's own code. An analytics assistant isn't a chatbot for fun facts. People act on what it says: they rewrite pages, pause campaigns, or stop worrying. When it's wrong, it's wrong in one of two costly ways: Both are worse than saying "I don't know". A useful analyst needs three habits that don't show up on general AI leaderboards: Public benchmarks are excellent at telling you which model knows more maths, writes better code, or calls functions correctly. But choosing a model for a product is a product decision , and it has three questions they can't answer for you: The only way to answer those is to test models on the actual work. That's what this benchmark does. InsightTrack Analyst Bench has 480 auto-graded cases across four tasks. The first three are Pulse's job today; the fourth, SQL, is the step Pulse doesn't take yet: writing a query when no built-in tool fits. | Task | What Pulse must do | The model gets | It's right when | |---|---|---|---| | Tool choice | Decide how to answer | A user question + InsightTrack's real 23-tool catalogue | It picks the right tool and arguments, or says "none" when no tool can do it | | Data reading | Understand a result | A tool result in InsightTrack's real output format | It gets the number right, or says not available instead of inventing one | | Diagnosis | Explain a change | 8 weeks of page traffic + Google results before and after | It says whether the change is real, which way it went, and the exact set of causes | | SQL | Dig deeper next on Pulse's roadmap | InsightTrack's DuckDB events and sessions tables | Its query, executed read-only, returns the right rows | Each task has 80 standard + 40 hard cases. The hard ones are where careless reasoning shows: getsite.com is a different site, but www.site.com is yours Here's what a diagnosis question looks like, simplified: … 495, 517, 347 . The last week dropped by 170 views, or 33%. The correct answer: the drop is real, but search didn't cause it. In InsightTrack's terms, {"significant": true, "direction": "drop", "causes": "unexplained by serp" } . The tempting wrong answer is "your rank dropped". A model that says that sends the user to fix SEO that isn't broken. That's the heart of the benchmark: can the model tell a real cause from a coincidence? No LLM grades another LLM: Every grader also labels how a model was wrong: invented a cause , called a real change noise , made up a number , refused an answerable question , used a tool that doesn't exist . That turns a score into something you can act on. All models ran through Kaggle's model proxy on the same task version. I picked them in pairs, so each comparison answers a practical question: | Model | Tool choice | Data reading | Diagnosis | SQL | Overall | Cost 480 cases | |---|---|---|---|---|---|---| | Claude Sonnet 5 | 100 | 100 | 100 | 100 | 100.0 | $2.08 | | GPT-5.5 | 100 | 100 | 96.7 | 100 | 99.2 | $3.23 | | Grok 4.20 reasoning | 100 | 96.7 | 100 | 95.0 | 97.9 | $1.21 | | Gemini 3.7 Flash | 100 | 100 | 100 | 90.8 | 97.7 | $1.07 | | Gemini 3.5 Flash-Lite | 98.3 | 88.3 | 49.2 | 88.3 | 81.0 | $0.20 | | Claude Haiku 4.5 | 98.3 | 82.5 | 44.2 | 86.7 | 77.9 | $0.63 | | Grok 4.20 no reasoning | 98.3 | 82.5 | 35.0 | 92.5 | 77.1 | $0.40 | | GPT-5.4 nano | 93.3 | 61.7 | 20.0 | 93.3 | 67.1 | $0.07 | Every model picked the right tool at least 93% of the time. So "can it call tools?" is no longer the interesting question for an analytics assistant. The real question is whether it understands what the numbers mean . On diagnosis, the top three models score 100%, GPT-5.5 scores 96.7%, and every cheaper model scores between 20% and 49%. Grok 4.20 with reasoning got every diagnosis case right. The same model without reasoning got 35% . It lost all 16 multi-cause cases, and 9 of the 12 decoys. Nothing else in the results moved this much. If your assistant answers "why did X change?", reasoning isn't a nice-to-have. The alarmist. Without reasoning, Grok invented a search cause 34 times. In one case, traffic rose from 437 to 538 views while the page slipped from 3 to 5 on Google. The prompt says plainly that a 1–2 place move is normal jitter and never a cause. Grok answered: {"significant": true, "direction": "spike", "causes": "rank drop" } It blamed a rank drop for a traffic rise . The right answer was unexplained by serp : this change didn't come from search. The shrugger. GPT-5.4 nano went the other way, calling real changes noise 73 times. A page fell from 209 to 110 views −47% right after sliding from 14 to 26 on Google. Nano answered: {"significant": false, "direction": null, "causes": } For an analytics product, this is the dangerous one. An alarmist wastes your afternoon. A shrugger tells you everything is fine while your traffic halves. In the hard cases, one keyword lost rank while another gained , and traffic fell. The correct answer names only the loss, since a gain can't explain a drop. Flash-Lite, Haiku, Grok without reasoning, and nano all scored 0 out of 8 . Haiku, for instance, blamed the drop on ai citation gained , a rule the prompt says applies only when traffic rises. Real pages rank for many keywords at once, so this is the everyday case, not an edge case. On data reading, all 7 models correctly said not available on the 24 standard questions the data couldn't answer. The traps in the hard set split them: A model that can't tell "I can't compute this" from "this can't be computed" isn't safe in either direction. Gemini 3.7 Flash scored 0/7 on JSON questions , and nearly everything else it got right. Every time, it wrote: php WHERE properties- 'name' = 'signup' That's fine in PostgreSQL. In DuckDB, - binds more loosely than = , so it parses as properties - 'name' = 'signup' and crashes. One pair of parentheses fixes it. It looks like a habit carried over from PostgreSQL, which is far more common in examples than DuckDB. Claude Haiku corrects itself mid-answer. In 9 SQL replies it wrote a query, then "Wait, let me reconsider…" , then a second query. The prompt asks for the query only, so the grader takes the first GPT-5.5 made exactly four mistakes on diagnosis, and all four are the same one. Each time, a page slid just past 10 10 to 11 twice, 10 to 12 once, 9 to 11 once : it fell off page one, so the answer is page one exit . GPT-5.5 said unexplained by serp every time. Rereading my prompt, I found two rules that clash: page one exit : "position 1–10 before; now below 10". I meant the jitter rule to cover rank drop and rank gain . GPT-5.5 took "never" literally. Claude Sonnet 5, Gemini 3.7 Flash and Grok with reasoning all read the intent. Neither reading is careless. One model follows the letter of an instruction and others follow its spirit, and that's worth knowing about any model you give rules to. It's also a reminder that a benchmark's prompt is code too . The next version will say "never counts as rank drop or rank gain ", and I'm reporting the score as it stands rather than quietly fixing it. Gemini 3.7 Flash reaches 97.7% at about half the cost of Claude Sonnet 5. GPT-5.5 was the most expensive run $3.23 and still finished just behind Sonnet. Below the top four there's a cliff: no model under $0.65 per run scored above 81%, and all of them fell apart on diagnosis. Gemini 3.7 Flash by default. It scored 100% on the three tasks that are about judgement, at half the price of the most accurate model. Its one weakness is fixable in code: wrap - comparisons in parentheses before running any generated SQL. Claude Sonnet 5 as the fallback for anything SQL-heavy, since it was perfect everywhere, and cheaper than GPT-5.5 for the same job. No cheaper model for diagnosis. An assistant that shrugs at a 47% drop costs more than it saves. The benchmark also backed a design choice InsightTrack already makes. In the product, Pulse's explain traffic change tool works out significance and causes in code , and the model only explains the result in plain English. These results show why: left to do that reasoning alone, every cheaper model got it wrong at least half the time. These lessons carry over to any assistant that works with numbers: Kaggle benchmark: https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench Dataset cases, sample tables, graders : https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench The product it's built from: https://github.com/NishikantaRay/InsightTrack https://github.com/NishikantaRay/InsightTrack Built with Kaggle Benchmarks https://github.com/Kaggle/kaggle-benchmarks . The task pattern follows its dataset evaluation.py example. It covers ground near BFCL and τ-bench tool calling , InfiAgent-DABench and DSBench data analysis , and Spider 2.0 and BIRD text-to-SQL . It differs by testing one real product's job end to end, and by scoring whether a model knows when nothing happened.