# I gave 8 LLMs my analytics product's AI job. The cheap ones either invent a reason or shrug. > Source: > Published: 2026-09-27 18:42:53+00:00 *This is a submission for the [Kaggle Benchmarking Challenge](https://dev.to/challenges/kaggle-2026-09-23)* I build [InsightTrack](https://github.com/NishikantaRay/InsightTrack), an open-source web analytics platform. It has an AI analyst called **Pulse**. You ask *"why did traffic to /pricing drop last week?"*, and Pulse picks the right tools, reads the numbers, and answers in plain English. Pulse needs a language model behind it. I wanted to choose that model with evidence, not vibes, so I turned Pulse's real job into a benchmark: **480 questions, graded automatically, with answers that come from InsightTrack's own code.** An analytics assistant isn't a chatbot for fun facts. People act on what it says: they rewrite pages, pause campaigns, or stop worrying. When it's wrong, it's wrong in one of two costly ways: Both are worse than saying "I don't know". A useful analyst needs three habits that don't show up on general AI leaderboards: Public benchmarks are excellent at telling you which model knows more maths, writes better code, or calls functions correctly. But choosing a model for a product is a **product decision**, and it has three questions they can't answer for you: The only way to answer those is to test models on the actual work. That's what this benchmark does. **InsightTrack Analyst Bench** has 480 auto-graded cases across four tasks. The first three are Pulse's job today; the fourth, SQL, is the step Pulse doesn't take yet: writing a query when no built-in tool fits. | Task | What Pulse must do | The model gets | It's right when | |---|---|---|---| | **Tool choice** | Decide how to answer | A user question + InsightTrack's real 23-tool catalogue | It picks the right tool and arguments, or says "none" when no tool can do it | | **Data reading** | Understand a result | A tool result in InsightTrack's real output format | It gets the number right, or says `not_available` instead of inventing one | | **Diagnosis** | Explain a change | 8 weeks of page traffic + Google results before and after | It says whether the change is real, which way it went, and the **exact** set of causes | | **SQL** | Dig deeper (next on Pulse's roadmap) | InsightTrack's DuckDB `events` and`sessions` tables | Its query, *executed* read-only, returns the right rows | Each task has **80 standard + 40 hard** cases. The hard ones are where careless reasoning shows: `getsite.com` is a different site, but `www.site.com` is yours Here's what a diagnosis question looks like, simplified: `… 495, 517, 347`. The last week dropped by 170 views, or 33%. The correct answer: **the drop is real, but search didn't cause it.** In InsightTrack's terms, `{"significant": true, "direction": "drop", "causes": ["unexplained_by_serp"]}`. The tempting wrong answer is "your rank dropped". A model that says that sends the user to fix SEO that isn't broken. That's the heart of the benchmark: **can the model tell a real cause from a coincidence?** No LLM grades another LLM: Every grader also labels **how** a model was wrong: *invented a cause*, *called a real change noise*, *made up a number*, *refused an answerable question*, *used a tool that doesn't exist*. That turns a score into something you can act on. All models ran through Kaggle's model proxy on the same task version. I picked them in pairs, so each comparison answers a practical question: | Model | Tool choice | Data reading | Diagnosis | SQL | **Overall** | Cost (480 cases) | |---|---|---|---|---|---|---| | Claude Sonnet 5 | 100 | 100 | 100 | 100 | **100.0** | $2.08 | | GPT-5.5 | 100 | 100 | 96.7 | 100 | **99.2** | $3.23 | | Grok 4.20 (reasoning) | 100 | 96.7 | 100 | 95.0 | **97.9** | $1.21 | | Gemini 3.7 Flash | 100 | 100 | 100 | 90.8 | **97.7** | $1.07 | | Gemini 3.5 Flash-Lite | 98.3 | 88.3 | 49.2 | 88.3 | **81.0** | $0.20 | | Claude Haiku 4.5 | 98.3 | 82.5 | 44.2 | 86.7 | **77.9** | $0.63 | | Grok 4.20 (no reasoning) | 98.3 | 82.5 | 35.0 | 92.5 | **77.1** | $0.40 | | GPT-5.4 nano | 93.3 | 61.7 | 20.0 | 93.3 | **67.1** | $0.07 | Every model picked the right tool at least 93% of the time. So "can it call tools?" is no longer the interesting question for an analytics assistant. The real question is whether it understands what the numbers *mean*. On diagnosis, the top three models score 100%, GPT-5.5 scores 96.7%, and every cheaper model scores between 20% and 49%. Grok 4.20 with reasoning got **every** diagnosis case right. The same model without reasoning got **35%**. It lost all 16 multi-cause cases, and 9 of the 12 decoys. Nothing else in the results moved this much. If your assistant answers "why did X change?", reasoning isn't a nice-to-have. **The alarmist.** Without reasoning, Grok invented a search cause 34 times. In one case, traffic *rose* from 437 to 538 views while the page slipped from #3 to #5 on Google. The prompt says plainly that a 1–2 place move is normal jitter and never a cause. Grok answered: ``` {"significant": true, "direction": "spike", "causes": ["rank_drop"]} ``` It blamed a rank *drop* for a traffic *rise*. The right answer was `unexplained_by_serp`: this change didn't come from search. **The shrugger.** GPT-5.4 nano went the other way, calling real changes noise 73 times. A page fell from **209 to 110 views (−47%)** right after sliding from #14 to #26 on Google. Nano answered: ``` {"significant": false, "direction": null, "causes": []} ``` For an analytics product, this is the dangerous one. An alarmist wastes your afternoon. A shrugger tells you everything is fine while your traffic halves. In the hard cases, one keyword lost rank while another *gained*, and traffic fell. The correct answer names only the loss, since a gain can't explain a drop. Flash-Lite, Haiku, Grok without reasoning, and nano **all scored 0 out of 8**. Haiku, for instance, blamed the drop on `ai_citation_gained`, a rule the prompt says applies only when traffic rises. Real pages rank for many keywords at once, so this is the everyday case, not an edge case. On data reading, all 7 models correctly said `not_available` on the 24 standard questions the data couldn't answer. The traps in the hard set split them: A model that can't tell "I can't compute this" from "this can't be computed" isn't safe in either direction. **Gemini 3.7 Flash scored 0/7 on JSON questions**, and nearly everything else it got right. Every time, it wrote: ``` php WHERE properties->>'name' = 'signup' ``` That's fine in PostgreSQL. In DuckDB, `->>` binds more loosely than `=`, so it parses as `properties ->> ('name' = 'signup')` and crashes. One pair of parentheses fixes it. It looks like a habit carried over from PostgreSQL, which is far more common in examples than DuckDB. **Claude Haiku corrects itself mid-answer.** In 9 SQL replies it wrote a query, then *"Wait, let me reconsider…"*, then a second query. The prompt asks for the query only, so the grader takes the first GPT-5.5 made exactly four mistakes on diagnosis, and all four are the same one. Each time, a page slid just past #10 (#10 to #11 twice, #10 to #12 once, #9 to #11 once): it fell off page one, so the answer is `page_one_exit`. GPT-5.5 said `unexplained_by_serp` every time. Rereading my prompt, I found two rules that clash: `page_one_exit`: "position 1–10 before; now below 10". I meant the jitter rule to cover `rank_drop` and `rank_gain`. GPT-5.5 took "never" literally. Claude Sonnet 5, Gemini 3.7 Flash and Grok with reasoning all read the intent. Neither reading is careless. One model follows the letter of an instruction and others follow its spirit, and that's worth knowing about any model you give rules to. It's also a reminder that **a benchmark's prompt is code too**. The next version will say "never counts as `rank_drop` or `rank_gain`", and I'm reporting the score as it stands rather than quietly fixing it. Gemini 3.7 Flash reaches 97.7% at about half the cost of Claude Sonnet 5. GPT-5.5 was the most expensive run ($3.23) and still finished just behind Sonnet. Below the top four there's a cliff: no model under $0.65 per run scored above 81%, and all of them fell apart on diagnosis. **Gemini 3.7 Flash by default.** It scored 100% on the three tasks that are about judgement, at half the price of the most accurate model. Its one weakness is fixable in code: wrap `->>` comparisons in parentheses before running any generated SQL. **Claude Sonnet 5 as the fallback** for anything SQL-heavy, since it was perfect everywhere, and cheaper than GPT-5.5 for the same job. **No cheaper model for diagnosis.** An assistant that shrugs at a 47% drop costs more than it saves. The benchmark also backed a design choice InsightTrack already makes. In the product, Pulse's `explain_traffic_change` tool works out significance and causes **in code**, and the model only explains the result in plain English. These results show why: left to do that reasoning alone, every cheaper model got it wrong at least half the time. These lessons carry over to any assistant that works with numbers: **Kaggle benchmark:** [https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench](https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench) **Dataset (cases, sample tables, graders):** [https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench](https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench) **The product it's built from:** [https://github.com/NishikantaRay/InsightTrack](https://github.com/NishikantaRay/InsightTrack) Built with [Kaggle Benchmarks](https://github.com/Kaggle/kaggle-benchmarks). The task pattern follows its `dataset_evaluation.py` example. It covers ground near BFCL and τ-bench (tool calling), InfiAgent-DABench and DSBench (data analysis), and Spider 2.0 and BIRD (text-to-SQL). It differs by testing one real product's job end to end, and by scoring whether a model knows when *nothing* happened.