{"slug": "i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or", "title": "I gave 8 LLMs my analytics product's AI job. The cheap ones either invent a reason or shrug.", "summary": "A developer built InsightTrack Analyst Bench, a 480-case auto-graded benchmark that tests eight language models on the real tasks behind Pulse, the AI analyst in the open-source InsightTrack web analytics platform. Claude Sonnet 5 scored a perfect 100.0 overall at $2.08 for the full run, while cheaper models such as Gemini 3.5 Flash-Lite (81.0, $0.20) and Claude Haiku 4.5 (77.9, $0.63) collapsed on diagnosis, scoring 49.2 and 44.2 respectively by inventing causes or misreading real traffic changes. The benchmark grades tool choice, data reading, diagnosis and SQL, and labels each failure mode so scores translate into actionable model-selection evidence.", "body_md": "*This is a submission for the [Kaggle Benchmarking Challenge](https://dev.to/challenges/kaggle-2026-09-23)*\n\nI build [InsightTrack](https://github.com/NishikantaRay/InsightTrack), an open-source web analytics\n\nplatform. It has an AI analyst called **Pulse**. You ask *\"why did traffic to /pricing drop last week?\"*, and Pulse picks the right tools, reads the numbers, and answers in plain English.\n\nPulse needs a language model behind it. I wanted to choose that model with evidence, not vibes, so I\n\nturned Pulse's real job into a benchmark: **480 questions, graded automatically, with answers that come from InsightTrack's own code.**\n\nAn analytics assistant isn't a chatbot for fun facts. People act on what it says: they rewrite pages,\n\npause campaigns, or stop worrying. When it's wrong, it's wrong in one of two costly ways:\n\nBoth are worse than saying \"I don't know\". A useful analyst needs three habits that don't show up on\n\ngeneral AI leaderboards:\n\nPublic benchmarks are excellent at telling you which model knows more maths, writes better code, or\n\ncalls functions correctly. But choosing a model for a product is a **product decision**, and it has\n\nthree questions they can't answer for you:\n\nThe only way to answer those is to test models on the actual work. That's what this benchmark does.\n\n**InsightTrack Analyst Bench** has 480 auto-graded cases across four tasks. The first three are Pulse's job today; the fourth, SQL, is the step Pulse doesn't take yet: writing a query when no built-in tool fits.\n\n| Task | What Pulse must do | The model gets | It's right when | \n|---|---|---|---|\n| **Tool choice** | Decide how to answer | A user question + InsightTrack's real 23-tool catalogue | It picks the right tool and arguments, or says \"none\" when no tool can do it | \n| **Data reading** | Understand a result | A tool result in InsightTrack's real output format | It gets the number right, or says `not_available` instead of inventing one | \n| **Diagnosis** | Explain a change | 8 weeks of page traffic + Google results before and after | It says whether the change is real, which way it went, and the **exact** set of causes | \n| **SQL** | Dig deeper (next on Pulse's roadmap) | InsightTrack's DuckDB `events` and`sessions` tables | Its query, *executed* read-only, returns the right rows | \n\nEach task has **80 standard + 40 hard** cases. The hard ones are where careless reasoning shows:\n\n`getsite.com` is a different site, but `www.site.com` is yours\nHere's what a diagnosis question looks like, simplified:\n\n`… 495, 517, 347`. The last week dropped by 170 views, or 33%.\nThe correct answer: **the drop is real, but search didn't cause it.** In InsightTrack's terms,\n\n`{\"significant\": true, \"direction\": \"drop\", \"causes\": [\"unexplained_by_serp\"]}`. The tempting wrong\n\nanswer is \"your rank dropped\". A model that says that sends the user to fix SEO that isn't broken.\n\nThat's the heart of the benchmark: **can the model tell a real cause from a coincidence?**\n\nNo LLM grades another LLM:\n\nEvery grader also labels **how** a model was wrong: *invented a cause*, *called a real change noise*,\n\n*made up a number*, *refused an answerable question*, *used a tool that doesn't exist*. That turns a\n\nscore into something you can act on.\n\nAll models ran through Kaggle's model proxy on the same task version. I picked them in pairs, so each\n\ncomparison answers a practical question:\n\n| Model | Tool choice | Data reading | Diagnosis | SQL | **Overall** | Cost (480 cases) | \n|---|---|---|---|---|---|---|\n| Claude Sonnet 5 | 100 | 100 | 100 | 100 | **100.0** | $2.08 | \n| GPT-5.5 | 100 | 100 | 96.7 | 100 | **99.2** | $3.23 | \n| Grok 4.20 (reasoning) | 100 | 96.7 | 100 | 95.0 | **97.9** | $1.21 | \n| Gemini 3.7 Flash | 100 | 100 | 100 | 90.8 | **97.7** | $1.07 | \n| Gemini 3.5 Flash-Lite | 98.3 | 88.3 | 49.2 | 88.3 | **81.0** | $0.20 | \n| Claude Haiku 4.5 | 98.3 | 82.5 | 44.2 | 86.7 | **77.9** | $0.63 | \n| Grok 4.20 (no reasoning) | 98.3 | 82.5 | 35.0 | 92.5 | **77.1** | $0.40 | \n| GPT-5.4 nano | 93.3 | 61.7 | 20.0 | 93.3 | **67.1** | $0.07 | \n\nEvery model picked the right tool at least 93% of the time. So \"can it call tools?\" is no longer the\n\ninteresting question for an analytics assistant. The real question is whether it understands what the\n\nnumbers *mean*. On diagnosis, the top three models score 100%, GPT-5.5 scores 96.7%, and every cheaper\n\nmodel scores between 20% and 49%.\n\nGrok 4.20 with reasoning got **every** diagnosis case right. The same model without reasoning got\n\n**35%**. It lost all 16 multi-cause cases, and 9 of the 12 decoys. Nothing else in the results moved this\n\nmuch. If your assistant answers \"why did X change?\", reasoning isn't a nice-to-have.\n\n**The alarmist.** Without reasoning, Grok invented a search cause 34 times. In one case, traffic\n\n*rose* from 437 to 538 views while the page slipped from #3 to #5 on Google. The prompt says plainly\n\nthat a 1–2 place move is normal jitter and never a cause. Grok answered:\n\n```\n{\"significant\": true, \"direction\": \"spike\", \"causes\": [\"rank_drop\"]}\n```\n\nIt blamed a rank *drop* for a traffic *rise*. The right answer was `unexplained_by_serp`: this change\n\ndidn't come from search.\n\n**The shrugger.** GPT-5.4 nano went the other way, calling real changes noise 73 times. A page fell\n\nfrom **209 to 110 views (−47%)** right after sliding from #14 to #26 on Google. Nano answered:\n\n```\n{\"significant\": false, \"direction\": null, \"causes\": []}\n```\n\nFor an analytics product, this is the dangerous one. An alarmist wastes your afternoon. A shrugger\n\ntells you everything is fine while your traffic halves.\n\nIn the hard cases, one keyword lost rank while another *gained*, and traffic fell. The correct answer\n\nnames only the loss, since a gain can't explain a drop. Flash-Lite, Haiku, Grok without reasoning, and\n\nnano **all scored 0 out of 8**. Haiku, for instance, blamed the drop on `ai_citation_gained`, a rule the\n\nprompt says applies only when traffic rises. Real pages rank for many keywords at once, so this is\n\nthe everyday case, not an edge case.\n\nOn data reading, all 7 models correctly said `not_available` on the 24 standard questions the data\n\ncouldn't answer. The traps in the hard set split them:\n\nA model that can't tell \"I can't compute this\" from \"this can't be computed\" isn't safe in either direction.\n\n**Gemini 3.7 Flash scored 0/7 on JSON questions**, and nearly everything else it got right. Every time,\n\nit wrote:\n\n``` php\nWHERE properties->>'name' = 'signup'\n```\n\nThat's fine in PostgreSQL. In DuckDB, `->>` binds more loosely than `=`, so it parses as\n\n`properties ->> ('name' = 'signup')` and crashes. One pair of parentheses fixes it. It looks like a\n\nhabit carried over from PostgreSQL, which is far more common in examples than DuckDB.\n\n**Claude Haiku corrects itself mid-answer.** In 9 SQL replies it wrote a query, then *\"Wait, let me reconsider…\"*, then a second query. The prompt asks for the query only, so the grader takes the first\n\nGPT-5.5 made exactly four mistakes on diagnosis, and all four are the same one. Each time, a page slid\n\njust past #10 (#10 to #11 twice, #10 to #12 once, #9 to #11 once): it fell off page one, so the\n\nanswer is `page_one_exit`. GPT-5.5 said\n\n`unexplained_by_serp` every time.\n\nRereading my prompt, I found two rules that clash:\n\n`page_one_exit`: \"position 1–10 before; now below 10\".\nI meant the jitter rule to cover `rank_drop` and `rank_gain`. GPT-5.5 took \"never\" literally. Claude\n\nSonnet 5, Gemini 3.7 Flash and Grok with reasoning all read the intent. Neither reading is careless.\n\nOne model follows the letter of an instruction and others follow its spirit, and that's worth knowing\n\nabout any model you give rules to. It's also a reminder that **a benchmark's prompt is code too**.\n\nThe next version will say \"never counts as `rank_drop` or `rank_gain`\", and I'm reporting the\n\nscore as it stands rather than quietly fixing it.\n\nGemini 3.7 Flash reaches 97.7% at about half the cost of Claude Sonnet 5. GPT-5.5 was the most\n\nexpensive run ($3.23) and still finished just behind Sonnet. Below the top four there's a cliff: no model\n\nunder $0.65 per run scored above 81%, and all of them fell apart on diagnosis.\n\n**Gemini 3.7 Flash by default.** It scored 100% on the three tasks that are about judgement, at half the\n\nprice of the most accurate model. Its one weakness is fixable in code: wrap `->>` comparisons in\n\nparentheses before running any generated SQL.\n\n**Claude Sonnet 5 as the fallback** for anything SQL-heavy, since it was perfect everywhere, and\n\ncheaper than GPT-5.5 for the same job.\n\n**No cheaper model for diagnosis.** An assistant that shrugs at a 47% drop costs more than it saves.\n\nThe benchmark also backed a design choice InsightTrack already makes. In the product, Pulse's\n\n`explain_traffic_change` tool works out significance and causes **in code**, and the model only\n\nexplains the result in plain English. These results show why: left to do that reasoning alone, every\n\ncheaper model got it wrong at least half the time.\n\nThese lessons carry over to any assistant that works with numbers:\n\n**Kaggle benchmark:** [https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench](https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench)\n\n**Dataset (cases, sample tables, graders):** [https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench](https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench)\n\n**The product it's built from:** [https://github.com/NishikantaRay/InsightTrack](https://github.com/NishikantaRay/InsightTrack)\n\nBuilt with [Kaggle Benchmarks](https://github.com/Kaggle/kaggle-benchmarks). The task pattern follows\n\nits `dataset_evaluation.py` example. It covers ground near BFCL and τ-bench (tool calling),\n\nInfiAgent-DABench and DSBench (data analysis), and Spider 2.0 and BIRD (text-to-SQL). It differs by\n\ntesting one real product's job end to end, and by scoring whether a model knows when *nothing* happened.", "url": "https://wpnews.pro/news/i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or", "canonical_source": "https://dev.to/nishikantaray/i-gave-8-llms-my-analytics-products-ai-job-the-cheap-ones-either-invent-a-reason-or-shrug-3f41", "published_at": "2026-09-27 18:42:53+00:00", "updated_at": "2026-09-27 19:01:21.246381+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools"], "entities": ["InsightTrack", "Pulse", "Claude Sonnet 5", "GPT-5.5", "Grok 4.20", "Gemini 3.7 Flash", "Gemini 3.5 Flash-Lite", "Claude Haiku 4.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or", "markdown": "https://wpnews.pro/news/i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or.md", "text": "https://wpnews.pro/news/i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or.txt", "jsonld": "https://wpnews.pro/news/i-gave-8-llms-my-analytics-product-s-ai-job-the-cheap-ones-either-invent-a-or.jsonld"}}