This is a submission for the Kaggle Benchmarking Challenge
I build InsightTrack, an open-source web analytics
platform. It has an AI analyst called Pulse. You ask "why did traffic to /pricing drop last week?", and Pulse picks the right tools, reads the numbers, and answers in plain English.
Pulse needs a language model behind it. I wanted to choose that model with evidence, not vibes, so I
turned Pulse's real job into a benchmark: 480 questions, graded automatically, with answers that come from InsightTrack's own code.
An analytics assistant isn't a chatbot for fun facts. People act on what it says: they rewrite pages,
campaigns, or stop worrying. When it's wrong, it's wrong in one of two costly ways:
Both are worse than saying "I don't know". A useful analyst needs three habits that don't show up on
general AI leaderboards:
Public benchmarks are excellent at telling you which model knows more maths, writes better code, or
calls functions correctly. But choosing a model for a product is a product decision, and it has
three questions they can't answer for you:
The only way to answer those is to test models on the actual work. That's what this benchmark does.
InsightTrack Analyst Bench has 480 auto-graded cases across four tasks. The first three are Pulse's job today; the fourth, SQL, is the step Pulse doesn't take yet: writing a query when no built-in tool fits.
| Task | What Pulse must do | The model gets | It's right when |
|---|---|---|---|
| Tool choice | Decide how to answer | A user question + InsightTrack's real 23-tool catalogue | It picks the right tool and arguments, or says "none" when no tool can do it |
| Data reading | Understand a result | A tool result in InsightTrack's real output format | It gets the number right, or says not_available instead of inventing one |
| Diagnosis | Explain a change | 8 weeks of page traffic + Google results before and after | It says whether the change is real, which way it went, and the exact set of causes |
| SQL | Dig deeper (next on Pulse's roadmap) | InsightTrack's DuckDB events andsessions tables |
Its query, executed read-only, returns the right rows |
Each task has 80 standard + 40 hard cases. The hard ones are where careless reasoning shows:
getsite.com is a different site, but www.site.com is yours
Here's what a diagnosis question looks like, simplified:
… 495, 517, 347. The last week dropped by 170 views, or 33%.
The correct answer: the drop is real, but search didn't cause it. In InsightTrack's terms,
{"significant": true, "direction": "drop", "causes": ["unexplained_by_serp"]}. The tempting wrong
answer is "your rank dropped". A model that says that sends the user to fix SEO that isn't broken.
That's the heart of the benchmark: can the model tell a real cause from a coincidence?
No LLM grades another LLM:
Every grader also labels how a model was wrong: invented a cause, called a real change noise,
made up a number, refused an answerable question, used a tool that doesn't exist. That turns a
score into something you can act on.
All models ran through Kaggle's model proxy on the same task version. I picked them in pairs, so each
comparison answers a practical question:
| Model | Tool choice | Data reading | Diagnosis | SQL | Overall | Cost (480 cases) |
|---|---|---|---|---|---|---|
| Claude Sonnet 5 | 100 | 100 | 100 | 100 | 100.0 | $2.08 |
| GPT-5.5 | 100 | 100 | 96.7 | 100 | 99.2 | $3.23 |
| Grok 4.20 (reasoning) | 100 | 96.7 | 100 | 95.0 | 97.9 | $1.21 |
| Gemini 3.7 Flash | 100 | 100 | 100 | 90.8 | 97.7 | $1.07 |
| Gemini 3.5 Flash-Lite | 98.3 | 88.3 | 49.2 | 88.3 | 81.0 | $0.20 |
| Claude Haiku 4.5 | 98.3 | 82.5 | 44.2 | 86.7 | 77.9 | $0.63 |
| Grok 4.20 (no reasoning) | 98.3 | 82.5 | 35.0 | 92.5 | 77.1 | $0.40 |
| GPT-5.4 nano | 93.3 | 61.7 | 20.0 | 93.3 | 67.1 | $0.07 |
Every model picked the right tool at least 93% of the time. So "can it call tools?" is no longer the
interesting question for an analytics assistant. The real question is whether it understands what the
numbers mean. On diagnosis, the top three models score 100%, GPT-5.5 scores 96.7%, and every cheaper
model scores between 20% and 49%.
Grok 4.20 with reasoning got every diagnosis case right. The same model without reasoning got
35%. It lost all 16 multi-cause cases, and 9 of the 12 decoys. Nothing else in the results moved this
much. If your assistant answers "why did X change?", reasoning isn't a nice-to-have.
The alarmist. Without reasoning, Grok invented a search cause 34 times. In one case, traffic
rose from 437 to 538 views while the page slipped from #3 to #5 on Google. The prompt says plainly
that a 1–2 place move is normal jitter and never a cause. Grok answered:
{"significant": true, "direction": "spike", "causes": ["rank_drop"]}
It blamed a rank drop for a traffic rise. The right answer was unexplained_by_serp: this change
didn't come from search.
The shrugger. GPT-5.4 nano went the other way, calling real changes noise 73 times. A page fell
from 209 to 110 views (−47%) right after sliding from #14 to #26 on Google. Nano answered:
{"significant": false, "direction": null, "causes": []}
For an analytics product, this is the dangerous one. An alarmist wastes your afternoon. A shrugger
tells you everything is fine while your traffic halves.
In the hard cases, one keyword lost rank while another gained, and traffic fell. The correct answer
names only the loss, since a gain can't explain a drop. Flash-Lite, Haiku, Grok without reasoning, and
nano all scored 0 out of 8. Haiku, for instance, blamed the drop on ai_citation_gained, a rule the
prompt says applies only when traffic rises. Real pages rank for many keywords at once, so this is
the everyday case, not an edge case.
On data reading, all 7 models correctly said not_available on the 24 standard questions the data
couldn't answer. The traps in the hard set split them:
A model that can't tell "I can't compute this" from "this can't be computed" isn't safe in either direction.
Gemini 3.7 Flash scored 0/7 on JSON questions, and nearly everything else it got right. Every time,
it wrote:
WHERE properties->>'name' = 'signup'
That's fine in PostgreSQL. In DuckDB, ->> binds more loosely than =, so it parses as
properties ->> ('name' = 'signup') and crashes. One pair of parentheses fixes it. It looks like a
habit carried over from PostgreSQL, which is far more common in examples than DuckDB.
Claude Haiku corrects itself mid-answer. In 9 SQL replies it wrote a query, then "Wait, let me reconsider…", then a second query. The prompt asks for the query only, so the grader takes the first
GPT-5.5 made exactly four mistakes on diagnosis, and all four are the same one. Each time, a page slid
just past #10 (#10 to #11 twice, #10 to #12 once, #9 to #11 once): it fell off page one, so the
answer is page_one_exit. GPT-5.5 said
unexplained_by_serp every time.
Rereading my prompt, I found two rules that clash:
page_one_exit: "position 1–10 before; now below 10".
I meant the jitter rule to cover rank_drop and rank_gain. GPT-5.5 took "never" literally. Claude
Sonnet 5, Gemini 3.7 Flash and Grok with reasoning all read the intent. Neither reading is careless.
One model follows the letter of an instruction and others follow its spirit, and that's worth knowing
about any model you give rules to. It's also a reminder that a benchmark's prompt is code too.
The next version will say "never counts as rank_drop or rank_gain", and I'm reporting the
score as it stands rather than quietly fixing it.
Gemini 3.7 Flash reaches 97.7% at about half the cost of Claude Sonnet 5. GPT-5.5 was the most
expensive run ($3.23) and still finished just behind Sonnet. Below the top four there's a cliff: no model
under $0.65 per run scored above 81%, and all of them fell apart on diagnosis.
Gemini 3.7 Flash by default. It scored 100% on the three tasks that are about judgement, at half the
price of the most accurate model. Its one weakness is fixable in code: wrap ->> comparisons in
parentheses before running any generated SQL.
Claude Sonnet 5 as the fallback for anything SQL-heavy, since it was perfect everywhere, and
cheaper than GPT-5.5 for the same job.
No cheaper model for diagnosis. An assistant that shrugs at a 47% drop costs more than it saves.
The benchmark also backed a design choice InsightTrack already makes. In the product, Pulse's
explain_traffic_change tool works out significance and causes in code, and the model only
explains the result in plain English. These results show why: left to do that reasoning alone, every
cheaper model got it wrong at least half the time.
These lessons carry over to any assistant that works with numbers:
Kaggle benchmark: https://www.kaggle.com/benchmarks/nishikantaray/insighttrack-analyst-bench
Dataset (cases, sample tables, graders): https://www.kaggle.com/datasets/nishikantaray/insighttrack-analyst-bench
The product it's built from: https://github.com/NishikantaRay/InsightTrack
Built with Kaggle Benchmarks. The task pattern follows
its dataset_evaluation.py example. It covers ground near BFCL and τ-bench (tool calling),
InfiAgent-DABench and DSBench (data analysis), and Spider 2.0 and BIRD (text-to-SQL). It differs by
testing one real product's job end to end, and by scoring whether a model knows when nothing happened.