{"slug": "i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time", "title": "I Asked ChatGPT to Analyze 3 Datasets. It Made the Same Mistakes Every Time", "summary": "In an experiment with three small datasets, OpenAI's GPT-5.6 Terra and GPT-5.6 Luna models made repeated data-analysis errors, including approving a backwards conclusion and inventing a correction that turned a right answer wrong, according to a report from KDnuggets. The review pass fixed a row count but failed to catch other mistakes, highlighting reliability issues for business teams using AI for data analysis.", "body_md": "# I Asked ChatGPT to Analyze 3 Datasets. It Made the Same Mistakes Every Time\n\nThe review pass fixed a row count and approved two wrong conclusions.\n\nWe ran an experiment: three small datasets, one AI model, and the questions a business team asks in a normal week — what's our average delivery time, which region is our best performer, how many athletes are in this file.\n\nThen we added a review pass. We handed the model its own answer back and told it the numbers were going into an exec deck, so verify everything.\n\nOne review pass caught a wrong row count and put a checkmark next to a conclusion that was backwards. The other invented a correction and turned a right answer into a wrong one.\n\nEverything below is reproducible. We used ** GPT-5.6 Terra** for the fast first pass and\n\n**for a separate set of unhurried runs on the same files. The code runs on**\n\n[GPT-5.6 Luna](https://developers.openai.com/api/docs/models/gpt-5.6-luna)**and**\n\n[Pandas](https://pandas.pydata.org/)**.**\n\n[SciPy](https://scipy.org/)## The Data\n\nFirst, we use the `shipment_tracking`\n\ndatatable, which is used in this [interview question](https://platform.stratascratch.com/coding/10563-multi-status-order-tracking-by-week?code_type=1&utm_source=blog&utm_medium=click&utm_campaign=kdn+ai+data+analysis+mistakes).\n\n**shipment_tracking** is one row per order. 40 orders from 40 different customers, all placed between January 1 and 21, 2024. Every row carries three dates that fill in as the order progresses: `ordered_date`\n\nfrom the start, `shipped_date`\n\nonce the parcel leaves the warehouse, and `delivered_date`\n\nonce it arrives.\n\n| order_id | user_id | ordered_date | shipped_date | delivered_date | order_amount |\n|---|---|---|---|---|---|\n| 1001 | 201 | 2024-01-01 | 2024-01-03 | 2024-01-05 | 89.99 |\n| 1002 | 202 | 2024-01-01 | 2024-01-03 | 2024-01-08 | 124.50 |\n| 1003 | 203 | 2024-01-01 | 2024-01-07 | 2024-01-10 | 56.25 |\n| 1004 | 204 | 2024-01-02 | 2024-01-02 | 2024-01-02 | 299.99 |\n| … | … | … | … | … | … |\n| 1040 | 240 | 2024-01-21 | 145.70 |\n\nLook at that last row: ordered, never shipped, never delivered. There are 18 like it out of 40 in the dataset.\n\nThe second file we are dealing with in this article is `regional_sales`\n\n, used in this [interview question](https://platform.stratascratch.com/coding/10550-five-year-sales-growth-regions?code_type=1&utm_source=blog&utm_medium=click&utm_campaign=kdn+ai+data+analysis+mistakes). **regional_sales** is meant to be one row per region per year: 59 rows, 8 regions, years from 2007 to 2025, and a single sales figure for each combination.\n\n| region_name | year | sales |\n|---|---|---|\n| latam | 2012 | 230.62 |\n| us_west | 2010 | 163.94 |\n| us_east | 2012 | 270.63 |\n| emea | 2010 | 150.00 |\n| … | … | … |\n| europe_north | 2020 | 300.00 |\n\nThat grain is broken in two ways, and neither one is visible in the column names. Six region-year combinations have more than one row. Three of the extra rows are exact duplicates, all in `us_west`\n\n, and four combinations hold conflicting figures: `apac`\n\n2015 appears as both 173.46 and 126.78, and `us_west`\n\n2012 appears four times with three different values. There is no single `us_west`\n\n2012 number to report. Coverage is uneven too, running from `apac`\n\nwith 15 years of history down to `latam`\n\nwith 1.\n\nThe third file is `olympics_athletes_events`\n\n, used in this [interview question](https://platform.stratascratch.com/coding/10184-order-all-countries-by-the-year-they-first-participated-in-the-olympics?code_type=1&utm_source=blog&utm_medium=click&utm_campaign=kdn+ai+data+analysis+mistakes).\n\n**olympics_athletes_events** is one row per athlete per event — which is the detail that matters later. Its 352 rows cover 336 athletes across 15 Games and 167 events, so 11 athletes appear more than once and one appears 6 times. The `medal`\n\ncolumn is filled for 120 rows, and a blank means that athlete did not win a medal in that event.\n\n| id | name | sex | age | height | team | noc | year | sport | medal |\n|---|---|---|---|---|---|---|---|---|---|\n| 3520 | Guillermo J. Amparan | M | Mexico | MEX | 1924 | Athletics | |||\n| 35394 | Henry John Finchett | M | Great Britain | GBR | 1924 | Gymnastics | |||\n| 21918 | Georg Frederik Ahrensborg Clausen | M | 28.0 | Denmark | DEN | 1924 | Cycling | ||\n| 110345 | Marinus Cornelis Dick Sigmond | M | 26.0 | Netherlands | NED | 1924 | Football | ||\n| … | … | … | … | … | … | … | … | … | … |\n| 999998 | John Testman | M | 30.0 | 180.0 | Canada | CAN | 2004 | Athletics | Bronze |\n\n## Mistake 1: Measuring Ship-To-Door When We Asked Order-To-Door\n\nWorking on `shipment_tracking`\n\n, we asked for the average delivery time, and this is the calculation that came back:\n\n```\ndf['delivery_days'] = (df['delivered_date'] - df['shipped_date']).dt.days\nprint(f\"Avg delivery: {df['delivery_days'].mean():.1f} days\")\n```\n\n#### Output\n\n```\nAvg delivery: 2.6 days\n```\n\nThe reply led with \"Average delivery time: 2.6 days.\"\n\nA customer waiting for a package experiences ordered-to-door, and that clock starts at checkout.\n\n```\norder_to_door = (df['delivered_date'] - df['ordered_date']).dt.days\nprint(round(order_to_door.mean(), 2))\n```\n\n#### Output\n\n```\n6.09\n```\n\nThe question we asked has one answer — 6.09 days — and the reply gave a number 2.4 times smaller.\n\nThis is the class of mistake to watch hardest, because there is no bug to find. The code runs, it is valid pandas, and it computes exactly what it claims to compute. The error lives in the choice of columns, so no test, no exception, and no type check will ever flag it. You catch it by reading the question and then reading the column names in the calculation, and by nothing else.\n\nBoth metrics are real and they measure different things: ship-to-door tells you how the warehouse is performing, and order-to-door tells you how long customers wait. We asked the second question and got the first number, and the one-line summary that people read before a meeting gives no sign of the swap.\n\n#### How to Catch the Error\n\nRead the question, then read the column names in the calculation underneath it. That is the only check that works here, because the code runs clean and no test will ever flag a valid subtraction between the wrong two dates.\n\n## Mistake 2: Writing Numbers That No Code Ever Computed\n\nThis one showed up in two different files. The same `shipment_tracking`\n\nreply closed with a caveat that reads like good practice:\n\nHeads up: Only 22 of 50 orders have delivery dates yet (28 still in transit/pending).\n\nThe file has 40 rows.\n\n```\nprint(len(df), df['delivered_date'].notna().sum(), df['delivered_date'].isna().sum())\n```\n\n#### Output\n\n```\n40 22 18\n```\n\nThe 22 is right. The 50 and the 28 came from nowhere: no code in that session computed either figure or printed either figure.\n\nWhat makes the sentence dangerous is that 50 minus 22 is 28, so it is internally consistent and externally false. A reader doing the arithmetic in their head finds nothing wrong.\n\nThe `regional_sales`\n\nrun failed the same way with more damage. Asked which region performs best, it reported APAC at \"\\$3.68M total sales (32% of all regional revenue)\" and signed off with \"the data is clear.\"\n\n```\nprint(round(df.groupby('region_name')['sales'].sum()['apac'], 2))\n```\n\n#### Output\n\n```\n3675.49\n```\n\nThe total is 3675.49 in whatever unit the file uses, and APAC's share is 30.4%. The reply inflated the magnitude roughly a thousandfold, attached a currency symbol to a column that carries no units, and rounded a share that was never computed. Reading the session log explained how: that run executed no code at all. It printed a pandas snippet and wrote numbers underneath it.\n\nRunning code is not sufficient protection either. One unhurried Luna model run did execute its queries and still wrote that APAC was \"more than 60% above US West and US East combined,\" when those two regions sum to 3575.70 against APAC's 3675.49 — a gap of 2.8%.\n\nIts other comparison in the same paragraph, 32% ahead of `europe_north`\n\n, was correct at 32.2%. One figure was measured and one was invented, side by side in one sentence.\n\nThat is the thing to hold on to about summaries. The numbers inside a code block are computed; the numbers in the paragraph around it are written. Nothing forces the two to agree.\n\n#### How to Catch the Error\n\nAsk whether the code actually ran, and check that every number in the prose appears somewhere in the output, because two of our runs presented code they never executed. In our example, check the grain before accepting any ranking: three duplicate rows inflate `us_west`\n\nby 30.3%, and the regions carry between 1 and 15 years of history, so dividing by years of data puts `us_west`\n\nfirst at 290.9 against APAC's 245.0 and reverses the headline.\n\n## Mistake 3: Reading a Trend From Orders That Have Not Arrived\n\nBack on `shipment_tracking`\n\n, we asked whether shipping was getting faster or slower. The fast pass said faster, and cited week 1 at 3.2 days against week 3 at 1.0.\n\nBoth numbers are real. The conclusion is backwards.\n\n```\ndf['week'] = df['ordered_date'].dt.isocalendar().week\nprint(df.groupby('week').agg(\n    orders=('order_id', 'size'),\n    delivered=('delivered_date', 'count'),\n    avg_days=('delivery_days', 'mean')).round(2))\n```\n\n#### Output\n\n| Week | Orders | Delivered | Avg. Days |\n|---|---|---|---|\n| 1 | 15 | 12 | 3.17 |\n| 2 | 15 | 7 | 2.29 |\n| 3 | 10 | 3 | 1.00 |\n\nThe file ends on January 21. Week 3 orders have had about 3 days to complete; week 1 orders had 17. Of week 3's 10 orders, 7 have no delivery date. The only week 3 orders with a delivery time are the ones that happened to be fast, because the slow ones haven't arrived to be measured.\n\nLater weeks look quicker because more of their evidence is missing. The average falls from 3.17 to 1.00 while unresolved orders climb from 20% to 70%.\n\nGiven the same file and no time pressure, the Luna model caught this unprompted and opened with a warning that the improvement was an illusion. Same trap, same data, opposite outcome.\n\n#### How to Catch the Error\n\nAsk what a blank means before an aggregate drops it for you. The absent delivery dates belonged to the newest and slowest orders, so dropping them manufactured a speedup. The giveaway is that unresolved orders climb from 20% to 70% across the same three weeks.\n\n## Mistake 4: Dropping 226 Blank Heights Without Saying So\n\nOn `olympics_athletes_events`\n\nwe asked whether height helps an athlete win a medal. The fast pass compared the two groups and stopped there.\n\n```\nmedalists = df[df['medal'].notna()]['height']\nothers = df[df['medal'].isna()]['height']\nprint(round(medalists.mean(), 1), round(others.mean(), 1))\n```\n\n#### Output\n\n```\n176.5 176.2\n```\n\nIts verdict: \"Height barely matters — medalists are only 0.3cm taller, so tall does not equal better at winning.\"\n\nThe arithmetic is right and the conclusion does not follow. That comparison ran on 126 of the file's 352 rows, because height is blank for the other 226, and pandas dropped every one of those rows without saying so. The mean of a column ignores its empty cells, so the sample quietly shrank by 64% between the question and the answer, and the reply never mentions it.\n\nThe second problem is what those blanks turn out to be.\n\n```\nprint(round(df[df['height'].notna()]['medal'].notna().mean() * 100, 1))\nprint(round(df[df['height'].isna()]['medal'].notna().mean() * 100, 1))\n```\n\n#### Output\n\n```\n54.0\n23.0\n```\n\nAthletes with a recorded height won a medal 54% of the time, and athletes without one won 23% of the time. A chi-square test on that relationship returns p = 9e-09, which means whether the value exists predicts the outcome far better than the value itself does.\n\nThe reason sits in the years. Of the 302 rows from before 2016, only 76 carry a height, and their medal rate is 26.5%. All 50 rows from 2016 onward carry a height, and their medal rate is 80%. In this file, having a recorded height, being recent, and winning a medal are close to the same fact, so the 126 rows the model tested lean heavily toward the one year where almost everyone medaled.\n\nThe useful answer to \"does height help\" is that this file cannot support one, and a stakeholder is better served by hearing that than by a 0.3cm difference. Every run we did dropped the blanks and analyzed what was left.\n\n#### How to Catch the Error\n\nCheck how many rows survived the calculation, because this comparison ran on 126 of 352 and never said so. Then ask whether the blanks are random: these belonged mostly to the earliest Games, and `NULL`\n\nmeans \"not yet delivered\" in one column and \"did not win a medal\" in another.\n\n## What Happened When We Asked It to Check Its Own Work\n\nFor each first-pass answer we opened a clean session, pasted that answer in full, attached the same file and sandbox, and asked it to verify every number for an exec deck.\n\nOn the `shipment_tracking`\n\nanswer, the review reported \"ONE ERROR in the Heads up section.\" It fixed 50 to 40 and 28 to 18, which was the right correction. It ran code to do it, and it counted the 18 undelivered orders correctly. Then it wrote this:\n\nAll three main metrics are correct:\n\n- Q1: 2.6 days\n- Q2: 45.5%\n- Q3: Getting faster (3.2 to 1.0 days)\n\nQ2 — the on-time rate against a 5-day target — was genuinely correct.\n\nQ1 is mistake 1 and Q3 is mistake 3. So the review approved a delivery time that answered a different question, and approved a trend created by the same 18 undelivered orders it had just finished counting. It had the number that explains the illusion on screen and never connected it to the claim two lines below.\n\nIts corrected reply was identical to the original apart from those two digits. It repaired the fabricated figure from mistake 2, left mistakes 1 and 3 standing, and the answer went out carrying a verification stamp.\n\nThe review of `olympics_athletes_events`\n\nwent further in the wrong direction. It opened with a real catch on a separate error — correctly spotting that the medal share had been computed per record when the question was about athletes, which is the grain problem from the data section — and it fixed that figure to 35.4%.\n\nThen it reached the height comparison from mistake 4. It never mentioned the 226 blank heights, which was the defect in that answer. Instead it reported that the true means were 176.4cm for medalists and 175.5cm for non-medalists, labeled the original 176.2 a \"Major\" error off by 0.7cm, and rewrote the conclusion to say that \"being taller does appear to correlate with winning medals.\"\n\nNo consistent grouping of this file produces 175.5. The 176.4 figure is roughly the medalist mean after duplicate rows are removed, so the review combined two incompatible groupings into one comparison and produced a difference that no single analysis yields. It then used that difference to reverse a verdict — moving from \"height barely matters,\" which the 126 usable rows do support, to a claim of correlation that those same rows reject at p = 0.87. That session also executed no code.\n\nLine up the four mistakes against what the review did with them. It fixed the fabricated numbers in mistake 2. It approved mistakes 1 and 3 without comment. On mistake 4 it missed the defect entirely, invented a replacement, and made the answer worse than the one it was reviewing. Every one of those verdicts arrived in the same confident tone, and nothing in the wording separated the correct ones from the wrong ones.\n\n## Conclusion\n\nThe mechanical work was strong throughout. The model parsed dates, wrote valid SQL and pandas, and in the unhurried runs produced analysis sharper than many analysts would write — including the censoring diagnosis in mistake 3.\n\nThe four mistakes have one thing in common. Each of them turned on something that was not on the screen: the question sitting behind the metric in mistake 1, the code that was never run in mistake 2, the orders that had not arrived yet in mistake 3, and the 226 heights nobody ever recorded in mistake 4. The model read the file it was given, and in all four cases the right answer depended on what the file left out. Knowing what a number is for is still the part you cannot hand over.\n\nRun the second pass for the arithmetic. Then work through those four checks yourself, because the review will tell you the numbers are correct either way.\n\nis a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Nate writes on the latest trends in the career market, gives interview advice, shares data science projects, and covers everything SQL.\n\n[Nate Rosidi](https://twitter.com/StrataScratch)", "url": "https://wpnews.pro/news/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time", "canonical_source": "https://www.kdnuggets.com/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time", "published_at": "2026-09-03 14:00:05+00:00", "updated_at": "2026-09-03 14:22:17.315600+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-safety"], "entities": ["OpenAI", "GPT-5.6 Terra", "GPT-5.6 Luna", "KDnuggets", "Pandas", "SciPy"], "alternates": {"html": "https://wpnews.pro/news/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time", "markdown": "https://wpnews.pro/news/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time.md", "text": "https://wpnews.pro/news/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time.txt", "jsonld": "https://wpnews.pro/news/i-asked-chatgpt-to-analyze-3-datasets-it-made-the-same-mistakes-every-time.jsonld"}}