cd /news/ai-agents/our-data-analyst-agent-lost-to-a-sin… · home › topics › ai-agents › article
[ARTICLE · art-171880] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model

A developer's team built a plan-and-execute data-analyst agent on a 1.5B local model and found it lost to a single-shot code baseline on DABench, scoring 0.55 versus 0.70 accuracy across 40 evaluation questions. The agent used roughly 5.6x the tokens and 3.3x the time of the one-shot approach, with self-repair adding only +0.025 and 15 questions stalling in repair loops. The team notes the agent protocol was designed after seeing failures on the eval set, making its 0.55 an optimistic figure, and recommends choosing the protocol on a dev set before evaluation.

recommendation

Benchmark a single-shot code baseline against your agent loop on a dev set before committing to a tool-calling or self-repair architecture, especially with small models.

more in playbook: Designing agent systems
by read4 min views8 publishedNov 18, 2026

We built a data-analyst agent: it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend.

Then we compared it with the simplest possible baseline: ask the same model once for a program, run it, done. The one-shot baseline won, 0.70 vs 0.55. This post explains how we measured that and why it happened.

Configuration What it does
Direct answer The LLM answers from the question text, no code
Keyword templates A classical rule-based solver, no LLM
Single-shot code One LLM call writes a program; we run it once
Agent v1 (tool calls) The agent must callrun_python and other tools
Agent v2 (main) Plan-and-execute; the reply is a program, failed runs are fed back for up to 3 repairs
Agent without self-repair v2, but a failed run ends the attempt
Configuration Accuracy [95% CI] easy / medium / hard Tokens per question
Direct answer 0.025 [0.00, 0.08] 0.06 / 0.00 / 0.00 702
Keyword templates 0.300 [0.15, 0.45] 0.75 / 0.00 / 0.00 0
Agent v1 (tool calls) 0.225 [0.10, 0.35] 0.19 / 0.33 / 0.17 6,775
Agent without self-repair 0.525 [0.38, 0.70] 0.88 / 0.58 / 0.00 1,944
Agent v2 (main) 0.550 [0.40, 0.70] 0.88 / 0.67 / 0.00 5,030
Single-shot code 0.700 [0.55, 0.85] 0.94 / 0.67 / 0.42 897

Accuracy with 95% bootstrap intervals on the 40 EVAL questions

On the same 40 questions, the agent minus single-shot difference is −0.15 [−0.275, −0.025]. The agent won 1 question and lost 7 (sign test p = 0.07). So the gap is consistent in direction, but 40 questions are not enough to call it significant at 5%.

1. The 1.5B model could not drive a tool-calling loop. In agent v1, 18 of the 40 questions failed because the model never actually ran any code: it wrote tool calls as plain JSON text, echoed the framework's retry prompt, or left chat-template fragments in its code. We added small, documented shims for each of these, and it still reached only 0.225.

2. Self-repair mostly went in circles. Agent v2 lets the model see its traceback and try again. On these questions that added only +0.025 over the same agent without repair (one extra correct answer). The most common v2 failure was a repair loop: 15 questions where the model kept fixing one error and introducing another until the 3-repair limit.

3. The hard questions need one clean program, not many patches. Single-shot code solved 5 of 12 hard questions. Both agent versions with plan-and-execute solved none. Splitting a hard question into steps gave the small model more places to go wrong.

Why each configuration failed — no code run, repair loops and execution errors

4. It costs more. The main agent used about 5.6× the tokens and 3.3× the time of single-shot code for lower accuracy.

Tokens per question against accuracy — single-shot code is cheapest and most accurate here

Agent v2's "reply is a program" protocol was designed after we saw v1 fail on the EVAL questions. That makes v2's 0.55 an optimistic number, and we label it that way in the report. Even with that advantage, it did not catch up with single-shot code. The lesson for us: choose the agent protocol on the DEV set before the evaluation run.

Generated code runs in a fresh python -I subprocess in a temporary folder, with a 30-second timeout, a 1 MB CSV limit and an allow-list of three tools. That guards against accidents. It is not a security boundary for hostile code, and we say so in the README.

These results are for a small local model. A larger model may well reverse the ranking; we have not tested that here.

Full project (Pydantic AI agent, sandbox, trace viewer web app, MATLAB version, report): Data analyst agent with code execution and self-correction — ProTechSC Academy. Video walkthrough: youtu.be/y9vpE8hbuhk.

We guide final-year and PhD research projects in engineering, AI and AI agents. WhatsApp +91 97907 09424.

── more in #ai-agents 4 stories · sorted by recency
── more on @dabench 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/our-data-analyst-age…] indexed:0 read:4min 2026-11-18 · —