{"slug": "our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1", "title": "Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model", "summary": "A developer's team built a plan-and-execute data-analyst agent on a 1.5B local model and found it lost to a single-shot code baseline on DABench, scoring 0.55 versus 0.70 accuracy across 40 evaluation questions. The agent used roughly 5.6x the tokens and 3.3x the time of the one-shot approach, with self-repair adding only +0.025 and 15 questions stalling in repair loops. The team notes the agent protocol was designed after seeing failures on the eval set, making its 0.55 an optimistic figure, and recommends choosing the protocol on a dev set before evaluation.", "body_md": "We built a **data-analyst agent**: it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend.\n\nThen we compared it with the simplest possible baseline: **ask the same model once for a program, run it, done.** The one-shot baseline won, **0.70 vs 0.55**. This post explains how we measured that and why it happened.\n\n| Configuration | What it does | \n|---|---|\n| Direct answer | The LLM answers from the question text, no code | \n| Keyword templates | A classical rule-based solver, no LLM | \n| **Single-shot code** | One LLM call writes a program; we run it once | \n| Agent v1 (tool calls) | The agent must *call*`run_python` and other tools | \n| **Agent v2 (main)** | Plan-and-execute; the reply *is* a program, failed runs are fed back for up to 3 repairs | \n| Agent without self-repair | v2, but a failed run ends the attempt | \n\n| Configuration | Accuracy [95% CI] | easy / medium / hard | Tokens per question | \n|---|---|---|---|\n| Direct answer | 0.025 [0.00, 0.08] | 0.06 / 0.00 / 0.00 | 702 | \n| Keyword templates | 0.300 [0.15, 0.45] | 0.75 / 0.00 / 0.00 | 0 | \n| Agent v1 (tool calls) | 0.225 [0.10, 0.35] | 0.19 / 0.33 / 0.17 | 6,775 | \n| Agent without self-repair | 0.525 [0.38, 0.70] | 0.88 / 0.58 / 0.00 | 1,944 | \n| Agent v2 (main) | 0.550 [0.40, 0.70] | 0.88 / 0.67 / 0.00 | 5,030 | \n| **Single-shot code** | **0.700 [0.55, 0.85]** | 0.94 / 0.67 / **0.42** | **897** | \n\n*Accuracy with 95% bootstrap intervals on the 40 EVAL questions*\n\nOn the same 40 questions, the agent minus single-shot difference is **−0.15 [−0.275, −0.025]**. The agent won 1 question and lost 7 (sign test p = 0.07). So the gap is consistent in direction, but 40 questions are not enough to call it significant at 5%.\n\n**1. The 1.5B model could not drive a tool-calling loop.** In agent v1, 18 of the 40 questions failed because the model never actually ran any code: it wrote tool calls as plain JSON text, echoed the framework's retry prompt, or left chat-template fragments in its code. We added small, documented shims for each of these, and it still reached only 0.225.\n\n**2. Self-repair mostly went in circles.** Agent v2 lets the model see its traceback and try again. On these questions that added only **+0.025** over the same agent without repair (one extra correct answer). The most common v2 failure was a **repair loop**: 15 questions where the model kept fixing one error and introducing another until the 3-repair limit.\n\n**3. The hard questions need one clean program, not many patches.** Single-shot code solved 5 of 12 hard questions. Both agent versions with plan-and-execute solved none. Splitting a hard question into steps gave the small model more places to go wrong.\n\n*Why each configuration failed — no code run, repair loops and execution errors*\n\n**4. It costs more.** The main agent used about **5.6× the tokens** and **3.3× the time** of single-shot code for lower accuracy.\n\n*Tokens per question against accuracy — single-shot code is cheapest and most accurate here*\n\nAgent v2's \"reply is a program\" protocol was designed **after** we saw v1 fail on the EVAL questions. That makes v2's 0.55 an **optimistic** number, and we label it that way in the report. Even with that advantage, it did not catch up with single-shot code. The lesson for us: choose the agent protocol on the DEV set *before* the evaluation run.\n\nGenerated code runs in a fresh `python -I` subprocess in a temporary folder, with a 30-second timeout, a 1 MB CSV limit and an allow-list of three tools. That guards against accidents. It is **not** a security boundary for hostile code, and we say so in the README.\n\nThese results are for a small local model. A larger model may well reverse the ranking; we have not tested that here.\n\n*Full project (Pydantic AI agent, sandbox, trace viewer web app, MATLAB version, report): [Data analyst agent with code execution and self-correction — ProTechSC Academy](https://www.protechscacademy.com/projects/agent-development/data-analyst-agent-with-code-execution-and-self-correction). Video walkthrough: [youtu.be/y9vpE8hbuhk](https://youtu.be/y9vpE8hbuhk).*\n\n*We guide final-year and PhD research projects in engineering, AI and AI agents. WhatsApp +91 97907 09424.*", "url": "https://wpnews.pro/news/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1", "canonical_source": "https://dev.to/protechscacademy/our-data-analyst-agent-lost-to-a-single-prompt-055-vs-070-on-dabench-with-a-15b-model-ohk", "published_at": "2026-11-18 15:36:24+00:00", "updated_at": "2026-11-18 15:45:36.260805+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "mlops"], "entities": ["DABench", "Pydantic AI"], "also_reported_by": [], "takeaway": "On 40 DABench questions a 1.5B model's single-shot code generation hit 0.70 accuracy versus 0.55 for a plan-and-execute agent with self-repair, at 897 versus 5,030 tokens per question.", "recommendation": {"advice": "Benchmark a single-shot code baseline against your agent loop on a dev set before committing to a tool-calling or self-repair architecture, especially with small models.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1", "markdown": "https://wpnews.pro/news/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1.md", "text": "https://wpnews.pro/news/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1.txt", "jsonld": "https://wpnews.pro/news/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on-dabench-with-a-1.jsonld"}}