# Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model

> Source: <https://dev.to/protechscacademy/our-data-analyst-agent-lost-to-a-single-prompt-055-vs-070-on-dabench-with-a-15b-model-ohk>
> Published: 2026-11-18 15:36:24+00:00

**Recommendation:** Benchmark a single-shot code baseline against your agent loop on a dev set before committing to a tool-calling or self-repair architecture, especially with small models.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

We built a **data-analyst agent**: it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend.

Then we compared it with the simplest possible baseline: **ask the same model once for a program, run it, done.** The one-shot baseline won, **0.70 vs 0.55**. This post explains how we measured that and why it happened.

| Configuration | What it does | 
|---|---|
| Direct answer | The LLM answers from the question text, no code | 
| Keyword templates | A classical rule-based solver, no LLM | 
| **Single-shot code** | One LLM call writes a program; we run it once | 
| Agent v1 (tool calls) | The agent must *call*`run_python` and other tools | 
| **Agent v2 (main)** | Plan-and-execute; the reply *is* a program, failed runs are fed back for up to 3 repairs | 
| Agent without self-repair | v2, but a failed run ends the attempt | 

| Configuration | Accuracy [95% CI] | easy / medium / hard | Tokens per question | 
|---|---|---|---|
| Direct answer | 0.025 [0.00, 0.08] | 0.06 / 0.00 / 0.00 | 702 | 
| Keyword templates | 0.300 [0.15, 0.45] | 0.75 / 0.00 / 0.00 | 0 | 
| Agent v1 (tool calls) | 0.225 [0.10, 0.35] | 0.19 / 0.33 / 0.17 | 6,775 | 
| Agent without self-repair | 0.525 [0.38, 0.70] | 0.88 / 0.58 / 0.00 | 1,944 | 
| Agent v2 (main) | 0.550 [0.40, 0.70] | 0.88 / 0.67 / 0.00 | 5,030 | 
| **Single-shot code** | **0.700 [0.55, 0.85]** | 0.94 / 0.67 / **0.42** | **897** | 

*Accuracy with 95% bootstrap intervals on the 40 EVAL questions*

On the same 40 questions, the agent minus single-shot difference is **−0.15 [−0.275, −0.025]**. The agent won 1 question and lost 7 (sign test p = 0.07). So the gap is consistent in direction, but 40 questions are not enough to call it significant at 5%.

**1. The 1.5B model could not drive a tool-calling loop.** In agent v1, 18 of the 40 questions failed because the model never actually ran any code: it wrote tool calls as plain JSON text, echoed the framework's retry prompt, or left chat-template fragments in its code. We added small, documented shims for each of these, and it still reached only 0.225.

**2. Self-repair mostly went in circles.** Agent v2 lets the model see its traceback and try again. On these questions that added only **+0.025** over the same agent without repair (one extra correct answer). The most common v2 failure was a **repair loop**: 15 questions where the model kept fixing one error and introducing another until the 3-repair limit.

**3. The hard questions need one clean program, not many patches.** Single-shot code solved 5 of 12 hard questions. Both agent versions with plan-and-execute solved none. Splitting a hard question into steps gave the small model more places to go wrong.

*Why each configuration failed — no code run, repair loops and execution errors*

**4. It costs more.** The main agent used about **5.6× the tokens** and **3.3× the time** of single-shot code for lower accuracy.

*Tokens per question against accuracy — single-shot code is cheapest and most accurate here*

Agent v2's "reply is a program" protocol was designed **after** we saw v1 fail on the EVAL questions. That makes v2's 0.55 an **optimistic** number, and we label it that way in the report. Even with that advantage, it did not catch up with single-shot code. The lesson for us: choose the agent protocol on the DEV set *before* the evaluation run.

Generated code runs in a fresh `python -I` subprocess in a temporary folder, with a 30-second timeout, a 1 MB CSV limit and an allow-list of three tools. That guards against accidents. It is **not** a security boundary for hostile code, and we say so in the README.

These results are for a small local model. A larger model may well reverse the ranking; we have not tested that here.

*Full project (Pydantic AI agent, sandbox, trace viewer web app, MATLAB version, report): [Data analyst agent with code execution and self-correction — ProTechSC Academy](https://www.protechscacademy.com/projects/agent-development/data-analyst-agent-with-code-execution-and-self-correction). Video walkthrough: [youtu.be/y9vpE8hbuhk](https://youtu.be/y9vpE8hbuhk).*

*We guide final-year and PhD research projects in engineering, AI and AI agents. WhatsApp +91 97907 09424.*
