Tags: #ai #python #machinelearning #datascience
If you spend any time on tech Twitter or LinkedIn right now, you’ve probably seen the hype around Multi-Agent AI. Frameworks like CrewAI, AutoGen, and LangGraph promise that if you divide tasks among specialized agents: a "Planner", a "Coder", a "Reviewer", and an "Explainer" the system will magically solve complex workflows with superhuman reasoning.
As a CS student, I wanted to see if the hype holds up in the real world. Does assigning distinct personas to multiple LLMs actually make data analysis more accurate than just asking a single AI? Or does it just burn API credits for fun?
I built an automated data-analyst pipeline, locked it down in Docker, and tested it on 50 benchmark questions across three public datasets (tips, penguins, and taxis) against two single-agent baselines using Groq's LLaMA 3.3 70B.
🔗 Project Repository: data-analyst-ai-team on GitHub The complete source code, benchmark setup, and implementation details are available in the repository.
Here is what I built, how I set up the evaluation, and the humble reality check the data gave me.
The System Architecture
Before evaluating anything, I set a strict security rule for myself: The LLM should never have direct shell or code execution access.
If you let an LLM generate code and run it directly on your host, you are practically asking for rm -rf / or security leaks. So I built an execution sandbox: The model never sees the entire raw CSV (which saves massive context tokens); it only receives a summary of column names, inferred data types, and the first few sample rows.
I then built three configurations (arms) to test:
**Baseline 1 (Single Agent, 1 Attempt):**
A single LLM receives the schema and question, writes Python/pandas code once, and runs it in the sandbox. If it throws an error or fails, that's it.
Baseline 2 (Single Agent, up to 3 Attempts): The single agent generates code. If the sandbox throws a Python traceback (e.g., KeyError or syntax error), the exact error is fed back to the same agent with: "That failed with [error]. Fix it." (Max 3 attempts).
Multi-Agent Team (4 Agents): (1) Planner: Analyzes the question and outlines the pandas steps.
(2) Coder: Writes the pandas script strictly adhering to the plan.
̐(3) Reviewer: Checks the code and sandbox output. If flawed, it rejects it and sends feedback back to the Coder (up to 3 iterations).
(4) Explainer: Takes the verified numeric result/plot and writes a clean, plain-English summary.
The 50-Question Benchmark
To keep myself honest, I didn't just cherry-pick 3 prompts and call it a day.
I wrote 50 ground-truth questions across four distinct categories:
Calculation (27): e.g., "What is the average total_bill?", "What percentage of diners are smokers?"
Lookup (11): e.g., "Which day had the highest average tip?"
Charts (8): e.g., "Draw a histogram of trip distances."
Unanswerable / Guardrails (4): e.g., "What is the average waiter rating?" (where the column does not exist).
The AI is scored on whether it properly raises a refusal instead of hallucinating a fake number.
I computed the exact ground-truth values using vanilla pandas, froze the questions file, and computed its SHA-256 fingerprint so I couldn't quietly edit questions after seeing errors.
The Results: A Reality Check
I ran all 50 questions across all 3 arms (150 total sandbox executions) logging latency, total tokens, USD cost, and retry counts.
Here are the primary results:
What happened here?
Baseline 2 got a perfect 100% score.
A single prompt with a dumb while-loop feeding Python error tracebacks back into the model beat the entire 4-agent collaborative team.
The Multi-Agent Team cost ~3.5x more
Passing context, plans, feedback, and reviews between the Planner, Coder, Reviewer, and Explainer burned 3.6x more tokens (3,543 vs. 978 mean tokens per question) and cost 3.5x as much money.
The Multi-Agent Team was noticeably slower.
Sequential LLM calls chained together meant average latency ballooned from ~2.4 seconds to ~4.0 seconds per query.
Failure Analysis: Why Did the Team Lose?
When I dug into the exact failed logs, I learned two big lessons about AI engineering:
On question tips-05, the user asked for the average tip percentage. The Coder agent actually computed the correct pandas series in the sandbox. But when the result reached the Explainer agent, the Explainer synthesized a paragraph saying: "The row-level percentages were computed, but the average was not explicitly summarized."
The multi-agent pipeline dropped the ball on synthesis even though the code ran fine. In contrast, the single agent just outputted result = df['tip'].mean() directly without losing context between agents.
Baseline 1 failed exactly 3 questions: tips-14, tips-15, and taxis-15. All three were chart questions. The model successfully generated the matplotlib figure, but forgot to assign a return variable result = "bar_chart".
Under Baseline 1, the sandbox flagged result not set, and it was scored as a failure.
Under Baseline 2, the sandbox returned ValueError: The code did not set 'result', the model saw the error, added result = True on attempt 2, and passed instantly.
You didn't need a Planner or Reviewer agent to fix this; you just needed execution feedback.
A Fun Plot Twist: The Scorer Bug
During testing, the Multi-Agent team initially scored 47/50 (94%).
When I manually inspected the diff for tips-08 ("How many rows have size equal to 6?"), the expected answer was 4.
Baseline output: 4 → Scorer marked PASS.
Team output: "Four rows have a size of 6." → Scorer marked FAIL.
My automated regex scorer was looking for digits (\b4\b), and completely overlooked written English number words. Because the multi-agent team has an "Explainer" agent that turns numbers into polite prose, the team was disproportionately penalized by the test harness!
Fixing the scorer to parse number words ("four" → 4) raised the team to 48/50.
Lesson: Always manually inspect your benchmark failure edge cases. Automated test harnesses can carry measurement biases.
Key Takeaways for Student & Junior Devs
If you are building LLM applications for hackathons, side projects, or internships:
Self-Correction > Agent Teams:
Before building a complex multi-agent hierarchy with LangChain or CrewAI, try a simple Generate → Execute in Sandbox → Retry on Error loop. In deterministic domains like code and SQL, runtime compiler/interpreter feedback is 10x more valuable than another LLM giving "advice".
Beware the Token Tax:
Every agent you add multiplies your token usage, latency, and failure points. Multi-agent coordination should only be introduced when different subtasks require genuinely different tools, isolated state, or human approval gates.
Guardrails work surprisingly well:
All 3 setups scored 4/4 (100%) on unanswerable questions. Prompting models with explicit instructions: "If the data cannot answer the question, raise CANNOT_ANSWER: [reason]" prevented hallucinations completely.
The full project code, FastAPI server, Docker sandbox setup, and frozen benchmark results are open source on GitHub.