{"slug": "i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly", "title": "I Built an AI Marketing Analyst. Then I Tested Whether It Actually Helped (Mostly, It Didn’t)", "summary": "An open-source project called Devil's Analyst, comprising 16 agent skills, 229 numbered analytical checks and 110 benchmark cases, found no meaningful performance gain when its analytical skills were given to an AI agent, with a mean difference of 0.01 across 216 runs on 36 draft-report cases. The test, run on 17 October 2026 using Claude Haiku 5.5 in Claude Code with the default Claude plugin-evaluation judge and three votes per grader, showed almost no difference in one broad evaluation, while a smaller follow-up found better results on some reporting and audit tasks but also exposed failures that required changes to the skills. The author, C. Kapsalis, concluded that neither result establishes a general performance improvement and published the repository at github.com/C-Kapsalis/devils-analyst.", "body_md": "In one of my test cases, September email revenue was up 11.2%. Every segment that existed in August was down by about a quarter. The entire increase came from a new VIP segment, and a dashboard showing only the total would have told exactly the wrong story.\n\nThat gap is why I built Devil’s Analyst. Getting an AI model to produce an answer is easy. Getting it to challenge the assumptions behind that answer, separate evidence from explanation and recommend an action that holds up to scrutiny is much harder.\n\nDevil’s Analyst is an open-source collection of analytical skills for AI agents. It covers marketing metrics, attribution, experiments, research, pricing, channel performance, Power BI models and reporting.\n\nOnce the skills, benchmark cases and graders were in place, I had one question: does giving an AI agent a structured body of analytical knowledge actually improve its work?\n\nI compared outputs with and without the relevant skills, using identical prompts and synthetic data. The results were mixed. One broad evaluation showed almost no difference. A smaller follow-up found better performance on some reporting and audit tasks, but also exposed failures that required changes to the skills.\n\nNeither result establishes a general performance improvement. But the experiment shows where the approach helped, where it didn’t, and what I still need to test. If you’re building skills, prompts or evaluations for AI agents, those details are the useful part.\n\nConsider an ordinary marketing report. A dashboard shows that conversion rate has declined. Someone identifies a recent landing-page redesign as the likely explanation. A recommendation follows: revert the redesign.\n\nThat chain hides three separate jobs: establish whether the decline is real, investigate what might explain it, and decide what action the evidence justifies.\n\nA correctly calculated metric does not establish its interpretation. A plausible explanation does not establish causality. An action based on an untested explanation remains untested.\n\nI wanted Devil’s Analyst to make the reasoning explicit: define the metric, validate the comparison, investigate competing explanations, examine the evidence and connect recommendations to something measurable.\n\nDevil’s Analyst is an open-source collection of 16 agent skills and a session hook. The project contains 229 numbered analytical checks, alongside formulas, diagnostic playbooks, adversarial questions and review procedures.\n\nIts workflows cover metric definitions, movement decomposition, experimental design, evidence checks, diagnosis, recommendations and review of existing reports. The review workflows also examine whether correct work should remain unchanged.\n\nThe intention is to make analytical reasoning easier to inspect and challenge. The skills should not make an agent produce longer reports or more warnings by default.\n\nRepository: [https://github.com/C-Kapsalis/devils-analyst](https://github.com/C-Kapsalis/devils-analyst)\n\nI compared two conditions:\n\nBoth arms received the same task and identical input data for each case. I used synthetic marketing data because I needed to know what the correct answer should be. Each benchmark case contains deliberately constructed analytical problems, supporting exports and, where appropriate, decoys that should not be flagged.\n\nThe repository contains 110 cases: 62 focused analytical traps, 36 draft reports with supporting exports, and 12 audits and reviews. Of these, 105 are generated by scripts that calculate expected answers; five early cases are hand-written.\n\nOn 17 October 2026, I ran all 36 draft-report cases. Each case received three runs per arm, producing 216 runs in total.\n\nThe model was Claude Haiku 5.5 in Claude Code. The evaluation used the default Claude plugin-evaluation judge, with three votes per grader and majority agreement determining the result.\n\nThe mean difference was 0.01. With three runs per case, that difference is too small to support a meaningful performance claim. The model without Devil’s Analyst already identified planted problems in 107 of 108 runs. These cases did not separate the two conditions very well on problem recall.\n\nOn 22 October 2026, I selected 17 cases: 13 focused analytical cases, two audit cases, one flawed report review and one clean report used as a control.\n\nEach arm received the same prompt and byte-identical fixture data. Arm A read the relevant skill and supporting references. Arm B received the same task without those materials.\n\nThere was one run per arm per case. I graded the outputs myself against the predefined graders for each case.\n\nThis result is more encouraging, particularly for auditing and reviewing existing work. It is also a much smaller experiment. I wrote the skills, cases and graders, knew which arm I was evaluating, and ran each case only once. It is a promising observation, not proof of a general improvement.\n\nOne case asked both agents to build a self-contained HTML dashboard comparing email performance in August and September 2026. The input contained 20 rows of synthetic email-send data, with one row per campaign send and segment.\n\nA new VIP repeat-buyer segment appeared in September. Its attributed revenue made total September revenue look higher, even though the two existing segments had declined.\n\nTotal revenue increased by 11.2%. The two comparable segments declined by 24.8%.\n\nArm A passed all five graders. It produced a revenue bridge, showed blended and like-for-like rates, separated hypotheses from findings, and attached verification steps to its recommendations.\n\nArm B passed two of the five graders. Its dashboard included some correct figures, but it did not make the distinction between the blended result and comparable segments sufficiently explicit.\n\nThe synthetic input also contained repeated weekly figures that required investigation. The stronger output identified the concern and recommended checking the underlying export before treating the totals as reliable.\n\nI included a clean-control case: an existing report without the planted problems the review skill was designed to identify.\n\nAn analytical reviewer that always finds something wrong creates unnecessary work and can undermine confidence in correct work.\n\nArm A concluded that the report was ready to send and passed all three graders. Arm B proposed three changes and passed one grader.\n\nBoth arms failed to distinguish declining market demand from a problem with the website’s SEO performance.\n\nThe skill-assisted arm did not solve the case. I traced the failure to a missing diagnostic step. The instructions required the agent to rule out measurement problems before investigating behaviour, but did not explicitly ask whether the whole market had moved.\n\nI added a general rule: before diagnosing a decline as an organisation-specific problem, investigate whether the category or competitors experienced the same movement.\n\nThe case was rerun with the revised skill, and the previously failing graders passed. This is useful engineering feedback, but not independent validation. I changed the instructions after seeing the failure, and the revised case was familiar to me.\n\nI changed general rules rather than inserting case-specific answers. Otherwise, the benchmark would become a memorisation exercise.\n\nThe data is synthetic. The planted problems make correct answers easier to establish, but real marketing data is messier and often contains questions that cannot be resolved from an export alone.\n\nThe first evaluation did not demonstrate a meaningful overall improvement. The 0.94 versus 0.93 mean score difference is too small to support that conclusion.\n\nThe second evaluation was small: 17 cases and one run per arm. A single changed result can materially affect the comparison.\n\nI was also the judge. I wrote the cases, skills and graders, and knew which arm produced each answer. Another reviewer could disagree with some assessments.\n\nThe experiment did not test the complete plugin-routing mechanism. Skill-assisted agents were directed to read relevant files. This did not test whether the session hook and routing mechanism would reliably select the right skill in an ordinary user session.\n\nNeither arm could execute code in the second evaluation. The skills recommend using code for calculations where appropriate, so this experiment did not test that part of the intended workflow.\n\nFinally, the results are specific to Claude Haiku in Claude Code. They do not establish that the same results will occur with other models, agents or platforms.\n\nThe next evaluation should use more repetitions per case and arm to distinguish consistent behaviour from variation between runs.\n\nI also want independent grading. Another reviewer should assess anonymised outputs without knowing which arm produced them.\n\nThe evaluation should separate whether the agent identified the problem, calculated correctly, avoided false alarms, distinguished observations from hypotheses, recommended an evidence-supported action and explained how to evaluate it.\n\nI also want cases created by people other than me. Finally, I want to test the complete agent workflow, including routing, data access and code execution.\n\nWe are giving AI agents access to more data systems, reporting tools and analytical workflows. Access to data does not automatically produce sound analytical judgement.\n\nAn answer can rely on an unstable metric definition, an unfair comparison or an explanation the evidence does not support. I built Devil’s Analyst to make those assumptions more visible and give an agent a structured way to challenge them.\n\nThe broad benchmark showed almost no difference. The smaller follow-up was encouraging on review tasks and exposed failures I have since fixed.\n\nI would rather publish an inconclusive result and explain what I learned than turn a small difference into a performance claim. The next step is to make the evaluation more independent, reproducible and difficult for my own assumptions to influence.\n\nIf you work with marketing data, I could use your help. I’m looking for benchmark cases written by someone other than me, and for reviewers willing to grade anonymised outputs. Open an issue on the repository or leave a response here.\n\n- Project: [https://github.com/C-Kapsalis/devils-analyst](https://github.com/C-Kapsalis/devils-analyst)\n\n- Benchmark methodology: [https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/METHOD.md](https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/METHOD.md)\n\n- Breadth evaluation: [https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-17-breadth.md](https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-17-breadth.md)\n\n- 17-case spot check: [https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/README.md](https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/README.md)\n\n- Dashboard case and outputs: [https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/dashboard-email.md](https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/dashboard-email.md)\n\n- Clean-control case: [https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/review-clean-control.md](https://github.com/C-Kapsalis/devils-analyst/blob/main/benchmarks/results/2026-10-22-spot-check/review-clean-control.md)\n\nThe benchmark uses synthetic data, not customer datasets. Results describe the specific models, prompts, environments and evaluation procedures documented above.\n\n*Disclosure: I am the author of Devil’s Analyst. It is an independent personal project and is not affiliated with or endorsed by my employer.*\n\n*Christoforos (Chris) Kapsalis is a technical author and marketer with a passion for building open-source tools that will hopefully help businesses in disadvantaged parts of the world or run by members of disadvantaged communities improve their workflows and maximize the value generated by their marketing efforts.*\n\n[I Built an AI Marketing Analyst. Then I Tested Whether It Actually Helped (Mostly, It Didn’t)](https://pub.towardsai.net/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly-it-didnt-31ec50f15cfb) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly", "canonical_source": "https://pub.towardsai.net/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly-it-didnt-31ec50f15cfb?source=rss----98111c9905da---4", "published_at": "2026-11-15 18:01:02+00:00", "updated_at": "2026-11-15 18:21:28.163180+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research"], "entities": ["Devil's Analyst", "C. Kapsalis", "Claude Haiku 5.5", "Claude Code", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly", "markdown": "https://wpnews.pro/news/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly.md", "text": "https://wpnews.pro/news/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly.txt", "jsonld": "https://wpnews.pro/news/i-built-an-ai-marketing-analyst-then-i-tested-whether-it-actually-helped-mostly.jsonld"}}