{"slug": "the-five-minute-ai-jury-that-still-needed-a-human", "title": "The five-minute AI jury that still needed a human", "summary": "Smartphonehoesjes.nl reported roughly 90 experiments in the first five months of 2026, a little over four a week, versus 100 to 120 in a prior full year, after shifting most experiment coding to Claude Code and routing country-level test data through an n8n flow into Airtable for AI-assisted analysis, according to a case the Dutch retailer submitted to the 2026 Experimentation Culture Awards. The setup had an AI proponent and opponent argue each result before an AI jury applied predefined business rules, with the company saying analysis took about five minutes, but the entry states the jury was often wrong when returns or extra metrics entered the decision, leaving the verdict to a human. The submission reports experiment volume rather than revenue lift and does not say how many of the 90 tests reached a valid conclusion; Noah Stokkink, who works on CRO and experimentation at Smartphonehoesjes.nl, describes the result as roughly 80% more experiments, and the company was a finalist in the awards' AI in Experimentation category, which Carwow won.", "body_md": "## Experiment summary\n\nSmartphonehoesjes.nl used Claude Code, n8n, Airtable and an AI debating panel to run about 90 experiments in five months. The company says analysis took about five minutes, while return rates and extra metrics still sent the verdict back to a human.\n\n- The Dutch ecommerce company reports about 90 experiments in the first five months of 2026, a little over four a week, compared with 100 to 120 in a prior full year.\n- Claude Code wrote most experiment code, while an n8n flow moved country-level test data into Airtable for AI-assisted analysis.\n- An AI proponent and opponent argued each result before an AI jury applied predefined business rules. The company says the analysis took about five minutes.\n- The entry reports experiment volume, not revenue lift, and says the jury was often wrong when returns or extra metrics entered the decision.\n- *Hacking Growth* ties tempo to test quality. The entry does not yet say how many of the 90 tests reached a valid conclusion.\n\n13 min read\n\nA conversion team can have more hypotheses than it can afford to build. The backlog grows, the same developer becomes the gatekeeper, and a test that looked urgent in Monday’s meeting is still waiting for code two Fridays later.\n\nAt [Smartphonehoesjes.nl](https://www.smartphonehoesjes.nl/), that build step had been the bottleneck for years. The Dutch retailer sells cases and accessories through its home market, Germany and France. Its public company history says the business began in Leusden in 2011 and now serves between 8,000 and 20,000 customers a day across its shops.\n\nWhat went under test was the experimentation workflow itself. Claude Code would build most variants. An automation would collect the result. Three AI roles would argue over the evidence. Humans would stay responsible for the parts that the system could not judge safely.\n\nIn a [case submitted to the 2026 Experimentation Culture Awards](https://experimentationcultureawards.com/#sh2026), Smartphonehoesjes.nl reports about 90 experiments in five months, against 100 to 120 in a prior full year. The same account says its automated analysis took about five minutes, and that the AI jury was often wrong when a test included returns or additional business metrics.\n\n## A queue grew behind the build step\n\nThe constraint is familiar to any growth manager whose test roadmap depends on a thin engineering lane. Ideas are cheap. Production-ready variants, trigger logic, instrumentation and clean readouts are not. Speeding up ideation alone can make the queue worse.\n\nSmartphonehoesjes.nl framed the problem as a full experiment lifecycle. Since January 2026, its CRO team had been deploying AI across Smartphonehoesjes.nl, Handyhuellen.de and Ploonk.fr. The intervention attacked two slow points. First, it moved most experiment coding to Claude Code. Second, it assembled an AI experiment analysis flow that could retrieve, structure and debate the test data.\n\nThe team gave each tool a defined job. Claude Code handled much of the variant building, while people still owned hypotheses, trigger logic, documentation and review. That division matters more than asking a model, in general terms, to improve conversion.\n\n[Noah Stokkink](https://nl.linkedin.com/in/noah-stokkink), who works on CRO and experimentation at Smartphonehoesjes.nl, identifies himself on LinkedIn as a co-submitter of the case, and his profile describes the result as roughly 80% more experiments. The awards page lists the company as a finalist in the AI in Experimentation category, one of 48 case submissions in 2026. Carwow won the category, and Fyxer, whose testing pace is covered in [How four engineers ran nearly a test a day](https://industrycontents.com/ai-experimentation-workflow-fyxer/), was another finalist.\n\n## Claude Code moved the bottleneck\n\nThe build layer did the most visible work. According to the company’s submission, Claude Code wrote the experiment code almost entirely. Team members supplied the hypothesis, watched the trigger logic and adjusted the output where needed. Two colleagues who had not previously built or analysed experiments began doing both.\n\nThat capability gain matters more than a simple hours-saved claim. One expert no longer sat in the middle of every test. More people could turn a defined hypothesis into a live variant, which expanded the number of parallel learning loops available to the team.\n\nA copyable version does not require an ML team. It requires a stable front-end pattern library, explicit trigger conditions, a test platform and review rules that say which changes can pass quickly and which need engineering attention. A model can be fast and still produce a variant that fires on the wrong page, conflicts with another campaign or records the wrong exposure event.\n\nThe highest-tempo growth teams increase throughput without treating throughput as proof. More experiments only help if assignment stays valid, the primary metric was chosen before the result appeared and the losing code is removed. The entry does not describe those quality controls, so the volume is a measure of capacity. How many of the 90 reached a valid conclusion is the natural next number.\n\n## A five minute jury argued both sides\n\nThe analysis layer is the more inventive part of the system. An n8n flow retrieved test data by country and structured it in Airtable. An AI proponent then argued for the treatment. An AI opponent argued against it. A third AI role acted as the jury and applied predefined business rules to reach a decision.\n\nThe team says that process took about five minutes. The debating pattern is useful because it forces the system to look for competing explanations. A conversion increase may disappear after mobile traffic is separated from desktop. A revenue gain may be carried by a handful of unusually large orders. A local win in France may hide a loss in Germany. The opponent role gives those questions a formal place before a verdict appears.\n\nThe business rules are equally important. The model was not asked to improvise what success meant after seeing the chart. A growth team can encode a simple decision contract before launch. Ship only if the primary metric clears the agreed threshold, guardrails stay within bounds and the result is consistent across the segments that matter commercially.\n\nTwo rules from *Hacking Growth* fit that contract. Sean Ellis and Morgan Brown recommend a 99 percent confidence level, which they say cuts the chance of a false winner from about one in 20 to one in 100. They also give an inconclusive result to the control, because a variant that fails to prove itself may still turn out to be a long term loser.\n\n## Ninety tests need the right denominator\n\nThe public figures need careful arithmetic. Ninety experiments in five months equals a run rate of 18 per month. A previous annual pace of 100 to 120 equals about 8.3 to 10 per month. On those raw numbers, the monthly run rate increased by roughly 80% to 116%.\n\nThe source headline says 80% more experiments, which fits the low end of that calculation. The body also calls the change roughly threefold, which the disclosed figures do not support. The safest statement is the raw one. The team ran about 90 experiments in five months after previously running 100 to 120 in a year.\n\nExperiment velocity is a legitimate growth operations metric because it measures how quickly a team can turn uncertainty into evidence. The entry publishes no combined lift in conversion, revenue, retention or profit from the 90 tests. It also gives no win rate, conclusive rate, sample sizes, test durations or number of tests stopped for quality reasons.\n\n## Where the 90 tests sit against Hacking Growth\n\nNinety experiments in about 21 weeks is a little over four a week. In *Hacking Growth*, Ellis and Brown say many leading growth teams run 20 to 30 experiments a week, and they present that as a pace reached by building up. Their own GrowthHackers team set a floor of three tests a week after finding it had run fewer than ten in a quarter, and called that a vigorous pace for a small team.\n\nThe authors attach a condition to speed. They want the highest pace a team can manage without sloppy execution or invalid results, and they warn that careless tests can do more harm than good, since every test takes the place of another candidate. For Smartphonehoesjes.nl, that makes the share of the 90 tests that reached a valid conclusion the figure to watch.\n\nThe book also asks for a written summary of every test: the variants and audience, key metrics, dates, the hypothesis, sample sizes, statistical confidence and power, confounding issues and the conclusion. Summaries go into a searchable knowledge base, so a team running at high tempo does not repeat a test by accident. Smartphonehoesjes.nl already pushes each country result into Airtable. The entry does not say whether that table doubles as the searchable record, but a fixed summary template would also give the proponent, the opponent and the jury the same fields to argue over every time.\n\n## The jury failed when the business got complicated\n\nThe most useful sentence in the case is the least promotional. When experiments included return rates or additional metrics, the AI jury was often wrong. Manual input was still required, and the system remained under development.\n\n*Hacking Growth* explains why that boundary appears. The authors tell teams to measure more than one metric and to track the downstream metrics an experiment touches, because a gain in one number can come at the expense of another. Returns are the downstream metric for a retailer. They arrive after the order is placed, so a fast verdict has to either wait for them or leave them out.\n\nA model can compare two conversion rates in seconds. It has a harder time deciding whether a short-term gain survives a return window, whether margin moved with revenue or whether a country-level effect should outweigh the blended total. Those questions concern causal structure and business value, and a five minute readout of two conversion rates does not settle them.\n\nThe team also reports a productive catch. During a navigation test, the AI noticed that users were clicking elements without links. A quick human walkthrough had missed the pattern, and the observation led to a follow-up iteration that was still running when the case was submitted. The account does not report the result of that iteration.\n\nThis is the right place for human review. Let automation compress retrieval, calculation and first-pass interpretation. Escalate tests when revenue and margin disagree, delayed outcomes enter the model, segment effects reverse or data quality checks fail. The aim is faster decisions with a visible exception queue.\n\n## What another growth team can copy\n\nStart with one bottleneck and one measurable service level. Smartphonehoesjes.nl began with build and analysis time. Another team might measure median days from approved hypothesis to launch, analyst hours per readout or the percentage of completed tests documented within 48 hours.\n\nKeep the first version narrower than the company’s full system. Use Claude Code experiments for low-risk front-end variants inside an existing design system. Send exposure and outcome data into a fixed analysis template. Ask one model role to make the case for shipping and another to attack that case. Apply explicit rules, then require a person to sign off whenever guardrails or delayed metrics are involved.\n\nThe next test should evaluate the workflow itself. Randomly assign eligible experiments to the AI-assisted process or the normal process. Compare time to launch, analyst minutes, instrumentation errors, decision reversals after human review and the share of tests that reach a valid conclusion. Revenue impact comes later, and decision error is the guardrail to track alongside speed.\n\nThe company’s clearest boundary came from its own AI jury: it could prepare a verdict in minutes, but return rates and competing business metrics still confused it. A faster experimentation programme is useful when people can see those exceptions and make the final call.\n\n| Experiment detail | What the reporting shows | \n|---|---|\n| Constraint | Experiment building had been a bottleneck, and analysis required specialist time. | \n| Intervention | Claude Code built most variants. n8n and Airtable structured country-level data for an AI proponent, opponent and jury. | \n| Period | The first five months of 2026 across Dutch, German and French ecommerce sites. | \n| Reported result | About 90 experiments in five months versus 100 to 120 in a prior year, a little over four a week. Analysis reportedly took about five minutes. | \n| Open questions | Win rate, sample sizes, a workflow control group and commercial uplift across the 90 tests are not published. The company says the AI jury was often wrong when returns or extra metrics were included. | \n\n## Sources and disclosure\n\n- The [Experimentation Culture Awards 2026 finalist entry](https://experimentationcultureawards.com/#sh2026) in the AI in Experimentation category, submitted by the company\n- Smartphonehoesjes.nl’s [official company history](https://www.smartphonehoesjes.nl/service/wie-zijn-wij/) and[careers profile](https://werkenbijsmartphonehoesjes.nl/over-ons/)\n- Case co-submitter [Noah Stokkink on LinkedIn](https://nl.linkedin.com/in/noah-stokkink)\n- Sean Ellis and Morgan Brown, *Hacking Growth* , on experiment tempo, testing rules and test summaries\n\nThe experiment mechanics and results come from Smartphonehoesjes.nl’s own award entry. The awards site says a jury scored each submitted case and interviewed the finalists by video call, and it lists 48 submissions for 2026. The entry does not include a control group for the workflow change, a commercial result across the 90 experiments or an error rate for the AI jury.\n\n## FAQ\n\n## What did Smartphonehoesjes.nl change?\n\nThe CRO team used Claude Code to build most experiment variants and an n8n plus Airtable workflow to prepare country-level test data for three AI roles that argued and judged the result.\n\n## What growth result was reported?\n\nThe company reports about 90 experiments in five months, compared with 100 to 120 in a prior full year. It did not publish a combined revenue or conversion result from those tests.\n\n## Did the AI make final decisions alone?\n\nNo. The company says humans remained in the loop and the AI jury was often wrong when return rates or additional metrics complicated the decision.\n\n## How does the pace compare with Hacking Growth?\n\nThe 90 tests work out at a little over four a week. Ellis and Brown say many leading teams run 20 to 30 a week and advise building pace alongside test quality.\n\n## What can another team copy?\n\nAutomate low-risk variant building and first-pass analysis, encode decision rules before results arrive, and route complex or conflicting metrics to a human reviewer.\n\nCompare more measured experiments in the [Signal Index](https://industrycontents.com/ic-lab/growth-signal-index/), browse the [Growth Signal archive](https://industrycontents.com/category/growth-signal/), or use the benchmark tools in the [IC Lab](https://industrycontents.com/ic-lab/).", "url": "https://wpnews.pro/news/the-five-minute-ai-jury-that-still-needed-a-human", "canonical_source": "https://industrycontents.com/smartphonehoesjes-ai-experimentation-workflow/?utm_source=rss&utm_medium=feed&utm_campaign=editorial", "published_at": "2026-11-09 11:00:00+00:00", "updated_at": "2026-11-09 11:19:19.339546+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools"], "entities": ["Smartphonehoesjes.nl", "Claude Code", "n8n", "Airtable", "Noah Stokkink", "Experimentation Culture Awards", "Carwow", "Fyxer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-five-minute-ai-jury-that-still-needed-a-human", "markdown": "https://wpnews.pro/news/the-five-minute-ai-jury-that-still-needed-a-human.md", "text": "https://wpnews.pro/news/the-five-minute-ai-jury-that-still-needed-a-human.txt", "jsonld": "https://wpnews.pro/news/the-five-minute-ai-jury-that-still-needed-a-human.jsonld"}}