Experiment summary #
Wine Access randomly split 23,970 subscribers between human, Claude and Claude-plus-marketer copy. The sales differences were inconclusive, but a cost-adjusted decision rule favoured AI.
- Wine Access wanted to know whether AI could replace or assist three salaried writers without weakening a twice-daily sales email.
- For two weeks, subscribers were randomly assigned to human copy, Claude copy or Claude copy edited by the marketing team.
- Purchase incidence, revenue and gross-profit differences were not statistically significant between the three groups.
- After annualising the two-week results and subtracting assumed labour and software costs, the researchers projected 8.48% more net profit for Claude and 9.36% more for the hybrid policy than for human writers.
- Those are modelled annual gains, not realised savings. The test was registered after it began, and the raw data, exact Claude version and full prompt are not public.
Before one of Wine Access’s twice-daily newsletters could sell a bottle, a salaried writer spent about three hours turning tasting notes, scores and price into a 200 to 600 word pitch. The Napa retailer employed three writers at a combined annual salary and benefits cost of $375,000.
That made the commercial question more demanding than whether Claude could write acceptable prose. Wine Access needed to know whether a cheaper production workflow would preserve the orders and gross profit created by its human team.
Its third randomised field trial gave an untidy answer. Claude did not significantly beat the writers on sales. It still came out ahead once the researchers counted what each workflow would cost to run.
The offer stayed fixed while the writer changed #
The peer-reviewed study by Chicago Booth researchers Jean-Pierre Dubé and Ariel Xu describes three field experiments. This story focuses on the last one, run from 24 February to 9 March 2025.
The Wine Access AI email test allocated 23,970 opted-in subscribers across three groups. The paper’s balance table reports 7,988 people with complete demographic data in the human arm, 7,987 in the Claude arm and 7,988 in the hybrid arm. Seven records lacked some demographic fields, but the researchers retained unsubscribers in the outcome analysis.
Wine Access chose the wines through its existing promotional calendar. Every production method received the same facts about the bottle, including winery, vintage, price, tasting notes and selling points. Claude also received one historical offer for a similar wine or vineyard. The product, title and buy box were held constant so the experiment changed the body copy rather than the proposition.
One group received copy written by the salaried team. A second received Claude’s draft. A third received Claude’s draft after the marketing team could edit it. The cost model kept human work in both AI policies for fact-checking, wine research and, in the hybrid case, editing.
This was a prompt-based Claude email experiment, not a custom-model project. Earlier Wine Access trials had used a Mistral 7B model trained on five years of selected emails plus synthetic examples. For the 2025 trial, the company dropped the training pipeline and the outside developer. The prompt carried bottle facts and one relevant historical email.
That distinction makes the mechanism copyable. A smaller retailer can assemble an input schema and a reference example without training a model. It still needs a source of verified product facts, a review step and a way to keep the treatment cells apart.
The sales result was a statistical draw #
The researchers measured every order made during the two-week campaign, not only purchases attributed to an email click. For each customer they calculated purchase incidence, bottles bought, revenue and gross profit after the cost of wine and other variable costs.
The human group generated $7.616 in gross profit per subscriber during the period. Claude added a point estimate of $0.166. The hybrid added $0.272. Revenue moved in the same direction, but neither difference was statistically significant at the study’s registered threshold.
Purchase incidence was 11.577% in the human cell. The Claude difference was exactly zero percentage points in the reported table, while the hybrid difference was 0.038 points. This AI email marketing RCT therefore does not show that the model persuaded more customers than the writers.
It does show that the AI workflows produced commercially comparable copy during the short test. Fewer than 0.4% of customers unsubscribed: 21 in the human arm, 37 in the Claude arm and 20 in the hybrid arm. The paper reports no unusual customer complaints across the three experiments.
The guardrails matter because a conversion result can hide brand damage. They are not proof that customers approved of AI. Subscribers were not told which system wrote the email, and the study did not test disclosure or long-term trust.
Costs picked a winner the sales test could not #
The study’s decision rule treated annual net profit as the objective. Researchers projected the two-week gross-profit estimates across 26 biweekly periods, then subtracted the labour and software assigned to each policy.
The human policy carried $375,000 in annual writer costs. The researchers modelled the Claude policy at $63,700 and the hybrid at $94,950. Both figures include a $1,200 software licence and reduced human time for checking, research or editing.
On that basis, annual net profit was projected at $5.07 million for the human workflow, $5.50 million for Claude and $5.55 million for the hybrid. The relative differences were 8.48% and 9.36%.
Those are AI marketing profit projections, not money Wine Access reported earning. The arithmetic assumes the two-week point estimates repeat for a year and that the company can reduce the writing team’s cost as modelled. The paper does not report a year-long deployment, realised headcount changes or actual savings.
The decision framework is still the most useful part of the LLM copywriting test. A conventional significance rule would say the company lacked evidence that AI improved sales. A commercial rule asks a different question: is the observed performance high enough, given the lower operating cost and the cost of choosing the wrong workflow?
The researchers used an asymptotic minimax-regret rule calibrated to the profit loss Wine Access was willing to tolerate. That let the company make a bounded decision with thousands of customers per cell instead of waiting for the roughly 191,000 observations per cell that the paper estimates a conventional 5% significance rule would require.
What a smaller team can reproduce #
A team does not need to copy the mathematics to copy the operating design. Choose the decision before the test. If the choice is between human, AI and hybrid production, define the same offer, audience, sending schedule and measurement window for every arm.
Feed every workflow the same approved facts. Log the human minutes spent researching, drafting, checking and correcting. Add model and tooling costs. Measure realised revenue or gross profit across all orders, not just the purchases that an attribution system credits to a click.
Keep unsubscribe, complaint, factual-error and correction rates beside the commercial metric. The Wine Access study says management initially rejected untrained GPT-4 copy as “something they would never send out.” The winning setup therefore depended on better model selection, grounded inputs and retained human checks, not a prompt alone.
Then separate the result you observed from the economics you modelled. A two-week randomised comparison can support a production decision. It cannot establish that annual demand, model quality, labour arrangements or customer tolerance will remain constant.
The evidence stops short of a staffing decision #
The trial has stronger causal evidence than most AI marketing stories. Assignment occurred at customer level, the product facts stayed fixed, pre-period characteristics were balanced and the outcomes included gross profit rather than engagement alone.
The biggest limitations sit where the headline economics begin. The third trial did not include a no-email control, so it compares production methods rather than the incremental value of sending the campaign. It was registered on 7 March, after the 24 February start, because the researchers say institutional approval was delayed.
The exact Claude version and full prompt are not published. Customer-level data are not available for independent replication. The test ran for two weeks at one US wine retailer, and the annual net-profit figures depend on constant effects and management’s staffing assumptions.
The next experiment should preserve a long-running holdout after deployment and measure realised labour hours, corrections, complaints, gross profit and retention. That would show whether the cost advantage survives routine operations without quietly moving work into review and repair.
| Experiment detail | Public evidence |
|---|---|
| Business problem | Preserve the sales performance of twice-daily email copy while reducing production cost. |
| Population | 23,970 opted-in Wine Access subscribers; 23,963 had complete demographic data in the balance table. |
| Arms | Human writers, Claude, and Claude copy edited by the marketing team. |
| Duration | Two weeks, from 24 February to 9 March 2025. |
| Measured outcomes | Purchase incidence, bottles, revenue, gross profit, unsubscribes and complaints. |
| Observed result | AI and hybrid point estimates were higher, but the differences from human copy were not statistically significant. |
| Modelled result | Projected annual net profit was 8.48% higher for Claude and 9.36% higher for hybrid after assumed labour and software costs. |
| Not published | Exact Claude version, full prompt, raw customer data, realised annual savings or a long-term holdout. |
Sources and disclosure #
- Jean-Pierre Dubé and Ariel Xu, Large Language Models and Creative Content Design , Quantitative Marketing and Economics, published 13 January 2026
- [Chicago Booth Review summary](https://www.chicagobooth.edu/review/ai-is-coming-marketing-department) of the field experiments
- Wine Access, [official company history and headquarters](https://www.wineaccess.com/our-story/)
The researchers designed and analysed the experiment with Wine Access’s support and report no competing interests. Industry Contents found no public customer-level dataset or independent replication. The paper is open access under a CC BY 4.0 licence.
FAQ #
What did Wine Access test? #
Wine Access randomly assigned subscribers to email bodies written by its salaried writers, generated by Claude, or generated by Claude and edited by the marketing team.
Did Claude sell more wine than the human writers? #
Claude and the hybrid workflow had higher revenue and gross-profit point estimates, but the differences from the human group were not statistically significant.
Why did the study select an AI policy? #
The decision rule used projected annual net profit. After subtracting assumed labour and software costs, the researchers projected 8.48% more net profit for Claude and 9.36% more for the hybrid workflow.
Were those annual gains realised? #
No. The figures annualise a two-week experiment and apply management’s assumptions about staffing and software. The study does not report a year-long deployment or realised savings.
What can another team copy? #
Hold the offer and source facts fixed, randomise the production workflow, count gross profit across all orders, include labour and software in the decision, and keep factual errors, complaints and unsubscribes as guardrails.
Compare more measured experiments in the Signal Index, browse the Growth Signal archive, or use the benchmark tools in the IC Lab.