9 min read
Spotify wrote its own test conversations, fixed the agent from the failures, and reports 5% more weekly users. Baselines are unpublished.
TL;DR #
- Spotify had no real multi-turn conversations to test a new recommendation experience with, so its team generated replayable synthetic ones.
- People checked a sample, failure traces fed an automated repair loop, and the new experience was compared with Spotify’s earlier session-refinement product.
- Across roughly 15 million users over two weeks, Spotify reports 14% more additional listening, 5% more weekly active users and a 5% lower skip rate.
- Spotify published the work and left out the markets, traffic allocation, absolute baselines, confidence intervals and statistical significance. The copyable lever is writing test conversations before launch and replaying them against every new version.
Imagine you have built a music assistant that people can talk to. Someone asks for indie soul, narrows the mood, refers to the third artist, then switches to a podcast. A test with a single prompt says little about whether the assistant remembers the thread. Enrico Palumbo, a senior research scientist at Sweden-founded audio streaming company Spotify and first author of its RecSys 2026 paper, and his colleagues faced exactly this at launch. A chatbot shows its worst failures only after people start talking to it, and waiting for launch means finding them in production.
Spotify had seed prompts from an existing single-turn experience and no multi-turn conversations to expose the new agent’s weak points. Its answer was to manufacture those conversations, make them replayable, and use the failures as input to a repair loop before putting the experience in front of listeners.
The paper, posted to arXiv on 16 September 2026, ends with a business result. In a two-week A/B test across several markets and device types, the Spotify conversational agent reached about 15 million users. Spotify reports 5% more weekly active users, 14% more additional listening and a 5% lower skip rate against its earlier experience. A neighbouring test at Pinterest, where AI wrote the search queries for images, ended with visits up 18%.
The missing data was the first experiment #
The work began with a cold start, the stage where no real usage data exists yet. How do you evaluate a conversational recommendation agent when the conversations it must handle have not happened? Spotify’s researchers listed five things that can fail across turns. They were refining content, retaining an instruction, resolving a reference, following a shift in intent and assembling a request delivered across separate inputs. Each had its own scoring rubric.
Spotify’s synthetic conversation testing pipeline had two stages. A language model turned a real single-turn prompt into an abstract multi-turn plan built to exercise one capability. A second model, playing the listener, then followed that plan while talking to the live recommendation agent, including its real tool calls and playlist output. Keeping the plan apart from the conversation mattered. A finished synthetic chat is hard to diagnose, while a plan is an explicit test case. Spotify says plans could be audited, versioned and replayed against a different agent version, and the same format could later hold real production conversations.
A team without a machine learning group can copy this part. Start with a short capability list, turn each item into a scoring rubric, and keep the planned interaction separate from the model’s reply. It can begin as a spreadsheet and a prompt runner, because the value comes from repeating the same tests.
Human checks kept the simulator honest #
Synthetic data can create a closed loop in which one model writes a conversation, another judges it, and both share the same blind spots. Spotify added a small human check before trusting the test set. About 100 conversations were each reviewed by five annotators. Spotify’s own research account reports that 98% exercised the intended capability, 93% were judged realistic, 94% showed reasonable user adaptations and 97% received an accurate verdict from the language model judge.
Those checks could not show that the synthetic conversations would match future listeners. They did show that the test harness was testing what its designers meant it to test. The resulting AI recommendation experiment then surfaced failures that a single average score would hide. Quality fell most sharply after four turns. Some reference-resolution failures came from wrong tool arguments that changed track order, and instruction-retention failures appeared when the agent summarised a request too tightly and dropped an earlier constraint. A model can understand a request while the system around it fails, through a faulty tool contract, memory summary or action order. The fix then lives in that workflow, and a bigger model would not necessarily solve it.
Failure traces became the repair queue #
Spotify then used the test set to automate part of prompt and tool improvement. For each query, the team sampled several agent plans at a higher temperature, a setting that makes the model’s output more varied. Mixed outcomes meant the system could succeed but behaved unreliably. Universal failure pointed to a missing capability. For unreliable cases, a coding agent compared successful and unsuccessful plans and looked for differences in tool choice, argument construction or action order. For capability gaps, it inspected failing traces and proposed a prompt or tool change such as a routing instruction or a tool-use example.
Proposals were batched, re-evaluated and reviewed by people before merging. In one run, Spotify says the loop improved pass@1, the share of test cases the agent passes on its first attempt, by 8% relative to a highly optimised manual prompt. Examples included deciding whether “give me more artists” should refine the current playlist, reading “hardcore” in a rock context, and using a full calendar year for a year-in-music request. The pattern worth taking away is a controlled change queue, where the candidate fix, the regression set and the human approval stay separate. For operators weighing which AI mistakes they can afford, see Seven leaders on when to constrain AI and when to let it fail.
The offline score earned an online test #
The offline score showed the agent was good enough to put in front of real listeners. The online comparison pitted a new experience that supported multi-turn interaction, informational requests and music sessions against a baseline limited to music session refinement. It ran for two weeks, covered several unnamed markets and multiple device types, and reached approximately 15 million users. Spotify reports 14% more additional listening, 5% more weekly active users and a 5% lower skip rate, and says it rolled the agent into production after the test. This Spotify A/B test is the source of every growth figure in the story.
Keeping the two measures apart protects the claim. The 8% gain in pass@1 describes system quality, while the listening, active-user and skip changes are product metrics from a live comparison. One helped the team reach the other.
The public record leaves gaps. Spotify’s paper and research post are first-party sources, and they give no market names, group sizes, absolute metric levels, confidence intervals, p-values or device breakdown. The treatment also changed several capabilities at once, so the test cannot say which part drove the effect.
How a smaller team can copy it #
A smaller software company cannot reproduce Spotify’s traffic or recommendation stack. It can reproduce the order of operations.
- Write five or fewer multi-turn capabilities in language a product manager can score.
- Convert real single-turn inputs into abstract dialogue plans instead of free-form synthetic transcripts.
- Replay those plans against the live tool chain and save the full execution trace.
- Have people audit a small random sample for realism, coverage and judge accuracy.
- Separate unreliable behaviour from missing capability before changing prompts or tools.
- Use the offline gate to earn an online test tied to listening, activation, conversion, retention or another business metric.
Synthetic plans test the requirements their authors thought of, and they find motives, phrasing and behaviour nobody encoded less well. Spotify’s next step is to add real production conversations and extend its taxonomy, and that handoff matters.
For European operators, Spotify is a useful case because the company was founded in Sweden and launched first across six European markets. The experiment ran across several markets too, though the paper does not say which. The lesson fits teams preparing multilingual launches, where collecting representative conversation data country by country is slow.
The next test #
Synthetic conversations exposed system failures, controlled fixes improved an offline score, and a large online comparison measured what listeners did. The next test is whether real production conversations confirm that the synthetic plans pointed at the right failures, and whether the 5% lift holds once baselines and significance levels are published.
Sources and disclosure #
- Spotify Research practitioner account , published September 25, 2026
- [RecSys 2026 paper and preprint](https://arxiv.org/abs/2609.30297) , posted September 16, 2026
- [Enrico Palumbo on LinkedIn](https://it.linkedin.com/in/enrico-palumbo-0272baa8) , first author and Spotify Senior Research Scientist
- [Spotify company timeline](https://newsroom.spotify.com/spotify-timeline/) for European provenance
Spotify researchers ran and published the experiment, so the growth figures come from the company. Independent replication has not appeared in the sources reviewed here.
Related reading #
- [Wolt got shoppers to try new restaurants, but its ad revenue dropped](https://industrycontents.com/wolt-recommendation-system/)
- [marko.pl’s AI recommendations got more clicks. Did they sell more?](https://industrycontents.com/ai-recommendation-testing-marko/)
FAQ #
Did synthetic conversations cause the 5% weekly user lift? #
The evidence does not isolate that step. The synthetic pipeline helped develop the treatment, while the online test compared the complete conversational experience with the earlier product.
Which metric was the primary one? #
The public account reports additional listening, weekly active users and skip rate, and it names none of them as the formal primary metric.
Can a team copy this without a machine learning function? #
A small team can copy the capability rubric, the dialogue plan format, the replay process, the human audit and the failure triage. Spotify’s automated self-improvement loop and its production-scale recommender need more engineering.
Read more Growth Signal experiments on the systems, comparisons and metrics behind reported growth.