Experiment summary #
Netflix ran an AI cold start experiment on its artwork: image embeddings in one arm, pooled interaction data in another, both together in a third. Each idea fell flat by itself. Only the pair moved its core member metrics in an online A/B test, though Netflix has kept the size of that lift to itself.
- New Netflix artwork arrived with almost no interaction history, so the personalisation system had little to go on.
- The team tested CLIP embeddings alone, one model shared across five artwork canvases alone, and both changes together.
- The first two variants were flat and statistically non significant on core online metrics. Only the combined model won.
- The online test ran for at least four weeks on every device platform. A later holdback lasting a month, during the TV redesign, also lifted discovery and streaming hours.
- Netflix disclosed the direction of the results and a 5.691% offline lift on the short panel canvas, but not online effect sizes, sample counts, allocation or cost.
Netflix had a picture problem ahead of its biggest TV home screen redesign in a decade. The new short panel artwork format was about to become the main canvas, and it was the one with the least history. The existing Netflix artwork personalization system knew which images members had clicked or played before. A fresh crop showed up as an opaque ID with almost nothing attached.
The usual fallbacks were exploration and popularity. Netflix could show new artwork more often to gather behaviour, or simply show whichever image worked for the crowd. Either way, personalisation would arrive late, right when the new interface needed it.
Netflix wanted to know whether visual understanding could improve discovery before interaction data piled up, and whether any gain would hold in a live test as well as on an offline scorecard.
Two sensible changes failed on their own #
In an official engineering account, Netflix researchers led by Emma Yanyang Kong describe the old setup as five separate artwork models. Each served one canvas, such as a billboard, a vertical box or a short panel. They learned from asset IDs and member interactions, yet none of them could tell that two crops showed the same actor, scene or colour palette.
The team tried two fixes. The first added CLIP embeddings, a 768 dimension image vector, to every asset. A new image could then borrow signals from artwork that looked like it before it had any history of its own. The second swapped the five canvas models for one unified model, so busy canvases could lend interaction data to quiet ones.
Both ideas made sense on paper. Netflix tested each against the five model production system. Variant one kept the five models and added image embeddings. Variant two pooled the canvases into one model but kept plain ID representations. Variant three put the unified model and the image embeddings together.
A plain treatment against control test would only have shown that the bundle worked. Separate arms showed that each ingredient did nothing on its own.
According to Netflix, the online A/B test ran on every device platform for at least four weeks. Variants one and two were flat and statistically non significant on core member metrics. Variant three alone produced a significant lift, and it went into production.
That pattern points to two separate bottlenecks. Image embeddings told the model what was in the artwork, yet a sparse canvas still had too few examples of how members reacted to those features. Pooling supplied the examples, yet a model that only knew IDs could not carry them over to an image it had never seen. The combined system had both the vocabulary and the history.
The offline number in the Netflix A/B test story #
Netflix reports a 5.691% relative offline lift for the combined model on the short panel canvas, where data was thinnest. It says this beat the gains of the two single change variants added together. That is the only precise effect size published for the artwork experiment.
The figure comes from an inverse propensity score evaluation, run outside the production A/B test. Netflix keeps a small slice of exploration traffic where artwork is drawn from a known distribution. Because the odds of showing each asset are logged at serving time, results can be reweighted to estimate how a candidate policy would have done without inheriting the bias of the system already in production.
Other teams can borrow this. Historical logs record what the current recommender chose to show, which is a skewed sample of what it could have shown. Train and judge a new policy on those logs and you can mistake agreement with the incumbent for quality.
A candidate has to pass that debiased offline gate before it gets any A/B traffic. Netflix also treats changes within plus or minus 1% as non significant on the offline metric. The post does not say how that band was set, how big the exploration slice is, or how many members and impressions sat in each online arm.
So 5.691% is an offline score, and Netflix has kept the online viewing and revenue values private. The number supports the direction of the decision and says nothing about its commercial size.
The redesign became the harder test #
The home screen launch turned into a natural stress test. The short panel format went from sparse to dominant almost overnight, so a model trained on that canvas’s own history would start the rollout with its weakest evidence exactly where traffic was heading.
Netflix shipped the combined model before the redesign and left a small control group on the old per canvas system for a month. It reports significant gains in its core discovery metric and in streaming hours, and says the effects were larger than in the earlier steady state test.
A holdback like this tells you more than a before and after chart. Both groups saw the same new interface and the same catalogue. The only difference was whether the artwork system could carry visual preferences and interaction history into the canvas that had suddenly become important.
Netflix does not publish the control allocation, member count, geographic mix, confidence intervals or exact gains. Nor does it tie the streaming effect to retention or revenue. Streaming hours sit closer to member value than artwork clicks do, but they are still a stepping stone metric.
What teams building multimodal recommendations can copy #
Most companies working on multimodal recommendations will never build Netflix’s embedding store or train a foundation model. What they can copy is the shape of the experiment.
Pick a contained cold start surface: a newly launched product, a new recommendation slot or a fresh content format. Keep the current ranking as control. Test the content representation alone, test signal pooling alone, then test the two together. The point is to find out whether you have a model problem or a data problem.
Ready made image or text embeddings, CLIP embeddings included, can supply the content representation. A shared feature table or warehouse view can pool history across related placements. A growth or product team will still need engineering help to wire those signals into ranking, but nobody has to train a foundation model to test the idea.
Measure what people do after they see the change, as well as similarity scores or predicted clicks. Log serving latency, inference cost, coverage and failure rate. And keep a lasting holdout whenever the treatment shapes the data it will later learn from, or the new policy can wipe out its own point of comparison.
Netflix also reused the same CLIP vectors for artwork ranking that responds to search queries. That reuse hints at a wider infrastructure lesson. Store representations once, version them and serve the same vectors to training and to live inference. It cuts the gap between training and serving and makes the next experiment cheaper, although Netflix has not said what building it cost.
The missing economics still matter #
Netflix says the winning model is live and helped the redesign, but it gives no online lift, traffic split, compute change or engineering effort. Without those, nobody outside can estimate payback or weigh the unified system against a simpler popularity fallback.
The write up still records two failed variants, a combined winner, an online comparison of at least four weeks and a later holdback lasting a month. It also keeps the 5.691% offline result clearly apart from the undisclosed online outcome.
The disclosure we would most like to see next links the discovery and streaming gains to a cost per extra streaming hour, with the old system kept alive long enough to measure retention. Until then, the result is about how the system was put together. Netflix fixed cold start by pairing what an image shows with how members had behaved, and the test showed neither half worked without the other.
| Experiment detail | Public evidence |
|---|---|
| Commercial problem | Personalise new artwork before it builds up interaction history. |
| Control | Five separate canvas models using learned asset IDs. |
| Variant one | Image embeddings added to the five existing models. |
| Variant two | One shared model across five canvases, still using IDs only. |
| Variant three | One shared model plus CLIP image embeddings. |
| Duration | At least four weeks online, then a one month redesign holdback. |
| Reported result | Only variant three lifted core online metrics significantly. The redesign holdback lifted discovery and streaming hours. |
| Not published | Online effect sizes, sample, allocation, confidence intervals, geography and cost. |
Sources and disclosure #
- Netflix Technology Blog, MAPS: Netflix’s Multimodal Asset Personalization at Scale , published 28 August 2026
- Netflix, [official company overview](https://about.netflix.com/en)
- Netflix Investor Relations, [operating markets and business context](https://ir.netflix.net/ir-overview/top-investor-questions/)
- CLIP, [original research paper](https://arxiv.org/abs/2103.00020)
Netflix designed, ran and reported these experiments. Industry Contents found no independent release of the underlying results. The engineering post names the variants, duration, direction of significance and production decision, but withholds online effect sizes and sample details.
FAQ #
What did Netflix test in its artwork personalization experiment? #
Netflix compared CLIP image embeddings alone, one unified model across all canvases alone, and both changes together against its old system of five artwork models.
Which variant won the Netflix A/B test? #
Only the combined model produced a statistically significant lift in core online member metrics. The two single change variants were flat and non significant.
Did Netflix publish the streaming lift? #
No. Netflix disclosed a 5.691% offline lift on the short panel canvas and said online discovery and streaming hours improved significantly, but kept the online effect sizes private.
How long did the experiment run? #
The first online A/B test ran on every device platform for at least four weeks. Netflix then ran a one month holdback during its TV home screen redesign.
How can another team run an AI cold start experiment like this? #
Test the AI feature, its supporting data change and the two combined as separate arms. Judge them on a business outcome with cost guardrails, and keep a holdout whenever the new system shapes the data it later learns from.
Compare more measured experiments in the Signal Index, browse the Growth Signal archive, or use the benchmark tools in the IC Lab.