6 min read
Deliveroo’s churn model cried wolf. Now a randomised holdout will show whether the sharper list works.
Deliveroo had a retention problem inside its retention system. Its model was built to catch restaurants that might leave the marketplace, yet it raised so many warnings that the people expected to act on them could never work through the list.
Growth teams know this failure well. A score can look useful on a dashboard while making the operating queue worse. When almost every account looks urgent, a retention manager still has no answer to who gets the next call.
In a first-party engineering account published on 22 September 2026, Deliveroo data scientist and economist Firat Yaman described how the company rebuilt that system around a scarce resource. That resource was the time of its restaurant account managers and retention team.
The old model optimised the wrong scarcity #
Deliveroo connects consumers, riders and restaurant, grocery and retail partners. Restaurant churn matters because the marketplace has already spent time and money finding and onboarding each partner. A preventable departure also shrinks the selection customers can choose from.
The previous system watched restaurant performance statistics and flagged accounts that resembled partners who later churned. Deliveroo says it was built on purpose for high recall, meaning it tried to catch most restaurants at risk of leaving.
Churn is rare, though. Catching almost every potential case meant creating a flood of false positive churn alerts, and the queue outgrew the retention team. Every call to a healthy restaurant carried an opportunity cost, because an account manager could have phoned a genuinely vulnerable partner instead.
The growth constraint was usable intervention capacity.
Deliveroo ranked risk to fit the outreach queue #
The replacement churn prediction model estimates a probability for every restaurant, ranks the accounts and applies an adjustable threshold. Deliveroo can move that threshold to keep the number of alerts in line with what its teams can realistically investigate.
That adjustment sounds small, yet it changes the system’s objective. Instead of asking how many eventual leavers the model can catch, the team asks whether the top of the list holds enough real risk to justify each contact.
Deliveroo also weighs a restaurant’s wider importance to the business when prioritising the queue. Yaman kept the features, weighting formula and intervention capacity private, so another marketplace could not rebuild the scoring system from the article. The copyable lever is simpler. Set the alert threshold from the team’s weekly capacity, then measure the opportunity cost of every false positive.
Time-aware validation protected the test #
Forecasting future churn invites data leakage. A model can look accurate when information from a later period slips into training or parameter selection.
Deliveroo used rolling window cross validation. Each configuration trained on historical observations and was validated on a later period. The team also chose precision over recall, since restaurant churn is infrequent and outreach capacity is limited.
For interpretability, the pipeline calculates Shapley values, a method that shows how much each input pushed a prediction up or down, and surfaces the feature that contributed most to each estimated churn probability. The model runs weekly in an Argo pipeline, a workflow tool that schedules automated jobs, and sends the risk score with supporting information to its internal users. This matters day to day. A probability can decide who enters the queue, while an explanation can help an account manager prepare the conversation. The article leaves open whether showing that explanation improves the quality of outreach.
Better precision still leaves retention unproven #
Deliveroo reports that offline testing cut the number of flagged restaurants by more than 90% and improved precision eightfold. Recall fell, as expected. The company argues the trade made sense because its team had no way to action the much larger pool the old system produced.
Those are company-reported classification results, and they stop short of showing that the system saved restaurants. Deliveroo withheld the model’s absolute precision, recall, sample size, evaluation period and confidence intervals. An eightfold lift could start from a low base, and fewer alerts can look efficient even when the remaining interventions leave behaviour untouched.
This is where the retention outreach experiment becomes the more important part of the work.
The holdout tests the whole retention system #
Deliveroo is randomly assigning restaurants flagged by the new model to treatment and control. The retention team and account managers see alerts for the treatment group. Alerts for the control group stay hidden. The stated hypothesis says treated restaurants will churn at a lower rate.
That comparison tests the combined pipeline. Prediction, prioritisation, human follow-up and the action offered to a partner all sit inside it. It cannot isolate which part creates any observed effect, yet it answers the business question a retention owner needs first. Does the system reduce churn compared with leaving the same type of at-risk account alone?
The control group also gives Deliveroo a cleaner check on calibration. Because those restaurants receive no alert-driven intervention, the team can compare their predicted churn probabilities with observed churn while the treatment leaves the outcome untouched.
As of the publication date, that experiment was still running. Deliveroo had yet to report its duration, audience size, market coverage, outreach script or result. Any claim that the model reduced churn would therefore be premature.
Audit the queue before retraining the model #
A growth team can use the operating lesson without reproducing Deliveroo’s machine learning stack. Start with the decision queue. Count how many alerts the team can action, how many prove useful, how many displace higher-value work and whether the action moves the metric the business owns.
Say a model produces 1,000 alerts but the team can investigate 100. Optimising recall across all 1,000 solves the wrong problem. Ranking, an adjustable capacity threshold and a randomised holdout make the system auditable in business terms.
Deliveroo’s account also marks the boundary between prediction and growth. Restaurant churn prediction can show where risk sits. Only a controlled intervention can connect an accurate score to retention, because the score alone cannot prove the risk is preventable or that the company chose the right response. Matas ran a similar holdout on dormant loyalty subscribers, while Wantedly found that a candidate ranking boost barely moved churn.
Sources #
Compare the evidence and operating lessons from more AI experiments in our Growth Signal Index, or browse benchmark tooling in the Industry Contents Lab.