Building a Next Best Action System with Offline RL

A developer built a music streaming Next Best Action system from scratch using offline reinforcement learning, modeling 5 user engagement states and 5 action channels (email digest, push notification, in-app banner, personalised mix, or do nothing) in a simulator with hardcoded transition probabilities. Training an agent to optimize click-through rate caused reward hacking, where the agent spammed push notifications across every user state because push gets opened more than email, leading the developer to adopt an LTV proxy that converts signals into predicted likelihood of long-term engagement.

The biggest thing companies like Spotify and Netflix care about is keeping users around. And when a user goes quiet, no streams, no opens, no activity for 30 days, the clock starts ticking. What do you do? Do you send an email? A push notification? A personalised playlist? Do nothing and hope they come back on their own? Pick the wrong channel and it might not just fail to re-engage them. It might actively push them out the door. So what’s the right action? And how do you even know if you got it right before you’ve already sent it to a million users? To answer that, I built a music streaming NBA system from scratch using offline reinforcement learning https://arxiv.org/abs/2005.01643 . Next Best Action systems are how companies decide what to do with a user at any given moment. Not just whether to contact them, but which channel, with what content, and when. Every time you get a “we think you’d like this” email or a push notification about a new release, that’s an NBA system making a decision. The goal is straightforward: given everything we know about this user right now, what single action is most likely to keep them engaged? This is a sequential decision problem. The action you take today changes the user’s state tomorrow. That’s what reinforcement learning RL https://ujangriswanto08.medium.com/what-is-reinforcement-learning-a-beginners-guide-to-reinforcement-learning-d6eadadcb668 is built for. The rest of this piece walks through what I built, what broke, and what I learned. So I built a simulator. The simulator models a simplified version of the re-engagement problem: 5 user engagement states from active daily listeners to users about to cancel , 5 action channels email digest, push notification, in-app banner, personalised mix, or do nothing , and a transition table that captures how users move between states depending on what action they receive. The transition probabilities are hardcoded based on intuition about how users actually behave. This is obviously a simplification. In production these transition probabilities would be learned from historical data using a learned dynamics model https://ujangriswanto08.medium.com/how-to-build-and-train-dynamics-models-in-model-based-reinforcement-learning-183f8fc7279a . The advantage of having a simulator is ground truth. The true user behaviour is known, which makes it possible to verify whether the policy actually learned the right thing. With a simulator and a set of actions defined, the next step is defining what a good decision actually looks like. That’s the reward function. The problem is that each channel reports success differently. Email has open rates and click-through rates. Push notifications have session starts and opt-out rates. In-app surfaces have impressions, clicks, and dwell time. These signals are in different units, on different timescales, and measuring different things. So how do I combine them into one number the agent can optimize for? I tried three reward functions. CTR, weighted sum, and an LTV proxy https://addepto.com/blog/customer-lifetime-value-prediction-machine-learning/ . Each one taught me something different about what can go wrong. Click-through rate is the default metric for a reason. It is immediate, easy to measure, and directly observable. But when I trained an agent to optimize for CTR, it learned to spam push notifications across every user state. Push gets opened more than email, so the agent just picked push every time. This is reward hacking: the agent did exactly what was asked. I just asked for the wrong thing. The obvious fix is to combine multiple signals. Weight the clicks, the streams, the saves. But here’s the problem: the weights are made up. Change 0.3 to 0.4 and the policy changes completely. There’s no principled reason for any of it. It’s fragile by design. The fix is to convert everything into one common currency: predicted likelihood of long-term engagement. That’s the LTV proxy. A push notification that gets opened but leads to opt-out scores poorly. A personalised mix that gets saved scores well, even if the immediate CTR is lower. The reward is now grounded in what the business actually cares about. In a normal RL setup the agent explores, collects experience, and updates its policy. But here there is no environment to interact with. I only have logs from the old system. The question is: can those logs be reused to train a smarter policy? Yes, but not with standard Q-learning. When training on a frozen dataset, Q-learning starts assigning high values to actions that were never in the logs. It can never actually test them, so those inflated estimates never get corrected. Q values just keep climbing. Conservative Q-learning https://arxiv.org/abs/2006.04779 CQL fixes this by adding one extra term to the standard Bellman loss. It pushes Q values down for actions not in the data, and pushes them back up for actions that are. The result is a policy that stays conservative. It never gets excited about actions it has never seen. In code it looks like this: q all actions = self.policy net states logsumexp = torch.logsumexp q all actions, dim=1 .mean q data = q all actions.gather 1, actions .squeeze 1 .mean cql penalty = logsumexp - q dataloss = bellman loss + alpha cql penalty logsumexp is a soft maximum over all actions. Minimizing it pushes all Q values down. q data is the Q value for the action actually in the logs. Subtracting it means data actions get pushed back up. Everything else stays low. alpha controls how aggressive the penalty is. Too low and overestimation creeps back in. Too high and the policy becomes so conservative it learns nothing useful. The most important part of building any model is knowing whether it actually works before it touches real users. This is what offline policy evaluation OPE https://edoconti.medium.com/offline-policy-evaluation-run-fewer-better-a-b-tests-60ce8f93fa15 tries to solve. Given only the logs from the old system, can the performance of the new policy be estimated without running it? I tried three approaches: Direct Method trains a reward model on the logs and asks it what the new policy would have gotten. Simple, but the model is almost always wrong. It is trained on data from the old policy, so it has never seen what happens when the new policy takes a different action. Inverse Propensity Scoring skips the model entirely. Instead it reweights the logged rewards by how likely the new policy would have been to take the same action. If the old system sent email 50% of the time but the new policy would only send it 10% of the time, those email observations get downweighted. The problem is when the two policies are very different, those weights get large and a few observations start dominating the whole estimate. Doubly Robust DR combines both. It uses the reward model as a baseline and IPS to correct wherever the model was wrong. It only fails if both are wrong at the same time, which is much less likely than either one failing alone. So I used DR as a deployment gate. A policy only moves forward if its DR estimate beats the behavior policy baseline. I evaluated four policies using the DR estimator: random, the behavior policy, vanilla DQN, and CQL. Both DQN and CQL beat the behavior policy baseline, which was the whole point. CQL scored lower than DQN, which surprised me at first. But CQL spreads probability across actions rather than betting everything on one channel. That conservatism looks worse on a simple average but is safer before it touches real users. The result that actually caught me off guard was random beating the behavior policy. Turns out this is a known result in OPE. When the behavior policy is biased enough, a uniform random policy can score higher on average. It means the old system was actively making bad decisions for certain users. This is a toy system. I know that. Here is what I would do differently if I had real data. The reward model is a linear regression, which only explains about 20% of the variance in LTV. A real system would use a much richer reward model, trained on years of user behaviour. The state space has five buckets. Real users are not five buckets. But the concepts transfer. Overestimation, reward hacking, and the need to evaluate offline before deploying are all real problems. If you want to dig into the code, the full project is on GitHub https://github.com/alyona-vishnoi/music-streaming-nba-rl . Kumar et al. 2020 — Conservative Q-Learning for Offline RL https://arxiv.org/abs/2006.04779 Levine et al. 2020 — Offline RL: Tutorial, Review, and Perspectives https://arxiv.org/abs/2005.01643 Fujimoto et al. 2019 — Off-Policy Deep RL without Exploration BCQ https://arxiv.org/abs/1812.02900 Building a Next Best Action System with Offline RL https://pub.towardsai.net/building-a-next-best-action-system-with-offline-rl-a6031325027d was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.