cd /news/machine-learning/deep-reinforcement-learning-for-cont… · home › topics › machine-learning › article
[ARTICLE · art-173225] src=stochastic.blog ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Deep reinforcement learning for control

A blog post on deep reinforcement learning for control reports that a Deep Q-Network (DQN) trained on Gymnasium's CartPole-v1 scored 45.80 against a random-policy baseline mean of 25.22 over 100 episodes, and a REINFORCE agent with a baseline reached 40.70 on the same environment. The post also measured random-policy floors on LunarLander-v2 (8-dimensional observation, 4 discrete actions) and Pendulum-v1 (3-dimensional observation, one continuous action), where random returns ranged from -1855.31 to -743.84 with a mean of -1221.86. It frames DQN's replay buffer and target network as fixes for the deadly triad of function approximation, bootstrapping and off-policy data, with Double DQN and dueling networks addressing upward bias in the max operator.

recommendation

Measure a uniformly random policy over 100 episodes per environment first, then report learned-policy returns against that floor, as the post did with CartPole's 25.22 baseline.

more in playbook: Evals and testing for LLM apps
read4 min views6 publishedNov 20, 2026
Deep reinforcement learning for control
Image: Stochastic (auto-discovered)

Stochastic Blog

In Post 35 we solved small control problems with a table. Every state got its own row, every action its own column, and value iteration filled the cells until the numbers stopped moving. That worked while the table fit in memory, but once a state is a vector of four floats or a continuous angle, the table stops existing, so this post builds the machinery that replaces it: neural networks that stand in for the table, policies that are optimized directly, and the trust-region and continuous-control methods that make those policies trainable. The concrete result we reach is a Deep Q-Network (DQN) on CartPole that scores 45.80 against a random baseline of 25.22, and a REINFORCE agent with a baseline that reaches 40.70 on the same environment.

The through-line for the whole post is the gap between random and learned. Every environment gives us a floor for free, the return a coin-flipping policy collects, and every method we build has to clear that floor by a margin we can measure. When a method fails to clear it, the gap tells us exactly how much the method still has to learn.

We work with three Gymnasium classic control environments. CartPole-v1 has a 4-dimensional observation and 2 discrete actions. LunarLander-v2 has an 8-dimensional observation and 4 discrete actions. Pendulum-v1 has a 3-dimensional observation and a single continuous action. All three generate episodes on demand, so there is no dataset to download and every rollout is cheap enough to run on a laptop CPU.

The floor we have to beat #

Before training anything, we need to know what random looks like. We ran 100 episodes per environment with a uniformly random policy and recorded the returns. CartPole returns ranged from 9.0 to 87.0 with a mean of 25.22, and the episode length matched the return exactly because every step pays a reward of 1.0. Pendulum returned between -1855.31 and -743.84 with a mean of -1221.86, and every episode ran the full 200 steps because the environment never terminates early. The two scales are not comparable, which is why we normalize advantages before any policy update.

The histogram makes the CartPole floor concrete. Random play clusters near 22 steps with a long right tail, so a learned policy that averages 45 is doing real work. The Pendulum distribution is a wide band of large negative numbers, spanning the full -1855.31 to -743.84 range, which tells us the reward scale is punishing and the policy has to learn to hold the pendulum upright rather than just survive.

The action counts confirm that random play is close to uniform. CartPole splits near 50/50 and LunarLander spreads across its four actions, so no action-prior correction is needed and no class weighting is needed. We hold out evaluation seeds so random-policy measurements never overlap with training seeds.

Deep value learning #

The first idea is to keep Q-learning but replace the table with a network. Function approximation in RL lets us handle continuous states, but it can trigger The deadly triad: function approximation, bootstrapping (updating toward the network's own estimate), and off-policy data (transitions from older policies) can combine to make the value estimates diverge. DQN stabilizes the triad with two tricks. A Replay buffer breaks the correlation between consecutive transitions by sampling random minibatches, and a Target network gives the bootstrap target a slowly moving anchor instead of a value that chases itself.

Double DQN fixes a subtler problem. The max over next-state values in vanilla DQN is biased upward because the same network both selects and evaluates the best action. Double DQN selects the next action with the online network and evaluates it with the target network, which removes most of that bias. Dueling networks split the output into a state value and a per-action advantage, so the network can learn that some states are good regardless of which action we take. Prioritized experience replay samples transitions with large errors more often, which focuses learning where it is most wrong.

The core of the update is the target calculation and the loss. We compute the current Q-value, pick the next action with the online net, evaluate it with the target net, and regress toward the bootstrapped target.

q = self.online(states).gather(1, actions)
with torch.no_grad():
    next_actions = self.online(next_states).argmax(dim=1, keepdim=True)  # Double DQN selection
    next_q = self.target(next_states).gather(1, next_actions)             # target net evaluation
    target = rewards + self.gamma * (1.0 - dones) * next_q
loss = self.loss_fn(q, target)

We trained a dueling double DQN on CartPole for 200 episodes with a replay buffer of 10,000 transitions, a batch size of 64, and a target sync every 200 steps. The last-20 training return averaged 26.15 and the mean Huber loss was 0.3094. That training figure sits near the random floor because training returns include epsilon-greedy exploration, while the held-out evaluation runs the greedy policy. On that held-out evaluation with 10 episodes the greedy agent scored 45.80 against the random baseline of 25.22. That gap is the whole lesson for value learning: the network learned a policy that survives roughly twice as long as random play, and the loss curve stayed flat enough that the target network and replay buffer did their job.

── more in #machine-learning 4 stories · sorted by recency
── more on @cartpole-v1 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/deep-reinforcement-l…] indexed:0 read:4min 2026-11-20 · —