DQN vs Q-Learning: Scaling RL with Neural Networks

Deep Q-Networks (DQN) replace the Q-table in Q-learning with a neural network that outputs Q-values for all actions, enabling generalization across similar states. To stabilize training, DQN uses experience replay to break state correlation and target networks to fix the moving Bellman target. The author implements DQN with JAX on a 4x4 gridworld, demonstrating how neural networks scale RL and form the foundation for modern LLM agent logic.

DQN vs Q-Learning: Scaling RL with Neural Networks The core technical shift is simple: Instead of q value = q table state, action , we use q values = neural network state . The network takes the state as input and outputs a Q-value for every available action. The Bellman equation and epsilon-greedy strategy remain identical; we've just changed how the values are stored and retrieved. The real power here is generalization. While a table only knows what it has explicitly seen, a neural network recognizes patterns. If three different states share similar features, the network can infer the value of an unvisited state based on its proximity to known ones. However, combining neural networks with RL is notoriously unstable. To make this work in a real-world AI workflow, two specific architectural fixes are required to prevent the model from collapsing: Experience Replay: In standard RL, consecutive states are highly correlated state 4 → 5 → 6 . Training on this sequence is like feeding a model the same image 32 times; it overfits and diverges. By storing transitions in a buffer and sampling random batches, we break this correlation. replay buffer.add state, action, reward, next state, done batch = replay buffer.sample batch size=32 Random sampling for stability Target Networks: The Bellman target depends on the network's own prediction of the next state. If you update the network, the target moves. It's like chasing a moving goalpost. The fix is to maintain two identical networks: an "online" network that learns and a "target" network that remains frozen for a set number of steps to provide a stable baseline. Stable target calculation using the frozen target network target = reward + gamma max target network next state Gradient update applied only to the online network loss = target - online network state action 2 I've been implementing this using JAX to handle the 4x4 gridworld. Even in a simple environment, the difference in how the agent "perceives" the state space via a three-layer MLP versus a table is a great deep dive into why DQN is the foundation for most modern LLM agent logic. for each episode: while not done: action = epsilon greedy online network, state next state, reward, done = env step state, action replay buffer.add state, action, reward, next state, done if len buffer = 200: batch = replay buffer.sample batch size=32 perform gradient descent on online network Next Proprietary vs Open-Weight: The Moat War → /en/threads/2518/