The core technical shift is simple:
Instead of q_value = q_table[state, action]
, we use q_values = neural_network(state)
.
The network takes the state as input and outputs a Q-value for every available action. The Bellman equation and epsilon-greedy strategy remain identical; we've just changed how the values are stored and retrieved. The real power here is generalization. While a table only knows what it has explicitly seen, a neural network recognizes patterns. If three different states share similar features, the network can infer the value of an unvisited state based on its proximity to known ones.
However, combining neural networks with RL is notoriously unstable. To make this work in a real-world AI workflow, two specific architectural fixes are required to prevent the model from collapsing:
Experience Replay: In standard RL, consecutive states are highly correlated (state 4 → 5 → 6). Training on this sequence is like feeding a model the same image 32 times; it overfits and diverges. By storing transitions in a buffer and sampling random batches, we break this correlation.
replay_buffer.add(state, action, reward, next_state, done)
batch = replay_buffer.sample(batch_size=32) # Random sampling for stability
Target Networks: The Bellman target depends on the network's own prediction of the next state. If you update the network, the target moves. It's like chasing a moving goalpost. The fix is to maintain two identical networks: an "online" network that learns and a "target" network that remains frozen for a set number of steps to provide a stable baseline.
target = reward + gamma * max(target_network(next_state))
loss = (target - online_network(state)[action]) ** 2
I've been implementing this using JAX to handle the 4x4 gridworld. Even in a simple environment, the difference in how the agent "perceives" the state space via a three-layer MLP versus a table is a great deep dive into why DQN is the foundation for most modern LLM agent logic.
for each episode:
while not done:
action = epsilon_greedy(online_network, state)
next_state, reward, done = env_step(state, action)
replay_buffer.add(state, action, reward, next_state, done)
if len(buffer) >= 200:
batch = replay_buffer.sample(batch_size=32)
Next Proprietary vs Open-Weight: The Moat War →