Sequence data shows up everywhere in natural language processing (NLP). Sentences are sequences of words. Time-stamped data—like sensor readings or stock prices—are sequences of numbers. To predict the next word in a sentence, translate languages, or detect sentiment, a model needs to “remember” what came earlier in the sequence. Ordinary neural networks, which treat each input independently, fail at this. They have no memory.
A typical fully connected (or dense) neural network expects input in a fixed shape—like a flattened image vector or a list of features. There’s no built-in idea of “earlier” or “later.” For sequences, this means losing the chain of context. In language, for example, “The cat sat on the...” could end with “mat,” “roof,” or “sofa.” The correct word depends on every word that came before.
A recurrent neural network (RNN) handles sequences directly. Imagine reading a sentence one word at a time. With each new word, you update your understanding, carrying forward what you’ve seen so far. An RNN does the same by passing information forward through the sequence, one time step at a time.
At each step, an RNN takes the current input and mixes it with its “memory” from the previous step (called the hidden state). It then produces a new hidden state to pass to the next time step.
The hidden state works like a running notebook. Each new word or data point lets you update, erase, or reinforce what you’ve written. This lets RNNs, at least in theory, learn patterns that depend on previous elements of the sequence.
Vanilla RNNs (the basic kind) have a big limitation: they struggle to learn relationships between distant parts of a sequence. If something from early in the data matters much later, the RNN often can't connect the dots.
This happens because of the vanishing gradient problem. When we train RNNs using a process called backpropagation through time, the “signal” that updates their parameters—the gradient—must pass backward through every time step. If the sequence is long, these gradients get multiplied by numbers slightly less than one, over and over. The effect: the gradient quickly shrinks toward zero, like a message fading in a long game of telephone. The network can’t learn connections that span many steps.
Picture a row of dominoes where each only gives a slight nudge. If you have a lot of dominoes, the force barely reaches the last one. Similarly, the signal in an RNN dies out, and the network “forgets” by the time it reaches a distant step.
Researchers developed new RNN variants specifically to tackle these memory issues. The Gated Recurrent Unit (GRU) is a popular choice. It adds internal mechanisms, called gates, to control how information moves through the network.
Gates are like smart switches. At each time step, they decide which information to keep, which to update, and what to forget. This helps the RNN keep important facts from earlier and ignore irrelevant data.
With this setup, GRUs let models carry forward important information across many time steps. They work much better for tasks where context really matters.
A standard GRU cell uses two gates: the update gate and the reset gate.
Here’s what happens inside a GRU cell at each time step:
Gates give the network fine control: it can “remember” relevant dependencies for as many steps as it needs, and “forget” what's unimportant.
Picture a GRU cell as a box with two sliders—one for each gate.
If you sketched it, you’d see arrows for input and previous hidden state feeding into two “gate” blocks (update and reset), then combining to create a candidate hidden state, which merges with prior hidden state based on the update gate’s position.
Vanilla RNNs overwrite their memory at every step. The GRU’s careful switches keep crucial context alive and ignore distractions.
Here's a minimal GRU in PyTorch. It runs a sequence of numbers through a single GRU cell so you can watch the hidden state evolve:
import torch
import torch.nn as nn
gru = nn.GRU(input_size=1, hidden_size=1, batch_first=True)
input_seq = torch.tensor([[[1.0], [2.0], [3.0], [4.0], [5.0]]]) # shape (1, 5, 1)
h0 = torch.zeros(1, 1, 1)
output_seq, hn = gru(input_seq, h0)
print("Output at each timestep:")
print(output_seq)
print("Final hidden state:")
print(hn)
Try changing the input sequence—or initial hidden state—and see how the hidden state tracks the “memory” of what’s happened so far.
GRUs are a common choice for sequence problems where you need memory, but want something faster and simpler than an LSTM (another variant, covered next). In actual NLP tasks, GRUs excel at:
GRUs have fewer gates than LSTMs, so they train faster and use fewer parameters. They’re strong when your sequences are of moderate length, or you want fast experimentation.
On the toughest tasks with really long-range dependencies, LSTMs can sometimes outperform GRUs. But for many practical problems, GRUs offer a great combination of speed and effectiveness. As a starting point for sequence models, they’re a strong default—much better than vanilla RNNs for real tasks.
You’ll learn about LSTMs next. For now, think of GRUs as a smart, efficient upgrade for learning from sequences—especially where basic RNNs would forget important context.
Using PyTorch, define a simple GRU layer with one input and hidden unit. Pass a sequence like [[0.1], [0.2], [0.3], [0.4]] through the GRU, printing the output at each timestep. Change one value in the input sequence and observe how the GRU's outputs change, illustrating how it retains memory of earlier inputs.
Coming up on Day 22: Sequence-to-Sequence Models