Munich, early 1990s. A grad student named Sepp Hochreiter is staring at a training run that just won’t behave. He’s working on recurrent neural networks — models built to learn from sequences, to hold onto something from a few steps back the way you hold the start of a sentence in your head while you’re still reading the end of it. Except the model isn’t holding onto anything. The further back the useful information sits, the less the network seems able to learn from it. Not a little less. Basically zero.
That’s the vanishing gradient problem, and in the early ’90s it was quietly wrecking a whole line of research. Recurrent neural networks (RNNs) were supposed to be the obvious tool for sequential data — language, speech, anything with a time axis — because they loop back on themselves, which gives them something like memory. In theory, an RNN could figure out that a word from ten steps ago changes what the current word means. In practice, it mostly couldn’t get past three or four steps before the signal was gone. People talk about the AI winter of that era as a funding and hype problem, and it partly was, but underneath that was a real technical wall nobody had a way over yet.
Hochreiter’s 1991 thesis — supervised by Jürgen Schmidhuber — was one of the first places this got properly diagnosed instead of just noticed. He traced the problem back to what actually happens mathematically when you backpropagate through many time steps: you end up multiplying a lot of small numbers together, over and over, and small numbers multiplied by themselves enough times don’t shrink, they disappear. It took another six years of work before the fix showed up in print. Hochreiter and Schmidhuber published “Long Short-Term Memory” in 1997, in Neural Computation, and the core idea was almost stubbornly simple: stop making information fight its way through repeated multiplication. Build it a separate lane instead — one where it mostly gets added to, not multiplied away — and let small, trainable gates decide what stays, what goes, and what actually gets used right now.
The paper didn’t take off right away. It sat around for years, cited here and there, before eventually becoming one of the most cited papers in deep learning history — the backbone under translation systems, speech recognizers, and text generators for a couple of decades, right up until attention mechanisms and Transformers took over the spotlight.
Picture a message being passed down a mile-long line of people, one person at a time. Person one memorizes a sentence, walks a few steps, whispers it to person two. Do that once and the sentence survives fine. Do it a hundred times and small slips start to pile up — a mumbled word here, a dropped clause there — until by the end of the line you’re not left with a degraded message, you’re left with nothing. Static.
That’s roughly what’s happening inside a plain RNN trying to learn from something several steps in the past. During training, the network works out how much each earlier input contributed to the final error, and doing that math means multiplying numbers together once for every time step in between. Multiply something smaller than one by itself a hundred times and it doesn’t just shrink — it vanishes. So the network’s ability to say “hey, that word from ten steps back mattered” is gone before the signal even makes it that far.
LSTMs get around this with an architectural trick that’s almost too simple for how well it works: instead of routing everything through a chain of multiplications at every step, add a second path — the cell state — that mostly adds to information rather than multiplying it down. It’s less like a game of telephone and more like a conveyor belt running the length of a factory. Workers at each station can pull something off or drop something new on, but the belt itself never stops and never resets to zero.
The center of an LSTM cell is that conveyor belt — the cell state, written Cₜ . It runs the full length of the unrolled network, one time step after another, and at each stop three gates decide what happens to it, what gets erased, what gets added, what gets shown to the outside world.
Before anything new gets written, the cell first has to decide what old information isn’t worth keeping anymore. That’s the forget gate’s job.
fₜ = σ(W_f · [hₜ₋₁, xₜ] + b_f)Let's go through it piece by piece:• xₜ is whatever's arriving right now - a word, a data point, a sensor reading.• hₜ₋₁ is the hidden state carried over from the last step, a compressed summary of everything relevant so far.• [hₜ₋₁, xₜ] just means those two are stitched together into one longer vector.• W_f and b_f are weights and a bias the network learns during training.• σ is the sigmoid function - it squashes any number down into a range between 0 and 1.• fₜ is the result: a bunch of numbers between 0 and 1, one per unit of cell state, where 0 means "forget this completely" and 1 means "hang onto it."
A quick made-up example helps here. Say the cell state currently holds one value tracking “the subject of this sentence is singular,” and it sits at Cₜ₋₁=0.9 . Then a period arrives, signaling the sentence just ended. If the forget gate outputs fₜ =0.1 for that unit, the network has essentially learned that once a sentence wraps up, whether the previous subject was singular or plural stops mattering, so it should mostly wipe that value before writing anything new over it.
Once the network’s decided what to forget, it needs to figure out what’s actually worth remembering from the new input. This happens in two parts, the input gate decides how much of the new information to let through, and a separate calculation proposes what that new information should look like.
iₜ = σ(W_i · [hₜ₋₁, xₜ] + b_i)Ĉₜ = tanh(W_c · [hₜ₋₁, xₜ] + b_c)Where,• iₜ is the input gate's output - again numbers between 0 and 1, controlling how much of the candidate information actually gets stored.• Ĉₜ(say "C-tilde") is the candidate cell state, the new information being proposed, produced through a tanh\tanh tanh function so values can land anywhere between -1 and 1 - meaning the update can push a memory up or down, not just add to it.• W_i, b_i, W_C, b_C are their own separate learned weights and biases.
Back to the running example: the word “cats” shows up, kicking off a new sentence, plural subject. The candidate calculation might output Ĉₜ =−0.8 (call that “plural”), and the input gate computes iₜ=0.95 the network’s basically saying yes, write this in, almost at full strength.
With the forget gate deciding what to drop and the input gate deciding what to add, the cell state gets updated by combining the two:
Cₜ = fₜ * Cₜ₋₁ + iₜ * ĈₜThis one line is really the whole memory mechanism:• fₜ * Cₜ₋₁ takes the old memory and multiplies it, element by element, by the forget gate - clearing out whatever's no longer needed.• iₜ * Ĉₜ takes the proposed new memory and scales it by how confident the input gate is.• Add the two together and you get Cₜ, the updated state.
Following the numbers through: Cₜ₋₁ = 0.9 , fₜ=0.1, so fₜ * Cₜ₋₁=0.09 — nearly wiped out. Then iₜ * Ĉₜ =0.95×(−0.8)=−0.76. Add them: Cₜ=0.09+(−0.76)=−0.67. The cell state has flipped from “singular” to “plural,” driven almost entirely by the new word, with just a faint trace of the old value still hanging around.
The cell state Cₜ is the network’s long-running internal memory, but it’s not used directly anywhere else. What actually gets passed along and used for predictions is the hidden state, hₜ - a filtered slice of the cell state, controlled by the output gate.
oₜ = σ(W_o · [hₜ₋₁, xₜ] + b_o)hₜ = oₜ * tanh(Cₜ)Where,• oₜ is the output gate, deciding what part of memory is relevant right this second.• tanh(Cₜ) squashes the cell state back into the -1 to 1 range, getting it ready to be filtered.• hₜ, the hidden state, is those two multiplied together.
If the model’s about to predict the verb following “cats,” it needs to know the subject is plural right now. Say the output gate computes oₜ =0.9, and tanh(Cₜ)≈−0.58. Then hₜ≈0.9×(−0.58)≈−0.52 — a clear enough signal, passed forward to the next step and to the output layer, that whatever verb comes next should agree with a plural subject (“are,” not “is”).
To make this less abstract, picture an LSTM trained to transcribe a handwritten 19th-century diary from scanned images — reading pen strokes, one at a time, and deciding what letter each one belongs to. It’s a brutally sequential task. At every step the model needs not just the current stroke but everything written before it in the word, sometimes the whole sentence, to make a good call.
Early in a word, the forget and input gates work together to hold onto the shape of a capital letter that started things off, keeping the hypothesis “this is a proper noun” alive even as lowercase strokes come in after it. Old diaries tend to trail off into loopy, ambiguous letterforms toward the end of a word, and this is where memory actually earns its keep — if the cell state has kept “Wednesday” partially formed from earlier strokes, the model can lean on that context rather than the ink alone when it hits an ambiguous loop, and correctly guess “y” instead of “e.” Then, at every full stop, the forget gate resets — the specific letter shapes from that sentence get mostly cleared out, while a slower-moving part of the cell state, tracking the diarist’s general handwriting style, slant, and pen pressure, sticks around and keeps informing predictions for the rest of the page.
That’s really the trick underneath all of this, the cell state isn’t one uniform kind of memory. Different parts of it can end up tracking information at completely different timescales — some flushed at every word break, others surviving a whole page — just because training discovered that certain forget-gate patterns cut down transcription errors more than others.
It’s easy to nod along to that explanation without really feeling it, so let’s walk through one real time step with real numbers — the exact moment the model hits that ambiguous final loop in “Wednesday” and has to decide between “y” and “e.”
• The setup: to keep the math readable, we'll simplify the hidden state and cell state down to a single number each (in a real model these would be vectors with hundreds of dimensions, but the arithmetic works the same way, just repeated across every dimension).• h_{t−1} = 0.6 — the hidden state coming in already carries a fairly strong signal that we're inside a capitalized proper noun.• x_t = 0.4 — the current input is a simplified feature encoding of the ambiguous loop stroke itself (a shape that could plausibly be read as either "y" or "e").• C_{t−1} = 0.7 — the cell state has been holding onto the partial memory of "Wednesday," built up from the strokes read so far.• Step 1 — The forget gate. First we check how much of that existing memory should survive. ○ z_f = W_f ⋅ [h_{t−1}, x_t] + b_f ○ With W_f = [0.5, −0.2] and b_f = 0.1: ○ z_f = (0.5 × 0.6) + (−0.2 × 0.4) + 0.1 = 0.3 − 0.08 + 0.1 = 0.32 ○ f_t = σ(0.32) ≈ 0.58 ○ A value around 0.58 means the model is leaning toward "keep most of this memory" — it isn't at a sentence boundary, so there's no strong reason to wipe the "Wednesday" context.• Step 2 — The input gate and candidate memory. Next, how much of the new stroke should get written in, and what would that new information actually be? ○ z_i = W_i ⋅ [h_{t−1}, x_t] + b_i ○ With W_i = [0.3, 0.7] and b_i = −0.1: ○ z_i = (0.3 × 0.6) + (0.7 × 0.4) − 0.1 = 0.18 + 0.28 − 0.1 = 0.36 ○ i_t = σ(0.36) ≈ 0.59 ○ z_C = W_C ⋅ [h_{t−1}, x_t] + b_C ○ With W_C = [0.4, 0.6] and b_C = 0.05: ○ z_C = (0.4 × 0.6) + (0.6 × 0.4) + 0.05 = 0.24 + 0.24 + 0.05 = 0.53 ○ C̃_t = tanh(0.53) ≈ 0.49• Step 3 — Update the cell state. Combine what survives with what's new. ○ C_t = f_t ∗ C_{t−1} + i_t ∗ C̃_t ○ C_t = (0.58 × 0.7) + (0.59 × 0.49) = 0.41 + 0.29 = 0.69 ○ Notice the cell state barely moved, from 0.7 to 0.69. The model isn't throwing away the "Wednesday" hypothesis just because one stroke is ambiguous — it's reinforcing it.• Step 4 — The output gate and hidden state. Finally, decide how much of that memory to actually expose for this prediction. ○ z_o = W_o ⋅ [h_{t−1}, x_t] + b_o ○ With W_o = [0.6, 0.1] and b_o = 0: ○ z_o = (0.6 × 0.6) + (0.1 × 0.4) = 0.36 + 0.04 = 0.40 ○ o_t = σ(0.40) ≈ 0.60 ○ h_t = o_t ∗ tanh(C_t) = 0.60 × tanh(0.69) ≈ 0.60 × 0.60 = 0.36
that final number, h_t ≈ 0.36, is a moderately strong positive signal getting passed on to the classifier that actually picks the letter. On its own it’s not overwhelming — the stroke really is ambiguous, after all — but it’s tilted firmly enough in one direction, carrying the weight of everything the cell state remembered about this being a capitalized weekday, that it tips the final decision toward “y.” The model isn’t reading the loop in isolation. It’s reading the loop plus the accumulated memory of “Wednesda-” that came before it, and that memory is what breaks the tie.
It’s worth remembering there’s really only one LSTM cell, with one set of learned weights (W_f, W_i, W_C, W_o and their biases). What looks like a chain of different cells across time steps t−1, t, t+1 and so on is actually the same cell, applied again and again, each time picking up the previous hidden state h_{t-1} and cell state C_{t-1} alongside whatever new input x_t just arrived. That’s what “unrolling” means — drawing that same repeated computation out flat, once per step, so you can actually see how information moves through time.
The cell state runs like a highway across the top of that unrolled diagram, carrying information from the first step to the last with only small, gated additions along the way — never the repeated multiplicative crushing that killed vanilla RNNs. That’s the whole insight, laid out across time: a structure built specifically so the gradient has somewhere to go without disappearing on the way.
LSTMs didn’t just patch a technical problem — they changed how people thought about building architectures at all. Instead of treating a network as one big undifferentiated pile of weights, the LSTM introduced the idea of deliberately engineering pathways for information — structures designed on purpose to let some things through easily and filter other things out. That idea kept echoing forward. Gated Recurrent Units (GRUs) took the LSTM’s gating idea and simplified it. Attention mechanisms pushed the “selectively focus on what matters” concept further, freeing it from having to process things strictly in order — a model could just look directly at any point in a sequence instead of relying on a state passed forward step by step. And Transformers, built almost entirely out of attention, eventually took over as the go-to architecture for most large-scale sequence work.
But getting replaced isn’t the same as being forgotten. The real legacy of the LSTM might be the lesson buried in its own twenty-year slow start: some ideas are right long before anyone has the computing power, the data, or the attention span to notice. Hochreiter and Schmidhuber built something to help information survive across long stretches of time, and their paper had to survive its own long stretch before its moment actually came. There’s something worth sitting with in that — that in research, like in memory itself, what matters isn’t being loud right away. It’s being built to last.
The Memory Machine: How Two Researchers Taught Neural Networks to Remember was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.