Sharpness & The Edge of Stability #
Sharpness & The Edge of Stability
This week in CS 312, we explored one of the most unintuitive concepts in pretraining: sharpness. An analysis of sharpness suggests that gradient descent should become unstable when sharpness exceeds a critical threshold. Yet in practice, language models routinely cross this threshold and continue learning. In this post, I'll explain what sharpness is, how sharpness appears in real training, and our unanswered questions about sharpness.
- Sharpness Basics
Sharpness measures the highest curvature of the loss function at a certain point. To get the curvature of the loss function we take the Hessian, the second derivative of the loss function, and get the largest eigenvalue of that matrix. The eigenvalue of the Hessian corresponds to how strongly the loss curves in that particular direction, so by getting the maximum eigenvalue we are measuring the steepest curvature (λ).
Two things matter about λ. First, it describes only one axis in a high dimensional space and is independent of the other axes. Second, λₘₐₓ often increases during training. As gradient descent reduces the loss, the parameters can move into regions of increasingly high curvature. This growth of λₘₐₓ is known as progressive sharpening and it’s still unclear why it happens, but it shows up in nearly every pretraining run across architectures.
- Sharpness in a Bowl
To better understand sharpness, let’s consider doing gradient descent on a convex bowl with the loss function L(x) = ½λx², where x measures its position relative to the bottom of the bowl and λ sets the curvature of the bowl. Gradient descent is written as xₜ₊₁ = xₜ − η∇L(xₜ) with learning rate η. We can observe that ∇L(x) = λx such that xₜ₊₁ = xₜ − η(λxₜ) = xₜ(1 − ηλ). The ηλ term is very important because it directs the update of our weights for xₜ₊₁.
When 0 < ηλ < 1, the weights move toward the bottom of the bowl because we are making fractional steps towards the bottom. When 1 < ηλ < 2, the weights overshoot to the other side, but land closer to the bottom each time. At ηλ = 2, the weights oscillate between opposite sides of the minimum without converging. At ηλ > 2, the weights land farther from the bottom each time and the zigzags grow in magnitude until we diverge. A model’s loss is more complicated, but we can locally approximate the loss along its sharpest direction with a bowl of curvature λ. As such, we should expect divergence in general model training once ηλ > 2. That’s why the bowl calculation makes us expect trouble when λ crosses 2/η, but in practice we don’t see divergence.
- Measuring Sharpness in LLMs
To examine sharpness in a real language model setting, I trained a 35M-parameter transformer (8 layers, hidden width 512) using full-batch gradient descent on a fixed dataset of 32 sequences of 1,024 tokens each. I used a constant learning rate of η = 0.01, giving a predicted stability threshold of λ = 2/η = 200.
Every 25 updates I measured λ and loss. I found that we climbed slowly for the first 200 updates, then quickly, and crossed λ = 200 at update 375. Then we the sharpness stopped increasing. For the rest of the run the weights wobble between λ = 220 and 280, with a median of 260. While loss occasionally spikes after crossing 200, we still see healthy training and loss decrease overall after crossing the sharpness threshold. This is known as the edge of stability, when our sharpness wobbles around a threshold rather than exploding or dropping down. In our run, the median sharpness of 260 is 30% above the stability threshold of 200. This raises an interesting question: why does my model stabilize so far beyond the predicted boundary?
- Stability and Progressive Sharpening
In order to understand why our loss does not diverge, we need to think about the loss landscape. A good visualization of the landscape is to think of it as a river-valley. In the river-valley landscape, the lowest loss is at the bottom of the valley along the river, but the valley is slanted downwards so the further along the river the lower your loss. Moreover, as you go along the river the walls of the valley tend to get more steep.
When running gradient descent on the river-valley landscape, our weights can either bounce along the sides of the valley or drop to the bottom of the river. As we continue along the river, the valley gets narrower and our weights bounce farther up the walls. The farther our weights land from the center of the river, the higher the loss and the steeper our walls. This creates a self-stabilizing feedback loop. As curvature increases, the weights begin oscillating more aggressively following gradient descent. Because the loss landscape is nonlinear, these oscillations can push the parameters toward regions of lower curvature, reducing the instability that caused them. Gradient descent is trying to lower the loss, yet across many different networks it also leads toward sharper regions. As the model gets better at fitting the data, the valley often gets narrower. We have explanations in simplified models, but we still don’t understand why the pattern is so widespread and persists across deep learning architectures.
The edge of stability reveals something surprising about deep learning: models can continue optimizing in regimes where simple mathematical models predict instability. My training run raises a more specific question: why did this transformer settle so far above the quadratic threshold? We still lack a complete understanding of the mechanisms governing sharpness during training despite explanations at simplified settings. Understanding these dynamics could help us design more effective optimization methods and potentially enable new approaches to training models.
Thank you to Suhas, Kaiyue, Radi, and others for reviewing this article.
—
References:
Hashimoto, T., & Kotha, S. (2026, October 13). Loss landscapes part 1: Basins, river valley, critical points [Lecture slides]. Stanford CS 312: Deep Learning Alchemy. https://deep-learning-alchemy.github.io/assets/slides/lecture_05.pdf
Hashimoto, T., & Kotha, S. (2026, October 15). Loss landscapes part 2: Edge of stability [Lecture slides]. Stanford CS 312: Deep Learning Alchemy. https://deep-learning-alchemy.github.io/assets/slides/lecture_06.pdf
Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., & Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. International Conference on Learning Representations. https://arxiv.org/abs/2103.00065
Damian, A., Nichani, E., & Lee, J. D. (2023). Self-stabilization: The implicit bias of gradient descent at the edge of stability. International Conference on Learning Representations. https://arxiv.org/abs/2209.15594