{"slug": "loss-landscapes-of-llms-the-map-beneath-gradient-descent", "title": "Loss Landscapes of LLMs: The Map Beneath Gradient Descent", "summary": "Shrijith Venkatramana, developer of the AI code review tool LiveReview, explains the concept of loss landscapes in large language models, describing how gradient descent navigates high-dimensional parameter spaces and why the geometry of trained LLMs is more complex than a simple bowl-shaped minimum. The post references research by Ian Goodfellow and colleagues showing that neural network optimization paths may lack the expected obstacles, offering insights into learning rates, initialization, and model merging.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nThere is a strange fact about training a large language model:\n\nA model with hundreds of billions of parameters is trained by repeatedly nudging a point in an unimaginably high-dimensional space downhill.\n\nThat sentence sounds almost absurd.\n\nImagine a landscape where every coordinate is a model weight. With 70 billion parameters, your \"position\" is a vector with 70 billion coordinates. The training objective assigns one scalar value to that position. Gradient descent looks at the local slope and says:\n\nMove this way.\n\nThen it does it again. And again. And again.\n\nThe resulting object is the **loss landscape**.\n\nFor developers, loss landscapes are more than a mathematical curiosity. They provide a useful mental model for understanding why learning rates explode, why initialization matters, why some architectures train dramatically better than others, why independently trained models can sometimes be merged or connected, and why the geometry of a trained LLM is much stranger than the familiar picture of a ball rolling into a single bowl.\n\nThe most interesting part is that the naive picture of \"find the lowest valley\" is increasingly misleading.\n\nStart with an ordinary function:\n\n```\ny = (x - 3)^2\n```\n\nPlot it and you get a bowl.\n\nThe minimum is at:\n\n```\nx = 3\nloss = 0\n```\n\nNow imagine two parameters:\n\n```\nL(w1, w2)\n```\n\nYou can plot this as a 3D surface. Every point `(w1, w2)`\n\ncorresponds to one model, and its height corresponds to the loss.\n\nNeural networks simply take this idea to an absurd scale.\n\nFor a model with parameters\n\n```\ntheta = [theta_1, theta_2, ..., theta_N]\n```\n\nthe training objective is:\n\n```\nL(theta)\n```\n\nwhere `N`\n\nmight be billions.\n\nSo the actual landscape has billions of dimensions.\n\nYou cannot draw it. But the mathematical object is perfectly well-defined.\n\nFor an autoregressive language model, a simplified training loss is cross-entropy:\n\n```\nL(theta) = - (1/T) sum_t log p_theta(x_t | x_<t)\n```\n\nThe model predicts the next token, we compare that distribution with the actual next token, and average the negative log probabilities.\n\nTraining asks us to solve approximately:\n\n```\nmin_theta L(theta)\n```\n\nThe remarkable engineering achievement of modern deep learning is that this apparently ridiculous optimization problem is actually tractable.\n\nAnd that realization was itself historically important.\n\nIn 2014, Ian Goodfellow, Oriol Vinyals, and Andrew Saxe investigated the optimization behavior of neural networks and found something contrary to the prevailing intuition: for several networks they examined, the loss along a straight path from initialization toward the trained solution did not exhibit the giant obstacles that one might expect from a highly non-convex function. Their observation helped motivate a different view of neural-network optimization: perhaps these landscapes are difficult in high-dimensional ways, but not necessarily because SGD is constantly trapped in horrible local minima. ([Google Research](https://research.google/pubs/qualitatively-characterizing-neural-network-optimization-problems/?utm_source=chatgpt.com))\n\nThat distinction becomes crucial once we get to LLMs.\n\nSuppose you're standing somewhere in the landscape.\n\nThe gradient is:\n\n```\ngrad L(theta)\n```\n\nIt points in the direction of steepest increase in loss.\n\nSo gradient descent takes:\n\n```\ntheta_new = theta - eta * grad L(theta)\n```\n\nwhere `eta`\n\nis the learning rate.\n\nThe simplest mental model is skiing downhill.\n\nBut there is an important subtlety.\n\nThe gradient tells you about **local slope**, not the shape of the entire mountain range.\n\nConsider:\n\n```\n          ___\n         /   \\\n    ____/     \\____\n```\n\nTwo places could have exactly the same gradient while having radically different curvature around them.\n\nThat leads naturally to the Hessian:\n\n```\nH = d^2 L / d theta^2\n```\n\nConceptually, the Hessian tells you how the slope itself changes.\n\nIn one dimension:\n\n```\nL(x) = x^2\n```\n\nhas:\n\n```\ndL/dx   = 2x\nd2L/dx2 = 2\n```\n\nThe curvature is positive everywhere.\n\nFor a neural network, the Hessian is an enormous matrix:\n\n```\nN x N\n```\n\nfor `N`\n\nparameters.\n\nFor a 70-billion-parameter model, explicitly constructing it would be laughably impractical.\n\nInstead, practitioners often reason about its **eigenvalues** or quantities related to them.\n\nVery roughly:\n\n``` php\nlarge positive eigenvalue  -> steep direction\nsmall eigenvalue           -> flat direction\nnegative eigenvalue        -> locally downhill in some direction\n```\n\nThis already gives us a much better picture of a trained model.\n\nIt isn't merely sitting at a point.\n\nIt is sitting somewhere inside a complicated geometry containing directions that are extraordinarily stiff and directions that barely matter.\n\nThere is a famous old intuition about non-convex optimization:\n\nThere must be countless terrible local minima, and SGD somehow has to avoid them.\n\nModern neural-network research complicated this picture.\n\nGoodfellow, Vinyals, and Saxe found surprisingly unobstructed paths between initialization and solutions for networks they studied. Later work became even more striking.\n\nIn 2018, Felix Draxler and collaborators and, independently, Timur Garipov and collaborators showed that different independently trained neural networks could be connected through low-loss curves in parameter space. Rather than finding isolated little valleys separated by huge mountains, one could often find a continuous path between solutions that stayed at low loss. ([Proceedings of Machine Learning Research](https://proceedings.mlr.press/v80/draxler18a.html?utm_source=chatgpt.com))\n\nImagine this:\n\n```\ntraditional intuition:\n\n      \\       /\n       \\_____/       \\_____/\n       minimum        minimum\n\n        ^ high barrier ^\n```\n\nversus:\n\n```\nactual high-dimensional geometry:\n\n       _________\n      /         \\____________\n ___ /                       \\___\n    A                         B\n```\n\nThere can be many distinct parameter vectors that all implement excellent solutions, with relatively easy paths connecting them.\n\nThis matters enormously for language models.\n\nTwo independently trained models can have completely different parameter vectors:\n\n```\ntheta_A != theta_B\n```\n\nwhile implementing broadly similar functions.\n\nAnd there is another complication: neural-network parameterizations contain huge amounts of redundancy.\n\nFor example, hidden units can sometimes be permuted without changing the function represented by the network. Rescaling symmetries and other parameterization effects create additional equivalent or near-equivalent representations.\n\nSo the question\n\n\"Where is the optimum?\"\n\nmay be much less meaningful than:\n\n\"What does the region of good solutions look like?\"\n\nThat is a much more interesting question.\n\nSuppose near a trained solution `theta*`\n\n, we perturb the parameters by a small vector `delta`\n\n.\n\nA second-order Taylor approximation gives:\n\n```\nL(theta* + delta)\n≈ L(theta*)\n  + grad L(theta*)^T delta\n  + 1/2 delta^T H delta\n```\n\nAt a well-trained solution:\n\n```\ngrad L(theta*) ≈ 0\n```\n\nso approximately:\n\n```\nL(theta* + delta)\n≈ L(theta*) + 1/2 delta^T H delta\n```\n\nNow imagine diagonalizing the Hessian.\n\nThen the loss increase can approximately be thought of as:\n\n```\nDelta L ≈ 1/2 sum_i lambda_i * delta_i^2\n```\n\nwhere `lambda_i`\n\nis the curvature along direction `i`\n\n.\n\nConsider three directions:\n\n```\nlambda_1 = 1000\nlambda_2 = 1\nlambda_3 = 0.000001\n```\n\nMove by the same amount in each direction.\n\nThe first direction produces a huge loss change.\n\nThe second produces a moderate change.\n\nThe third essentially does nothing.\n\nThis is the intuition behind **flat directions**.\n\nA billion-dimensional model can therefore have a tiny collection of very sensitive directions embedded inside an enormous space of comparatively forgiving directions.\n\nThat is one reason the parameter count alone tells us almost nothing about how difficult optimization is.\n\nSuppose a model has `10^11`\n\nparameters.\n\nEven if only one part in a million corresponded to strongly curved directions, that would still be:\n\n```\n10^11 / 10^6 = 10^5\n```\n\nor roughly 100,000 highly sensitive dimensions.\n\nAnd that leaves roughly 99,999,900,000 other directions.\n\nThis is why \"the model has billions of parameters\" does not imply that optimization is equivalently difficult in billions of independent ways.\n\nThe geometry is highly anisotropic.\n\nLearning rate schedules suddenly become much less mysterious when viewed geometrically.\n\nConsider the simplest quadratic:\n\n```\nL(x) = 1/2 * lambda * x^2\n```\n\nGradient descent gives:\n\n```\nx_new = x - eta * lambda * x\n```\n\nor:\n\n```\nx_new = (1 - eta * lambda) x\n```\n\nFor this to converge rather than explode, roughly:\n\n```\n|1 - eta * lambda| < 1\n```\n\nwhich implies:\n\n```\n0 < eta < 2/lambda\n```\n\nSo the maximum stable learning rate depends on curvature.\n\nNow replace the single `lambda`\n\nwith the largest Hessian eigenvalue:\n\n```\nlambda_max\n```\n\nand you get the rough intuition:\n\n```\neta must be small enough for the stiffest direction\n```\n\nThis explains a frustrating phenomenon engineers routinely encounter.\n\nYou can have a model where most directions are beautifully flat, yet one pathological direction is enormously steep.\n\nThe optimizer cannot simply say:\n\n\"Most of the landscape is flat, so let's take huge steps.\"\n\nThe steep direction gets to veto that decision.\n\nThis is one reason optimization systems spend so much effort on learning-rate schedules, warmup, normalization, optimizer state, gradient clipping, and parameterization.\n\nThey are, in various ways, attempts to make movement through the landscape numerically manageable.\n\nSuppose early training contains badly scaled gradients.\n\nJumping immediately to the final learning rate can move the parameters an enormous distance through the landscape before the model has settled into a useful region.\n\nWarmup effectively says:\n\n```\nstart cautiously\n      ↓\nobserve the geometry through gradients\n      ↓\nincrease step size\n```\n\nIt is not literally measuring the Hessian at every step, but geometrically it is doing something compatible with the idea that optimization dynamics change dramatically during training.\n\nYou will often hear:\n\nFlat minima generalize better.\n\nThere is a real phenomenon behind this statement, but the slogan is too simplistic.\n\nHao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein popularized practical visualization techniques for neural-network loss landscapes. Their 2018 work showed how architecture and optimization choices affect the observed geometry and introduced **filter normalization** to make visual comparisons more meaningful. ([ML Anthology](https://mlanthology.org/neurips/2018/li2018neurips-visualizing/?utm_source=chatgpt.com))\n\nThe core insight is intuitive.\n\nSuppose two solutions have identical training loss:\n\n```\nSolution A:   steep bowl\n\nSolution B:   broad basin\n```\n\nA small parameter perturbation may barely affect B but substantially hurt A.\n\nThat sounds like B should be more robust.\n\nBut there is a serious technical wrinkle:\n\n**sharpness depends on parameterization and scale.**\n\nSuppose we multiply one layer's weights by 10 and compensate by dividing another layer's weights by 10.\n\nThe represented function can remain essentially unchanged while the raw parameter-space curvature changes.\n\nSo saying:\n\n\"This minimum has Hessian eigenvalue 500 and that one has eigenvalue 100\"\n\ndoes not automatically tell you that the first function is less robust.\n\nYou have to specify the geometry being measured.\n\nThis is an important general lesson:\n\nParameter space is not function space.\n\nTwo parameter vectors that look wildly different can implement similar functions.\n\nTwo parameter vectors that are close in Euclidean distance can sometimes implement meaningfully different functions.\n\nThat distinction is particularly important for LLMs, because developers increasingly do operations directly on weights:\n\n```\nfine-tuning\nLoRA\nweight interpolation\nmodel merging\ncheckpoint averaging\ndistillation\ncontinual pretraining\n```\n\nAll of these interact with parameter-space geometry.\n\nNow we can translate the geometry back into everyday LLM work.\n\nSuppose training suddenly does this:\n\n```\nstep 1000   loss = 3.8\nstep 1001   loss = 4.0\nstep 1002   loss = 5.7\nstep 1003   loss = NaN\n```\n\nOne useful interpretation is that optimization has entered a region where the chosen step size is incompatible with the local geometry.\n\nThe cause could involve:\n\n```\nlearning rate\ngradient scale\nnumerical precision\nactivation statistics\noptimizer state\ndata distribution\nnormalization\n```\n\nbut the geometric symptom is simple:\n\n```\nstep too large relative to local curvature\n```\n\nSGD does not observe the exact population gradient.\n\nIt observes an estimate:\n\n```\ng_hat = g + noise\n```\n\nA larger batch generally reduces the variance of this estimator.\n\nThat means the optimizer experiences a different effective dynamical system.\n\nOne way to visualize it:\n\n```\nsmall batch:\n\n        noisy path\n       /\\/\\__/\\/\\___\n      /\n\nlarge batch:\n\n      smooth path\n     /────────────\n```\n\nThis noise is not necessarily undesirable.\n\nIt can affect which parts of the landscape the optimizer visits and which solutions it eventually reaches.\n\nThis is one reason optimization hyperparameters are not merely numerical plumbing. They can change the trajectory through the landscape itself.\n\nResidual connections are a particularly revealing example.\n\nA plain deep network can require every layer to learn a useful transformation.\n\nA residual block can instead learn approximately:\n\n```\nf(x) = x + delta(x)\n```\n\nwhere `delta(x)`\n\nis a correction.\n\nThe identity path creates a much easier route for information and gradients.\n\nLi et al.'s loss-landscape experiments helped visualize the broader phenomenon: architectural choices can alter the geometry of optimization, not merely the number of parameters or FLOPs. ([ML Anthology](https://mlanthology.org/neurips/2018/li2018neurips-visualizing/?utm_source=chatgpt.com))\n\nThis is one reason the history of deep learning is partly the history of making the optimization landscape easier to traverse.\n\nImagine training a frontier model costs:\n\n```\n$50M\n```\n\nand an optimization improvement reduces the required number of training steps by 10%.\n\nVery roughly:\n\n```\n$50M * 0.10 = $5M\n```\n\nThat is before considering engineering capacity, cluster availability, electricity, scheduling, opportunity cost, and failed runs.\n\nA seemingly abstract improvement to optimization geometry can therefore be worth millions of dollars.\n\nThe economics of frontier training makes the landscape a systems problem.\n\nA better optimizer, initialization, normalization scheme, architecture, or learning-rate schedule is effectively a way of making the billion-dimensional terrain cheaper to cross.\n\nThe most useful mental shift is this:\n\n**Training an LLM is probably not best understood as searching for one magical global minimum.**\n\nThe picture is closer to finding a good region in an enormous, structured space of solutions.\n\nYou can imagine:\n\n```\n                       high loss\n                          /\\\n             ____________/  \\________\n            /                         \\\n      _____/                           \\_____\n     /                                         \\\n    A===============================B\n           low-loss region\n```\n\nThe \"equals\" line is not necessarily a straight interpolation.\n\nGaripov et al. demonstrated that low-loss curves could connect solutions that looked separated by barriers under naive linear interpolation. Draxler et al. likewise found essentially barrier-free paths between independently trained solutions in several settings. ([NeurIPS Papers](https://papers.nips.cc/paper/2018/hash/be3087e74e9100d4bc4c6268cdbe8456-Abstract.html?utm_source=chatgpt.com))\n\nThat gives us a striking reinterpretation of several modern LLM techniques.\n\nWhen you fine-tune a base model, you're moving through the landscape.\n\nWhen you train two different fine-tunes, you're landing at different places in the landscape.\n\nWhen you merge models, you're betting that useful solutions occupy sufficiently compatible regions of parameter space.\n\nWhen you average checkpoints, you're betting that nearby points lie within a useful basin.\n\nWhen you change the optimizer, you're changing the dynamics by which you travel.\n\nAnd when you scale the model, you are not merely adding more capacity.\n\nYou are changing the dimensionality and geometry of the object being optimized.\n\nThis is why loss landscapes are such a useful concept for developers: they connect seemingly unrelated engineering decisions into one underlying question.\n\nWhat kind of terrain are we asking gradient descent to navigate?\n\nThe most important thing to take away is not a particular Hessian formula or visualization technique.\n\nIt is the mental model.\n\nA neural network is a point in a gigantic parameter space.\n\nThe loss function turns that space into a landscape.\n\nThe gradient tells you which way is locally uphill.\n\nThe optimizer chooses how aggressively to move.\n\nThe Hessian describes local curvature.\n\nArchitecture changes the terrain.\n\nBatch size changes the noise in your navigation.\n\nLearning rate determines whether your steps are cautious exploration or giant leaps.\n\nAnd surprisingly, good solutions may form broad, connected regions rather than isolated \"perfect minima.\"\n\nOnce you start seeing LLM training this way, a lot of seemingly arbitrary choices become geometrically legible.\n\nThe next time a training run diverges, a fine-tune behaves unexpectedly, or two checkpoints refuse to combine nicely, ask yourself:\n\n**What does the landscape around this model probably look like?**\n\nAnd perhaps the more interesting frontier question is:\n\n**As models scale from billions to trillions of parameters, what properties of their loss landscapes actually change—and which ones remain remarkably invariant?**\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/loss-landscapes-of-llms-the-map-beneath-gradient-descent", "canonical_source": "https://dev.to/shrsv/loss-landscapes-of-llms-the-map-beneath-gradient-descent-e5b", "published_at": "2026-09-02 17:35:51+00:00", "updated_at": "2026-09-02 17:53:55.284034+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Shrijith Venkatramana", "LiveReview", "Ian Goodfellow", "Oriol Vinyals", "Andrew Saxe", "Google Research"], "alternates": {"html": "https://wpnews.pro/news/loss-landscapes-of-llms-the-map-beneath-gradient-descent", "markdown": "https://wpnews.pro/news/loss-landscapes-of-llms-the-map-beneath-gradient-descent.md", "text": "https://wpnews.pro/news/loss-landscapes-of-llms-the-map-beneath-gradient-descent.txt", "jsonld": "https://wpnews.pro/news/loss-landscapes-of-llms-the-map-beneath-gradient-descent.jsonld"}}