{"slug": "adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible", "title": "Adam and AdamW: The Optimizer That Made Modern LLM Training Possible", "summary": "Shrijith Venkatramana, an engineer building LiveReview, explains how the Adam and AdamW optimizers became essential to modern LLM training. The post details how Adam, introduced in 2014 by Diederik Kingma and Jimmy Ba, combines momentum and adaptive learning rates to handle noisy gradients, and how AdamW fixed regularization issues. Venkatramana highlights that Adam's influence earned it an ICLR Test of Time award by 2025.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nMost people learn neural networks by staring at the model.\n\nWeights. Attention. MLPs. LayerNorm. Tokenizers. Context windows.\n\nBut when you actually train an LLM, there is another piece of machinery making billions of decisions every second:\n\n**the optimizer.**\n\nA 70-billion-parameter model does not \"learn\" because gradient descent tells it which direction is better. It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates.\n\nFor the last decade, the dominant answer has largely been some form of **Adam**, and increasingly **AdamW**.\n\nThe interesting part is that Adam is not some mysterious LLM-specific invention. The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era. Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto.\n\nThree years later, the Transformer paper used Adam directly in its training recipe.\n\nThen came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization.\n\nBy 2025, Adam was sufficiently influential to receive an ICLR Test of Time award.\n\nSo what exactly is Adam doing?\n\nAnd why is AdamW usually what you actually want when training a Transformer?\n\nSuppose your neural network has parameters\n\n```\ntheta = [theta_1, theta_2, ..., theta_N]\n```\n\nand your training batch produces a loss `L`\n\n.\n\nBackpropagation gives you\n\n```\ng = dL/dtheta\n```\n\nThe simplest possible optimizer is gradient descent:\n\n```\ntheta <- theta - alpha * g\n```\n\nwhere `alpha`\n\nis the learning rate.\n\nThat looks almost embarrassingly simple.\n\nAnd that is indeed roughly what people did before adaptive optimizers became dominant.\n\nThe problem is that the gradients of a neural network are not nicely behaved.\n\nImagine two parameters:\n\n```\ng_1 = 0.001\ng_2 = 10\n```\n\nA single learning rate has to deal with both.\n\nIf you choose `alpha = 0.001`\n\n, parameter 1 barely moves:\n\n```\ndelta_1 = -0.001 * 0.001 = -0.000001\n```\n\nwhile parameter 2 gets:\n\n```\ndelta_2 = -0.001 * 10 = -0.01\n```\n\nAnd this situation is not exotic.\n\nDifferent parameters can have wildly different gradient scales. Some receive dense gradients every step. Others receive sparse or intermittent signals. Some directions in parameter space are noisy. Others are remarkably consistent.\n\nSo the fundamental problem is:\n\nHow do we turn a raw gradient into a sensible update for each individual parameter?\n\nMomentum gives one answer.\n\nAdaptive methods give another.\n\nAdam essentially combines both.\n\nAdam stands for **Adaptive Moment Estimation**.\n\nThe easiest way to understand it is to imagine that every parameter maintains two small pieces of memory.\n\nThe first remembers:\n\n\"What direction have gradients generally been pointing?\"\n\nThe second remembers:\n\n\"How large have those gradients generally been?\"\n\nFor every parameter, Adam maintains:\n\n```\nm = moving average of gradients\nv = moving average of squared gradients\n```\n\nMore precisely:\n\n```\nm_t = beta1 * m_(t-1) + (1 - beta1) * g_t\n\nv_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2\n```\n\nTypically:\n\n```\nbeta1 = 0.9\nbeta2 = 0.999\n```\n\nThe interpretation is surprisingly intuitive.\n\n`m`\n\n: momentum\nSuppose gradients over five steps are:\n\n```\n+1\n+1\n+1\n+1\n+1\n```\n\nThen the moving average also points strongly positive.\n\nNow imagine:\n\n```\n+1\n-1\n+1\n-1\n+1\n```\n\nThe signs keep cancelling.\n\nAdam therefore distinguishes:\n\n```\nconsistent signal\n```\n\nfrom\n\n```\nnoisy oscillation\n```\n\nThis is momentum.\n\n`v`\n\n: gradient scale\nNow suppose a parameter frequently gets gradients around `10`\n\n, while another gets gradients around `0.01`\n\n.\n\nTheir squared gradients differ by a factor of:\n\n```\n10^2 / 0.01^2 = 100 / 0.0001 = 1,000,000\n```\n\nAdam remembers this.\n\nThat allows it to normalize the effective update.\n\nIgnoring some details for a moment, the update looks like:\n\n```\ndelta_theta ~= -alpha * m / sqrt(v)\n```\n\nSo if a parameter has persistently large gradients, its denominator is large.\n\nIf its gradients are consistently tiny, its denominator is small.\n\nAdam is therefore doing something qualitatively like:\n\nmove in the direction supported by recent gradients, but normalize the step according to how volatile/large those gradients have been.\n\nThat is the core idea.\n\nIt is not just \"gradient descent with momentum.\"\n\nIt is **per-parameter adaptive step sizing**.\n\nThere is an immediately obvious problem with the equations above.\n\nAt initialization:\n\n```\nm_0 = 0\nv_0 = 0\n```\n\nSuppose the very first gradient is:\n\n```\ng_1 = 1\n```\n\nThen:\n\n```\nm_1 = 0.1\n```\n\nbecause:\n\n```\nm_1 = 0.9 * 0 + 0.1 * 1\n```\n\nBut the actual observed gradient was `1`\n\n, not `0.1`\n\n.\n\nThe exponential moving average starts biased toward zero because its history is artificially filled with zeros.\n\nAdam therefore uses bias correction:\n\n```\nm_hat_t = m_t / (1 - beta1^t)\n\nv_hat_t = v_t / (1 - beta2^t)\n```\n\nand the actual update becomes:\n\n```\ntheta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)\n```\n\nThat little correction matters most early in training.\n\nFor example, with:\n\n```\nbeta1 = 0.9\nt = 1\n```\n\nwe have:\n\n```\n1 - beta1^t = 1 - 0.9 = 0.1\n```\n\nso:\n\n```\nm_hat_1 = 0.1 / 0.1 = 1\n```\n\nExactly what we wanted.\n\nThe full Adam algorithm therefore has only a handful of moving pieces:\n\n```\nm_t = beta1 * m_(t-1) + (1-beta1) * g_t\nv_t = beta2 * v_(t-1) + (1-beta2) * g_t^2\n\nm_hat = m_t / (1-beta1^t)\nv_hat = v_t / (1-beta2^t)\n\ntheta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)\n```\n\nThat's basically it.\n\nA remarkable amount of modern deep learning sits on top of those few equations.\n\nThe timing here is worth appreciating.\n\nKingma and Ba submitted the Adam paper in **December 2014**.\n\nAt that point, the dominant deep-learning world looked very different. Recurrent networks, convolutional networks, and SGD-style training were central. The Transformer did not yet exist.\n\nThen, in 2017, Vaswani and colleagues published **Attention Is All You Need**.\n\nThe Transformer paper didn't invent some new optimizer specially designed for attention. It simply used Adam:\n\n```\nbeta1 = 0.9\nbeta2 = 0.98\nepsilon = 1e-9\n```\n\nwith a warmup-and-decay learning-rate schedule.\n\nThat is historically significant because the Transformer went on to become the basic architecture underneath the modern LLM ecosystem.\n\nIn other words, one of the most consequential architecture papers in modern AI essentially plugged an existing adaptive optimizer into a radically different neural architecture.\n\nAnd it worked spectacularly well.\n\nThere is a useful practical lesson here:\n\nThe optimizer does not have to understand the semantics of the architecture.\n\nAdam has no idea whether a parameter belongs to:\n\n```\nQ projection\nK projection\nV projection\nMLP\nembedding table\nlayer normalization\n```\n\nIt simply sees gradients and maintains statistics about them.\n\nThat abstraction is part of its power.\n\nSuppose two parameters receive:\n\n```\nParameter A:\ngradients ≈ [0.1, 0.2, 0.15, 0.1]\n\nParameter B:\ngradients ≈ [10, 20, 15, 10]\n```\n\nParameter B has gradients roughly 100x larger.\n\nWith vanilla SGD:\n\n```\ndelta_B ≈ 100 * delta_A\n```\n\nAdam partially cancels that scale difference because its denominator tracks gradient magnitude.\n\nYou can think of Adam as making the optimizer less sensitive to the arbitrary units in which different parts of the network happen to express their gradients.\n\nThat is especially attractive in giant heterogeneous models.\n\nThere is a catch.\n\nAdam needs to store two additional tensors:\n\n```\nm\nv\n```\n\nfor every parameter.\n\nSo if your model has `N`\n\nparameters, Adam needs roughly:\n\n```\n2N extra values\n```\n\nIf those optimizer states are stored in FP32:\n\n```\n4 bytes/value\n```\n\nthen optimizer state alone costs:\n\n```\n2 * 4 * N = 8N bytes\n```\n\nConsider a 7B parameter model:\n\n```\n7,000,000,000 * 8 bytes\n= 56,000,000,000 bytes\n≈ 56 GB\n```\n\nJust for the two Adam moment tensors.\n\nNot model weights.\n\nNot activations.\n\nNot gradients.\n\nNot KV cache.\n\nJust:\n\n```\nm + v\n```\n\nFor a 70B model:\n\n```\n70B * 8 bytes ≈ 560 GB\n```\n\nThis is one reason optimizer engineering becomes a systems problem at LLM scale.\n\nYou can easily have a situation where the matrix multiplications themselves are perfectly GPU-friendly, but your optimizer state is forcing enormous distributed-memory and communication overhead.\n\nThere are several ways modern systems deal with this:\n\n```\nFSDP / ZeRO-style sharding\noptimizer-state partitioning\nCPU/NVMe offload\n8-bit optimizer states\nfused optimizer kernels\nmixed precision\n```\n\nBut Adam's conceptual simplicity hides a surprisingly expensive implementation reality.\n\nFor example, suppose your parameters are BF16:\n\n```\n2 bytes / parameter\n```\n\nbut your Adam moments are FP32:\n\n```\n8 bytes / parameter total for m and v\n```\n\nThe \"small\" optimizer logic now consumes roughly four times as much memory as the model parameters themselves.\n\nThat is why optimizer state can become a first-class architectural concern in large training systems.\n\nThis is probably the most important distinction to understand in practice.\n\nPeople often use the terms:\n\n```\nL2 regularization\nweight decay\n```\n\nas though they are interchangeable.\n\nFor ordinary SGD, they can effectively be equivalent.\n\nFor Adam, they are **not**.\n\nSuppose we add an L2 penalty to the loss:\n\n```\nL' = L + (lambda / 2) * ||theta||^2\n```\n\nThe gradient becomes:\n\n```\ng' = g + lambda * theta\n```\n\nNow notice what Adam does to `g'`\n\n.\n\nIt doesn't simply subtract:\n\n```\nalpha * lambda * theta\n```\n\nfrom the weights.\n\nThe regularization term goes into the adaptive machinery:\n\n``` php\ng' -> m -> v -> normalization\n```\n\nSo the shrinkage of a parameter becomes entangled with Adam's gradient statistics.\n\nThat produces a surprising effect.\n\nTwo parameters with the same weight magnitude can receive different effective regularization depending on their gradient history.\n\nSuppose:\n\n```\ntheta_1 = 1\ntheta_2 = 1\n```\n\nand the only difference is that:\n\n```\nsqrt(v_1) = 0.1\nsqrt(v_2) = 10\n```\n\nThe same regularization contribution gets normalized very differently.\n\nSo the thing you thought was:\n\n\"shrink every weight by some amount\"\n\nhas turned into something closer to:\n\n\"shrink weights according to how the optimizer's adaptive statistics happen to scale their gradients.\"\n\nThat is not the same operation.\n\nIn 2017, Ilya Loshchilov and Frank Hutter proposed a simple fix.\n\nDon't put weight decay inside the gradient.\n\nDo it separately.\n\nInstead of conceptually doing:\n\n```\ng <- g + lambda * theta\nAdam(g)\n```\n\nAdamW does:\n\n```\nAdam(g)\n\ntheta <- theta - alpha * lambda * theta\n```\n\nor, equivalently:\n\n```\ntheta <- (1 - alpha * lambda) * theta\n```\n\nNow the optimization step and the shrinkage step are decoupled.\n\nThat is the entire conceptual breakthrough.\n\nIt sounds tiny.\n\nIt isn't.\n\nThis means the optimizer controls:\n\n```\nHow should the model move to reduce the loss?\n```\n\nwhile weight decay controls:\n\n```\nHow strongly should parameters be pulled toward zero?\n```\n\nThose are different jobs.\n\nAdamW keeps them separate.\n\nSuppose:\n\n```\ntheta = 2\nalpha = 0.001\nlambda = 0.1\n```\n\nThen AdamW's direct decay contribution is:\n\n```\nalpha * lambda * theta\n= 0.001 * 0.1 * 2\n= 0.0002\n```\n\nSo the weight gets multiplied by:\n\n```\n1 - 0.0001\n= 0.9999\n```\n\nper optimization step, ignoring the gradient update for illustration.\n\nAfter 10,000 steps, that multiplicative factor becomes approximately:\n\n```\n0.9999^10000 ≈ e^(-1) ≈ 0.368\n```\n\nSo repeated tiny decay can become very substantial.\n\nThis is a useful way to think about weight decay:\n\nIt is not a tiny penalty applied occasionally. It is a multiplicative force acting at every optimization step.\n\nAnd that is why seemingly boring hyperparameters like `weight_decay=0.1`\n\ncan have a large effect over a long training run.\n\nAt this point, the practical picture looks something like this:\n\n```\nforward pass\n     |\n     v\ncompute loss\n     |\n     v\nbackprop\n     |\n     v\ngradient g_t\n     |\n     +----> Adam exponential moving averages\n     |             |\n     |             v\n     |        m_t, v_t\n     |             |\n     |             v\n     |       adaptive update\n     |\n     +----> AdamW weight decay\n                   |\n                   v\n               parameters\n```\n\nThere are several consequences worth keeping in your head.\n\nAdam does not eliminate the need to tune the learning rate.\n\nThe optimizer normalizes gradients, but `alpha`\n\nstill determines the global scale of movement.\n\nA useful mental model is:\n\n```\nAdam decides:\n    \"How large should this parameter's step be relative to its gradient history?\"\n\nLearning rate decides:\n    \"How aggressive should the entire optimizer be?\"\n```\n\nThat is why learning-rate schedules remain central in LLM training.\n\nThe Transformer paper, for example, used a warmup followed by inverse-square-root decay rather than holding the learning rate constant.\n\n`beta1`\n\ncontrols gradient-memory timescale\nThe moving average\n\n```\nm_t = beta1*m_(t-1) + (1-beta1)*g_t\n```\n\nhas an effective memory on the order of roughly:\n\n```\n1 / (1 - beta1)\n```\n\nsteps.\n\nSo:\n\n```\nbeta1 = 0.9\n```\n\nmeans roughly a ten-step memory scale.\n\nThat is not an exact cutoff; it is an intuition for the EMA timescale.\n\nLikewise:\n\n```\nbeta2 = 0.999\n```\n\ncorresponds to a much longer memory:\n\n```\n~1000 steps\n```\n\nfor the second-moment estimate.\n\nThis is why changing beta values is not just changing some arbitrary constants.\n\nYou're changing the temporal horizon over which the optimizer interprets gradient behavior.\n\n`epsilon`\n\nis mostly a numerical stabilizer\nThe denominator is:\n\n```\nsqrt(v_hat) + epsilon\n```\n\nThe `epsilon`\n\nprevents division by something vanishingly small.\n\nIn many practical regimes, it is not the dominant behavioral hyperparameter.\n\nBut in low-gradient or low-precision regimes, its interaction with numerical scale can matter.\n\nA common misunderstanding is:\n\n\"Adam uses second-order information.\"\n\nNot really.\n\nIt tracks a **second moment of gradients**:\n\n```\nE[g^2]\n```\n\nbut it does not construct the Hessian:\n\n```\nH = d^2L/dtheta^2\n```\n\nand does not estimate the full curvature matrix.\n\nAdam is still a first-order optimizer.\n\nIts sophistication comes from using historical statistics of first-order information.\n\nIf you are debugging LLM training, AdamW is not an implementation detail.\n\nIt can directly influence:\n\n```\ntraining stability\nloss curves\nsample efficiency\ngeneralization\nmemory footprint\ndistributed-training architecture\nhyperparameter sensitivity\n```\n\nA few practical examples:\n\nYou might immediately suspect:\n\n```\nbad initialization\nbad normalization\nbad data\nexploding gradients\n```\n\nBut the optimizer configuration is also part of the system.\n\nA learning rate that is perfectly reasonable under one optimizer can behave differently under another.\n\nWeight decay becomes interesting.\n\nBecause AdamW separates optimization from regularization, you can reason about:\n\n```\nlearning rate\n```\n\nand\n\n```\nweight decay\n```\n\nas two separate control knobs.\n\nThat conceptual separation is much cleaner than treating \"L2 regularization\" as something buried inside the gradient.\n\nCheck the optimizer state.\n\nFor a 7B model:\n\n```\nAdam moments ≈ 56 GB in FP32\n```\n\nThat number alone can explain a lot of apparently mysterious infrastructure decisions.\n\nThis is an increasingly interesting research question.\n\nThe optimal AdamW weight decay is not necessarily a universal constant that you can blindly copy from a smaller model.\n\nRecent work has explicitly studied how the optimal weight decay changes with model size, dataset size, and training dynamics.\n\nIn other words, once you're operating at serious scale, \"just set AdamW to 0.1\" is more cargo cult than theory.\n\nThe cleanest mental model I know is this:\n\nA neural network is trying to optimize an absurdly high-dimensional function using noisy measurements.\n\nThe raw gradient says:\n\n```\n\"Here is what today's minibatch thinks you should do.\"\n```\n\nAdam says:\n\n```\n\"Fine. But I also remember what the gradients have been doing lately.\"\n```\n\nIt keeps track of:\n\n``` php\ndirection  -> m\nscale      -> v\n```\n\nand uses those statistics to construct an adaptive update.\n\nAdamW then says:\n\n```\n\"And separately, I want the parameters to decay.\"\n```\n\nThat separation turns out to matter.\n\nSo the evolution is roughly:\n\n```\nSGD\n  |\n  +-- momentum\n  |\n  +-- adaptive scaling\n        |\n        v\n      Adam\n        |\n        +-- decoupled weight decay\n              |\n              v\n            AdamW\n```\n\nAnd the reason this matters for LLMs is not that Adam is mathematically glamorous.\n\nIt is that **training billion-parameter models is fundamentally an optimization-and-systems problem**.\n\nThe model might contain 70 billion parameters, but every one of those parameters is being updated by a tiny piece of state maintained over the entire training trajectory.\n\nThat makes the optimizer part of the model's computational machinery.\n\nThe next time you see:\n\n```\noptimizer = AdamW(...)\n```\n\nyou are not looking at five lines of boilerplate.\n\nYou are looking at a compact algorithm that is simultaneously doing:\n\n```\nmomentum\nadaptive normalization\nbias correction\nparameter updates\nregularization\n```\n\nfor billions of variables, potentially millions of times.\n\nThat is a rather extraordinary amount of machinery hiding behind one constructor.\n\nWhen you train or fine-tune an LLM, how much attention do you actually pay to the optimizer compared with the model architecture and data?\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible", "canonical_source": "https://dev.to/shrsv/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible-4f3o", "published_at": "2026-08-30 18:26:00+00:00", "updated_at": "2026-08-30 18:52:55.878161+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "neural-networks", "ai-research", "ai-infrastructure"], "entities": ["Shrijith Venkatramana", "LiveReview", "Adam", "AdamW", "Diederik Kingma", "Jimmy Ba", "Geoffrey Hinton", "ICLR"], "alternates": {"html": "https://wpnews.pro/news/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible", "markdown": "https://wpnews.pro/news/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible.md", "text": "https://wpnews.pro/news/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible.txt", "jsonld": "https://wpnews.pro/news/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible.jsonld"}}