Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems.
Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Most people learn neural networks by staring at the model.
Weights.
Attention.
MLPs.
LayerNorm.
Tokenizers.
Context windows.
But when you actually train an LLM, there is another piece of machinery making billions of decisions every second: the optimizer.
A 70-billion-parameter model does not "learn" because gradient descent tells it which direction is better.
It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates.
For the last decade, the dominant answer has largely been some form of Adam, and increasingly AdamW.
The interesting part is that Adam is not some mysterious LLM-specific invention.
The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era.
Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto.
Three years later, the Transformer paper used Adam directly in its training recipe.
Then came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization.
By 2025, Adam was sufficiently influential to receive an ICLR Test of Time award.
So what exactly is Adam doing?
And why is AdamW usually what you actually want when training a Transformer?
1.
First, forget Adam: what problem is the optimizer solving?
Suppose your neural network has parameters and your training batch produces a loss .
Backpropagation gives you The simplest possible optimizer is gradient descent: where is the learning rate.
That looks almost embarrassingly simple.
And that is indeed roughly what people did before adaptive optimizers became dominant.
The problem is that the gradients of a neural network are not nicely behaved.
Imagine two parameters: A single learning rate has to deal with both.
If you choose , parameter 1 barely moves: while parameter 2 gets: And this situation is not exotic.
Different parameters can have wildly different gradient scales.
Some receive dense gradients every step.
Others receive sparse or intermittent signals.
Some directions in parameter space are noisy.
Others are remarkably consistent.
So the fundamental problem is: How do we turn a raw gradient into a sensible update for each individual parameter?
Momentum gives one answer.
Adaptive methods give another.
Adam essentially combines both.
2.
Adam's key idea: keep a memory of the gradient Adam stands for Adaptive Moment Estimation.
The easiest way to understand it is to imagine that every parameter maintains two small pieces of memory.
The first remembers: "What direction have gradients generally been pointing?" The second remembers: "How large have those gradients generally been?" For every parameter, Adam maintains: More precisely: Typically: The interpretation is surprisingly intuitive. : momentum Suppose gradients over five steps are: Then the moving average also points strongly positive.
Now imagine: The signs keep cancelling.
Adam therefore distinguishes: from This is momentum. : gradient scale Now suppose a parameter frequently gets gradients around , while another gets gradients around .
Their squared gradients differ by a factor of: Adam remembers this.
That allows it to normalize the effective update.
Ignoring some details for a moment, the update looks like: So if a parameter has persistently large gradients, its denominator is large.
If its gradients are consistently tiny, its denominator is small.
Adam is therefore doing something qualitatively like: move in the direction supported by rece