This article focuses on how an optimizer converts an already computed gradient into a parameter update. It does not discuss whether that gradient was estimated using the full dataset, one sample, or a mini-batch. The optimizer is responsible for updating the trainable parameters of the model.

SGD

Stochastic Gradient Descent moves each parameter in the direction opposite to its current gradient:

The negative gradient is the direction of the steepest local decrease for a sufficiently small step. The learning rate controls the distance moved. A rate that is too small wastes computation, while one that is too large can overshoot useful regions or make training diverge. Different parameters can have gradients with very different magnitudes, but SGD multiplies all of them by the same learning rate. More precisely, the loss can have steep and flat directions that combine changes to many parameters. Using one learning rate in every direction can cause SGD to oscillate across high-curvature directions while progressing slowly along low-curvature directions. Plain SGD stores no per-parameter optimizer state.

SGD with momentum

Momentum maintains an accumulated gradient that combines the current gradient with previous gradients:

If gradients repeatedly point in a similar direction, their contributions accumulate and the optimizer moves faster in that direction. If a gradient alternates across a narrow valley, positive and negative contributions partially cancel, reducing oscillation. The momentum coefficient $\beta$ is commonly close to $0.9$. Momentum stores one additional value per parameter.

NAG

Nesterov Accelerated Gradient (NAG) first moves to the point where the previous momentum is expected to take the parameters:

It then updates the momentum with the gradient evaluated at that look-ahead point and applies the resulting direction:

The difference from ordinary momentum is which gradient is added when computing $m_t$. Ordinary momentum adds $g_t$, evaluated at the current parameters $\theta_{t-1}$. NAG adds $\widetilde g_t$, evaluated at the future point where the previous momentum would move the parameters. This lets the gradient correct the direction before the complete momentum step is applied and can reduce overshooting. Its importance for deep-network optimization was examined by Sutskever et al. Like ordinary momentum, NAG stores one additional state value per parameter for the momentum buffer:

AdaGrad

AdaGrad gives every parameter its own effective learning rate. It accumulates the squared gradients:

A parameter that repeatedly receives large gradients builds a large denominator and takes smaller steps. A parameter with rare or small gradients retains larger effective steps, which makes AdaGrad useful for sparse features. However, $r_t$ only increases, so the effective learning rates continually decrease and can become too small during a long training run. AdaGrad stores one additional state value per parameter for the accumulated squared gradients.

RMSProp

RMSProp replaces AdaGrad's cumulative sum with an exponential moving average of squared gradients:

Old squared gradients gradually decay instead of remaining in the denominator forever. Each parameter is still scaled according to the recent magnitude of its gradients, but its effective learning rate no longer has to shrink monotonically. RMSProp was presented in Geoffrey Hinton's neural-network lecture notes. It stores one additional state value per parameter for the moving average of squared gradients.

Adam

Adam combines momentum with RMSProp-style scaling. It tracks an exponential moving average of gradients, called the first moment, and an exponential moving average of squared gradients, called the second raw moment:

Both averages start at zero and are biased toward zero during the first steps. Adam corrects this initialization bias:

It then uses the first moment as a smoothed direction and the second moment to scale each coordinate:

The original defaults are $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=10^{-8}$. Adam stores two state values per parameter, twice as many as SGD with momentum.

AdamW

Weight decay directly shrinks the parameters. With decay coefficient $\lambda$, a decoupled AdamW update is:

In common implementations, the default decay coefficient is $\lambda=0.01$ although it should be tuned for the model and training setup. Weight decay discourages unnecessarily large weights by continuously pulling them toward zero. Gradients can still push useful weights away from zero, producing a balance between learning and decay. AdamW stores the same two state values per parameter as Adam. Decoupled weight decay does not require another state value.

Muon

Some good posts:

• https://jeremybernste.in/writing/deriving-muon
• https://kellerjordan.github.io/posts/muon

Muon stores one additional state value per optimized parameter element in its momentum buffer. The Newton-Schulz iterations use temporary working memory during each update but do not add another persistent state value.

Miscellaneous

Different parameter groups can use different optimizers. For example, the weights of linear layers could be updated with Adam while the remaining parameters use SGD.

Different optimizers can also be used at different stages of training. For example, training can begin with SGD, which requires less optimizer-state memory than Adam, to reduce GPU memory usage. It can later switch to Adam when adaptive per-parameter updates become more valuable.