Learning Rate Annealing

Few takes on lr annealing:

Near a minimum, the true gradient becomes small, but minibatch noise remains. Updates are then influenced more by differences between batches than by a consistent downhill direction. Lowering the learning rate makes these noisy movements smaller, allowing them to average out over time and helping the model settle more precisely.

This is primarily a consequence of stochastic optimization. With exact full-dataset gradients, a fixed learning rate can converge to a minimum as long as it is stable for the local curvature. With minibatches, however, gradient noise creates a floor below which optimization cannot settle easily at a fixed learning rate.

Increasing the batch size can have a related effect because it produces a less noisy gradient estimate. Learning-rate decay instead reduces the distance that each noisy estimate can move the parameters. The two approaches are related, but not identical.