Learning Rate Warmup

Few takes on lr warmup:

At initialization weights and learned representations are mostly random. Even similar examples may activate the network differently. Their parameter gradients may therefore be poorly aligned. A minibatch gradient can be strongly influenced by which examples happened to be sampled. A large update may move too far in that noisy direction of a batch.

Without warmup, a large learning rate can cause weight updates to overshoot the local minima entirely. This shoots the weights into high loss zones, causing numerical overflow also called gradient explosion.

Adaptive optimizers like Adam use a running average of past gradients to scale updates. During the very first steps, these running averages are uninitialized or highly biased. Warmup restricts step sizes until the optimizer has collected enough gradient history to calculate stable, reliable update directions

When the data distribution is highly diverse, a small minibatch represents only a narrow subset of the full distribution. Its gradient may therefore reflect the needs of that subset rather than the best direction for the overall dataset. With a very large learning rate, the model makes a large update based on this incomplete information, effectively overfitting temporarily to the current minibatch. Subsequent batches may demand corrections in different directions, causing oscillation or instability, so smaller initial updates help prevent the model from overreacting to noisy minibatch gradients.

If the dataset is ordered by domain rather than shuffled, learning-rate warmup can unintentionally give different domains different influence. Samples from the first domain are processed while the learning rate is very small, so their gradients produce much smaller parameter updates than gradients from domains encountered later. Consequently, the model may initially learn much less from the first domain, especially in one-pass or short training. This does not mean it gains no knowledge at all, and the problem may disappear if later epochs revisit that domain at the full learning rate; nevertheless, shuffling or balanced sampling is important to avoid this position-dependent bias.

Warmup has an opportunity cost. Under a fixed training budget, extending it reduces the number of steps performed at the target learning rate. Once the model’s activations, gradients, and optimizer statistics are sufficiently stable, additional warmup may slow optimization without providing a corresponding benefit. In practice, the goal is therefore to find the shortest warmup that reaches the target learning rate without degrading stability or final model quality.