One way to think about learning rate is as a navigation parameter. It decides how far the optimizer moves across the loss landscape after each gradient estimate.

Mental model

• Huge learning rate. The step size is larger than the basin you are in. The optimizer can jump over good minima, bounce around, and never settle. Loss may oscillate or diverge.
• Low learning rate. The optimizer can get trapped in the first basin it falls into. That basin might be a bad, sharp minimum that generalizes poorly. The step size may not give the optimizer enough energy to climb out and find a better, flatter basin.
• Learning rate just right. The step size is high enough to escape bad basins, but low enough not to overshoot good ones.

By training stage

• Early training. The model is close to random initialization and surrounded by many possible basins. A higher learning rate helps it explore broadly, skip past sharp local minima, and find the region where better flatter basins live. At this stage, training is partly about choosing which valley to descend into.
• Mid training. The optimizer has found a useful region. The goal is to narrow down to a good basin inside that region. Learning rate is still meaningful, but it should usually be lower because the search is more local.
• Late training. The optimizer is already inside a basin. The goal is to walk toward the bottom with smaller, more precise steps. A high learning rate here can bounce the model out of the basin that earlier compute was spent finding.

Catastrophic forgetting

A basin is a valley in the loss landscape. During pretraining, a larger learning rate is useful because the model starts from random initialization and needs broad exploration. If it gets stuck in a bad local minimum too early, training can waste a lot of compute in the wrong region.

A pretrained model already sits in a good region of the loss landscape. More precisely, it sits in a wide basin that gives low loss across diverse pretraining data. That basin is flat relative to the pretraining objective. Gradients for next-token prediction are small because, from the pretraining objective's point of view, there may be little left to improve nearby.

Fine-tuning changes the objective. The same parameter point can look flat through the pretraining objective, but sloped through the fine-tuning objective. The general basin is not featureless. It contains subregions that are better for specific tasks. When we switch loss functions, we put on different glasses. Through the old glasses, the model may still look like it is in a flat valley. Through the new glasses, the same location can have new slopes, hills, and task-specific directions.

Catastrophic forgetting happens when fine-tuning follows the new objective so aggressively that it leaves the old good basin. A point that is low for the new task can be high for the pretraining objective. In practical terms, the model improved on the fine-tuning loss but forgot capabilities that depended on the old representations.

The ideal fine-tuning endpoint is good through both glasses. It should improve the new task while staying inside the pretrained basin. Learning rate controls how strongly we respect that constraint. A high learning rate can jump out of the basin and overwrite useful representations. A very low learning rate may fail to adapt to the new task. The useful range is often narrow, which is why learning rate is one of the most important fine-tuning hyperparameters to sweep.

Low learning rate does not make forgetting impossible. Given enough steps, even small updates can walk the model out of the pretrained basin. The protection is learning rate plus early stopping. A lower learning rate buys time to notice, through evaluation, that general pretraining quality is degrading before the model has moved too far. A high learning rate can overshoot before evaluation has a chance to catch it.

Sometimes forgetting matters less. If the target task is narrow and self-contained, the old capabilities may not matter much. But if the new task still depends on features learned during pretraining, destroying those features can hurt the new task too, even if the fine-tuning loss looked good for a while.