At the start of training, a model often learns simple rules first. Those simple rules can make the loss go down quickly because they cover a large part of the input space, in the same spirit as the curse of dimensionality. The model can get a large gain by learning patterns that apply across many regions of the hypercube. Some patterns are common across many examples, so the model effectively sees them thousands or millions of times in different forms. Grammar, common facts, formatting, and basic reasoning patterns can fall into this category.

The remaining cases are harder. A model may learn something like the first $70\%$ of the useful simple rules quickly, then spend much more time on the last $30\%$ of harder rules. The remaining errors often come from rare, difficult, or highly specific rules. Those appear in fewer examples, so the model gets fewer chances to learn them. Learning rare cases needs more optimization steps because each individual example only nudges the model a little. Loss curves we want to minimize can therefore look like a hockey stick.

There is also a geometric reason training slows down. Early in training, the model is far from a good minimum, so gradients are often large and point in more consistent directions. Near a minimum, gradients are smaller and different batches can point in more conflicting directions. The signal-to-noise ratio of the gradient decreases, so each optimizer step becomes less useful.

Saturation

One useful way to analyze training progress is to plot the loss improvement from each doubling of training steps. If the improvement per doubling keeps shrinking near the end of training, the marginal return from additional compute is getting smaller. This helps decide whether training for twice as many steps is worth it, or whether it is better to change something more fundamental, such as the model architecture, data mixture, optimizer settings, or training objective.

Debugging loss shapes

If the loss plateaus too early, one possible reason is bad or unlucky data ordering, especially in mini-batch training. If early batches are biased, non-representative, or sorted by class, difficulty, or domain, the model can initially move toward a poor region of parameter space. Later samples may then not be enough to move it out easily.