Gradient Clipping

Large gradient norms are ambiguous. They can be real signal, or they can be a symptom of a bad batch (batch contains outliers or labels are wrong), loss spike, or numerical instability. The norm alone does not tell us which case we are in.

Clipping accepts the tradeoff. Sometimes it shrinks a useful update, but it can prevent a rare bad update from damaging the run.

If the clipping threshold is too small, useful gradients get scaled down too often, which slows training. If the threshold is too large, clipping almost never activates, so large gradient spikes can still make training unstable.

Gradient norm clipping preserves the gradient direction and only caps its magnitude. If the norm is below the threshold, nothing changes. If it is above the threshold, the whole gradient is scaled down.

Layer normalization alone is not enough. For example, let $y = W\cdot\mathrm{LN}(x)$. The weight gradient is $\nabla W = \nabla y\cdot\mathrm{LN}(x)$. LayerNorm controls $\mathrm{LN}(x)$, but it does not control $\nabla y$. If $\nabla y$ is large, then $\nabla W$ can still be large.