Normalization layers control activation scale. They make training less sensitive to initialization, learning rate, and changes in activation magnitude across layers.

Imagine each layer receives a vector of activations. During training, earlier layers keep changing, so the next layer sees inputs whose mean and variance drift around. That makes optimization harder: the layer is trying to learn while its input coordinate system keeps moving.

What normalization preserves

Normalization does not make every activation the same. It shifts and rescales activations into a standard coordinate frame.

For example, if an activation is a vector $x=[x_1,x_2]$, LayerNorm first subtracts the mean of its coordinates, then divides by their standard deviation. The absolute location and overall size change, but the relative pattern remains: which feature is higher, which is lower, and by how much relative to the others.

This means vectors such as $[10,12]$, $[100,102]$, and $[1000,1002]$ become almost the same after normalization because they have the same internal pattern. But $[10,12]$ and $[12,10]$ remain different because the high and low features are swapped.

So normalization changes the question from «where is this activation in absolute space?» to «what is the pattern of highs and lows inside this activation?» Learnable scale and shift parameters can then put the normalized activation into whatever scale and offset are useful for the model.

Why learn a scale and shift if the next weights could adapt to standardized inputs? The problem is that fixed normalization forces the next operation to always receive zero-mean, unit-scale inputs, even if a different scale or mean would work better. Maybe a standard deviation of $0.2$ is better than $1$, or maybe a positive mean is useful before a mean-sensitive activation such as sigmoid. For example, inputs with mean $1000$ make sigmoid output almost $1$, while inputs with mean $-1000$ make it output almost $0$. If the next operation already has a bias, the normalization shift can be redundant because two consecutive shifts collapse into one. But when normalization is followed by an activation or a bias-free operation, the learned shift and scale give the model back control over the distribution it wants to use.

Batch norm

Batch normalization normalizes activations using statistics computed across the batch. In image models this works well because examples usually have the same shape. A batch of images has a regular tensor layout, so each channel statistic is estimated from many comparable positions across many examples.

This is a poor fit for language models. Text sequences have variable lengths, and token positions are not equally populated. Imagine a batch with one million examples where most sequences have length $1$, but one sequence has length $1000$. The first token position has about one million examples contributing to its batch statistic. Positions $2$ through $1000$ have only the long sequence contributing. Those later positions get a terrible estimate because their statistic is effectively computed from one example.

Layer norm

Layer normalization normalizes within each individual example, usually across the hidden dimension. It does not need other examples in the batch, so it works with variable sequence lengths and batch sizes.

RMS norm

RMSNorm is a simpler variant of LayerNorm. Instead of subtracting the mean and dividing by the standard deviation, it normalizes by the root mean square of the hidden activations. It controls scale but does not recenter the activations.

This makes RMSNorm cheaper. Mean subtraction has low arithmetic intensity: it moves memory but does little math, so it is often memory-bound and does not use tensor cores well. Removing it can improve hardware utilization, especially in transformer blocks where small memory-bound operations can become visible overhead.

Pre-norm and post-norm

In transformer blocks, normalization can be placed before or after the main sublayer. Pre-norm means the attention or MLP sees normalized input:

Post-norm means the normalization is applied after the residual update:

Pre-norm is usually easier to train in deep transformers because the residual path stays close to an identity path. Gradients can move backward through the residual connection without always passing through the normalization operation.

Post-norm can make the block output scale cleaner, but it is often less stable at depth. For modern LLMs, pre-norm or variants of it are the common choice.