Weight initialization sets the starting values of the trainable parameters. Those values affect activation scale, gradient scale, and whether different neurons learn different features.

Why initialization matters

Initialization decides the model's starting signal scale. If weights are too small, activations and gradients shrink as they move through layers. If weights are too large, they can grow, destabilize updates, or push activations into saturated regions.

Zero initialization

For a linear layer $Y = XW$, the backpropagation formulas are $\frac{\partial L}{\partial X}=GW^\mathsf{T}$ and $\frac{\partial L}{\partial W}=X^\mathsf{T}G$, where $G=\frac{\partial L}{\partial Y}$.

In a stack of linear layers without bias, if every $W$ is initialized to zero, every hidden activation $X$ becomes zero. The output gradient $G$ can be nonzero, but $X^\mathsf{T}G$ is still zero, so all weight matrices receive zero gradient.

If biases exist, only the last bias can receive a nonzero gradient at first. Its gradient comes directly from the output gradient $G$. But after backpropagation multiplies by a zero weight matrix, $\frac{\partial L}{\partial X}$ becomes zero, so biases in earlier layers also receive zero gradient. The activation between two linear layers does not change this. Once its upstream gradient is zero, multiplying by the activation derivative still gives zero, so the layers before it remain blocked.

If only the last layer has zero weights while earlier layers are nonzero, the last layer can still learn. Its input $X$ can be nonzero, so $X^\mathsf{T}G$ can update its weights, and $G$ can update its bias. But the gradient sent to earlier layers is $GW^\mathsf{T}=0$, so all layers before it receive zero gradient for both weights and biases.

Constant nonzero initialization

Creates a symmetry problem. Every neuron in a layer receives identical inputs and computes the exact same output, causing all neurons to update identically during backpropagation and rendering multiple neurons redundant.

Uniform random initialization

Uniform random initialization samples each weight independently from some interval $[l,r]$. This breaks symmetry, so neurons in the same layer no longer receive identical updates.

A general interval can have nonzero mean, which shifts activations in one direction before training starts. For initialization, it is usually cleaner to use a symmetric interval $[-a,a]$, whose mean is zero.

Randomness alone is not enough. If $a$ is too small, activations and gradients can shrink across layers. If $a$ is too large, they can grow or push nonlinearities into saturated regions. The useful question is therefore not only whether weights are random, but what scale the random distribution should have.

For a non-symmetric uniform distribution $W\sim U(l,r)$, the variance is

For the symmetric case $W\sim U(-a,a)$, we substitute $l=-a$ and $r=a$

So for a symmetric uniform initialization, choose the variance $v$ first, then convert it into the bound $a=\sqrt{3v}$.

The usual formulas come from trying to keep variance stable across layers. Let $fan_{in}$ be the number of inputs to one neuron. For that neuron, $x_i$ is one input value, $w_i$ is the weight attached to that input, and $y$ is the value before the activation function:

So the neuron adds $fan_{in}$ terms. The first term is $x_1w_1$, the second is $x_2w_2$, and so on. If these terms are not strongly correlated, the variance of the sum is roughly the sum of their variances:

For centered inputs and weights, each term $x_iw_i$ has variance $\operatorname{Var}(X)\operatorname{Var}(W)$. Since there are $fan_{in}$ such terms,

To keep the activation scale from growing or shrinking, we want $\operatorname{Var}(y)\approx\operatorname{Var}(X)$. That gives $\operatorname{Var}(W)\approx 1/fan_{in}$.

The backward pass gives a similar constraint in the other direction. Let $fan_{out}$ be the number of outputs from the layer. From $\frac{\partial L}{\partial X}=GW^\mathsf{T}$, the gradient for one input is also a sum:

Using the same variance approximation, this gives

To keep gradient scale from growing or shrinking as it moves backward, we want $\operatorname{Var}(\frac{\partial L}{\partial X})\approx\operatorname{Var}(G)$. That gives $\operatorname{Var}(W)\approx 1/fan_{out}$.

Xavier initialization balances the forward constraint $1/fan_{in}$ and the backward constraint $1/fan_{out}$ by using their average scale.

Xavier uniform assumes the activation does not strongly change the variance after the linear layer. It chooses

so the symmetric uniform bound is

For ReLU-like activations, if pre-activations are roughly centered around zero at the start of training, about half of them are negative and get set to zero. That reduces the signal variance, so He initialization uses roughly twice the Xavier forward variance.

He uniform usually uses the forward-preserving choice

so the symmetric uniform bound is

A backward-preserving variant uses

and therefore

Normal random initialization

Normal initialization uses the same variance targets, but samples from a normal distribution instead of a uniform interval:

Here $v$ is the variance. For Xavier normal, use $v=2/(fan_{in}+fan_{out})$. For He normal in the usual forward-preserving mode, use $v=2/fan_{in}$.

Normal and uniform initialization mostly differ in how they sample weights with the same target variance. Uniform initialization is bounded, so it cannot produce rare extreme weights. Normal initialization puts more weights close to zero, but has tails unless it is truncated. In many basic Xavier and He settings, the target variance matters more than whether the samples come from a uniform or normal distribution.

Truncated normal initialization keeps the normal shape near zero, but resamples values that are too far into the tails. This avoids rare large initial weights while still putting most weights near zero.

Bias initialization

Biases are often initialized to zero. This is usually fine because random weights already break symmetry between neurons. Zero bias does not make all neurons compute the same function by itself.