Dropout randomly sets part of an activation tensor to zero during training. It is a regularizer. The model cannot rely on every feature always being present, so it is pushed to learn less brittle representations.

Train and eval behavior

During training, each value is kept with probability $1-p$ and dropped with probability $p$. The kept values are divided by $1-p$:

where $m$ is a random mask with values $0$ or $1$.

The scaling keeps the expected value the same. For one activation $x$:

That is why in evaluation dropout does nothing but in training samples a mask and rescales the kept activations.

Where to put dropout

Dropout should usually come after the activation because some activations turn an input of zero into a nonzero output. For example, $\operatorname{sigmoid}(0)=0.5$, so masking a value before sigmoid does not actually remove it. Masking after the activation always makes the dropped feature zero. With ReLU the two placements can be equivalent because $\operatorname{ReLU}(0)=0$.

Do not dropout the logits

Applying dropout after the final logits usually does not make sense. Logits are the direct inputs to the loss. If we randomly zero them, we are corrupting the model's answer right before the objective reads it.

References

• Analysis of Dropout