Neural network architectures are built from components that transform representations and control how information flows through the model. This article collects short explanations of these components, starting with dropout.

Dropout

Dropout randomly sets part of an activation tensor to zero during training. It is a regularizer. The model cannot rely on every feature always being present, so it is pushed to learn less brittle representations.

During training, each value is kept with probability $1-p$ and dropped with probability $p$. The kept values are divided by $1-p$:

where $m$ is a random mask with values $0$ or $1$.

The scaling keeps the expected value the same. For one activation $x$:

That is why in evaluation dropout does nothing but in training samples a mask and rescales the kept activations.

In distributed training, replicated activations that must remain identical need the same dropout mask on every replica. Different shards of one activation instead use independent mask entries, so together they form one independent mask for the full tensor.

Dropout should usually come after the activation because some activations turn an input of zero into a nonzero output. For example, $\operatorname{sigmoid}(0)=0.5$, so masking a value before sigmoid does not actually remove it. Masking after the activation always makes the dropped feature zero. With ReLU the two placements can be equivalent because $\operatorname{ReLU}(0)=0$.

Applying dropout after the final logits usually does not make sense. Logits are the direct inputs to the loss. If we randomly zero them, we are corrupting the model's answer right before the objective reads it.

• Analysis of Dropout
• Improving neural networks by preventing co-adaptation of feature detectors, the original Dropout paper.

Residual connection

A residual block is a group of layers with a shortcut connection that adds the original input to the layers' output. If the layers compute $F(x)$, the block returns $F(x)+x$.

During the backward pass, the gradient can flow through both paths. Let $y=F(x)+x$ and let $J_F(x)$ be the Jacobian of $F$. Then

The first term is the gradient through $F$, and the second is the gradient through the identity shortcut.

Suppose the desired output for input $x$ is the underlying mapping $H(x)$. Instead of asking the stacked layers to learn $H(x)$ directly, a residual block lets them learn a residual function $F(x)$ relative to the input. The original mapping is then obtained by adding the input back:

The idea is that optimizing the residual mapping can be easier than optimizing the original, unreferenced mapping. If the identity mapping is already a good solution, so that $H(x)=x$, the stacked layers only need to push $F(x)$ toward zero. This can be easier than learning the identity function from scratch through multiple nonlinear layers. The Deep Residual Learning paper introduced this approach.

Multi-head attention (MHA)

MHA gives every attention head its own query, key, and value projections. The heads can learn different relationships, then their outputs are concatenated and mixed. This is the attention mechanism introduced in Attention Is All You Need.

Grouped-query attention (GQA)

GQA keeps many query heads but groups them so that several query heads share one key and value head. It reduces key/value cache memory and bandwidth while retaining more query-head diversity than MQA. See GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

Multi-query attention (MQA)

MQA gives every query head its own projection but shares a single key and value head across all queries. This greatly reduces key/value cache size and decoding bandwidth, at the cost of less independent key/value representations. It was introduced in Fast Transformer Decoding: One Write-Head is All You Need.

Mixture of experts (MoE)

An MoE layer contains several expert networks and a router. For each token, the router selects only a small number of experts, such as the top one or two, and the layer combines their outputs.

This lets the model increase its parameter count without running every parameter for every token. The main challenges are routing tokens evenly across experts and communicating tokens to the workers that own the selected experts.

MoE also creates a smaller effective batch for each expert because every token is sent only to a subset of experts. Each expert therefore performs smaller matrix multiplications, which can be less efficient. A larger global batch is often needed to provide enough tokens to each expert and keep the same computations well saturated.

Some MoE layers also include a shared expert that processes every token alongside the selected routed experts. The shared expert can learn common features, while the routed experts specialize. For example, Qwen3.5-35B-A3B has 256 routed experts, selects 8 of them per token, and adds 1 shared expert. Thus 257 expert modules exist in total if the shared expert is counted, while 9 expert branches are active for each token.

Activation functions

Activation functions add nonlinearity to neural networks. Without them, stacking linear layers would still produce only one linear transformation.