Parallelisms

August 2026 – Vladislav Kruglikov

Synchronous parallelism is limited by stragglers. A slow worker or pipeline stage delays the others at the next collective or dependency, so the slowest participant often determines step time. Workloads should therefore be balanced by measured runtime, not only by operation or layer count.

Data parallel
Tensor parallel
Sequence parallelism
Context parallelism
Expert parallelism
Pipeline parallelism
ZeRO-1
ZeRO-2
ZeRO-3
FSDP

Critical path and latency hiding

The critical path is the chain of dependencies that determines the earliest possible completion time. Delaying any operation on it directly delays the whole step.

In TP, local computation → all-reduce → next computation, so the next operation waits for communication. In DP, gradient communication ∥ backward computation for earlier layers. If communication finishes before its result is needed, its latency is hidden; otherwise, the uncovered remainder joins the critical path.

Overlap requires asynchronous communication, independent GPU work, separate execution resources, safe buffers, and synchronization only when the result is needed. Asynchronous launch alone does not prove overlap because communication may contend with computation or merely move the wait later; verify it with a profiler.

Data parallel

In data parallelism, every worker holds a copy of the model and processes a different shard of the batch. The forward pass needs no communication for sample-independent layers. During backward, workers compute local gradients. Only gradients of model weights, meaning all trainable parameters, need to be all-reduced. Activations and their gradients remain local. This happens before the optimizer step so every model replica receives the same update. Once a bucket of parameter gradients for later layers is ready, DDP starts its asynchronous all-reduce while the GPU uses activation gradients to continue backward through earlier layers. The effective batch size is the sum of all local batch sizes.

Gradient bucketing

Gradient bucketing, also called gradient bucketization, collects parameter gradients into buffers and communicates each buffer only when it is ready. The broader idea is also called tensor or gradient fusion, message aggregation, collective coalescing, or communication batching.

The Hockney communication model, also called the $\alpha$ - $\beta$ model, separates fixed collective latency from transfer time:

T(m)=\alpha+\beta m

Here, $\alpha$ is the fixed latency per collective, $m$ is the message size, and $\beta$ is approximately the inverse bandwidth. For $K$ small collectives containing $M$ total bytes,

T_{\text{small}}\approx K\alpha+\beta M

Fusing them into one collective gives

T_{\text{fused}}\approx \alpha+\beta M

A larger bucket means fewer collectives and better bandwidth utilization, but uses more temporary memory and starts communication later, which can reduce overlap.
A smaller bucket starts communication earlier and can improve overlap, but pays more collective-latency overhead.

Gradient bucketing therefore amortizes collective latency while preserving as much communication-computation overlap as possible.

Let $G$ be the total byte size of all parameter gradients and $N$ the number of workers. Forward communication is $0$ bytes. Backward all-reduces a $G$ -byte buffer. With ring all-reduce, each worker sends $2(N-1)G/N$ bytes and receives the same amount.

Regular BatchNorm uses statistics from each local shard and requires no forward communication, but this changes its semantics. Synchronized BatchNorm preserves global-batch statistics by communicating during the forward pass and also requires communication during backward.

Tensor parallel

Tensor parallelism splits each weight matrix across workers. Every worker processes the same batch but computes a different part of each layer. Unlike data parallelism, communication moves activations and their gradients between workers, while each worker keeps and updates only its local parameter shard.

Linear layer

For $Y=XW$ , split $W$ across output columns as $W=[W_1,\ldots,W_N]$ . Worker $i$ computes $Y_i=XW_i$ , so forward needs no communication while the output remains sharded. If a later operation needs the full output, workers all-gather the output shards; the weights remain sharded. In backward, workers all-reduce their partial input gradients, while each computes only its local weight gradient.

For row parallelism, split $W$ across input rows and $X$ across the same input features. Worker $i$ computes a partial output $Y_i=X_iW_i$ , so forward all-reduces the partial outputs to obtain the full $Y$ . Backward needs no communication while the input gradient remains sharded. If an earlier operation needs the full input gradient, workers all-gather its shards; the weights remain sharded.

MLP

For $\operatorname{MLP}(X)=\operatorname{Down}(\operatorname{act}(\operatorname{Up}(X)))$ , make the up projection column-parallel. Each worker computes one shard of the expanded activation and applies the elementwise activation locally. Make the down projection row-parallel with matching shards, then all-reduce its partial outputs.

This column-to-row layout keeps the intermediate activation sharded and avoids an extra all-gather between the two linear layers. Forward needs only the final all-reduce; backward similarly needs one all-reduce for the input gradient of the up projection.

MHA

Make the QKV projections column-parallel across attention heads, so each worker owns a subset of heads. Attention heads are independent, so each worker can run FlashAttention for its local heads without all-gathering Q, K, V, or the head outputs.

Make the output projection row-parallel over the same head shards. Each worker multiplies its local attention output by the matching weight shard, then workers all-reduce the partial outputs. This avoids an intermediate all-gather and leaves only the final all-reduce in forward.

Embedding and output-vocabulary matrices can also be vocabulary-parallel by splitting the vocabulary dimension across workers. Embedding lookup all-reduces the locally owned lookup results. The output head keeps logits sharded and computes cross-entropy with small reductions instead of all-gathering the full vocabulary logits.

Tensor-parallel forward usually has two all-reduces per transformer block. One follows attention, and one follows the MLP. Each is a hard dependency. The next operation cannot start until communication finishes, and there is usually no independent forward computation to hide it behind. This exposed waiting time accumulates across all layers, so TP needs a fast interconnect such as NVLink or NVSwitch.

Let $A$ be the byte size of one block's residual activation and $N$ the number of TP workers. With two ring all-reduces per block, each worker sends $4(N-1)A/N$ bytes in forward and receives the same amount. Backward has two more all-reduces with the same volume. These costs repeat for every block and microbatch.

Doubling the tensor-parallel degree doubles aggregate peak FLOP/s and ideally halves compute time, but the model's total FLOPs stay the same. Roughly,

T_{\text{step}}\approx \frac{T_{\text{compute}}}{\mathrm{TP}}+T_{\text{comm}}(\mathrm{TP})

Therefore, TP8 is not automatically twice as fast as TP4.

Moving from TP4 to TP8 can even reduce throughput. If the local matrix multiplications are already small, splitting them again lowers hardware utilization while collective communication volume and latency increase.

Sequence parallelism

Sequence parallelism is paired with tensor parallelism. It splits intermediate activations along the sequence dimension. Any tokenwise operation can theoretically use sequence sharding. QKV can be sharded across both sequence and feature dimensions, but this requires a 2D device grid with separate sequence- and tensor-parallel axes. Giving each worker one token shard and one weight shard leaves the cross-products missing. Because SP shards tokens rather than samples, its memory saving still applies at microbatch size 1 and can make such a workload fit when these activations are the bottleneck. Inputs to operations outside the TP region, such as LayerNorm, dropout, and residual additions, stay sharded because each token can be processed independently. After a row-parallel layer, reduce-scatter replaces the all-reduce. When the next TP region needs the full sequence, an all-gather unshards it.

The reduce-scatter plus all-gather moves the same number of bytes as the all-reduce it replaces, but activation memory for the sharded region falls by the TP degree. Unlike context parallelism, Megatron-style sequence parallelism does not shard attention across the sequence, so it does not solve long-context attention memory.

Ulysses sequence parallelism

Ulysses keeps the sequence sharded through QKV projection. Before attention, an all-to-all gives each worker all tokens but only for a subset of heads, so no worker materializes all tokens for all heads. Since attention heads can be computed independently, each worker computes its local heads, then another all-to-all restores sequence sharding. Its degree is therefore limited by the number and divisibility of attention heads.

Context parallelism

Ulysses can scale only down to one attention head per worker. That worker still holds the full sequence of Q/K/V for the head. If even this does not fit, context parallelism must also split a single head's attention over tokens.

Each worker keeps queries for its local token block and rotates K/V blocks through a ring. It computes attention one block at a time and combines the partial softmax results exactly, so no worker needs the full sequence of K/V or the full attention matrix. K/V transfer can overlap with blockwise attention computation.

See this detailed analysis, the original sequence-parallelism paper, and the Ring Attention paper.

Expert parallelism

Expert parallelism distributes a mixture-of-experts layer's expert MLPs across workers. A router selects the top- $k$ experts for each token. An all-to-all sends token activations to the workers owning those experts, each worker computes its local experts, and another all-to-all returns the outputs for combination. Backward reverses the same communication; expert weights and their gradients stay with their owners.

Concatenating all experts into one dense tensor-parallel matrix would require masking unselected experts with zeros, but a dense matrix multiplication still computes those zeros and loses MoE's sparse-compute benefit. Expert parallelism instead shards at expert boundaries and runs only the selected experts. Tensor-parallelizing a selected expert avoids wasted compute but adds collectives, so it is mainly useful when one expert does not fit on a worker.

Uneven routing creates stragglers and memory hotspots, so MoE training needs load balancing or expert-capacity limits. Too few tokens per local expert also creates small, inefficient matrix multiplications, so implementations commonly use grouped GEMM to process multiple experts together.

Pipeline parallel

Pipeline parallelism places consecutive groups of layers on different GPUs. At each stage boundary, neighboring stages use point-to-point send and receive operations to pass activations forward and activation gradients backward.

The batch is split into microbatches that move through the stages like an assembly line. Once the pipeline is full, all GPUs work simultaneously on different microbatches. The fill and drain periods create pipeline bubbles. With $P$ balanced stages and $M$ microbatches, ideal utilization is approximately $M/(M+P-1)$ , so $M$ should be much larger than $P$ .

GPipe

GPipe runs forward for every microbatch and then runs backward in reverse order. It pipelines well, but each stage must retain activations for many microbatches until their backward passes, so activation memory grows with $M$ .

1F1B

After warmup, 1F1B alternates one forward microbatch with one backward microbatch. This keeps fewer microbatch activations live and reduces peak memory, while the pipeline's fill and drain bubbles remain.

Interleaved 1F1B

Interleaved 1F1B gives each GPU multiple virtual pipeline stages and interleaves their microbatches. This reduces the bubble compared with basic 1F1B, but adds point-to-point communication and scheduling complexity. See the Megatron Core schedule.

Zero-bubble

Zero-bubble pipeline parallelism splits backward into input-gradient and weight-gradient computation. Input gradients stay on the critical path, while weight gradients can fill otherwise idle slots, allowing bubbles to approach zero with a suitable schedule and enough memory.

DualPipe

DualPipe sends microbatches through the pipeline in both directions and overlaps forward and backward computation with communication. This reduces bubbles but requires a more complex bidirectional schedule.

PipeDream

PipeDream avoids globally flushing the pipeline by updating stages asynchronously. It improves utilization but requires weight stashing and can train different microbatches with different weight versions.

ZeRO-1

ZeRO-1 is data parallelism with optimizer states partitioned across workers. For Adam, optimizer state consists of the first-moment buffer, the second-moment buffer, and a small step counter. Mixed-precision training commonly also keeps an FP32 master copy of each weight, which is usually counted with optimizer-state memory:

Every worker runs the usual forward and backward with a complete model replica.
Every gradient contributes to the global gradient. Reduce-scatter gives each worker the reduced gradient shard for the parameters it owns. Like DDP, this communication is bucketed and can run asynchronously while the GPU computes gradients for earlier layers.
Each worker uses its local optimizer states to update its owned parameter shard, roughly $1/N$ of the model.
Workers all-gather the updated parameter shards, restoring the complete identical model on every worker.

Communication volume remains the same as DDP because a ring all-reduce internally consists of a reduce-scatter and an all-gather. Standard DDP uses both phases for gradients. ZeRO-1 reduce-scatters gradients, updates the local parameter shards, and then all-gathers the updated parameters. It repurposes the second phase rather than adding another collective.

ZeRO-2

ZeRO-2 shards gradients in addition to optimizer states. As a gradient bucket becomes ready during backward, it is reduce-scattered. Each worker keeps only the reduced gradients for its $1/N$ parameter shard and releases the rest. ZeRO-1 and ZeRO-2 materialize the same temporary local gradient bucket when configured with the same bucket size, but ZeRO-1 retains full-model gradient storage. ZeRO-2 therefore stores roughly $G/N$ persistent gradients plus one temporary bucket instead of $G$ persistent gradients.

The optimizer step still normally runs after backward, not as each parameter gradient appears. Gradient accumulation may add contributions from later microbatches, while global gradient-norm clipping needs the norm over all gradient shards before any weight is updated. The latter requires only an all-reduce of local squared norms, not an all-gather of the gradients.

Compared with ZeRO-1, the main drawback is extra bookkeeping for gradient ownership, bucket lifetimes, and distributed operations.

ZeRO-3

ZeRO-3 shards parameters in addition to gradients and optimizer states. Each worker persistently stores roughly $1/N$ of each state. The largest layer must still fit on one worker when materialized; otherwise its computation must also be sharded, usually with tensor parallelism.

Before a layer runs in forward, workers all-gather its parameter shards, compute with the temporarily complete layer, and release parameters that will not be reused soon.
Backward usually all-gathers the layer parameters again, computes parameter and activation gradients, then reduce-scatters the parameter gradients to their owners.
Each worker updates only its owned parameter shard. No full-model all-gather is needed after the update because parameters are gathered layer by layer in the next forward pass.

FSDP

Fully Sharded Data Parallel is PyTorch's ZeRO-style API, not a separate parallelism strategy. Module boundaries determine communication granularity and temporary memory use.

FSDP1 wraps modules and stores each module's parameters in flat buffers used for sharding and communication.

Flat parameter

A flat parameter is one contiguous tensor formed by concatenating many parameters. The original parameters become views into its slices, so FSDP can shard and communicate one large buffer instead of many small tensors.

FULL_SHARD is ZeRO-3. It reshards parameters after forward and all-gathers them again for backward.
SHARD_GRAD_OP keeps full parameters through backward, skipping that all-gather at the cost of more memory, then reshards them after backward.
NO_SHARD replicates all states and all-reduces gradients like DDP.
HYBRID_SHARD applies FULL_SHARD within a group, commonly one node, and replicates across groups.
_HYBRID_SHARD_ZERO2 similarly applies SHARD_GRAD_OP within a group and replicates across groups.

FSDP2 shards each parameter in place as a DTensor, preserving parameter identities and avoiding FSDP1's flat-parameter abstraction.

YaFSDP is a sharded data-parallel framework optimized for transformer-like models. It reduces communication and memory-operation overhead; its authors report up to 20% faster LLM pre-training than PyTorch FSDP and better performance under high memory pressure.

It preallocates and reuses weight and gradient buffers across alternating transformer layers, avoiding repeated allocation and copying.
Gradients are written directly into communication buffers.