An Empirical Model of Large-Batch Training introduces the gradient noise scale, a useful way to think about the largest batch size that still improves training efficiency.

Batch size and data parallel communication

In data parallelism, each worker runs forward and backward on its local batch, then workers communicate gradients. The compute work grows with batch size. Doubling the batch roughly doubles the forward and backward work for that step. The gradient payload does not grow with batch size because it has one value per model parameter, not one value per activation or training example.

Therefore, an all-reduce communicates roughly the same number of bytes for a batch of 32 and a batch of 256. A larger batch makes each step more compute-heavy while the communication per step stays constant. This can improve compute-communication overlap and reduce the total number of gradient synchronizations needed to reach a target loss.

If increasing batch size by a factor of $X$ also reduces the required number of optimizer steps by roughly $X$, the same amount of example-level work is completed in fewer steps. Instead of paying for $X$ gradient all-reduces, the larger batch pays for one. It saves roughly $X-1$ communication rounds. The backward computation per example is not eliminated. The saving comes from fewer optimizer steps and fewer fixed communication costs.

Why batch size cannot grow forever

The true gradient is the average gradient over all training examples. Computing it exactly is usually too expensive, so training uses a mini-batch estimate instead. A small batch gives a noisy estimate. A larger batch averages more examples and produces an estimate closer to the true gradient.

This trades computation for gradient accuracy. A small fraction of the data can already capture most of the useful gradient signal. Capturing the remaining small improvement may require a disproportionately large number of extra examples. This is only a loose Pareto-style intuition, not a universal 80/20 law. The exact relationship depends on the data distribution, model, and point in training.

The gradient noise scale roughly measures the randomness in gradient estimates. Below that scale, gradient noise dominates and larger batches can make useful progress faster. Above it, the noise is already sufficiently suppressed, so extra examples make the gradient only slightly more useful.

Think of the gradient noise scale as the effective number of examples needed in one step to make good progress. If the gradient noise scale is roughly 1,000, a batch near 1,000 can tame most of the noise:

• A batch of 10,000 is far above that noise floor. It adds compute while contributing little additional gradient information.
• A batch of 100 is below the noise scale. Its gradient estimate is much noisier, so training is less efficient.

This can also be visualized on a loss-surface contour plot, where the level curves connect points with equal loss. The true gradient points downhill. Mini-batch gradients are noisy arrows around that direction. Once a large batch already points close to the true gradient, doubling the batch may move the arrow only a few degrees closer while doubling the compute. The exact angle depends on the problem, but the noise falls only with the square root of batch size, which explains the diminishing return.

This also helps explain why reinforcement learning can use global batch sizes of millions while simple handwritten digit classification can use batches of tens or hundreds. Reinforcement-learning gradients are highly stochastic: they depend on sampled actions, environment transitions, rewards, and long trajectories. More samples are needed to suppress that noise. Digit classification has a simpler supervised signal and often lower gradient variance, so much smaller batches can already give a useful estimate. The exact batch size still depends on the model, dataset, and training stage.

Critical batch size and diminishing returns

Informally, the critical batch size is the largest batch size that still gives nearly linear training speedup. If doubling the batch almost halves the optimizer steps needed to reach a target loss, scaling is still efficient. Past the turning point, doubling the batch no longer comes close to halving those steps, so the extra compute gives little speedup.

This refers to optimization efficiency, not steps per epoch. For a fixed dataset, doubling batch size always halves the number of steps in an epoch by definition. Hardware throughput can also saturate when a device is full, but that is a separate limit. The optimization critical batch size can be lower than the hardware-saturation batch size.

The critical batch size is the scale at which increasing batch size stops giving proportional improvement in training time to a target loss. Below it, a larger batch can reduce the number of steps enough to offset the extra computation. Above it, each larger batch adds forward and backward work without reducing the steps to target loss by the same factor. Training then becomes slower or less compute-efficient, even though it performs fewer gradient all-reduces.

To preserve similar optimization behavior while scaling batch size in the useful regime, the learning rate is often scaled roughly linearly with batch size. This rule is empirical and only applies over a limited range. It must be checked for the model and training schedule rather than extended to arbitrarily large batches.

References

• An Empirical Model of Large-Batch Training
• Paper PDF