Memory allocation and fragmentation

GPU allocation is expensive, so PyTorch uses a caching allocator. When a tensor is freed, its memory usually stays reserved by PyTorch instead of returning immediately to the GPU driver. A later tensor can reuse that cached block without another expensive driver allocation.

Dynamic tensor shapes make reuse harder. Changing batch size, sequence length, or image resolution creates requests of different sizes. If the workload gradually reaches its largest expected shape, the allocator can retain many smaller blocks along the way. In the worst case, requests that grow linearly up to size $N$ leave cached blocks totaling $O(N^2)$, rather than a block near $O(N)$ for the largest request. Warming up with the largest expected input first gives later smaller requests a large cached block to reuse or split. This helps only for memory that is not live concurrently, and the exact behavior depends on allocator settings and request sizes.

Fragmentation

Fragmentation is wasted memory caused by the sizes and positions of allocated blocks. Suppose memory contains 100 bytes:

• Allocate 30 bytes. [30][70 free]
• Allocate 40 bytes. [30][40][30 free]
• Free the first 30 bytes. [30 free][40][30 free]

There are 60 free bytes in total, but a request for 50 contiguous bytes still fails. The largest contiguous free block is only 30 bytes. The allocator may report substantial reserved memory even when none of its reusable blocks is large enough for the next tensor.

Resources

• PyTorch CUDA memory management
• Efficient Deep Learning Systems seminar