GPUs are expensive, so optimizing their performance has a direct financial impact: models train faster, inference throughput increases, fewer GPU hours are required, and overall compute costs decrease. There is also a less obvious strategic advantage: reducing GPU hours shortens development cycles, helping you publish research or release new models ahead of your competitors.

Optimizing random parts of a system and hoping for better efficiency is gambling and usually a waste of time. You may get lucky and solve the problem quickly, but you will not understand why the change worked. More often, you will repeat experiments without making progress.

I prefer a data and knowledge driven approach where metrics show what is happening, while fundamentals help understand why. With that, sort of, radar, performance problem is solvable. It becomes a question of execution time and implementation effort. Even when progress is slow, every step moves toward the goal. There is no random try this or that.

Table of Contents

• FLOP
• FMA
• FLOPS/s
• HFU
• MFU
• Arithmetic intensity
• System balance
• Roofline
• Manual causal profiling
• How to search for improvement

FLOP

Floating point operation. Can be one of floating point addition, subtraction, multiplication or division operation.

import torch
import torch.utils.flop_counter


n = 1024
device = 0
a = torch.rand(n, n, dtype=torch.float16, device=device)
b = torch.rand(n, n, dtype=torch.float16, device=device)

flop_counter = torch.utils.flop_counter.FlopCounterMode(display=False)
with flop_counter:
    a @ b

analytical_flops = 2 * n ** 3
empirical_flops = flop_counter.get_total_flops()

assert analytical_flops == empirical_flops

FMA

Fused multiply add computes $a \times b + c$ as one instruction. 1 FMA equals to 2 FLOPS.

FLOPS/s

Floating point operations per second is how many floating point operations were performed in 1 second.

import torch


n = 1024
device = 0
warmup_steps = 4
measure_steps = 8

for _ in range(warmup_steps):
    a = torch.rand(n, n, dtype=torch.float16, device=device)
    b = torch.rand(n, n, dtype=torch.float16, device=device)
    a @ b

starter = torch.cuda.Event(enable_timing=True)
ender = torch.cuda.Event(enable_timing=True)

latencies = []
for _ in range(measure_steps):
    a = torch.rand(n, n, dtype=torch.float16, device=device)
    b = torch.rand(n, n, dtype=torch.float16, device=device)
    torch.cuda.synchronize()
    starter.record()
    a @ b
    ender.record()
    torch.cuda.synchronize()
    latency = starter.elapsed_time(ender) / 1e3
    latencies.append(latency)

analytical_flops = 2 * n ** 3
average_latency = sum(latencies) / len(latencies)
flops = analytical_flops / average_latency
tflops = flops / 10 ** 12
print(tflops) # 11.440554750942827

HFU

Hardware FLOPS utilization measures the achieved FLOPS/s as a percentage of the theoretical peak FLOPS/s of the hardware:

Suppose we use activation checkpointing and, in the extreme case, store no intermediate activations. During the backward pass, every required activation must be recomputed. This introduces many additional operations, but HFU does not distinguish recomputation from the original forward and backward work. Matrix multiplication is usually highly optimized, so these additional operations can still use the GPU efficiently and produce a high HFU.

Suppose the original forward pass requires 50 TFLOPs, recomputing the activations wastes another 450 TFLOPs, and the backward pass requires 300 TFLOPs. Divide the total by the 10-second step time to get the achieved TFLOP/s. Then divide that throughput by the theoretical peak of 100 TFLOP/s and multiply by 100 to express HFU as a percentage:

The distinction goes further. Consider a matrix multiplication:

Using the common convention that counts one multiplication and one addition for every reduction element, the algorithmic work is:

Algorithmic FLOPs describe the useful mathematical operation, but the hardware may execute more FLOPs. The operations actually issued by the device are called hardware FLOPs. Dividing this count by execution time gives achieved hardware FLOP/s, which is the numerator of HFU.

For a simplified example, suppose the selected Tensor Core kernel processes matrices in $128\times128\times128$ tiles. Tensor Core instructions use smaller fixed MMA shapes internally, but the larger tile is a useful way to model how a kernel partitions the matrix. Now suppose we multiply two $130\times130$ matrices. Since 130 is not divisible by 128, the kernel needs two tiles along each dimension. Conceptually, the matrices are padded with zeros from 130 to 256 elements in each dimension.

The useful algorithmic work is:

In this simplified tiled implementation, the hardware work can be as large as:

The zero padding does not change the result, but the hardware still performs FMAs for padded elements. Because matrix multiplication kernels are usually highly optimized, this extra work can keep the compute units busy and produce a high HFU even while making the workload less efficient. HFU therefore measures utilization, not usefulness.

MFU

Model FLOPS utilization measures how much of the theoretical peak FLOPS/s is spent on useful model work:

MFU uses an implementation-independent definition of useful work. Its numerator is determined by the mathematical operations required by the model, rather than by how a particular implementation executes them. The measured MFU can still change between implementations because a faster implementation reduces the step time.

Consider the activation-checkpointing example again. The original forward pass requires 50 TFLOPs and the backward pass requires 300 TFLOPs, so the useful model work is 350 TFLOPs. The additional 450 TFLOPs spent recomputing activations are excluded. With a 10-second step time and theoretical peak performance of 100 TFLOP/s, MFU is:

The same execution has an HFU of 80% because the hardware executes 800 TFLOPs in total. The 45-percentage-point difference is caused by recomputation that keeps the GPU busy without performing additional useful model work.

Now consider the simplified padded matrix multiplication. Multiplying the two $130\times130$ matrices requires $4{,}394{,}000$ useful FLOPs. Because the Tensor Core kernel works with fixed $128\times128\times128$ tiles, it pads the dimensions to 256 and may execute as many as $33{,}554{,}432$ hardware FLOPs for the resulting $256\times256\times256$ tiled computation. Since MFU and HFU use the same execution time and theoretical hardware peak, they are related by:

If the padded matrix multiplication reaches 80% HFU, its useful-work utilization is:

The GPU is highly utilized, but only a small fraction of its theoretical peak performance contributes to the requested $130\times130$ multiplication. Strictly speaking, for one matrix multiplication this is algorithmic FLOPS utilization rather than model FLOPS utilization. MFU applies the same idea to all useful operations in a model.

Under consistent FLOP-counting conventions, MFU is always less than or equal to HFU. Both metrics use the same step time and theoretical peak, but HFU counts every operation executed by the hardware, while MFU counts only useful model operations. They are equal when the implementation performs no recomputation, padding, or other redundant work counted by HFU.

Arithmetic intensity

Arithmetic intensity (AI) is the ratio of the number of floating-point operations to the amount of data moved:

Assume that all calculations use FP16, so each element occupies 2 bytes. The examples below use an idealized traffic model that counts each input matrix or vector once and ignores cache effects, reloading, and other implementation details.

Consider the element-wise addition of two vectors $a,b\in\mathbb{R}^{N}$:

The two input vectors require $2N\cdot2=4N$ bytes to read. The output vector requires $N\cdot2=2N$ bytes to write. Therefore, the total data movement is $6N$ bytes. The addition performs $N$ FLOPs, so its arithmetic intensity is:

This means that the operation performs only one-sixth of a FLOP per moved byte, or equivalently, moves 6 bytes for every FLOP.

Now consider multiplying two square matrices $A,B\in\mathbb{R}^{N\times N}$:

Under the same idealized traffic model, reading the two input matrices moves $2N^2\cdot2=4N^2$ bytes, and writing the output matrix moves another $N^2\cdot2=2N^2$ bytes. The total data movement is therefore $6N^2$ bytes. Matrix multiplication performs $2N^3$ FLOPs, counting one multiplication and one addition for each term in each dot product. Its arithmetic intensity is:

Unlike vector addition, matrix multiplication can reuse values from the input matrices many times. Tiled implementations exploit this reuse, so the idealized arithmetic intensity above is a simple baseline rather than a complete model of actual memory traffic.

System balance

It is useful to think of a GPU as a factory connected to a warehouse. The compute units are the factory, device memory is the warehouse, and memory bandwidth is the speed at which raw materials can be delivered. A kernel performs well only when these two pipelines supply and consume data at compatible rates.

The system-balance point is the arithmetic intensity at which the hardware's compute throughput and memory-bandwidth throughput are equal:

For a kernel with arithmetic intensity $\mathrm{AI}$:

At the balance point, neither pipeline has spare capacity. Below it, memory cannot deliver data quickly enough to sustain peak compute throughput. Above it, the memory system can supply data faster than the compute units can process it, so the compute units themselves are the limiting resource. Arithmetic throughput is the bottleneck.

When $\mathrm{AI}<\mathrm{AI}_{\text{balance}}$, the memory pipeline is the bottleneck. Data arrives more slowly than the compute units could consume it, so arithmetic units may remain idle while waiting for memory. Increasing peak compute throughput alone usually does little; increasing effective memory bandwidth or reducing data movement is more useful.

Kernel fusion often helps by keeping intermediate values in registers or shared memory instead of writing and rereading them from device memory. This reduces bytes moved and increases arithmetic intensity. However, fusion is not guaranteed to improve performance. Excessive fusion can increase register pressure, reduce occupancy, or create a less efficient kernel.

When $\mathrm{AI}>\mathrm{AI}_{\text{balance}}$, the memory system can supply data faster than the compute units can process it. The arithmetic units stay busy, while some memory bandwidth may remain unused, so computation is the bottleneck. Improvements should target faster math instructions, better tiling, higher occupancy, or reduced instruction overhead.

The following values are approximate system-balance points for dense FP16 Tensor Core computation. They are not the arithmetic intensity of vector addition or matrix multiplication. Each value is a hardware threshold: the ratio of the GPU's peak dense FP16 Tensor Core throughput to its peak memory bandwidth. A kernel must perform roughly this many FLOPs per moved byte for the memory pipeline and the Tensor Cores to operate near their limits at the same time, so neither resource spends most of its time waiting for the other:

For FP16 vector addition, the arithmetic intensity is $1/6$ FLOP/byte regardless of the vector length. On a T4, whose balance point is approximately 201 FLOP/byte, this is far below the transition point. Under this traffic pattern, increasing the vector length does not make the operation compute-bound. Some algorithms are therefore inherently memory-bound. Their implementation can be improved, but upgrading only the GPU's compute throughput will usually provide little benefit unless memory bandwidth also increases.

For the idealized square matrix multiplication above, $\mathrm{AI}=N/3$. Setting this equal to the system-balance point gives the matrix dimension at which the two pipelines are approximately matched:

For example, on a T4, the operation crosses the balance point when:

The corresponding balance-point dimensions for the other GPUs are:

The same matrix multiplication can therefore be compute-bound on one GPU and memory-bound on another. This leads to an important hardware-aware design principle: neural-network architectures and tensor dimensions can, in principle, be chosen for a specific accelerator. In modern neural networks, however, the matrix dimensions are usually large enough that their main GEMMs are compute-bound across all of the GPUs in this table. The system-balance analysis still matters for smaller projections, unusual shapes, and other operations with lower arithmetic intensity.

Now we know whether the kernel is compute-bound or memory-bound. However, this does not tell us whether its performance is already good enough or whether further optimization is worthwhile. The roofline model helps answer that question.

Roofline

Roofline allows us to understand the upper bound on FLOP/s for an ideal algorithm running on a particular accelerator. Given the algorithm, we can estimate its arithmetic intensity, and given the hardware, we can estimate the maximum FLOP/s achievable at that arithmetic intensity. We have already learned how to measure the actual FLOP/s, so we can plot that measured point on the roofline and see how far it is from the upper bound.

The Roofline is different for each GPU because accelerators have different peak compute throughput and memory bandwidth, which gives them different operations-per-byte thresholds. The ridge point, also called the system-balance point, separates the graph into the memory-bound region on the left, the balanced transition point in the center, and the compute-bound region on the right.

To construct the Roofline, place arithmetic intensity on the horizontal axis, from zero to a sufficiently large value. For each arithmetic-intensity value, compute the maximum FLOP/s that an ideal algorithm could achieve. This performance is capped either by memory bandwidth or by the hardware's peak compute throughput:

Using the dense FP16 Tensor Core throughput and memory bandwidth from the specifications above gives the following Roofline curves. The horizontal axis is logarithmic arithmetic intensity, and the vertical axis is the corresponding upper bound on dense FP16 Tensor Core throughput:

Any point on a Roofline plot can move horizontally or vertically:

• Horizontal movement changes arithmetic intensity. Since $\mathrm{AI}=\text{FLOPs}/\text{bytes moved}$, it requires changing the amount of useful computation, the amount of data movement, or both. This usually involves changing the algorithm or the way the computation is organized. However, changing an algorithm does not necessarily move the point horizontally. If two implementations perform the same FLOPs and move the same number of bytes, they have the same arithmetic intensity even if their memory-access order or operation scheduling differs.
• Vertical movement keeps arithmetic intensity constant while increasing achieved FLOP/s. This means removing overhead and executing the same FLOP/byte workload faster—for example, through better kernel implementations, tiling, fusion, parallelism, or instruction scheduling. FLOPs and bytes may both change, but if they change by the same factor, their ratio remains unchanged and the point moves vertically.

NVIDIA Nsight Compute can generate Roofline charts directly from kernel profiles.

Roofline does not tell us whether the algorithm is efficient in terms of latency or asymptotic complexity. An algorithm with poor asymptotic complexity may still remove implementation overhead and reach close to the Roofline ceiling. Roofline only shows the gap between the observed performance and the peak performance available for that particular arithmetic intensity.

As rough heuristics for the ratio of observed performance to the Roofline ceiling:

• Above 90% is essentially optimal, with no obvious optimization left. This is usually achievable only for very regular kernels.
• Above 70% is very strong and is typical of high-quality GEMMs, fused kernels, and well-tuned attention implementations.
• Above 50% is common for real workloads and is often good enough unless the kernel dominates runtime.
• Above 30% may be acceptable for a non-dominant kernel. Further optimization is worthwhile mainly when the kernel is on the critical path.
• Below 10% is poor in many cases and usually justifies investigation, although tiny operations and latency-critical inference can make such utilization unavoidable.

How to search for improvement

Amdahl's law estimates the maximum speedup of an entire system when only one part is improved. Suppose a system takes 100 ms, with 10 ms spent in one part and 90 ms in everything else. Improving the 10 ms part by a factor of two gives:

The whole system becomes only about 5% faster, even though that part became twice as efficient. Even an infinitely fast 10 ms part could improve the total runtime by at most 10%.

Optimization should be data-driven rather than a gamble. Measure the current FLOP/s and the fraction of total runtime spent in the kernel. Then estimate a realistic improvement—for example, a 10% efficiency increase—and use Amdahl's law to determine whether the resulting end-to-end speedup is worth the effort. Use the Roofline model to estimate the remaining performance gap. This keeps optimization focused on small, justified steps toward the final goal.

References

• Making Deep Learning Go Brrrr From First Principles
• ML Engineering
• GPU metrics
• Another post on gpu metrics