Operator fusion combines multiple operations into one kernel. Without fusion, each operation writes its intermediate result to HBM, and the next operation reads it back. Those HBM reads and writes can take far longer than the small amount of arithmetic in pointwise operations. A fused kernel can keep intermediate values on chip and write only the final result to HBM, reducing memory traffic and often improving runtime.

References

• Making Deep Learning Go Brrrr From First Principles