Profiling in PyTorch (Part 3): Attention is all you profile

Hugging Face · 1 month ago · 4

PyTorch's profiler documentation on attention mechanisms was extended to show how different implementations of the attention operation appear in performance traces. The naive in-place attention implementation launches five GPU kernels and takes 1.955 ms, while the math backend of scaled dot product attention launches twenty kernels and takes 7.239 ms due to upcasting to FP32 and materializing intermediate matrices. Different SDPA backends optimize attention by fusing multiple operations into single kernels while maintaining numerical safety, with trade-offs between speed and precision visible in profiler traces.