Flash Attention

Also known as: FlashAttention

Flash Attention is an IO-aware implementation of scaled dot-product attention that fuses the attention computation into a single GPU kernel and tiles operations to avoid materializing the full N×N attention matrix in high-bandwidth memory. By keeping intermediate activations in fast SRAM, Flash Attention achieves the same mathematical result as standard attention while using substantially less GPU memory and running significantly faster — enabling training and inference over much longer context lengths without algorithm-level changes.