FlashAttention
A walkthrough of how FlashAttention reduces GPU memory I/O: tiling, online softmax rescaling, the algorithm line by line, I/O complexity, block-sparse attention, and experimental results.
Badges come from GitHub once the author claims this feed.
A walkthrough of how FlashAttention reduces GPU memory I/O: tiling, online softmax rescaling, the algorithm line by line, I/O complexity, block-sparse attention, and experimental results.