FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Episode seven of Topic 3 stays inside the GPU for the sequel. FlashAttention ended attention's memory commute, and the same author's audit found the chip still far from busy — so FlashAttention-2 re-divides the labor instead of the math: less slow non-matmul bookkeeping, a single attention head's work split across many thread blocks so long-sequence runs fill the machine, and warps exchanging less through shared memory. Maya maps the three findings onto a moving crew — the tape gun, the empty stairwells, the over-the-shoulder pass — while Leo reads the stopwatch: about twice the already-fast kernel, fifty to seventy-three percent of theoretical peak on A100s, and end-to-end GPT-style training near seventy-two percent model FLOP utilization. They argue whether scheduling counts as research, agree the schedule has become part of the algorithm, and close on the harder truth: a partition tuned for one chip is a hypothesis about the next. Sources: • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning: https://arxiv.org/pdf/2307.08691 • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: https://arxiv.org/pdf/2205.14135