Optimize CUDA Kernel for Transformer Attention

The Challenge

Implement a fused attention kernel in CUDA that outperforms naive attention by at least 2x on an A100 GPU. Bonus for flash-attention style tiling. Benchmarks required.

Required Skills

What to Submit