

PyTorch scaled dot product attention
PyTorch's torch.nn.functional.scaled_dot_product_attention (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for torch>=2.1.1 when an implementation is available.
For now, Transformers supports SDPA inference and training for the following architectures:
* Bart
* GPTBigCode
* Falcon
* Gemma
* Llama
* Phi
* Idefics
* Whisper
* Mistral
* Mixtral
* StableLm
* Starcoder2
* Qwen2