Enabling Flash Attention 2
What is the difference between using Flash Attention 2 viamodel = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "sdpa")vsmodel = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "flash_attention_2")when Pytorch SDPA support FA2 according todocs?@marcsun13
2024-05-29T09:54:33Z
@ybelkadaCan you shed some light on this?
2024-07-03T14:41:43Z
@varadhbhatnagarwere you able to figure out the difference?