Phi3 Mini 4k Instruct Flash Attention not found
When loading  “microsoft/Phi-3-mini-4k-instruct” using transformers.pipeline with “load_in_4bit=True” I get a warning stating:WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.920b6cf52a79ecff578cc33f61922b23cbc88115.modeling_phi3:`flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'.

WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.920b6cf52a79ecff578cc33f61922b23cbc88115.modeling_phi3:Current `flash-attenton` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.Perhaps I have loaded it incorrectly? I have accelerate in my environment
2024-05-10T18:04:10Z
+1 ı have same problem.flash-attentionpackage not found, consider installing for better performance: No module named ‘flash_attn’.Currentflash-attentondoes not supportwindow_size. Either upgrade or useattn_implementation='eager'.microsoft/Phi-3-mini-128k-instruct"
2024-05-10T21:02:20Z
Solution for me was to install the missing package.I thought flash attention was handled in accelerate.But I think phi3 implements their own version using a specific package.You can just do pip install flash-attn
2024-05-10T22:16:57Z
Collecting flash-attnUsing cached flash_attn-2.5.8.tar.gz (2.5 MB)Installing build dependencies … doneGetting requirements to build wheel … errorerror: subprocess-exited-with-errornote: This error originates from a subprocess, and is likely not a problem with pip.error: subprocess-exited-with-error× Getting requirements to build wheel did not run successfully.│ exit code: 1╰─> See above for output.note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-11T10:17:42Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.