Load Phi 3 small on Nvidia Tesla V100 - Flash Attention
Hi,I would like to inquire about the possibility of uploading and fine tuning a Phi 3 8k small. When I load the model, I get an error about missing Flash attention. If I want to install the given package, I get this error :RuntimeError: FlashAttention is only supported on CUDA 11.6 and above.  Note: make sure nvcc has a supported version by running nvcc -V.


      torch.__version__  = 2.3.1+cu121But I have the required version of pytorch and CUDA (torch 2.3.1 and cuda 12.1)Is it because I am using a Tesla V100 graphics card? Is there any way to load the model also with this graphics card?I found this in the documentation for the Phi 3 mini on Huggingface:If you want to run the model on:
NVIDIA V100 or earlier generation GPUs: call AutoModelForCausalLM.from_pretrained() with attn_implementation="eager"Does this also apply to the Phi3 Small 8k?? Beacause when I try to load it, the error occursmodel = AutoModelForSequenceClassification.from_pretrained("path", num_labels=num_labels ,attn_implementation="eager" )

AssertionError: Flash Attention is not available, but is needed for dense attentionOr should I try the ONNX version or it is just for inference?Thank you.
2024-07-31T09:16:08Z
v100 is Volta generation GPU and has Compute Compatibility 7.0, which is not supported by flash-attention. You need to have Ampere or later GPUs.Based ontransformers/src/transformers/models/phi3/modeling_phi3.py at 47c29ccfaf56947d845971a439cbe75a764b63d7 · huggingface/transformers · GitHubyou should be able to run it in “eager” mode. Make sure you have the latest transformers library
2024-07-31T12:30:00Z
@antonpolishkoThank you for response.Yes, I have the latest transformers library and I tried this:model = AutoModelForSequenceClassification.from_pretrained("path", num_labels=num_labels ,attn_implementation="eager" )But still I have the error:AssertionError: Flash Attention is not available, but is needed for dense attentionDo you have some idea what to do?
2024-08-06T07:36:04Z
I opened an issue on github at trnasformers. Unable to load model in eager mode.