Best practices to use models requiring flash_attn on Apple silicon macs (or non CUDA)?
There are any number of models on HuggingFaces that seem torequireflash_attn, even though my understanding is most models can actually work fine without it. A few examples:huggingface.comicrosoft/Florence-2-large · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coqnguyen3/nanoLLaVA-1.5 · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.What is the best practice to get them working on Apple M2/M3 laptops (ideally teally with Metal support)? Obviously flash_attn won’t be available, but there is still plenty of value in working with models locally on a laptop before they need the higher efficiency of flash_attn and CUDA.I’ve found a few directional hints, but none of them have worked:In theory you should be able to monkey patch out the exception triggered in transformers.dynamic_module_utils but I cannot get that to workIn theory you should be able to FLASH_ATTENTION_SKIP_CUDA_BUILD=TRUE pip install flash-attn==2.5.8 but that fails to build (due to some strange issue with os.rename not working on Mac OS).Has anybody gotten these models working? Is there a general solution that Huggingface can implement to allow these models to run / train (even if it isn’t very efficient) on non CUDA devices?
2024-07-16T04:37:13Z
Just as I posted it, I found atleast one solution (the monkey patch approach) that works!huggingface.coqnguyen3/nanoLLaVA-1.5 · Handling `flash_attn` Dependency for Non-GPU...I encountered an issue when running the code from the repo without a GPU. The original code requests the flash_attn module, which is specific to CUDA execution. This can cause problems for users wh...Can something like this be built into transformers so we don’t have to do it everytime?
2024-08-23T20:05:26Z
thanks for sharing, it works! was also facing the same problem