Combining GPT-Neo and Flash Attention 2
First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature, and make sure your hardware is compatible with Flash-Attention 2. More details are available here concerning the installation.
Make sure as well to load your model in half-precision (e.g. torch.float16).
To load and run a model using Flash Attention 2, refer to the snippet below:
thon