Are there any plans for replacing attention in transformers?
As I’m currently doing a lot of work on replacing quadratic attention with linear attention mechanisms on pretrained transformer models that don’t yet have it implemented, I was wondering; Are there any plans to port efficient attention substitutes to different models and make them easily swappable in the transformers library?
2022-04-08T12:20:05Z
Short answer: no.Longer answer: this blog post (especially point 3, cc@patrickvonplaten):huggingface.co~Don't~ Repeat YourselfWe’re on a journey to advance and democratize artificial intelligence through open source and open science.We do implement models rather independently in the library, hence we do have some linear attention variants available, including:NyströmformerYOSOLongFormerBigBirdThere’s also a FLAX implementation available of Performer, which you can find here:transformers/examples/research_projects/performer at main · huggingface/transformers · GitHub
2022-04-08T12:26:52Z
Ok thanks! I think these models are great, but was thinking of way the attention could be abstracted, so that I could for example load a checkpoint that was trained using quadratic self attention and then fine-tune using linear attention just by specifying some argument. I suppose this could be an idea for a separate library.
2024-07-11T22:31:14Z
GitHubGitHub - HazyResearch/zoology: Understand and test language model...Understand and test language model architectures on synthetic tasks. - HazyResearch/zoologyThis repo seems like a good foundation for this use case.