Improvements with SWA
Has anyone tried SWA with transformers ? Would be interesting to see how much gains does it provide over just AdamW. I’m thinking of adding it to theTrainerif there are significant gains since Pytorch natively supports it.
2020-08-25T10:33:59Z
SWA is a wrapper around any optimizer, so you could try it with AdamW. Fair warning though, I have never been able to reproduce any results of the original paper of finding it actually helped with anything when using it in fastai (maybe why it took two years to have it inside PyTorch). It’s also supposed to work best with a cyclical schedule (and you average the weights at the end of the cycles) which are not used in transfomers.
2020-08-25T10:37:37Z
Yes I saw. It’s not exclusive to SGD. But the PR and thetorch contribimplementation doesn’t mention this aspect of reproducibility issue. Many people have seen improvements as per the blog. I wonder how their LR scheduling is different from cyclical LR (in what ways is it better ?). Btw why are cyclical schedulers not used with transformers ?
2020-08-25T10:42:26Z
prajjwal1:Btw why are cyclical schedulers not used with transformers ?They are not SOTA (or take a longer time to be) since 2 years ago. It’s just that fastai and transformers models are the only wide place that knows itIn the case of transformers, I know the GPT models just don’t train as well with another schedule. I don’t know if all the BERT models are just following that trend but I’d guess they also need that kind of schedule to train properly.
2021-01-06T20:40:39Z
I was planning to add SWA to transformers by working on a PR. After seeing this thread, I am a lot skeptical about starting working on it.@sguggerI have some questions for you. Any answer will be much appreciated.You have said that it works well with cyclical learning rates. When I look at thepaperand also the image below, I see that constant learning rate also achieves better results.Did you experiment transformers with SWA and constant learning rate?@prajjwal1Do you have any other insights about SWA with transformers?FYI: I also opened afeature requestto get more insights from others.
2021-01-12T06:06:26Z
I also couldn’t produce better results with SWA. Thanks for your input@sgugger