Does the default weight_decay of 0.0 in transformers.AdamW make sense?
Hi,I have a question regarding the AdamW optimizer defaultweight_decayvalue.In theDocswe can clearly see that the AdamW optimizer sets the default weight decay to 0.0.Given that the whole purpose of AdamW is to decouple the weight decay regularization, is my understanding that the results anyone can get with AdamW and Adam if both are used withweight_decay=0.0(this is, without weight decay) should be exactly the same.Therefore, shouldn’t make more sense to have the default weight decay for AdamW > 0?Thank you so much!
2020-09-18T12:57:12Z
Even though I agree about the default value (it should probably be 0.01 as in thePyTorchimplementation), this probably should not be changed without warning because it breaks backwards compatibility. It was also implemented intransformersbefore it was available in PyTorch itself.I guess it is implemented in this way, because most of the time you decide in the initialization which parameters you want to decay and which ones shouldn’t be decayed, such as here:github.comhuggingface/transformers/blob/a75c64d80c76c3dc71f735d9197a4a601847e0cd/examples/contrib/run_openai_gpt.py#L230-L237optimizer_grouped_parameters = [{"params": [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],"weight_decay": args.weight_decay,},{"params": [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], "weight_decay": 0.0},]optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
2020-09-18T13:25:14Z
In general the default of all optimizers for weight decay is 0 (I don’t know why pytorch set 0.01 for justAdamW, all other optimizers have a default at 0) because you have to opt-in for weight decay. Even if it’s true thatAdamandAdamWbehave the same way when the weight decay is set to 0, I don’t think it’s enough to change that default behavior (0.01 is a great default otherwise, that is the one we set in fastai for theLearnerafter countless experiments, but I think it should be set in a higher-level API, not the optimizer itself).And like@BramVanroysaid, it would be such a breaking change that even if we really wanted to change that default, we probably wouldn’t.