How to exclude layers in weight decay
My code is written in pytorch, thus I use torch.optim.adam as my optimizer. However, I need to do use Adam wright decay with some layer excluded. To be more specifically, I am trying to reproduce this tensor flow codeoptimizer = AdamWeightDecayOptimizer(*some parameter setting here,*    exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])What should I do to exclude [“LayerNorm”, “layer_norm”, “bias”] in weight decay in pytorch? Could I use tensor flow optimizer in pytorch?Thank you.
2021-10-18T13:48:59Z
You can look at how we do this for the Trainerhere.