Is there an easy way to apply layer-wise decaying learning rate in huggingface trainer for RobertaMaskedForLM?
I am pre-training RobertaMaskedForLM on my own custom dataset. I wanted to implement the layer-wise learning rate decay given inhttps://github.com/aws-health-ai/multi_domain_lm#learning-rate-controlcorresponding to the paper -An Empirical Investigation Towards Efficient Multi-Domain LanguageModel Pre-training. Is there an easy way to incorporate this decay of learning rate with layer depth towards input usingtransformers.Trainer?
2020-11-14T04:14:27Z
I have the same question
2020-11-16T13:57:46Z
There is nothing in the lib for this, but you can pass your own optimizer and scheduler.
2022-04-05T09:01:27Z
Hello, I have the same question. I’m fine-tuning RoBERTa large for RE(Relation Extraction) task andthe paperI referenced usedlayer decay.It seems like I have to custom my own optimizer and scheduler for layer-wise learning rate decay. Could you tell me how you implemented your own scheduler?