What is "scheduled LR warm-up"?
Hi everyone!I am optimizing the finetuning of my T5 model and come across this entry on the HF website (Optimization)where it is recommended to “use scheduled LR warm-up” with Adafactor when training T5. What is that and how can I implement the latter? Did anyone do this before?Many thanks in advance.