Linear learning rate despite lr_scheduler_type="polynomial"
Hello,While fine-tuning my network, I would like to set up a polynomial learning rate scheduler by settinglr_scheduler_type="polynomial"andlearning_rate=0.00005.However, when I visualize the learning rate on the wandb dashboard, I’m observing a linear decrease of the learning rate instead of polynomial. (Screenshot below)What might be causing the issue? I’ve tested without setting thelearning_rateand the behavior was exactly the same.image872×622 22.8 KB
2021-09-01T19:57:56Z
Polynomial scheduler is not really supported via this argument, as it requires an additionalpowerkeyword argument that defaults to 1 and which can’t be set via this API. You should thus set the scheduler directly in the Trainer.
2021-09-02T12:37:07Z
@sguggerI previously tested with lr_scheduler_type=“polynomial” andlr_scheduler_type="linear"to compare the outcome and I ended up getting different accuracy results, which makes me wonder - if the polynomial scheduler is not supported via this argument, shouldn’t I get exactly the same results?Thank you for the info, I was able to set the scheduler directly in the Trainer.
2021-09-02T12:38:15Z
Did you use the exact same seed as well?
2021-09-02T12:57:34Z
Yes, the seed was set usingset_seed(42).