How is the AdafactorScheluder suppose to be used?
Hi,I recently saw my transformer model having divergence issues and I saw a paper that uses Adafactor and wanted to try it out. The docs are fantastic but they don’t mention how often or how the Adafactor scheduler actually works. How is that suppose to be used? When do I call the scheduler in my code?from transformers.optimization import Adafactor, AdafactorSchedule
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
lr_scheduler = AdafactorSchedule(optimizer)all my code is my custom training so idk when lr_scheduler is suppose to be called. Usually it depends on the model or I try to start of with the convenientReduceLROnPlateau.When usinglr=Nonewith [Trainer] you will most likely need to useAdafactorSchedulescheduler as following:from transformers.optimization import Adafactor, AdafactorSchedule optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) lr_scheduler = AdafactorSchedule(optimizer) trainer = Trainer(..., optimizers=(optimizer, lr_scheduler))related:Optimization — transformers 4.7.0 documentationT5 Finetuning Tips - #34 by brando
2021-12-19T16:22:34Z
Apologies for the direct ping, but can you help me get the right person to help me with this?@sgugger?Thank you in advance!
2021-12-20T13:48:01Z
I think@stasor@patrickvonplatenhave more experience with Adafactor.Note that it won’t stay in the library forever: merging it was overspreading ourselves a little bit too much in optimizers territory and we now realize we don’t have the manpower to properly maintain it. So you should use a version from another library to be future-proof
2021-12-20T19:54:56Z
Well, I hacked togetherAdafactorSchedulesinceAdafactoruses an internal scheduler and provides no access to it. We needed the workaround to prevent HF Trainer from failing when it callsget_last_lr.Later someone pointed out that my hack was incomplete since it only reported the schedule for a single param group if I remember correctly.As@sguggermentioned it’s best for you too seek out an external-to-transformers solution, sinceAdafactoris scheduled to be removed in transformers-v5. I think our original copy came from fairseq.
2023-02-13T08:59:23Z
Something is not clear to me - Should the optimizer (adafactor) be given to the TrainingArguments or to the Trainer?
2024-01-08T08:06:57Z
I have a same confusion