How to ignore attributes of TrainingArguments?
I am trying to subclass Huggingface’s Trainer and overwrite it with customoptimizerandlr_schedulerfrom transformers import TrainingArguments

training_args = TrainingArguments(
    "rugpt3-headers",
    num_train_epochs=3,
    per_device_train_batch_size=3,
    evaluation_strategy="epoch",
    logging_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

from transformers import Trainer, AdamW
from torch.optim.lr_scheduler import MultiStepLR

class MyTrainer(Trainer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
    
    def create_optimizer(self):
      self.optimizer = AdamW(model.parameters(),lr = 1e-7)
    
    def create_scheduler(self, num_training_steps):
      self.lr_scheduler = MultiStepLR(self.optimizer, milestones=[1,3], gamma=0.5)

trainer = MyTrainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)
trainer.train()Even though I did not specifylearning_rateinTrainingArguments, it has a default value of5e-7. My attempt to overwrite theoptimizerandscheduleris not successful because of that. After my training was completed, I usedtensorboardto check which learning rate was used and it is still5e-07even though I thought I overwrote it.How to overcome this issue? I want it to set up my learning rate based ondef create_optimizer(self):
     self.optimizer = AdamW(model.parameters(),lr = 1e-7)
    
def create_scheduler(self, num_training_steps):
     self.lr_scheduler = MultiStepLR(self.optimizer, milestones=[1,3], gamma=0.5)I wrote above and ignore the defaultlearning rateofTrainingArguments
2021-07-29T13:06:31Z
Are you sure you are looking at the right logs? The learning_rate from the TrainingArguments is only used once in thecreate_optimizermethod and you overrode it, so there is no reason for it to be used.Also, make sure you are on the latest version of Transformers, just in case.
2021-07-29T13:19:46Z
so am I doing it correctly?also, is there any way for me to get or print the current learning rate during the training process?for example:1 epoch: learning rate - 2e-072 epoch: learning rate - 1e-0……
2021-07-29T14:03:52Z
You should have the learning rate printed at each log step, you can adjust that variable if needed be!
2021-07-30T08:02:29Z
I swear I tried every possible combination of the code. I tried logging_step, eval_step, with both set up at step and batch, but nothing prints out the learning rate. They print everything starting from loss, training time, and all that but learning rate. They don’t print out the learning rates. How can i do that? How can I get them to print the learning rate