How to use Adaptive Learning rate during training?
Hi,I am trying to train a Siamese network and want to incorporate adaptive learning rate during training.So I have a SiameseTrainer which is a subclass of Trainer class. My question is am I doing something wrong that I get the errorTypeError: Object of type Tensor is not JSON serializablein this line of my code:train_result = trainer.train(resume_from_checkpoint=checkpoint)My code runs for few epochs and then crashes. So what I am doing is below:def compute_metrics(p: EvalPrediction):
  correct_predictions = p.predictions[1] if isinstance(p.predictions, tuple) else p.predictions
  labels = p.label_ids
  return {
    "MSE": np.mean((correct_predictions-labels)**2)
  }

optimizer = Adafactor(
            model.parameters(), 
            scale_parameter=True, # If True, learning rate is scaled by root mean square
        )
lr_scheduler = AdafactorSchedule(optimizer)

trainer = SiameseTrainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=data_collator,
            compute_metrics=compute_metrics,
            tokenizer=tokenizer,
            callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
            optimizers=(optimizer, lr_scheduler)
)When I comment out theoptimizers=(optimizer, lr_scheduler)parameter and run the code, it does not crash in the middle of epochs.Appreciate any help!
2021-11-05T09:45:35Z
Hi, anzamanI am stuck on the same problem as you do. Have You fixed this one yet?This is mine error.image1480×565 44 KB##########UPDATEI found a way to trick it by adding logging_steps=9999999 in the trainer argument to make it not log between training , but is there any better solutions for this one.
2022-02-11T17:11:57Z
your trick worked for me too.  do you know if there are any updates that allow using adaFactor in training and successfully log without trippingTypeError: Object of type Tensor is not JSON serializable
2022-08-20T19:27:41Z
I have the same problem.
2023-06-02T14:41:19Z
I ran into an issue like this when I inadvertently logged a metric in the trainer that was a Tensor instead of just a float. I did something likeaverage_scores.mean(), which returned a Tensor object, which was somehow affixed toTrainer.stateand then broke json-saving. I changed the metric computation toaverage_scores.mean().item()and this fixed the issue.