Increasing validation loss even with small learning rate - RoBERTa
I am using RoBERTa viaTrainer.hyperparameter_search(transformers 3.5.x) to optimize a couple model params. For some reason, even at very small learning rates (e.g.,5e-6), myvalidation_lossincreases epoch-over-epoch.def hyperparameter_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-3, log=True),
        "other_weight": trial.suggest_float("other_weight", 0.1, 1.0, log=False)
    }

hp_search_output = my_trainer.hyperparameter_search(
    hp_space=hyperparameter_space, 
    direction="minimize",
    study_name = "2021-02-26_roberta_3_epochs",
    n_trials=100,
    compute_objective=lambda x: x["eval_loss"],
)Batch size is 16 with a total of ~80,000 input arrays of length 512.Any suggestions for fixing/investigating this?FWIW, this is similar to this unanswered question here:Increasing loss during LM fine-tuning on custom datasetModelsHello, 
During the fine-tuning of camembert-base on a custom dataset using the Trainer API, the loss starts to increase quickly (compared to its value at step 0) and then decreases slowly, without ever reaching its value before training. What’s stranger is that lowering the learning rate results in a higher peak, whereas I expected the opposite to happen. Has anyone experienced a similar issue? 
For information, the dataset is quite small (35k items of 15-300 words each for training, 6k for vali…