Hyperparameters LiLT with custom RoBERTa training
After creating my version of LiLT with custom RoBERTa, I’m trying to perform training process over my own dataset for token classification problem.I’m not an expert in hyperparameters. For this  scenario, I’ve been using the following hyperparameters:max_steps=20000,
per_device_train_batch_size=2,
per_device_eval_batch_size=2,
learning_rate=1e-5,
evaluation_strategy="steps",
eval_steps=100This is how my training results are looking like (unfortunately, google colab only saved till step 17500):StepTraining LossValidation LossPrecisionRecallF1Accuracy170000,0030,0203610,9561470,9559050,9560260,996853171000,0030,0202920,9548890,9602130,9575440,996866172000,0030,0200270,9575330,9599590,9587450,99696173000,0030,0203560,957360,9559050,9566320,996849174000,0030,0202750,9591680,9584390,9588030,996934175000,00210,0199030,9625290,9569180,9597150,997002Could I improve those results by tuning the hyperparameters somehow? Perhaps increasing max_steps?
2024-03-25T16:21:08Z
Nice!The Trainer class supports hyperparameter tuning:Hyperparameter Search using Trainer API. This allows you to run various training runs across various settings of hyperparameters.
2024-03-26T08:25:00Z
What a nice feature! I’ll definitely dig into that.Just another quick question related to my training loss and validation loss chart. Doesn’t it show a tiny underfitting? The little I know about training results is, at times, when the validation loss is greater than the training loss may indicate that the model is underfitting.But I don’t really know when the difference between validation loss and training loss starts being significant.