Longformer model evaluating all the datapoints to negative on evaluation data set after training
I am using transformers library to download longformer-base-4096 model and training on my own dataset for classification using Trainer API. So, during evaluation, all the data is categorized to negative class. I am suspecting gradient vanishing problem. If anyone can respond to how this can be handled. Below are the training arguments which are passed to Trainer class.training_args = TrainingArguments(output_dir=’./results’,do_train=True,do_eval=True,num_train_epochs=1,per_device_train_batch_size=1,per_device_eval_batch_size=1,warmup_steps=500,weight_decay=0.01,evaluate_during_training=True,logging_dir=’./logs’,)
2021-06-09T18:50:53Z
@Harikawere you able to handle the vanishing gradient problem? If yes how?