Train BERT with sentence embeddings
Hi,I’m trying to use calculated sentence embeddings by average pooling of chunks of a long sentence as input to train a model based on the AutoModelForSequenceClassification class. I used the “inputs_embeds” parameter to pass the embeddings to the model, but something strange is happening. Metrics do not change over time. These are the values that practically remain in the 30 epochs:{‘eval_loss’: 0.48057085275650024,‘eval_f1’: 0.3008849557522124,‘eval_roc_auc’: 0.5,‘eval_accuracy’: 0.0,‘eval_precision’: 0.17708333333333334,‘eval_recall’: 1.0,‘eval_hammingLoss’: 0.8229166666666666,‘eval_runtime’: 0.7474,‘eval_samples_per_second’: 149,856,‘eval_steps_per_second’: 149,856,‘epoch’: 30.0}Does anyone have any tips on how to train BERT using embeddings as input?