How to train your own corpus without labels
Hi, I’d like to fine-tune BERT for my product catalog corpus which contains a lot of out-of-vocabulary words like brand names. By fine-tuning, I mean transfer learning and not training from scratch.I have been following this [Fine-tuning a pretrained model — transformers 4.5.0.dev0 documentation] tutorial and see that this requires labels. As you can imagine, my use case if connected to information retrieval and search and does not contain any y_labels. All I want is unique vector embedding out of my trained model.How should I approach this problem using HF Trainer module?
2021-05-25T11:32:07Z
hey@awaiskaleemif i understand correctly, i that what you’re looking for is to fine-tune the language model on your corpus. this will generally produce mask-filling that more accurately captures the relations in your corpus and for BERT you can check out theMasked language modelingsection of this tutorial:Google Colaboratoryonce the language model is fine-tune, you can save the weights and then load them usingAutoModel.from_pretrainedto generate your embeddings.
2021-05-25T13:05:11Z
Thanks! That’s what we were looking for.