Generating sentence embeddings from pretrained transformers model
Hi, I have a pretrained BERT based model hosted on huggingface.huggingface.comicrosoft/SportsBERT · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.How do I generate sentence vectors using this model? I have explored sentence bert but it doesn’t allow you to use custom trained models. I have also seen Bert as a client. It works but for my current scenario, I was wondering if there’s something which could be done without running a server for converting to vectors.
2021-01-22T08:31:42Z
I think the best method is to go with sentence-bert. Indeed, you can use your own model, just try reproducing what they do in thepaper.You add a pooling layer at the output of your model, from the paper:We experiment with three pooling strategies: Using the output of the CLS-token,computing the mean of all output vectors (MEAN-strategy),  and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.Finally, you might want to fine-tune your model for comparison, too:In order to fine-tune BERT / RoBERTa(your model), we create siamese and triplet networks  (Schroff  et  al.,2015) to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity.Hope this help!