Using Roberta for Sentence2Vec
Hey, I’ve been trying to train Sentence2Vec embeddings and I’ve been wondering what do you think about my approach.I would be glad to learn about different possible pitfalls in my approach and how to solve them.What do I have?I have a small unique corpus of about 4 million sentences in my test languageI have a smaller subset (150k) of labeled sentences of whether the sentence is toxic or not.For this question’s sake assume there isn’t any pretrained model or another existing corpus I can use.What am I trying to achieve?The target is to be able to cluster sentences that have similar meanings together.Way of ActionTrain a RoBERTa language modelFine tune it for classification of toxic or notUse a hidden layer of that classifier as an embedding modelCluster using embeddings and HDBSCAN.That’s about it, I tried to keep it as clear as I can.Thanks for anyone that read up until here!!
2021-04-11T09:42:00Z
What is your main goal? Is it to have a classifier for  toxic vs non-toxic sentences? Or to cluster sentence embeddings by semantic meaning?For creating sentence embeddings I would recommendsentence transformers. It’s an extensions of regular huggingface transformers but optimized for creating text embeddings.
2021-04-11T10:07:53Z
Cool!Will take a look at that!Is there a recommended way to finetune embeddings created bysentence-transofmers?
2021-04-11T10:31:03Z
Yes there is sample code in the repoWould also recommend to check out the repo homepagehttps://www.sbert.net/for good explanations of different use cases.