What are some recommended pretrained models for extracting semantic feature on single sentence?
Hi, I am more a CV guy and recently get interested in doing a nlp project.In this project, one part might involve extracting sentence-level semantic representation from a pretrained model.In computer vision, one standard way to extract feature of an image or a video snippet could beusing Resnet pretrained on Imagenet or I3D pretrained on Kinetics datasets, respectively.I want to do the similar thing but in nlp domain. I wonder if there are some recommended models pretrained on specific dataset for me to try?As far as my limited understanding, models trained on datasets which aim to to tell if two sentences are semantically equal could be a direction (e.g. QQP, STS-B ). But it needs a pair of sentences, my case is just feeding one sentence (or one block of sentences), not in a pair format. Any suggestion? Thanks!
2020-12-12T04:59:22Z
Hi! IMO, Bert could be comparable to ResNet as the baseline. (you can uselast_hidden_statevariable ofBertModeljust like the global-pooled features of ResNet) Then, newer models like Roberta and many more could be comparable to EfficientNet etc.
2020-12-12T08:29:45Z
Seems like you are looking for the Sentence Transformers library which trains Siamese BERT (etc.) networks on NLI data. That means that you can indeed pass one sentence to get a sentence embedding. They also have a few finetuned models that use cross-encoders instead. Those are obviously slower but lead to better performance on downstream tasks such as STSb.github.comUKPLab/sentence-transformersSentence Embeddings with BERT &amp; XLNet. Contribute to UKPLab/sentence-transformers development by creating an account on GitHub.
2020-12-12T16:04:53Z
Thanks for reply. And it seems sentence-BERT , LaBSE, and Universal Sentence Encoder are other some choices for sentence embeddings.
2020-12-14T03:57:06Z
Benchmark-wise speaking, I have some new idea : sinceSuperGLUEis one of the  most difficult (multi-)task on language understanding. And since T5 is the current SOTA on this benchmark so we can also try embedding vectors from T5.Previously, this may not be straightforward to extract (since T5 is encoder-decoder), but the latest master version of Huggingface now containsT5 encoder’s only modelwhich we can directly extract the vector of the pretrained model. (Thanks to@agemagician) … So this is interesting choice IMO