Can we directly use the embeddings from masked language models?
Hello there,I have a short conceptual question. I know can train amasked language modelfrom scratch. By doing so withhuggingface, I should be able to obtain a model that is very good at … filling the[mask]token!But what about theembeddings? are they any good for clustering for instance? Note that I amNOTfine-tuning the MLM model in any way. I am only interested in the embeddings that come from the MLM task itself.Any suggestions or papers greatly appreciated.Thanks!