Is there a possibility to use MLM modelling for pretraining for autocasualLM model like MPT or falcon? If yes, Has someone tried it? Are there any relevant code bases which I can use?
I wanted to use Masked language modelling as method to pretrain text on domain specific tasks. But for open source LLMs like MPT - They only work on autocasualLM. Is there  a way to do MLM on MPT 30B?