Costumizing MASKed tokens
Hello,I want to further pre-train a RoBERTa model using a customized set of masked tokens (e.g., masking a percentage of high information words or never masking stopwords). Is there a way to do this?This post asked a similar question, but there is no answer:How to pre-train a model using a custom mask strategy?🤗TransformersIs there any way to pre-train a model (i.e. RoBERTa), using my own corpus, but with a custom mask strategy? For instance, I want to look up a dictionary to choose which words to mask.
2023-09-27T18:23:52Z
Were you able to solve it?