BOS tokens for mBERT tokenizer
Default mBERT tokenizes a sentence as['[CLS]', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']. I want to change this behaviour and add a language specific token after theCLStoken like this:['[CLS]', '__en__', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']I knowTemplateProcessingcan be used to achieve this if the language token doesn’t changefrom tokenizers.processors import TemplateProcessing
tokenizer._tokenizer.post_processor = TemplateProcessing(
    single=f"{_lang_token} $A [SEP]",
    pair=f"{_lang_token} $A [SEP] $B:1 [SEP]:1",
    special_tokens=[("[SEP]", tokenizer.convert_tokens_to_ids("[SEP]")), 
                    (_lang_token, tokenizer.convert_tokens_to_ids(_lang_token))],
)But in my case, the language token changes with every batch. What is the best way to add these tokens? CreatingTemplateProcessingobjects every time seems inefficient.
2021-04-14T10:13:07Z
@sguggerany suggestions?