Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library
I can’t seem to create a “PreTrainedTokenizerFast” object from my originaltokenizerstokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a [PAD] token, and does not have any special tokens at all.tokenizer = ByteLevelBPETokenizer()
    tokenizer.preprocessor = pre_tokenizers.BertPreTokenizer()
    tokenizer.normalizer = normalizers.BertNormalizer()
    tokenizer.train_from_iterator(docs, vocab_size=16_000, min_frequency=15, special_tokens = ["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
    tokenizer._tokenizer.post_processor = processors.BertProcessing(
        ("[SEP]", tokenizer.token_to_id("[SEP]")),
        ("[CLS]", tokenizer.token_to_id("[CLS]")),
    )
    tokenizer.enable_truncation(max_length=256)
    tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")
    fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)The result of printing the fast_tokenizer is:PreTrainedTokenizerFast(name_or_path='', vocab_size=16000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={})Whichmodel_max_lenandspecial_tokensare wrong in it. Also, there is no pad_token and pad_token_id in the fast_tokenizer object. (the warning for pad_token for example:Using pad_token, but it is not set yet.) Have I done anything wrong, or is this not supposed to happen?The versions of libraries I’m using:'tokenizers                    0.10.3',
'transformers                  4.10.0.dev0'I have also tested with these versions:'tokenizers                    0.10.3', 
'transformers                  4.9.2'
2021-08-30T18:15:09Z
When usingPreTrainedTokenizerFastdirectly and not one of the subclasses, you have to manually set all the attributes specific to Transformers: themodel_max_lengthas well as all the special tokens. The reason is that theTokenizerhas no concept of associated model (so it doesn’t know the model max length) and even if it has a concept of special tokens, it doesn’t know the differences between them, so you have to indicate which one is the pad token, which one the mask token etc.