Padding not transferring when loading a tokenizer trained via the tokenizers library into transformers
Reposting this here from thetransformers forumbecause I got no answer there:Hi,I trained a simple WhitespaceSplit/WordLevel tokenizer using thetokenizerslibrary. I added padding by callingenable_padding(pad_token="<pad>")on the Tokenizer instance. Then I saved it to a JSON file and then loaded it into transformers usingthe instructions here:fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")When using thetokenizers.Tokenizerobject directly,encodecorrectly adds the padding tokens. However, if I try padding when tokenizing using thePreTrainedTokenizerFastinstance, I get the exception:ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.Sure enough, if I follow the instructions and add the pad token as a special token, it works. But I want the tokenizer to workout of the box, exactly as the equivalenttokenizer.Tokenizerinstance does, including in terms of padding behavior.Why is this not the case? Why do I have to enable padding for thetokenizer.Tokenizerinstance, and thenagainfor thePreTrainedTokenizerFastinstance? Am I doing something wrong or missing something?To reproduce the problem, you can use the code below. Most of it is from thetokenizers Quicktour, so you’ll need to download the data files as per the instructions there (or modifyfilesif using your own files). The rest is from the official transformers docs onhow to load a tokenizer fromtokenizersintotransformers:from tokenizers import BpeTrainer, Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from transformers import PreTrainedTokenizerFast

files = [f"data/wikitext-103-raw/wiki.{split}.raw" for split in ["test", "train", "valid"]]
sentences = ["Hello, y'all!", "How are you 😁 ?"]

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.pre_tokenizer = Whitespace()

tokenizer.train(files, trainer)

# Enable padding
tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")

# Now use this tokenizer to tokenize a couple of sentences.
output = tokenizer.encode_batch(sentences)

# The output is padded, as it should be:
print(output[0].tokens)
# ['Hello', ',', 'y', "'", 'all', '!']
print(output[1].tokens)
# ['How', 'are', 'you', '[UNK]', '?', '[PAD]']

# But now let's say we load the tokenizer into transformers- let's try loading it directly from the tokenizer object:

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)

# Tokenize two strings of different token length with padding
fast_output = fast_tokenizer(sentences, padding=True)This gives us the error:Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2548, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2634, in _call_one
    return self.batch_encode_plus(
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2816, in batch_encode_plus
    padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2453, in _get_padding_truncation_strategies
    raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.We can resolve the issue by explicitly specifying the special tokens when initializing thePreTrainedTokenizerFast:fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer,  pad_token="[PAD]", unk_token="[UNK]")

# Now padding works as expected
fast_output = fast_tokenizer(sentences, padding=True)

print(fast_output[0].tokens)
# ['Hello', ',', 'y', "'", 'all', '!']
print(fast_output[1].tokens)
# ['How', 'are', 'you', '[UNK]', '?', '[PAD]']The code above uses thetokenizer_objectparameter to load the fast tokenizer as aPreTrainedTokenizerFastinstance, but as you can confirm for yourselves, the same behavior occurs if you first save the tokenizer to file, then load it intoPreTrainedTokenizerFastusing thetokenizer_fileparameter instead.Bottom line:I don’t understand, if the padding information is already in the tokenizer (or in the saved tokenizer config file), why I should need to explicitly specify the padding token again when transferring the tokenizer. This introduces a lot of totally unnecessary friction into what should be a painless process. The tokenizer object/config should be self-contained. I should not have to re-hardcode what the padding token is in the code that loads it intotransformers, if that information is already encapsulated in the tokenizer object or its saved config file, any more than I should need to specify the vocab file, or the pretokenizers to use, etc. That’s the whole point of the tokenizer object/config file: to uniquely determine the behavior the tokenizer.Am I doing something wrong, or is this just how this works?