WordLevel Tokenization with GPT2?
I am thinking of pretraining a GPT2 model from scratch where the tokens must be of whole words (e.g. using theWordLeveltokenizer) instead of subwords (e.g. using theByteLevelBPETokenizertokenizer). This is because subwords don’t make sense for my application.Will theGPT2LMHeadModelandGPT2Tokenizerbe able to accept word level tokenization?Because aTokenizer(WordLevel())tokenizersaves to a single json filetokenizer = Tokenizer(WordLevel())
tokenizer.train(files=["words.txt"])
tokenizer.save('wordlevel.json')but theGPT2Tokenizerreads from 2 filesand is based on a byte-level Byte-Pair-Encoding.tokenizer = GPT2Tokenizer(
    os.path.join("vocab.json"),
    os.path.join("merges.txt"),
)If byte-level byte-pair  encoding must be used, canByteLevelBPETokenizerbe configured to do word level tokenization?Thanks
2023-03-26T13:01:33Z
Hey Athena,I also want to generate tokens that must be whole words and I have the same problem you had 2 years ago! Did you end up figuring out a way to solve this problem? I know it has been a long time but it would really help me out a lot!Thanks in advance!