Use a pretrained ByteLevelBPETokenizer on text
HiI am asking whether there’s a simple way to tokenize a piece of text “I will go to the bedroom” to BPE " I will go to the bed ##room" without training a tokenizer from scratch.
2020-07-17T14:20:26Z
Hi@abdallah197!There is a bunch of pre-trained tokenizers in thehuggingface/transformerslibrary that you can use directly, without having to train anything. You won’t have any control over how the tokens are split though, as this is based on what the tokenizer learned during training, and the size of its vocabulary.bedroomisn’t really a rare word, so often it will have its own token in the vocabulary.Your example looks like a WordPiece (instead of BPE), given the##in##roomwhich is very specific to this kind of tokenizers. You can try to use those fromBERTin the library to see if anything fits your needs, for example with:from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

tokenizer.tokenize("I will go to the bedroom")
# ['I', 'will', 'go', 'to', 'the', 'bedroom']

tokenizer.tokenize("I will go to the Bedroom")
# ['I', 'will', 'go', 'to', 'the', 'Bed', '##room']