Train wordpiece from scratch
Hi,I am pre training a Bert model from scratch. For that I first need to train a wordpiece tokenizer, I am using BertWordPieceTokenizer for this.My question:Should I train the tokenizer on the whole corpus which is huge, or training it on a sample is enough?Is there a way to tell the tokenizer to take train only on a sample?Thanks.
2021-09-09T11:41:14Z
Kamel:Should I train the tokenizer on the whole corpus which is hugeYes. WithHuggingFace Tokenizers, it takes seconds. From the README: “Takes less than 20 seconds to tokenize a GB of text on a server’s CPU”.
2021-09-09T11:56:35Z
Thanks again Nielsr