Initialize Vocabulary for Unigram Tokenizer
I have reviewed the documentation and code, but I was unable to find the specific method used for initializing the set of tokens in the Unigram tokenizer. Typically, it is initialized by running a BPE tokenizer, but I would like to confirm this information. If anyone could provide a link to the code or documentation or offer an answer regarding this matter, it would be greatly appreciated.