Documentation of SentencePieceBPETokenizer?
I cannot findSentencePieceBPETokenizerin the official documentation, therefore I ask this question.I refer to thesource codeofSentencePieceBPETokenizerto write my codes to train a custom Tokenizer based onSentencePieceBPETokenizer.Question 1:do the following codes make sense?from tokenizers import SentencePieceBPETokenizer

special_tokens = ["<unk>", "<pad>", "<cls>", "<sep>", "<mask>"]
tokenizer = SentencePieceBPETokenizer()
tokenizer.train_from_iterator(
    get_training_corpus(),
    vocab_size = 400000,
    min_frequency = 10, 
    show_progress = True,
    special_tokens = special_tokens
)whereget_training_corpus()is a function to get sentences in batch from a list of strings.I notice that there areTrainerclasses, but there is noSentencePiece. Also, from theTokenizerdocumentation, it specifies the attributes oftrain_from_iterator()areiteratorandtrainerobject, but my attributes follow theTrainerclass instead.Question 2:No error occurred, can I assume the attributes are correct?Question 3:What is the use ofinitial_alphabetandlimit_alphabet? The documentation does not tell clearly. Any example?p.s. I knowshow_progress = Truedoes not work in Jupyter notebook.p.s. usingtokenizers 0.13.3