Issues with BPE tokenizer
I attempted to train the BPE Hugging Face tokenizer for Pashto from scratch, but it is not decoding words correctly.
2024-01-24T15:56:05Z
No one can help you with such limited information.I am unfamiliar with Pashto. Does the official documentation say anything about BPE?Is it possible that Pashto has already been pre trained with a different tokenizer?Post your code so we can understand what the software is trying to tell you.
2024-01-24T16:09:38Z
I solve the issues.