Unable to register my own tokenizer
I have tried to register my own tokenization model based on sentencepiece using CustomAITokenizer.register_for_auto_class(“AutoTokenizer”) . But I am falied to do so.RANITBAG/CustomAItokenizer at main. This the repo link. Can anyone help me in this?