tokenizer
#24
by
ulasarikaya
- opened
there isn't a specific ModernBertTokenizer like there is for DistilBert (transformers.DistilBertTokenizer).
could anyone clarify if there's an equivalent tokenizer available for ModernBert, or do I have to use a generic one like AutoTokenizer?
thanks
It uses the PreTrainedTokenizerFast tokenizer. You can see this in tokenizer_config.json. Auto tokenizer will choose that tokenizer class.