Tokenizer from own vocab

🤗Tokenizers
1

Hi everyone.

I would like to build a tokenizer that uses as vocabulary a list of pre-defined tokens I have. Would that be possible with the Transformers library and, in that case, how could I do it?

Thanks.

1 Like