Writing custom tokenizer and wrapping it in tokenizer object
how to write my own tokenizer and wrap it in huggingface tokenizer object?we will follow this procedure instead of using any pretokenizer and get a tokenizer object like if we use one pretrained tokenizer?huggingface.coByte-Pair Encoding tokenization - Hugging Face NLP CourseWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2023-06-25T08:25:15Z
Did you find a solution?
2023-06-26T00:16:21Z
No, Nobody seems to be interested in it. and I am kind of disappointed. let me know if you find something. I am still stuck on this issue.