Custom DistilBertTokenizer training
Hi AllI am trying to find how to build a custom tokenizer for DistilBert all the examples I saw just use the pre-trained tokenizer.huggingface.coDistilBERT — transformers 3.5.0 documentationCan someone point me to how to build my custom model?Thanks in advance.
2020-11-12T13:03:03Z
Hi, this is probably where you can start if you want to build a fast tokenizer:https://huggingface.co/docs/tokenizers/python/master/quicktour.htmlcc@anthonyand@Narsil
2020-11-13T08:10:13Z
Thanks for the answer!. I will try the example it is a bit different and I wanted to make sure I have something more similar to the DistilBert.Thanks!
2020-11-13T11:48:41Z
I still do not see how to build this model as distilBertTokenizer any ideas?