Issue with tokenizer.tokenize
when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong
2020-11-05T08:19:12Z
There is some discussion in thistheradandthis. Perhaps it helps?
2020-11-16T19:21:35Z
See alsothis postin the forum.
2020-11-16T19:23:36Z
where is the wrongThat’s a new one. I haven’t seen that expression before. LOL