Xlm-Roberta Tokenizing
Hello everyone. How would you solve the following problem:I have  a word and a sentence with this word, I also have the position of this word. How would you universally for all languages find the position of a tokenized word in a tokenized sentence?I really have a problem with this one, it’s mainly with a Chinese language
2021-01-19T05:59:50Z
cc@Narsil,@anthony
2021-01-19T08:52:13Z
Hi@Seva,If you usetokenizers(which are FastTokenizers withintransformers.) then you have offsets whenever you callencode(if you usetransformersyou need to actually calltokenizer._tokenizer.encodeto call the underlyingtokenizersobject).tokenizer = AutoTokenizer.from_pretrained('roberta-large')

encoded = tokenizer._tokenizer.encode("This is a test")
encoded.offsets
# [(0, 0), (0, 4), (5, 7), (8, 9), (10, 14), (0, 0)]
# <s>,  "This", "is", "a", "test", </s>So from that you should be able to recover your information.This information is available throughFast Tokenizermethod calls too.
2021-01-19T17:24:03Z
You can have a look atthe doc forBatchEncodingintransformerswhich is the type returned from the tokenizers methods (encode_plus,batch_encode_plus,__call__).There are a bunch of helpers available to easily map between characters, words, and tokens.