Issues with offset_mapping values
Hi guys, I am trying to work with a FairSeq model converted tobut I have some issues with tokenizer. I am trying to fine-tune it for POS tagging so I have the text split to words already and I want to use the offset_mapping to detect first token for each word. I do it like this:tokenizer = RobertaTokenizerFast.from_pretrained('path', add_prefix_space=True)
ids = tokenizer([['drieme', 'drieme'], ['drieme']],
    is_split_into_words=True,
    padding=True,
    return_offsets_mapping=True)The tokenization looks like this:['<s>', 'Ġd', 'rieme', 'Ġd', 'rieme', '</s>']But the output from the command looks like this:{
  'input_ids': [[0, 543, 24209, 543, 24209, 2], [0, 543, 24209, 2, 1, 1]],
  'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]],
  'offset_mapping': [
    [(0, 0), (0, 1), (1, 6), (1, 1), (1, 6), (0, 0)],
    [(0, 0), (0, 1), (1, 6), (0, 0), (0, 0), (0, 0)]
  ]
}Notice the offset mapping for the worddriemein the first case. First word has mappings(0, 1)and(1, 6). This looks reasonable, however the seconddriemeis(1, 1)and(1, 6). Suddenly, there is1at the first position. This1is there for all but first word for any sentence I try to parse. I feel like it might have something to do with handling the start of the sentence vs all the other words, but I am not sure how to solve this issue so that I have proper offset mappings. What am I doing wrong?
2021-03-09T23:11:35Z
Thanks for reporting, it’s definitely a bug. Could you open an issue ontokenizerswith your snippet?
2021-04-10T08:11:16Z
Having similar issue as well. Is there a proper fix yet?
2021-04-10T08:26:10Z
Don’t know about the fix, but I solved it by detecting the special character that is used at the start of each word. In my case this was Ġ. I used this instead of offset mapping to detect the start of the word.
2022-02-15T13:13:18Z
Hello,Is there any update on this topic? I am encountering the same issue with any tokenizer that has theadd_prefix_space=Trueflag, which seems to be needed for example for RoBERTa tokenizers. A few more examples to exemplify the issue, usingwords = ['hello', 'world', 'foo', 'bar', '32', '31290485']:Using a non-RoBERTa tokenizer doesn’t show the issue and works as expected:tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer(words, is_split_into_words=True, return_offsets_mapping=True)
# This returns:
# 'input_ids': [101, 7592, 2088, 29379, 3347, 3590, 21036, 21057, 18139, 2629, 102]
# 'offset_mapping': [(0, 0), (0, 5), (0, 5), (0, 3), (0, 3), (0, 2), (0, 3), (3, 5), (5, 7), (7, 8), (0, 0)]
# Running tokenizer.add_prefix_space would raise an AttributeErrorin the above, as expected, special tokens have offsets (0, 0), beginning of words are (0, n), with n being the word/chunk length, continuations of words are of the form (n, m) withn > 0being the start of the chunk, andm - nthe chunk length.However, for RoBERTa gives the following:tokenizer = AutoTokenizer.from_pretrained('distilroberta-base', add_prefix_space=True)
tokenizer(words, is_split_into_words=True, return_offsets_mapping=True)
# This returns
# 'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 2]
# 'offset_mapping': [(0, 0), (0, 5), (1, 5), (1, 3), (1, 3), (1, 2), (1, 3), (3, 5), (5, 8), (0, 0)
tokenizer.add_prefix_space
# This returns TrueThe special tokens are still marked as (0, 0) offset, the first word is still of the form (0, n), the chunks from second on are still of the form (n, m), but each word beginning after the second is now of the form (1, n), with n still being the word/chunk length. So this looks to be the only exception to the rule that the length of the chunk ism - n, and that’s not good. I suppose the beginning1is because, as the flag name suggests, a space is added before the words and not taken into account when running the offsets_mapping, but as suggested above that looks like a bug.For my purpose (running NER) I can probably workaround the above by checking explicitly if the word is not the first word in the sentence, but the situation gets weirder when the chunks should be just 1 character long (so that, with the above logic, the offsets_mapping should then be something like (1, 1), (1, 2)). For example, usingwords = ['hello', 'world', 'foo', 'bar', '32', '31290485', 'üù'], I am getting:{'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 952, 4394, 3849, 9253, 2],
'offset_mapping': [(0, 0), (0, 5), (1, 5), (1, 3), (1, 3), (1, 2), (1, 3), (3, 5), (5, 8), (1, 1), (0, 1), (1, 2), (1, 2), (0, 0)]}contradicting what I was expecting from above; I guess in this case, first the space that is implicitly added beforeüùgets the offset (1, 1), and then the letterü(which doesn’t have any space before it anymore) correctly gets the offset (0, 1), but then somehow the offset forù(which should indeed be (1, 2)`, being the second half of a two-character word) gets… doubled?Edit: the weirdness in the last part seems to be explained by how the tokenizer works on Unicode characters, encoding them by piece. Soüis actually encoded asÃ¼andùis encoded asÃ¹, as explained by runningtokenizer.convert_ids_to_tokenson the output. It is made all a bit cleaner by runningtokenizer(' '.join(words))and looking at its result, where indeed the tokens are duplicated, but also the offsets are correctly duplicated:{'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 952, 4394, 3849, 9253, 2],
'offset_mapping': [(0, 0), (0, 5), (6, 11), (12, 15), (16, 19), (20, 22), (23, 26), (26, 28), (28, 31), (32, 33), (32, 33), (33, 34), (33, 34), (0, 0)]}I guess that for now I’ll stick to usingtokenizer(' '.join(words))instead oftokenizer(words, is_split_into_words=True)…