Tokenizer.encode not returning encodings
How do you get token encodings? The method isn’t working in this case.I need the token offsets in order to translate my labels to a normal list of token tags (my labels are in this format [{start_index: int, end_index: int, tag: str} … ]Thank you!Maybe it’s because this is a pre-trained fast tokenizer?
2021-10-08T14:03:13Z
You need to use the tokenizer directly on your text, not theencodemethod:tokenizer("hello world")
2021-10-09T02:03:14Z
Thank you Sylvain! That worked, I also had to pass return_offset_mapping=Trueimage1227×202 26.2 KB