Token Offsets in Rust vs. Python
Hi,I’m working on Scala bindings for the (Rust) Tokenizers library. While writing tests, I realized that the offsets for certain strings seem to differ when using the Python bindings and when calling the underlying Rust implementation directly.This can also be seen directly in thequicktour docswhen looking at the offsets of a “”:print(output.offsets[9])
# (26, 27)println!("{:?}", output.get_offsets()[9]);
// (26, 30)I suspect the difference is due to how strings are represented as bytes vs. characters, i.e. callinglen()on a ruststrreturns the number of bytes:let input = "😁";
println!("{}", input.len());
// 4There’s alsoOffsetTypewhich might be related but I haven’t been able to figure out yet where this is handled on the Python side.Any hints appreciated. Thanks!
2023-04-27T20:17:11Z
I just realized that there’s alsoencode_char_offsetsand the theencodemethod in Python calls it.