Best way to get the closest token indices of input of char_to_token is a whitespace
For many text datasets, often the character spans start or end at a whitespace, and if you put that into char_to_token , it will return None since the whitespace does not correspond to a token.Is there a way to get the closest token on either side of the white space? Or an indicator if the character span is out of range [in cases where chunking and striding where the character span may not be in the overflow tokens],I tried to see if there’s some variable in the class or function that I can play around with in the source codegithub.comhuggingface/transformers/blob/v4.26.1/src/transformers/tokenization_utils_base.py#L537raise ValueError("token_to_chars() is not available when using Python based tokenizers")if token_index is not None:batch_index = batch_or_token_indexelse:batch_index = 0token_index = batch_or_token_indexspan_indices = self._encodings[batch_index].token_to_chars(token_index)return CharSpan(*span_indices) if span_indices is not None else Nonedef char_to_token(self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0) -> int:"""Get the index of the token in the encoded output comprising a character in the original string for a sequenceof the batch.Can be called as:- `self.char_to_token(char_index)` if batch size is 1- `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1But it seems that it goes into the RUST part of the tokenizers library.My work around is to convert every character in the span to token indices and then take the max and min.