code: def tokens_to_indices(self,
                            tokens: List[Token],
                            vocabulary: Vocabulary,
                            index_name: str) -> Dict[str, List[TokenType]]:
          
          raise NotImplementedError
docs: Takes a list of tokens and converts them to one or more sets of indices.
          This could be just an ID for each token from the vocabulary.
          Or it could split each token into characters and return one ID per character.
          Or (for instance, in the case of byte-pair encoding) there might not be a clean
          mapping from individual tokens to indices.