Is detokenize available in transformer lib?
I’ve searched on doc but couldn’t find any hint.Generally,detokenizeis the inverse of thetokenizemethod, and can be used to reconstrct a string from a set of tokens.from transformers import TFBertTokenizer

tf_tokenizer = TFBertTokenizer.from_pretrained("bert-base-uncased")

# something like
tf_tokenizer.encode([string]) # o/p: ids / token
tf_tokenizer.decode([1,2,3]) # o/p: stringAvailable in some extentgithub.comkeras-team/keras-nlp/blob/b35e83fbc1/keras_nlp/models/bert/bert_tokenizer_test.py# Copyright 2023 The KerasNLP Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests for BERT tokenizer."""

import os

import pytest
import tensorflow as tf
from absl.testing import parameterizedThis file has been truncated.show originalgithub.comtensorflow/text/blob/dc5983e1be3140130f4eb70e981c7cb0fecadd85/tensorflow_text/python/ops/tokenization.py#L179...   def detokenize(self, input):...     return tf.strings.reduce_join(input, axis=-1, separator=" ")>>> text = tf.ragged.constant([["hello", "world"], ["a", "b", "c"]])>>> print(SimpleDetokenizer().detokenize(text))tf.Tensor([b'hello world' b'a b c'], shape=(2,), dtype=string)"""__metaclass__ = abc.ABCMeta@abc.abstractmethoddef detokenize(self, input):  # pylint: disable=redefined-builtin"""Assembles the tokens in the input tensor into a string.Generally, `detokenize` is the inverse of the `tokenize` method, and canbe used to reconstrct a string from a set of tokens.  This is especiallyhelpful in cases where the tokens are integer ids, such as indexes into avocabulary table -- in that case, the tokenized encoding is not veryhuman-readable (since it's just a list of integers), so the `detokenize`method can be used to turn it back into something that's more readable.Args:
2023-03-27T12:40:29Z
cc@Rocketknight1
2023-04-24T13:09:32Z
Hi@innat, and sorry for the delay! I don’t think our TF in-graph tokenizers support decoding/detokenization. However, our main tokenizers do. So you could do something likefrom transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

tokenizer.decode([1, 2, 3])This should work for most purposes - do you have a usecase for wanting to dodetokenization inside a TF graph? We’re very interested if so, because we assumed people would generally not need that!