How can I check the implementation of tokenizer.decode()
tokenizer = AutoTokenizer.from_pretrained("../../../Models/Qwen/Qwen2-7B-Instruct")  # tokenizer: Qwen2TokenizerFast
input = "如何学习大模型"
input_ids = tokenizer.encode(input)  # 将文本编码为 ID
ids_tokens = {tokenizer.convert_ids_to_tokens(input_id): tokenizer.decode([input_id]) for input_id in input_ids}
print(type(tokenizer))
print(ids_tokens)

>>>
<class 'transformers.models.qwen2.tokenization_qwen2_fast.Qwen2TokenizerFast'>
{'å¦Ĥä½ķ': '如何', 'åŃ¦ä¹ł': '学习', 'å¤§': '大', 'æ¨¡åŀĭ': '模型'}Hi!When I use the above code and output the result, I see ‘‘å¦Hä1/2K’’: ‘how’’ in the ids_tokens, and I’m confused, so I debug tokenizer.decode() and want to see how ‘å¦Ĥä1/2K’ is converted to ‘how’,But I see that there is no implementation of decode() in transformersmodelstokenization_qwen2_fast.py, and there is no implementation of _decode(), how can I check the implementation of tokenizer.decode().Thank you in advance🤗
2024-09-30T03:07:14Z
As is the case with most libraries,class inheritanceis in place here so that methods and elements of the inherited class are also available. That means you have to go back and debug parent classes and even more parent classes. Like this.# https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2/tokenization_qwen2_fast.py
class Qwen2TokenizerFast(PreTrainedTokenizerFast):# https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_fast.py
class PreTrainedTokenizerFast(PreTrainedTokenizerBase):…Maybe, but it’s faster to go find a code that is working well, find the difference between it and your code, and then do some trial and error. It could be a model-dependent problem. In this case, I wonder if the decoding part is suspect.I don’t even know what the correct code is.It shouldn’t be a character code issue like it used to be, but there is a slight quirkiness to each tokenizer.
2024-09-30T07:21:14Z
Thank you so much!I looked at the parent class and didn’t find an implementation of _decode(), only found an unimplemented _decode() in SpecialTokensMixin() of PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin) with the following code:def _decode(
        self,
        token_ids: Union[int, List[int]],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        **kwargs,
    ) -> str:
        raise NotImplementedErrorI see that PreTrainedTokenizerFast is a “Fast” implementation based on the Rust libraryTokenizers, and I’ll take another look at the Tokenizers code
2024-09-30T07:36:27Z
You trace back to that further parent classes. You’ll get to the ancestor class someday.There may be a nice feature on github, but I don’t use it, so I don’t know about it.
2024-09-30T07:42:51Z
I haven’t heard of it，so could you tell me the name of this feature on github?
2024-09-30T08:00:19Z
No, I just thought it might be possible.We’ll have to do it this way in Python.Python documentation9. ClassesClasses provide a means of bundling data and functionality together. Creating a new class creates a new type of object, allowing new instances of that type to be made. Each class instance can have ...GeeksforGeeks – 15 Feb 24Built-In Class Attributes In Python - GeeksforGeeksA Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.print(ClassName.__bases__)orstackoverflow.comInspect python class attributespython, class, attributes, introspectionasked byJakob Simon-Gaardeon12:00AM - 22 Nov 10 UTCPython documentationinspect — Inspect live objectsSource code: Lib/inspect.py The inspect module provides several useful functions to help get information about live objects such as modules, classes, methods, functions, tracebacks, frame objects, ...
2024-09-30T08:12:55Z
Thanks for the reply, I will try what you say