Why does tokenizer.apply_chat_template() add multiple eos tokens?
TheLLaMA-3.1-8B-Instruct modeluses the chat template defined below:"chat_template": "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}"To the best of my knowledge, the segment below adds aneos_tokento the end of every conversation turn:{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}This is my first time working with multi-turn conversation data, and I am wondering why aneos_tokenis added to the end of every turn. Wouldn’t training on data like this give the model a mistaken understanding that text can be generated even after theeos_token?Or does this not matter during inference because the LLM programmatically cuts generation once theeos_tokenhas been generated?
2024-09-12T14:47:45Z
The models have been trained with the EOS token positioned in this way to create the illusion of a multi-turn conversation. Therefore, sticking to the prompt template used during the instruction finetune will cause the model to also stick to its expected behaviour.The EOS token then effectively helps the model to understand that whomever was speaking is now done.In some examples I have found that omitting the EOS token in my query caused the model to attempt tocompletemy query. Whereas adding the EOS token caused the model toreplyto my query.As with all things, experiment an see what happens.
2024-09-13T02:20:58Z
@swtbIt’s interesting that you noticed this difference:swtb:In some examples I have found that omitting the EOS token in my query caused the model to attempt tocompletemy query. Whereas adding the EOS token caused the model toreplyto my query.I’ve been processing what you’ve said, and your explanation that theeos_tokenof theLlama-3.1-8B-Instructmodel serves a different purpose from that of theLlama-3.1-8Bbase model does make intuitive sense to me.As you’ve suggested, it would seem that structuring multi-turn conversations in this way would induce the model to learn that theeos_tokenmarks the end of a turn, rather than the end of a model generation.I’ve begun thinking that this may be why the base model and instruct models have differenteos_tokenby default. The base model has theeos_tokenset to:While the instruct model has theeos_tokenset to:Perhaps this was an intentional choice by Meta, so as to avoid“overwriting”the information the Llama model had learned about the<|end_of_text|>token? By setting a differenteos_tokenand ensuring that thechat_templatemade use of<|eot_id|>, perhaps they were able to preserve what was previously learned about the<|end_of_text|>token while inducing the behavior they desired.If this interpretation seems off in any way, please let me know!
2024-09-19T12:00:27Z
@ChahnwooI think you are on the right lines. If I wanted to train certain behaviour into a model using special tokens and markers I would definitely use a new token rather than reusing and old one.We for sure want to leverage the base models language understanding for the instruction tuning. Almost as an additive task that relies on the deeper knowledge in the pretraining.
2024-09-20T00:07:27Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.