Mistral trouble when fine-tuning : Don't set pad_token_id = eos_token_id
Hello everyone,I was working with Mistral Instruct 7B and realized that when fine-tuning it, I had a model that keeps generating undefinitely. While following very common guides and feedbacks, I realized that a common mistake done was to define the pad_token as the eos_token. This leads to a dramatic result as the DataCollator will mask every pad_token to -100 labels.Hence, the fine-tuned model never sees the eos_token and keeps generating undefinitely. Here is the extract of the DataCollator function:labels = batch[“input_ids”].clone()if self.tokenizer.pad_token_id is not None:labels[labels == self.tokenizer.pad_token_id] = -100batch[“labels”] = labelsI would advise to fix it quickly, either by communicating on it or making a difference between pad and eos id using their mask values.So far, my way to do was to set the pad_id to a very large value and modifiy the data collator to replace by -100 this value.Those guides are making the mistake:https://www.youtube.com/watch?v=zbKz4g100SQ&ab_channel=brevhttps://www.youtube.com/watch?v=RzSDdosu_y8&t=18s&ab_channel=PromptEngineeringHope that helps and could raise a PR
2024-04-02T22:38:09Z
Hi. I’m also fine tuning Mistral-7b Instruct model on a mental health dataset (~90k rows) using NVIDIA A100 GPUs. I am running into two issues:1- The model keeps generating text until it reaching the max_new_token limit.2- When it does stop, it stops abruptly, without finishing the sentence/paragraph/word.I am not sure what value to assign to the pad_token_id? Could you please guide me on what value worked for you?
2024-04-03T12:46:48Z
Hello !I did it in three steps:First pre-tokenize and pad the dataset outside of the data collator function with pad_id = eos_tokenApply a function which set the attention mask_id value to 1 for the first eos_token_id in the sequenceTrain the model on the tokenized examples
2024-04-03T15:52:16Z
Hi. Thanks for responding back. Actually I’m not using the data collator function. Here’s my  code snippet.base_model = “mistralai/Mistral-7B-Instruct-v0.2”eos_token=“[/INST]”bnb_config = BitsAndBytesConfig(load_in_4bit= True,bnb_4bit_quant_type= “nf4”,bnb_4bit_compute_dtype= torch.bfloat16,bnb_4bit_use_double_quant= True,llm_int8_enable_fp32_cpu_offload=True)model = AutoModelForCausalLM.from_pretrained(base_model,quantization_config=bnb_configdevice_map=“auto”,)model.config.use_cache = False # silence the warnings. Please re-enable for inference!model.config.pretraining_tp = 1model.gradient_checkpointing_enable()Load tokenizertokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)tokenizer.padding_side = ‘right’tokenizer.add_eos_token = Truetokenizer.max_new_tokens=2000tokenizer.max_length=200tokenizer.max_new_length=200tokenizer.pad_token_id=2041tokenizer.pad_token = tokenizer.unk_tokeneos_token_id=tokenizer.eos_token_idmodel.config.pad_token_id = tokenizer.pad_token_idLet me know your comments. Thanks!
2024-04-07T14:51:56Z
A simple solution is to use another token as the pad token, i.e.tokenizer.pad_token = tokenizer.unk_tokenThat way, you still train on the eos tokens, but not on the paddings.
2024-04-07T15:36:45Z
Rmote6603:1- The model keeps generating text until it reaching the max_new_tokenRmote6603:1- The model keeps generating text until it reaching the max_new_tokenany solution for stoping the model before its max token length
2024-05-18T07:09:30Z
Hey could you a figure out a solution for this? Please let me know if you did, thanks!
2024-08-28T00:45:50Z
That is simple but could result in some unexpected behavior because:Unknown Words:The<unk>(unknown) token is designed to represent out-of-vocabulary words—words that were not seen during training or are not included in the model’s vocabulary.Semantic Meaning:The<unk>token carries a semantic implication that something is unknown, which is different from the purpose of padI recommend creating a new padding token, that way we don’t have to modify any code in the DataCollatorForLanguageModelling class, while ensuring that we get a correct attention mask, ensuring correct generation. Also, don’t forget to update the vocabulary of the model you are using, to prevent CUDA assertion triggers.tokenizer.add_special_tokens({'pad_token': '<pad>'})
model.resize_token_embeddings(len(tokenizer))
2024-08-28T22:17:22Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.