Loading pre-trained models with AddedTokens
I have pre-trained a"meta-llama/Llama-2-7b-chat-hf"model using thetransformerslibrary. Since my model uses additional tokens, I added them to the tokeniser before training and fine-tuned the “embed_tokens” module of the network. My training code looked like this:tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf",trust_remote_code=True, token=REDACTED_TOKEN)
  tokenizer.add_special_tokens({ "additional_special_tokens":[AddedToken("<|move|>"),
                                                              AddedToken("<|endmove|>"),
                                                              AddedToken("<|end|>")]})

  model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map=device_map,
    token=REDACTED_TOKEN
  )
  model.resize_token_embeddings(len(tokenizer))
  peft_config = LoraConfig(
      lora_alpha=lora_alpha,
      lora_dropout=lora_dropout,
      r=lora_r,
      bias="none",
      modules_to_save= ["embed_tokens", "lm_head"],
      task_type="CAUSAL_LM",
  )The model trained and saved successfully. However, when trying to load it usingAutoModelForCausalLM.from_pretrained, I get the following error:Error(s) in loading state_dict for LlamaForCausalLM:
size mismatch for model.embed_tokens.modules_to_save.default.weight: copying a param with shape torch.Size([32003, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
size mismatch for lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([32003, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096])I appreciate the error is due to the fact that the fine-tuned model has three additional tokens and that causes a mismatch, but how should I load a pre-trained model with a different input shape like mine?I looked into the transformers API docs for a way to load models with AddedTokens, but I couldn’t find anything. I read a blog post mentioning that passingignore_mismatched_sizes=Trueto the from_pretrained function would solve the issue, but it didn’t work for me.
2024-10-14T14:07:38Z
was this ever solved? i have basically the exact problemRuntimeError: Error(s) in loading state_dict for MistralForCausalLM:size mismatch for model.embed_tokens.weight: copying a param with shape torch.Size([32002, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).size mismatch for lm_head.weight: copying a param with shape torch.Size([32002, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
2024-10-14T14:14:48Z
This is a post dealing with a completely different issue, but sometimes there are bugs specific to the version of PEFT, so you might want to try changing the version of each library.My recent experience has been that some programs do not work if the transformers library is newer than 4.44.LLaMa3.1 8B Instruct Prompt Tuning for Text Classification doesn't improve test accuracyModelsHello everyone! 
I am trying to fine-tune LLaMa3.1 8B Instruct for text classification with prompt tuning from peft library (prepending a trainable matrix before input embeddings). For instance, I am using the QNLI dataset to classify whether a question and sentence are in entailment. 
I have preprocessed the dataset so that each data looks like the following: 
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

<|eot_id|><|…