Llama2 pad token for batched inference
From what I can tell, the recommended approach is usually to set thepad_tokenas theeos_tokenafter loading a model. However, when running batched inference with Llama2, this approach fails.To reproduce:from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "meta-llama/Llama-2-7b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")

# Define PAD Token = EOS Token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = model.config.eos_token_id

# use different length sentences to test batching
sentences = [
    "Hello, my dog is a little",
    "Today, I",
    "I love",
    "Repeat after me: I love you.",
    "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    "You are a human that uses a lot of expletives. For each one of your responses to the user you use expletives liberally. If you find yourself questioning whether it's appropriate or not you're to say fuck that noise and keep using your expletives regardless. Human: Tell me a story. You:"
]


inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)
print(inputs['input_ids'].shape)

output_sequences = model.generate(**inputs, max_new_tokens=20, do_sample=True, top_p=0.9)

print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))RuntimeError: probability tensor contains either `inf`, `nan` or element < 0Potential solution: I’ve found that setting thepad_token=bos_tokenactually fixes the issue and allows for batched inference:# Define PAD Token = BOS Token
tokenizer.pad_token = tokenizer.bos_token
model.config.pad_token_id = model.config.bos_token_idI’m wondering if this is something special to the Llama2 model or not recommended for some particular reason?Thanks in advance for any clarification!
2023-07-25T09:32:41Z
tannonk:Maybe this would help you out:Llama2
2023-08-20T15:12:15Z
I find the following working very well:tokenizer.pad_token = "[PAD]"
tokenizer.padding_side = "left"I used to use what you had, but I found that doing batch inference with that inference gives different results compared to sequential inference, which is not supposed to happen.
2023-09-07T13:54:41Z
Works like a charm, thanks!
2023-10-23T02:25:47Z
I got the same error using this code with llama-2-7b-chat-hf, however, llama-2-13b-chat-hf doesn’t yield such an error
2023-11-24T23:35:10Z
After setting up efficient batching.  What I think is done is an old trick.  Concatenate a bunch of strings with an eos_token inbetween into one long continuous string, then chunk it.I suspect this is what was done to train the model, truncating the last piece, thereby no need for a pad token.
2024-03-31T06:25:04Z
Hi@LaferriereJC, I recently consider using this technique, which can be called as “sequence packing” I guess, but I can’t find much useful code for this. Could you please share some materials if you know any? Thanks in advance!
2024-03-31T08:49:09Z
Wrote a custom dynamic programming solution here for 100% efficent batchinggist.github.comhttps://gist.github.com/thistleknot/93c79696e25d0b89ff9ed829a02fbd9bmamba v3.py#https://gist.githubusercontent.com/thistleknot/raw/mamba_trainer.py
#SimplerMambaSSM
#https://colab.research.google.com/drive/1g9qpeVcFa0ca0cnhmqusO4RZtQdh9umY#scrollTo=2lECw6S4N7cn
"""
NanoGPT initial base
Mamba

Implemented Papers
* Symbolic Discovery of Optimization Algorithms
* Exponentially Faster Language ModelingThis file has been truncated.show original