How to set the padding configuration with Huggingface's GenerateMixin's generate method?
stackoverflow.comHow to set the padding configuration with Huggingface's GenerateMixin's generate method?huggingface-transformers, huggingface-tokenizersasked byNasheed Yasinon02:38AM - 16 Sep 23 UTCI am trying to batch-generate text 16 at a time. While tokenizing I left pad all my sequences and set the pad_token as equal to the eos_token. Since I don’t see a link between the generate method and the tokenizer used to tokenize the input, how do I set it up?Here is a smallcode snippetof what I am trying to do:from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.padding_side = "left" 
tokenizer.pad_token = tokenizer.eos_token # to avoid an error
model = GPT2LMHeadModel.from_pretrained('gpt2')

device = 'cuda' if torch.cuda.is_available() else 'cpu'

texts = ["this is a first prompt", "this is a second prompt"]
encoding = tokenizer(texts, padding=True, return_tensors='pt').to(device)
with torch.no_grad():
    generated_ids = model.generate(**encoding)
generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)Two sequences of unequal length look like this rn:[[2, 2, 2, 2, 323, 244, 34, 55, 1, 1, 1, 1]
 [2, 2, 23, 3225, 323, 244, 34, 55, 51, 61, 41, 612]]Where1is the<unk>token and2iseos_tokenHere is the current behavior when we batch generate:The generated text is padded on the right.The padding token used bygenerate()is the<unk>token of the model’s tokenizer.I want to be able to:Change the padding side toleft.Use a different token for padding. For instance theeos_tokenthat I already set the tokenizer to use in the snippet above.
2023-09-26T10:48:50Z
I think your snippet should already work correctly. Are you seeing any errors?
2023-09-26T11:09:13Z
No, I do not face errors in the above snippet. I will edit the question to better explain what I want.
2023-09-26T12:45:24Z
Use a different token for padding. For instance theeos_tokenthat I already set the tokenizer to use in the snippet above.I see. You can passpad_token_idto thegeneratecall to do this. For example:generated_ids = model.generate(**encoding, pad_token_id=tokenizer.eos_token_id)Or you can set it in the model’s generation config:model.generation_config.pad_token_id = tokenizer.eos_token_idAs for this part:Change the padding side toright.I’m still a bit confused about what you’re asking. During generation, the padding is already on the right in the outputs you posted. Or you’re saying you want the tokenizer to pad on the right?
2023-09-26T14:05:25Z
As for this part:Change the padding side toright.I’m still a bit confused about what you’re asking. During generation, the padding is already on the right in the outputs you posted. Or you’re saying you want the tokenizer to pad on the right?Oops I meantleft.Morning grogginess I guess. I am editing the Original post now.
2023-09-26T14:09:01Z
Finally, would I be right to say that the args generally taken by thetokenizerto pad, determine the side of padding, length to pad until, maybe truncation (although I do not see the utility of this), e.t.c. can all be passed as args to thegenerate()method?
2023-09-26T14:34:26Z
nasheed:Oops I meantleft.I don’t think there’s any way to do this - if you take a lookhere, you see that if a sequence in the batch is finished, the HF code will set the next token to the pad token and then concatenate it to the end of the finished sequence (in other words, pad on the right).However, this is totally fine. When you de-tokenize the outputs, the tokenizer will remove all the pad tokens regardless of which side they’re on. Is there a reason you want the pad tokens created during generation to be on the left (except perhaps for consistency with the tokenizer)?Finally, would I be right to say that the args generally taken by thetokenizerto pad, determine the side of padding, length to pad until, maybe truncation (although I do not see the utility of this), e.t.c. can all be passed as args to thegenerate()method?No, not all of them. Justpad_token_id,bos_token_id, andeos_token_id. Kwargs passed togenerateget passed alongto update theGenerationConfigobject (you can see the classhereand the args it can take). TheGenerationConfigonly has a few tokenizer-related args.
2023-09-26T15:19:48Z
I see. Part of why I’d like such a feature is maybe completeness (as you drew the parallel to the tokenizer class). But I can also imagine the case where you are calling generate within a loop and are iteratively feeding the generated sequences from the previous step as input to the next. The only way to do this right now is tobatch_decode​ and then retokenize the sequences withleft​ padding. (Since padding needs to beleftwhen batch generating.PS: maybe a method likeprep_for_batch_generation​ on theAutoTokenizerclass would be nice.Much love,Nasheed