Padding to the left of the inputs, GPT2LMHeadModel gives different answer
When I input the tokenized stringHello, my dog is cute, GPT2LMHeadModel predictsHello, my dog is cute. I'm not sure if she's a puppy.If I add four 0s to the left of the token IDs and use the attention mask[0,0,0,0,1,1,1,1,1,1,1], the prediction changes to!!!!Hello, my dog is cute hello hello hello hello hello hello hello hello hello hello.I’d thought that these two inputs should lead me to the same result?I have this notebook to reproduce the above experimentGoogle Colab
2023-02-21T16:04:09Z
Hi@wangkuiyiIn your examples, you forgot to consider theposition_ids– regardless of the padding you insert, you want theposition_idsthat correspond to a given word to be invariantFurther resources:Glossary: position IDsHow GPT2, the model itself, infers position IDs if you don’t pass them(spoiler: it’s based on yourinput_ids)Howgenerateattempts to handle it(spoiler: it uses theattention_maskas a guide)
2023-02-21T23:51:56Z
Thank you, Joao@joaogante!