How to actually use padding in Lllama Tokenizers
I have been struggling to get padding to work properly with llama based models.It seems like llama by default does not use a pad token. Does this mean that you simply can’t have batch_size > 1 ?But some suggestions on github include to set pad_token = eos_token. But the issue with that is that pad_token_id is actually set in the generation_configgeneration_config.json · lmsys/vicuna-13b-delta-v1.1 at mainpad_token_id is set to 0. Why is there an inconsistency? Does this matter should I also set pad_token_id to eos_token_id?Currently I simply set pad_token = eos_token, but keep the pad_token_id as 0 and am noticing poorer performance with batch inference as compared to single inference.
2023-06-11T03:23:15Z
@vikalexhow did you manage to solve this issue ? help please !
2023-06-16T17:46:01Z
I just set eos_token to pad_token and it doesn’t cause any issues as long as the model knows both of them and has a good stopping criteria.