Llama-2 generation_config top_p=0.6
Thegeneration_config.jsonfor the Llama-2-hf models explicitly settemperature=0.9andtop_p=0.6, (e.g.https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/generation_config.json). I am wondering why this is the case. In my experience,top_p=0.6easily leads to very repetitive text. Wouldn’t it be better to just not include these in thegeneration_config.json?@ArthurZ@osanseviero
2023-08-08T09:00:32Z
These parameter were specifically set by the Llama team, and probably come from their experiments with the model!
2023-08-08T09:45:58Z
Thanks for the response! I guess you probably won’t know the answer to this, but is it possible that the temperature and top_p were somehow swapped on accident? In the official Llama repo, it seems that a temperature of 0.6 and top_p of 0.9 are used, (https://github.com/facebookresearch/llama/blob/main/example_text_completion.py,https://github.com/facebookresearch/llama/blob/main/example_chat_completion.py). Whereas thegeneration_config.jsonhas them swapped, temperature of 0.9 and top_p of 0.6.
2023-08-08T10:21:34Z
Ah in that case I would need to look at the commits, they might have been swapped ! Thanks for noticing!