Right choice of padding side for Mistral
Mistral with flash attention 2 and right padding · Issue #26877 · huggingface/transformers (github.com)From the above discussion, I understand that -Duringmodel.generatethe padding side should be leftDuringmodel.forward, the padding side could either be left or right. Both are usually supported with HF.My question though is, ifmodel.generateis going to give “not so good” results using right padding, isn’t model forward going to affect w/ the same problem?