Can I compute `eval_loss` and `bleu` score simultaneously for decoder only transformers
I’m trying to fine-tune LLaMA, and I want to evalute both theeval_lossand thebleuscore during training, where the former needs teacher-forcing while the latter does not.I find theSeq2SeqTrainersimultaneously executesmodel(**inputs).lossto compute evaluation loss andmodel.generate(**inputs)to compute generated tokens.However, theinputsare different across the two function calls, asmodel(**inputs).lossrequires the inputs to includelabelswhilemodel.generate(**inputs)requires the inputs not to includelabels.Take the sentenceI love you, do youas an example. I train LLaMA withI love you,as context anddo youas target. When evaluating, I want to inspect:log[p(do|I love you,)] + log[p(you|I love you, do)], i.e.eval_loss, which requiresI love you, do youasinput_idsthe generation results fromp(*|I love you,), and computingbleuscore based on the generated results. This only requiresI love you,asinput_ids.How to resolve such a conflict so as to evaluate botheval_lossandbleuscore inSeq2SeqTrainer.prediction_step()?