How to **properly** prompt the decoder?
I have a pretrained Encoder + Decoder model (Pegasus), and want to fine-tune it as described inthis article.Specifically, they use the following process:Summary generation using entity prompts1770×1254 136 KBIn other words, they prepend amanualprompt to the generation of the model itself.My question relates to the Decoder input. Specifically, I want to fine tune the model so that it takes the prompt (entity chain), and generates a summary from that point onwards.For instance:<s> [ENTITYCHAIN] Frozen | Disney [SUMMARY] $tok_1 $tok_2 $tok_3 ...
=========================================== ^^^^^^ ^^^^^^ ^^^^^^
This is not generated                       Generate from hereHowever, as you would expect, the model is generating predictions for each token in the entity chain, which I do not need. But most importantly, thelossis being computed by also factoring in the predictions related to the entity chain. This clearly undermines the purpose of training, since it confuses the model, because it should learn toonly generate the summary, and not the entity chain (which is already given as a prompt).As I was saying, what I want is to give a prompt (entity chain) to the decoder, and make it generate a summary, while being able to attend to the extra information from the prompt. Of course, the loss should only be computed among the generated tokens, excluding the prompt tokens.By looking into themodel documentation, I don’t seem to find an option to do this. Any ideas?