T5 fine tuning, loss difference when using labels and decoder_input_ids
hello,i am fine tuning T5 for summarization on the news summary dataset.i am encountering a weird problem when calculating the loss of T5 while training. i get different results when i train computing the loss passing only the ‘labels’ parameter and when i pass both ‘labels’ and ‘decoder_input_ids’.i find the documentation a bit misleading. in the ‘Training’ section, it saysthe forward function automatically creates the correct decoder_input_idshereafter you can see the different loss plots i get:loss720×360 16.6 KBthe green line (copper-rain) is when i do not pass any ‘decoder_input_ids’, i.e. in the dataset classdef __getitem__(self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = self.labels[idx]the blue line (clean-cosmos) is when i do pass the ‘decoder_input_ids’ along with the ‘labels’ (following the example inthis notebook), i.e.def __getitem__(self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = self.labels[idx]
        item['decoder_input_ids'] = item['labels']do you kindly have any explanations for this behaviour?
2020-10-12T17:08:53Z
How are you preparinglabels?decoder_input_idsare shifted to the right and start withpadtoken.This is how you should preparedecoder_input_idsfor T5decoder_input_ids = labels.new_zeros(labels.shape)
decoder_input_ids[..., 1:] = labels[..., :-1].clone()
decoder_input_ids[..., 0] = self.pad_token_idConsider usingexamples/seq2seqhere, for seq2seq experiments, it does lot’s of things for you including this
2020-10-12T17:45:46Z
thank you for your help!i see. indeed, now i get exactly the same results as by passing ‘labels’ only (since i set the same seed). i was biased by the fact that i was getting a much lower loss passing the labels to the decode_input_ids as is. do you have any clue why i get this behaviour?i wonder decode_input_ids is not useful at all when passing labels to the trained model. i don’t really understand why in the notebook i linked they give it as a parameter.