Using Pegasus Model for Transfer Learning is generating garbage summaries
Hi ,I am using thePegasusForConditionalGenerationfromtransformerslibrary for transfer learning and generate summaries of chats based on samsum dataset. But, the output generated is repetitive and not related at all to the context.Also, gettinglossasnan. Can someone help me regarding what went wrong?Please note that the same code works perfectly fine forT5ForConditionalGeneration.Is there any difference in the implementation method?Should I try with other Loss Metrics or Optimizer?
2020-09-18T12:03:03Z
Could you post the command you are using for fine-tuning ?
2020-09-18T12:24:33Z
I’m using a Python Code for the same. A snippet of which I’m sharing below -for _ in range(epochs):
            self.model.train()
            train_loss = 0
            for idx, data in tqdm(enumerate(self.train_loader)):
                self.optimizer.zero_grad()
                output = self.model(input_ids = data["input_ids"], attention_mask = data["attention_mask"], lm_labels = data["lm_labels"])
                loss, prediction_scores = output[:2]  
                train_loss += loss.item()
                loss.backward()
                self.optimizer.step()
                if((idx % 1000) == 0):
                    print("loss: ", loss.item(), " train_loss: ", train_loss/(idx+1))As the name of each variable suggests, I’vePegasusForConditionalGenerationin the variableself.model,AdamOptimizer inself.optimizerandself.train_loaderis of typeDataLoader.