Pegasus on qa task
Hello everyone,I try to use pegasus on squad dataset. I am usingnotebookfor pegasus fine-tuning on xsum.I use all hyper parameters as in the above notebook. After train, I got following loss values./usr/local/lib/python3.7/dist-packages/transformers/optimization.py:562: UserWarning: This overload of add_ is deprecated: add_(Number alpha, Tensor other) Consider using one of the following signatures instead: add_(Tensor other, *, Number alpha) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:1005.) exp_avg_sq_row.mul_(beta2t).add_(1.0 - beta2t, update.mean(dim=-1))[2000/2000 46:21, Epoch 2000/2000]StepTraining LossValidation Loss10011.51540011.9715682009.58380010.4005963007.7829008.8124674001.6538001.6030775000.0060000.3677626000.0495000.2712217000.0006000.2711988000.0003000.2827409000.0002000.31101210000.0002000.30887911000.0001000.31637412000.0001000.32943613000.0001000.32699614000.0202000.29115715000.0024000.27254316000.0001000.29193417000.0359000.30823518000.0001000.30875519000.0001000.31109620000.0000000.311382CPU times: user 1h 1min 52s, sys: 18min 49s, total: 1h 20min 41s Wall time: 46min 26sTrainOutput(global_step=2000, training_loss=1.8480298345236805, metrics={‘train_runtime’: 2783.2468, ‘train_samples_per_second’: 0.719, ‘total_flos’: 0, ‘epoch’: 2000.0, ‘init_mem_cpu_alloc_delta’: 8192, ‘init_mem_gpu_alloc_delta’: 0, ‘init_mem_cpu_peaked_delta’: 0, ‘init_mem_gpu_peaked_delta’: 0, ‘train_mem_cpu_alloc_delta’: -4224876544, ‘train_mem_gpu_alloc_delta’: 2288025088, ‘train_mem_cpu_peaked_delta’: 4242300928, ‘train_mem_gpu_peaked_delta’: 8677123584})My question is what values for hyperparameters (optimization, learning rate etc.) I should use for a QA task? And the training loss shown in TrainOutput does not match with the loss values on table. What can the reason be?
2021-05-17T20:03:15Z
hey@helloworld123-lab, are you sure that pegasus can be used for a reading comprehension task like SQuAD? my understanding is that it is designed for abstractive tasks like summarization (although i’d be interested to hear otherwise!).as an alternative, i’d suggest checking out the official question-answering tutorial here:Google Colaboratory
2021-05-17T21:16:59Z
Thank you@lewtunfor your reply. Actually I’m not sure about that. I could not find a source about this either. However, I checked the model inhere, and I wonder how this could happen with pegasus or bart.
2021-05-18T11:31:56Z
ah so that notebook is about T5 whichdoessupport question-answering via it’s special text2text formulation. perhaps a better question from me would be: what specific task are you trying to solve?if you want to use BART, my suggestion would to see if one of the checkpoints on the hub is compatible with the tutorial i linked to earlier
2021-05-19T01:37:13Z
Thank you, I am just a beginner and working on QA. I just started training the notebook you shared with Bart by passing its checkpoints.I have one more questionI just saw some notebooks on Bert2bert/RobertaShared etc. on cnn-dailymail for summarization. Can I use such models on QA systems?
2021-05-19T11:49:43Z
as far as i know, you can’t use models that are fine-tuned for summarisation directly for question-answering as the tasks are quite different:in the summarisation case, the model is provided with(document, summary)tuplesin the question answering case, the model is provided with(question, document, answers)tuples (with potentially more subfields likeanswer_startif we’re dealing with the extractive case where we need to identify a span of text where the answer exists)so the main problem i see with naively using a summarisation model for QA is that the model does not expect a question / query in its inputs and so will not generate an answer.btw if you’re looking to build an end-to-end QA system, there’s a neat library calledhaystackthat is based ontransformersand provides a lot of nice functionality to store documents, query them etc:https://haystack.deepset.ai/