GPT2 for QA Pair Generation
I was wondering if it were possible to somehow train GPT2 to generate question-answer pairs in a particular domain?
2020-08-19T09:08:31Z
I’ve tried this with seq2seq models. I have worked on qa pair generation (separately)  using T5 with descent results. You can find ithere.One way we can do this with GPT-2 is prepare our input like thisOur context is42 is the answer to life, the universe and everything, answer is42and target question isWhat is the answer to life, universe and everything ?Theninput text:context: 42  is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42and prepare the attention mask such that, there will be no attention fromquestion: ...part, so model won’t look into future tokens and calculate loss only on thequestion: ...part. And it inference time we will feed only the context part and ask the model to generate the question.This just one one way I can think of the of my mind. Feel free to correct me if this is wrong.
2020-08-19T18:37:53Z
@valhallaThanks for your response. That’s an interesting approach! Does that still require humans to create training “context” strings for gpt2?
2020-10-12T19:51:20Z
@valhallaIf I understand this correctly:The input text will look likecontext: 42  is the answer to life, the universe and everything.  question: What is the answer to life, universe and everything ? answer:  42Mask out thequestionpart so the new text will look likecontext: 42  is the answer to life, the universe and everything.  <BIG MASK> answer: 42That is what gets fed as input text into the GPT2 modelDoes this mean I define thelabelsinto the model as the text that is masked?
2020-10-13T07:25:40Z
By mask, I meantattention_mask, theattention_maskshould be zero on the text you want to predict, so the model won’t peek into future.So if you want to generate question and answer, then the question and answer tokens should have 0in attention mask.
2020-10-13T11:19:41Z
Ah yes, sorry for my misunderstanding. So we mask out the parts we want to predict by setting theattention_maskof those tokens to 0.With these tokens masked inattention_mask, do we then pass it and the input string to GPT2 and train it with the language model head with no labels?
2020-10-13T15:06:14Z
You’ll still need to passlabelsfor training.Training will be same as training any GPT-2 model, only difference is theattention_mask
2020-10-13T15:53:54Z
If I only wanted to generate questions, would I set theattention_maskfor those tokens to 0 and use their text as thelabels? Something like:from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
def my_data_collator(text_str):
    encoded_results = tokenizer(text_str, padding=True, truncation=True, return_tensors='pt',
                                     return_attention_mask=True)
    enncoded_results['attention_mask'] = set_my_attention_mask(encoded_results) #function to set attention mask to 0 on tokens in the question:... part of text_str
    label_ids = get_my_label_str(encoded_results['input_ids']) #function to return list of token ids for question:... part of text_str

    batch = {}
    batch['input_ids'] = encoded_results['input_ids']
    batch['past'] = None
    batch['attention_mask'] = encoded_results['attention_mask']
    batch['position_ids'] = None
    batch['head_mask'] = None
    batch['inputs_embeds'] = None
    batch['labels'] = label_ids
    batch['use_cache'] = True
    return batch

text_str = 'context: 42 is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42'Andbatchwould get passed to aGPT2LMHeadModel?
2020-10-13T16:09:11Z
This seems correct. One more thing to add, you can calculate loss only on thequestion: ...part.To do this setlabelsto -100 for tokens before thequestion:part, so cross entropy will ignore it.Also you won’t need to explicitly set some arguments (position_ids,head_masketc) toNone.They are by defaultNoneso it’s okay if don’t pass them. Will make the code more cleaner.
2022-03-23T17:27:21Z
@valhallaif we set the context labels to -100, this will make the model ignore the context while training. In other words, the generation of the questions won’t be based context-based. Am I right?