Based on HF documentation, unnaswerable questions from Squad 2.0 don't make it into train/val data
Hi, I wanted to finetune Electra on my own Squad 2.0-style dataset so I looked at the following documentation to figure out what the data format should be.huggingface.coFine-tuning with custom datasets — transformers 3.5.0 documentationIt seems in the walkthrough that only answerable questions actually make it into the training/validation datasets. In the JSON files, if a question cannot be answered, the “answers” array is empty. However in the walkthrough, this is how a (context, question, answer) triplet gets added to the data:for answer in qa['answers']:
        contexts.append(context)
        questions.append(question)
        answers.append(answer)Because it’s iterating through the “answers” array, if i’m not mistaken, the questions that are unanswerable will never get added to the data.
2020-11-25T13:07:52Z
@valhalla@sguggerNot sure if you two are the right people to tag but thought I’d start somewhere!
2020-11-25T13:10:03Z
Asking because I’m not sure how to feed the model unanswerable questions in training, since the example in the doc just seems to ignore them - and it’s a bit part of squad 2
2020-12-03T07:01:03Z
Hi melody, I believe you are right.Conceptually I think we need to set target logits (both start and end) to be all zeros for all unanswerable questions. Also need to set/finetune threshold based on predicted logits whether the question is unanswerable or not (Need to modify the official example a bit)
2020-12-03T14:01:26Z
Note that this is not the “official” example, but a simplified version for a tutorial. The official example is inexamples/question-answering(will be further simplified very soon as I’m working on a PR) and does take into account the unanswerable questions.