Q&A evaluation: Mismatch in the number of predictions (775) and references (835)
I just ran the code provided intransformers/examples/pytorch/question-answering at main · huggingface/transformers · GitHubusing a custom dataset, which however corresponds to the Sqad dataset format, split in train, test, and validation sets, and privately uploaded as a HF Dataset.The training went well, however evaluation crashed with the following:File “…/envs/hf-pt/lib/python3.10/site-packages/evaluate/module.py”, line 432, in computeself.add_batch(**inputs)File “…/lib/python3.10/site-packages/evaluate/module.py”, line 512, in add_batchraise ValueError(error_msg) from NoneValueError: Mismatch in the number of predictions (775) and references (835)I tried running the code with and without the flag --version_2_with_negative which then leads to following stack trace:File …/utils_qa.py", line 209, in postprocess_qa_predictionsscore_diff = null_score - best_non_null_pred[“start_logit”] - best_non_null_pred[“end_logit”]UnboundLocalError: local variable ‘null_score’ referenced before assignmentAny idea on what could cause this error?
2023-07-10T02:31:11Z
I got a similar problem. Have you solved it?
2023-07-11T17:40:11Z
I have settled it. In my case, there are some duplicate IDs of questions in the validation set. So I recommend you check whether the id of your dataset is unique.
2023-08-09T15:05:45Z
i am trying to summarize the articles in the cnn daily mail datasetit is still giving me…ValueError: Mismatch in the number of predictions (145) and references (218)
2023-09-28T17:25:29Z
I got a similar problem when I computed the WER in my program. I found that the input values to evaluation function need to have same shape in the first dimension. So I turned the input values into list type, like this:wer_metric.compute(predictions=a, references=b)    >>>    wer_metric.compute(predictions=[a], references=[b])It works on me.