mT5 Question/Answering fine tuning is generating empty sentences during inference
mT5-small Question Answering training is converging to high accuracy, high validation accuracy, near-zero low loss; however, when testing the model on trained questions, I am always receiving empty answers.Experiment Language: ArabicDataset used: Arabic SQUADOptimizer tested: Adam or AdamW with learning rate: 3e-4loss function: tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)Epochs tested: 5, 30I tried to train the model with 30 epochs and the same result is obtained:Output:   <extra_id_0>Output:  [0, 250099, 1]It is very strange that the model is converging to high accuracy and low loss and I am getting empty sentences during inference. I validated the dataset questions and answers and they are correct.Below are some important code snippets:def preprocess_function(examples):
    padding = "max_length"
    max_length = 200
    inputs = [ex for ex in examples["question"]]
    targets = [ex for ex in examples["text"]]
    model_inputs = tokenizer(inputs, max_length=max_length, padding=padding, truncation=True)
    labels = tokenizer(targets, max_length=max_length, padding=padding, truncation=True)
    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

data_collator = DataCollatorForSeq2Seq(
            tokenizer,
            model=model,
            label_pad_token_id=tokenizer.pad_token_id,
            pad_to_multiple_of=64,
            return_tensors="np",
        )

tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            collate_fn=data_collator,
            batch_size=8,
            shuffle=True,
        )

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5) 
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=Adam(3e-5), loss=loss, metrics=['accuracy'])Output:below are the output of model.fit() for the first 5 epochsaccuracy	▁████epoch	▁▃▅▆█loss	█▁▁▁▁val_accuracy	▁▆███val_loss	█▂▁▁▁Run summary:accuracy	0.96812best_epoch	4best_val_loss	0.21643epoch	4loss	0.35643val_accuracy	0.97813val_loss	0.21643Sample question:Q:  [‘ما هي قيمة العقد بين شركة Under Armor و Notre Dame؟’]A:  قيمته 100 مليون دولارA:  [259, 42501, 3234, 966, 548, 36270, 259, 36136, 1]Input:  ما هي قيمة العقد بين شركة Under Armor و Notre Dame؟Input:  [1415, 7383, 2588, 23283, 402, 27419, 5373, 259, 11319, 8427,259, 117220, 341, 259, 37126, 34600, 2273, 1]Output:   <extra_id_0>Output:  [0, 250099, 1]
2023-08-08T12:22:00Z
I have this problem too (on Persian language). Did you solve it?
2024-06-02T15:20:37Z
I have the same problem too in French language