Different evaluation results during and after training: Wav2Vec2 finetuning
Hello,With the notebookFine-Tune Wav2Vec2 for English ASR withTransformers, I notice different results for evaluation during training with compute_metrics and after training on the same test dataset. Please find the details below.To save time, I select part of the dataset withtimit[‘train’] = timit[‘train’].select(range(1000))timit[‘test’] = timit[‘test’].select(range(500))Without changing, I get the following training results.Screenshot 2023-01-25 at 13.31.48718×362 16.8 KBWith load_best_model_at_end=True in TrainingArguments, the best model is loaded as:Loading best model from ./checkpoint-2500 (score: 0.605413556098938).This score is valid_loss, and the corresponding WER is 0.439.With the loaded model, I run the map() function to evaluate. Then I get the resultsTest WER: 0.386I expect to get the almost same results. If you have any ideas, please leave a comment.