Question about validation and testing loss
Hello,Just double checking my numbers here. Would these numbers for testing and validation be indicative of overfitting? I feel like I might be on the verge but would very much appreciate a second opinion.Thanks!Training LossEpochStepValidation LossPrecisionAccuracyF10.58940.45000.47100.83810.77470.75840.38630.810000.30000.82260.87370.88580.22721.215000.19730.95930.93330.93290.16391.620000.16940.90670.93670.94030.12632.025000.11280.96570.95970.96030.07532.430000.13050.96140.9670.96790.06192.835000.12460.96330.96970.9705
2022-04-18T03:15:37Z
More than validation loss this might be the case where there is an overlap in the training and validation data.Go through the validation data carefully and check if the data has somehow crept into the training data.
2022-04-18T03:34:43Z
Thanks for commenting and it’s nice to meet you! I’m positive my testing/validation data were defined. I augmented my data set with back-translation. I removed my duplicates but maybe that could be it?  Would you mind explaining your thoughts a bit more?Cheers!
2022-04-18T03:46:59Z
Ok. If you are using data augmentation then you need to make sure two things.Do the augmentation after the train test split. That way you make sure that train data does not accidentally creep into the test set.Only run the augmentation on the training data.Here is a reference to thispython - Data augmentation before splitting - Stack Overflow.Now that I think about it this can mean overfitting as well. The validation results look too good. You can try one more thing which is increasing the epochs. If the F1 still keeps on decreasing that would mean a high chance of overfitting. Please relook into the way you are using the augmented data.
2022-04-18T03:58:16Z
Ok great! The accuracy and f1 both stagnate after 3 epochs. I’ll read up. Thanks for the ideas
2022-04-19T06:52:28Z
Just fyi: we typically make a distinction between a test set and a validation set. They are not the same thing. A validation (or development) set is used during training to intermediately evaluate the model, to probe the current performance every x steps/epochs. This way we can quickly see if something is wrong (data pollution, overfitting). A test set is a hold-out set that we only ever useaftertraining (not during training, not even for evaluation, also not for cross-validation training or hyperparameter search). It should be explicitly beheld outto ensure that the trained model is not biased towards it at all.Tl;dr: validation/dev set: to monitor training; test set: hold-out for final evaluation (yes, it is confusing)
2022-04-19T13:23:21Z
Thanks for mentioning it. It was a typing accident lol. Nonetheless, I appreciate your explanation