Variance Based Samples Weighting for Supervised Deep Learning
Machine Learning (ML) aims at approximating functions defined on a measured space with a model. A relevant choice of distribution for the training data set can improve the performances of a given ML model. We claim and empirically justify that an ML model yields better results when the data set focuses on regions where the function to learn is steeper. We first traduce this assumption in a mathematically workable way. Then, theoretical derivations allow to construct a methodology that we call Variance Based Samples Weighting (VBSW). VBSW uses local variance of the labels to weight the training points. This methodology is general, scalable and cost effective. It is validated on Deep Learning models like Bert [10] or ResNet [14] by significantly increasing their performances for various Natural Language Processing and image classification tasks.