mixup: Beyond Empirical Risk Minimization (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper classifier dnn cnn high dimensions class boundaries mixing interpolation latent beta regularizer regularization generalization adversarial examples smooth
Neural Networks often draw hard boundaries in high-dimensional space, which makes them very brittle. Mixup is a technique that linearly interpolates between data and labels at training time and achieves much smoother and more regular class boundaries.

OUTLINE:
0:00 - Intro
0:30 - The problem with ERM
2:50 - Mixup
6:40 - Code
9:35 - Results

https://arxiv.org/abs/1710.09412

Abstract:
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

Authors: Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we'll look at Mix-Up Beyond Empirical Risk Minimization by Hongyi Cheng, Mustafa Sis, Jan Endauphin and David Lopez-Paz. So this paper is actually pretty simple, but it introduces a technique that apparently helps with training classifiers, and I haven't seen it used in practice, so there must be at least something to it. It is ultimately very simple. So usually you input a data point X into your neural network in deep learning. So f of X, that's your neural network. Your neural network has parameters theta. You get some output y hat, and along with the X you also have a y, a true label, and then you have a loss function that compares what you output with your true label, and then you just try to make that loss smaller. You want to adjust your parameters so next time you see data point X, its output will be a little closer to the true label y. And we call this empirical risk minimization, because you don't actually think that your X comes from some data distribution D, like the space of all natural images or the space of all of language, but what you actually have is a data set of a finite amount of data that you can sample X and Y from. So instead of minimizing your true risk, you minimize your empirical risk, the empirical misc-renimization right here. Now what's the problem with that? The problem is that you can get overly confident about your data points and nothing else, and that will hurt your generalization. So if you have a data point, let's say right here, and another one right here, your network is basically, so this is class 1, this is class 2, your network is going to maybe make decision boundaries like this, and like this, where it says okay, here is class 1 and here is class 2. But it's very conceivable that here it says, here is class 4, and over here is class 7, and right here through is class 9, and by the way, here, class 4 again. So the empirical risk minimization leaves everything in between the data points open. Now what this paper proposes is that we should not only train our classifier on these data points, but on all the data points sort of in between the two. And this is the mix-up data points. So this data point here might be constructed, if this is A and this is B, from 0.1 times B, right, and plus 0.9 times A, because it's mostly A and it's a little bit B. And now you think, what are the labels here if A belongs to class 1 and B belongs to class 2? And of course the label of this data point is 0.1 times the class of B, which is 2, plus 0.9 times the class of A, which is 1. Ultimately, because what you do is you input a class like class number 2, if you want to input this into a machine learning model, you don't just say it's class number 2. What you input is a distribution that basically has zeros everywhere. So these small things, they're 0, 0, 0, 1, 0. And this here is at class number 2. So this would be class number 1, class number 2, class number 3, right? You input a distribution like this if you want to express class number 2. Now in our sample right here, what we would input as a label is simply a mix between class 1, so 0.9, so 0.9 of class 1, 0.1 of class 2, and then 0 everywhere else. So this would be our label for the data point that we construct right here. This would be our, sorry, the top one would be our data point. Formally, you take two data points and you mix them using this lambda mixing factor. That'll give you a new data point that's in between the other data points. And you take the two corresponding labels and you mix them accordingly as well. And that will give you the label for that data point. And now your model will learn to basically smoothly interpolate. So you will teach your model. So the model on the left here is class number 1, right? That's class number 1. The thing on the right is class number 2. This here is half of class 1 and half of class 2. So the model basically learns a smooth interpolation where the situation that's here on top is probably not going to happen anymore. But what it would do is it would sort of create these iso lines around class 2 and around class 1 where it's sort of smoothly getting less and less sure about the class of the data points. But on the way, it is always either class 1 or class 2. And they say that can help the generalization performance. And it's visible. Why? Right? It's just the only thing that's not clear from the beginning is that this kind of interpolation actually makes sense. Because this means we sort of linearly interpolate between two images. So if we have two images, we just take half of one and half of the other. And that will be not a natural image. It will be kind of a blurry thing. Otherwise, you know, all our problems would be solved. And we could just linearly classify things. But in any case, in practice, it actually seems to help. Probably because interpolations of two images, linear interpolations, are still much more like something like a natural image than any random noise you could come up with. So they say this in code right here. Code is pretty simple. Simply want to mix the two things. And the mixing factor, this lambda here, comes from a beta distribution. And they use a beta, I believe, of 0.4 or something. Just want to quickly show you. This is the red line here. So the red line, as you can see, mostly, most of the time, they're going to either sample the thing on the very left or the thing on the very right. That means they either sample the first or the second data point. But some of the time, they actually sample something in the middle. And it's fairly uniform in the middle. So it appears like a good distribution to sample from if you want to sample these mixing coefficients. And by adjusting the actual number of alpha and beta here, you can determine how many times you sample the original data points versus how many times you sample something in the middle. OK. On this toy data set right here, they showcase what mixup can do. So in a classic model, you have the orange and the green data points. And blue is basically where the classifier believes it's class one. You see this very hard border here. It's quite a hard border. Now, you only have two classes here. And so the hard border is sort of a problem in itself, because if you think of, for example, adversarial examples, all they have to do is basically get over that one inch. And the classifier is already super duper sure it's the orange class. Whereas if you use mixup, your border is much, much, much more fuzzy. It's like, yeah, it's only really sure here. And out here everywhere. But in the middle, it's sort of like, I don't know. And so that's kind of a more desirable situation. Now, of course, this here works particularly in this in this linear 2D setting. But as we can see, the same reasoning applies to sort of higher, higher layers and higher dimensionality data points. I have seemed to lost the ability to zoom. Oh, no, it's back. OK. And that's basically it for this paper. This is all they do. They propose this method and then they test it. They say something interesting here that mixup converges to the classical method as alpha approaches zero. So that would push your beta distribution basically in the middle all the way down. And you would only sample from the very left or the very right. So you can smoothly interpolate between this mixing and the classic method. They so their main results are we apply this to classifiers. And what I like is, since again, is also a classifier. So the discriminator is a classifier. They also apply it to GANs and they outperform on stabilized the classic training on GANs. They show that it's more robust towards adversarial attacks because it's not so sure about intermediate things. And they generally outperform other methods. But also they do this nice investigation here where they measure the prediction error of in between data. And what it means is they say a prediction is counted as a miss if it does not belong to YI or YJ. So you have a sample right here, XI and a sample right here XJ. And you look at what the classifier says in between the two data points. So you just interpolate the two data points and just measure what the classifier says. And whenever the classifier either says YI or YJ, either either label of those two data points, you count it as correct and you only counted as incorrect if it says something else. And you can see here if you train with the classic method ERM, these errors happen much more often. That's exactly the situation I pointed out at the beginning where in the high dimensions, it can occur that all sorts of decision boundaries sneak here in between the two data points. And by interpolating between them during training, you sort of much reduce that. You reduce that effect a lot. Now, this they also say that the gradient norm of the gradients of the model with respect to input in between training data, it happens the same thing. The norm of the gradients in the middle is also much, much lower. And this investigation I find pretty cool. I have to say I've seen mix up in practice, so it might be useful. I've read a paper where they basically say, oh, it was a big transfer paper. Yeah, where they basically say it is useful if you have, for example, if you have little data and a big model, so you can sort of regularize the model and is also useful to know that they did test this with dropout. So they compared it with dropout. And the conclusion is basically that this is something else than dropout. So it's not doing the same thing. Dropout, of course, it means you drop out some of the data points in intermediate activations. And that sort of gives you a noisy version of the data point. This here can actually be combined with dropout, which means that it gives you an additional benefit. You see right here, most of the best numbers happen when you use mix up plus dropout. So it seems to be just an additional regularization on top of dropout. Pretty cool, pretty cool investigation also. All right. So if you like this, I invite you to read the paper. If you like the video, please subscribe and like and comment. And yeah, have a nice day. Bye bye.
 Hi there! Today we'll look at Mix-Up Beyond Empirical Risk Minimization by Hongyi Cheng,
 Mustafa Sis, Jan Endauphin and David Lopez-Paz.
 So this paper is actually pretty simple, but it introduces a technique that apparently helps with training classifiers,
 and I haven't seen it used in practice, so there must be at least something to it.
 It is ultimately very simple.
 So usually you input a data point X into your neural network in deep learning.
 So f of X, that's your neural network. Your neural network has parameters theta.
 You get some output y hat, and along with the X you also have a y, a true label,
 and then you have a loss function that compares what you output with your true label,
 and then you just try to make that loss smaller.
 You want to adjust your parameters so next time you see data point X, its output will be a little closer to the true label y.
 And we call this empirical risk minimization,
 because you don't actually think that your X comes from some data distribution D,
 like the space of all natural images or the space of all of language,
 but what you actually have is a data set of a finite amount of data that you can sample X and Y from.
 So instead of minimizing your true risk, you minimize your empirical risk, the empirical misc-renimization right here.
 Now what's the problem with that?
 The problem is that you can get overly confident about your data points and nothing else,
 and that will hurt your generalization.
 So if you have a data point, let's say right here, and another one right here,
 your network is basically, so this is class 1, this is class 2,
 your network is going to maybe make decision boundaries like this, and like this,
 where it says okay, here is class 1 and here is class 2.
 But it's very conceivable that here it says, here is class 4, and over here is class 7,
 and right here through is class 9, and by the way, here, class 4 again.
 So the empirical risk minimization leaves everything in between the data points open.
 Now what this paper proposes is that we should not only train our classifier on these data points,
 but on all the data points sort of in between the two.
 And this is the mix-up data points.
 So this data point here might be constructed, if this is A and this is B,
 from 0.1 times B, right, and plus 0.9 times A, because it's mostly A and it's a little bit B.
 And now you think, what are the labels here if A belongs to class 1 and B belongs to class 2?
 And of course the label of this data point is 0.1 times the class of B, which is 2,
 plus 0.9 times the class of A, which is 1.
 Ultimately, because what you do is you input a class like class number 2,
 if you want to input this into a machine learning model, you don't just say it's class number 2.
 What you input is a distribution that basically has zeros everywhere.
 So these small things, they're 0, 0, 0, 1, 0.
 And this here is at class number 2.
 So this would be class number 1, class number 2, class number 3, right?
 You input a distribution like this if you want to express class number 2.
 Now in our sample right here, what we would input as a label is simply a mix between class 1,
 so 0.9, so 0.9 of class 1, 0.1 of class 2, and then 0 everywhere else.
 So this would be our label for the data point that we construct right here.
 This would be our, sorry, the top one would be our data point.
 Formally, you take two data points and you mix them using this lambda mixing factor.
 That'll give you a new data point that's in between the other data points.
 And you take the two corresponding labels and you mix them accordingly as well.
 And that will give you the label for that data point.
 And now your model will learn to basically smoothly interpolate.
 So you will teach your model.
 So the model on the left here is class number 1, right?
 That's class number 1.
 The thing on the right is class number 2.
 This here is half of class 1 and half of class 2.
 So the model basically learns a smooth interpolation where the situation that's here on top
 is probably not going to happen anymore.
 But what it would do is it would sort of create these iso lines around class 2
 and around class 1 where it's sort of smoothly getting less and less sure about the class of the data points.
 But on the way, it is always either class 1 or class 2.
 And they say that can help the generalization performance.
 And it's visible. Why? Right?
 It's just the only thing that's not clear from the beginning is that this kind of interpolation actually makes sense.
 Because this means we sort of linearly interpolate between two images.
 So if we have two images, we just take half of one and half of the other.
 And that will be not a natural image.
 It will be kind of a blurry thing.
 Otherwise, you know, all our problems would be solved.
 And we could just linearly classify things.
 But in any case, in practice, it actually seems to help.
 Probably because interpolations of two images, linear interpolations,
 are still much more like something like a natural image than any random noise you could come up with.
 So they say this in code right here.
 Code is pretty simple.
 Simply want to mix the two things.
 And the mixing factor, this lambda here, comes from a beta distribution.
 And they use a beta, I believe, of 0.4 or something.
 Just want to quickly show you. This is the red line here.
 So the red line, as you can see, mostly, most of the time,
 they're going to either sample the thing on the very left or the thing on the very right.
 That means they either sample the first or the second data point.
 But some of the time, they actually sample something in the middle.
 And it's fairly uniform in the middle.
 So it appears like a good distribution to sample from if you want to sample these mixing coefficients.
 And by adjusting the actual number of alpha and beta here,
 you can determine how many times you sample the original data points versus how many times you sample something in the middle.
 OK.
 On this toy data set right here, they showcase what mixup can do.
 So in a classic model, you have the orange and the green data points.
 And blue is basically where the classifier believes it's class one.
 You see this very hard border here.
 It's quite a hard border.
 Now, you only have two classes here.
 And so the hard border is sort of a problem in itself, because if you think of, for example, adversarial examples,
 all they have to do is basically get over that one inch.
 And the classifier is already super duper sure it's the orange class.
 Whereas if you use mixup, your border is much, much, much more fuzzy.
 It's like, yeah, it's only really sure here.
 And out here everywhere.
 But in the middle, it's sort of like, I don't know.
 And so that's kind of a more desirable situation.
 Now, of course, this here works particularly in this in this linear 2D setting.
 But as we can see, the same reasoning applies to sort of higher, higher layers and higher dimensionality data points.
 I have seemed to lost the ability to zoom.
 Oh, no, it's back.
 OK.
 And that's basically it for this paper.
 This is all they do.
 They propose this method and then they test it.
 They say something interesting here that mixup converges to the classical method as alpha approaches zero.
 So that would push your beta distribution basically in the middle all the way down.
 And you would only sample from the very left or the very right.
 So you can smoothly interpolate between this mixing and the classic method.
 They so their main results are we apply this to classifiers.
 And what I like is, since again, is also a classifier.
 So the discriminator is a classifier.
 They also apply it to GANs and they outperform on stabilized the classic training on GANs.
 They show that it's more robust towards adversarial attacks because it's not so sure about intermediate things.
 And they generally outperform other methods.
 But also they do this nice investigation here where they measure the prediction error of in between data.
 And what it means is they say a prediction is counted as a miss if it does not belong to YI or YJ.
 So you have a sample right here, XI and a sample right here XJ.
 And you look at what the classifier says in between the two data points.
 So you just interpolate the two data points and just measure what the classifier says.
 And whenever the classifier either says YI or YJ, either either label of those two data points,
 you count it as correct and you only counted as incorrect if it says something else.
 And you can see here if you train with the classic method ERM, these errors happen much more often.
 That's exactly the situation I pointed out at the beginning where in the high dimensions,
 it can occur that all sorts of decision boundaries sneak here in between the two data points.
 And by interpolating between them during training, you sort of much reduce that.
 You reduce that effect a lot.
 Now, this they also say that the gradient norm of the gradients of the model with respect to input in between training data,
 it happens the same thing. The norm of the gradients in the middle is also much, much lower.
 And this investigation I find pretty cool. I have to say I've seen mix up in practice, so it might be useful.
 I've read a paper where they basically say, oh, it was a big transfer paper.
 Yeah, where they basically say it is useful if you have, for example, if you have little data and a big model,
 so you can sort of regularize the model and is also useful to know that they did test this with dropout.
 So they compared it with dropout. And the conclusion is basically that this is something else than dropout.
 So it's not doing the same thing. Dropout, of course, it means you drop out some of the data points in intermediate activations.
 And that sort of gives you a noisy version of the data point.
 This here can actually be combined with dropout, which means that it gives you an additional benefit.
 You see right here, most of the best numbers happen when you use mix up plus dropout.
 So it seems to be just an additional regularization on top of dropout.
 Pretty cool, pretty cool investigation also. All right. So if you like this, I invite you to read the paper.
 If you like the video, please subscribe and like and comment. And yeah, have a nice day. Bye bye.