Adversarial Examples Are Not Bugs, They Are Features
machine learning deep learning adversarial examples adversarial samples pgd projected gradient descent vulnerabiliby security artificial intelligence MIT geometry classifier deep neural network attack convolutional neural networks research robust features robust classifier robust network neural network
Abstract:
Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

Authors: Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry

https://arxiv.org/abs/1905.02175
 Hi there! Today we're looking at Adversarial Examples Are Not Bugs, They Are Features by Andrew Elias et al. So this paper is pretty interesting as a catchy title and we'll try to kind of dissect what it says. So first of all, in the abstract they say adversarial examples have attracted significant attention, but the reasons for their existence and pervasiveness remain unclear. So if you don't know what an adversarial example is, an adversarial example is basically the following. Say you have an image classifier, right? Classifier, boom, neural network, image here, and the image is of a, let's say, a cat. Right, this is my best attempt at a cat, bang, cat. And you feed it through the classifier and the classifier says cat. Now if you perturb this image, if you derive an image from it and you perturb it just very slightly, very subtly, so you introduce some pixels here, there, here, there, right, you change some pixels in a very targeted way, and you feed that new image through here, then the classifier will say dog or something really, you can make it say anything like airplane or, I don't know, sky or whatever you want. So these are called adversarial examples. And it's true, their existence and pervade, the reasons for their existence and pervasiveness remain unclear. They say we demonstrate that adversarial examples can be directly attributed to the presence of non-robust features. So they're basically, their paper is about these non-robust features and they define later what they mean exactly. But here they say features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. And this is pretty neat. So the fundamental idea, as I understand it, and I'm going to take this away right here, that if you have images, let's say here of cats, and I'm going to draw another one over here, if you have an image, say, of cats, there is multiple features in this image and the feature is something that the classifier can pick up on and kind of learn to, this is a horrible cat, learn to classify images from. So features that we humans generally use are a cat has ear, ear, eyes, whiskers, right, and the general relationship to each other of these things. This is what constitutes a cat. And that's how we classify it. But also they say there are other features that are also very indicative, right? If you think what differentiates a cat from a dog and a dog here, let's pick fluffy ears, also eyes, yeah, not going to go further with the dog too much. What differentiates a cat from a dog? And we, of course, we would say, well, the head shape is different and the ears are different and the relationship to them, to each other are different, but it could also be, and this is a simplistic right now, right? But it's also that cats, for example, have different fur than dogs. And yeah, being overly simplistic here, but bear with me. So let's say in our hypothetical world, cats have fur that goes like this, left to right, right? Every hair is basically vertical, sorry, horizontal. If you look at it like that and dog fur, on the other hand, is always like this, right? This is vertical, right? Top to bottom. And so the classifier might just as well pick up on the fur direction in order to classify images, right? Since all cats have that type of fur and all dogs have that other type of fur, the classifier might just as well pick up on that, right? And to us humans, we don't really pay attention to these things because they're minute, right? You don't look at the directions of individual hairs to classify in an animal to cat or dog. You would much rather go for these kind of large features like where are the ears, how do they look and so on. But a classifier, there's actually you can make an argument that the classifier would more likely pick up on the fur direction, right? In order to in order to classify since we're using convolutional neural networks and they're generally neighborhood pixel neighborhood operators. It can much easier pick up on these patterns than it can on the general relationship of the of the large features. So if a classifier now learns that cats always have fur like this and dogs always have fur like that, what we could do is we can go over here to the dog and change its fur, right? Change in the image, change its fur to this direction. Now, to us humans, that would still very much look like a dog because the fur direction is almost imperceptible. But to the classifier that has only learned, hey, a cat always has this type of fur and the dog always has that type of fur. That new image would totally look like a cat. Right. So this paper argues exactly that this paper argues that in the data set, there are features and these are real features like this. This actually could be the case that cats fur is always like that and dogs fur is always like this. It could be the case and the classifier could pick up on this. Right. And then the adversarial examples, the reason why they exist is because the classifier has picked up on these imperceptible features. And so by changing the features, we can change the classifiers decision. And without changing the image in a large scale. So they they say that they make this hypothesis and they kind of they say, OK, we established a widespread existence in standard data sets. So they kind of give supporting evidence for their hypothesis. And then they say, finally, we present a simple setting, which is a theoretical setting where we can rigorously tie the phenomena we observe to a misalignment between the human specified notion of robustness and the inherent geometry of the data. All right. So it's kind of different pieces of the of this paper. And we're going to look at them in succession. So the introduction, we largely skip, except that their main claim here is specifically we claim that adversarial vulnerability is a direct result of our models, sensitivity to well generalizing features in the data. So that's the core point, I think, is well generalizing features, which is what we mentioned. These are features that actually describe the data well, but but features that are kind of imperceptibly small to humans or that don't fit our notion of robustness. All right. So they go on and they define more clearly what they mean here. Here, whenever we talk of a feature, right? Remember, we had the our classifier here, then we input an image and the image is called X. Right. And that classifier, usually, if we look at it closer, consists of multiple layers of interconnected neurons, whatever. And the last layer will be an output layer into different classes. Right. And so the features, when they say a feature, what they mean specifically is the last here, the last representation before it goes into the classifier. So the way you would classify them and here they just establish a two class setting. The way you would establish that is you have feature one, feature two, feature three, and you have a weight vector W1 for each feature W2, W3. You make the inner product and that will give you a Y hat. Basically, if that is high, you say it's class one. If that is low, you say it's class minus one. So the classes here are plus one and minus one, just to make things simple. So but you see the features are basically what comes out after these layers, what is then used to make a linear classification. This last thing is basically just a logistic regression. So you can think of the features as the output of the neural network, but before it goes into the classifier. So a feature basically since then, it's linearly classified. If the feature is high, it will give a signal for one class. And if a feature is low, it will give a signal for the other class, depending on, of course, if this W is negative or positive. All right, so they say we call a feature row useful. And if this thing holds here, what is this thing? This thing means so the expectation over the dates. So generally in the data set, this must hold Y times the feature. So why is the class? And remember, it's plus or minus one. And the feature, as we've seen, is some some number Y times a feature must be higher than some some number. So what does it mean when a product is high? It means either both are high or both are low. So they're correlated. That's what that means. So basically, this is says a feature F is useful if whenever it an example, X is of class one, if it's class class one or let's if it if Y is one plus one, then F is high. And whenever Y is minus one, then F is low, which means it's high in the negative direction. Right. So this is this is our this is intuitive. Right. If a feature is useful, it means it should say one thing in samples of class one, then it should say another thing in samples of class two. Then I can actually use the feature to make a decision when it's, you know, very correlated with the class. So that, you know, that makes perfect sense. So that's kind of when is a feature useful if it correlates with the class label? Yes. Cool. But the usefulness simply any feature basically that classifier will extract will be useful. That's an assumption we can make. Otherwise, the classifier wouldn't extract it. So the neural network here, that's an assumption, will only extract useful features. Right. Because the non-useful features, there would simply be no reason for it to extract them because they don't contribute to solving the task, because they're not correlated with an output class. Right. So next, they define robust, robustly useful features. So in addition to being useful, they're now also robust. What does it mean? Again, we want a correlation of why and the feature to be higher than some constant. But not only the feature of the image X, but the feature of the image X that has been perturbed by a small perturbation. So and we take the infinum here over a class of perturbations. Of course, this class of perturbations is exactly the adversarial perturbations. Basically, what this means is it says that however we try to perturb X, right, and the infinum here means that the minimum correlation, however we try to make the feature not correlated with Y, however much we try, we can't get it lower than some some gamma, some number, right? We can't we can't get it down. So whatever we try to make the feature bad for the classifier, basically, we can't. If this holds for a feature, if this is the case, then we call that feature a robust feature. Right. That feature is robustly useful if it correlates, no matter how hard we try to make it not correlate. And of course, a non robust features, so a useful non robust feature is a feature which is useful. You see here is useful. But is not gamma robust feature for any gamma. So it is a feature that is useful like the cat fur. Right. So this here, an example of this would be that the cat's eyes and ear position. Right. We can't just make a small perturbation for the image and make the ears be somewhere completely else. That's just that would require a large perturbation of the image. So the position of the ears and eyes are pretty robust features. But here the cat's fur, no matter how no matter how small we we make this this gamma, we can always kind of change the fur to make the feature not to make the feature not useful. Right. If we can change the cat fur into a dog fur and the dog fur into a cat fur, then the feature will become not useful anymore. Because we can, you know, we can we can change that arbitrarily for any image and then the classifier will have no clue. It can't be like, well, this fur could be of any of any class. Right. So the feature is not useful anymore. So this is a non robust feature. The technique you can say any feature that is useful but not robust is a non robust feature. All right. So this is kind of the definition of what robust and non robust features are. Yeah. Remember, maybe remember robust features like position of the ears and their shape and non robust features would be which direction are the individual hairs in the fur going. Right. And in our world where cat fur is going different ways than dog fur. So they now go into experimental evidence for their for their hypothesis. And here you have to understand they do two experiments which give pretty good indication that their hypothesis is actually correct. And what you have to understand before this is is two things. First of all, here you basically you just have to assume that they already they have some procedure where they can do the following where they can take an image of the training data set and they can decompose it into its robust and non robust features. Right. Don't I mean don't ask yet how they do this. But they can decompose it into these two parts. Right. So that's assumption one. They have a procedure that can actually do that. And then number two is what they what they do here is basically the general theme of these experiments is they they have a training data set. Right. This is the original training. They create a derived version of it. So let's put a tick here. This is a derived version of the data set. Then they train a regular neural network with that. So what you can do with a neural network if you train one. All right. What you usually do is you feed images X you feed images in it gives you some output Y hat and you say well but I know why is the true label. So I feed an image of a cat that the network says airplane. You say well but this should be a cat. So please make this why more to be more to be. Please make this why had more be like why. And then you have a loss function here. You say this is wrong. Please correct this. You back propagate and all the network in here will update to make that a bit more likely. That's how you train usually in our network. Now what you can do is if you want to become robust adversarial examples you can do what is called adversarial training which means that you have the same network here. But of each of the training data points you create a derived version an adversarial example to that to this X you feed the adversarial examples through the network together with the original examples. Then this will give you some why hat to and then you say but this should also be equal to why. Basically you train the classifier also on adversarial examples right. Since the hypothesis is if you train on an image data set then you can teach the classifier about that data set right. Like you do with the regular data set say well OK I can now just train on adversarial examples and my classifier will be able to better classify these correctly right. This usually works it's called adversarial training and it's been a kind of standard method to make your classifier robust. They don't do that here. They don't do this. They simply want to say OK we now have we have a regular training procedure right like this except for what we change is here the training data set. We change this to in one case for example only robust images. So we've changed all the X to be only robust and we do the regular training procedure. And then we evaluate that resulting classifier here this thing we evaluate that. How does that behave. It's kind of a new approach where you modify the date the original data set. So what did they do. First of all they decompose this training data set into a version that is only robust features right. We assume we have such a procedure. We then train a regular neural network on that right. We train a regular neural network on this on this data set and what we get is two things. First of all good standard accuracy. What does good standard accuracy mean. It means that we we can test it on what's called the unmodified test set. So the the test set the original test set of the data set the test set belonging to this training data set. We can test it on that and it works just fine. Right. So that basically means that the robust features are predictive of the of the kind of they generalize well. It means that if I train a classifier only on robust features that can actually classify well to to the to the test set. Right. So that means that's standard accuracy standard accuracy is how well do I classify the test set just an unmodified test set. So they also obtain good robust accuracy which means that what is robust accuracy. Robust accuracy means your accuracy on adversarial examples of the test set. And usually classifiers are vulnerable to this classifier is usually obtained good standard accuracy but bad robust accuracy. But if I only train my classifier on what they call robust features then I all of a sudden retain good standard accuracy. But I also get good robust accuracy which means that. It gives pretty good support to their hypothesis that the adversarial examples are abusing the fact that the classifiers learn the non robust features. Since if I don't have any non robust features it means my classifier can't learn any non robust features which in turn means my classifier isn't vulnerable to adversarial attacks because they would abuse the fact that the classifier has learned about the non robust features. So that's pretty good evidence for their hypothesis. Second thing they do is they now create this on this modified data set where they only have non robust features. Right. So the only thing they have is non robust features. Again they train a standard neural network. They train just a regular neural network on that and they also get good standard accuracy. So this means that also the non robust features as we seen like the cats fur direction can lead to you generalize well to the test set since in the test set also the cats will have that property. But you get bad robust accuracy and this gives further support to their hypothesis if you train a classifier on only non robust features. They are features because they generalize well but they are very vulnerable because they're non robust. Right. So the classifier that has learned about non robust features is vulnerable. They didn't do a third experiment which I find pretty cool where they take they take the training image and of course it's an unmodified training image. So it's robust features will basically say this is a dog. It's non robust features will also say this is a dog because it's a training image of a dog. And what they then do is they derive from this dog an adversarial example towards the cat class. Right. So what does it mean in their hypothesis if their hypothesis is correct. It now means that the robust features still say it's a dog. We can also see this here right. The kind of big shape of the image still is a dog to us humans. But the non robust features will say it's a cat. Right. This hinges on their hypothesis that adversarial examples actually abuse the non robust features. Right. They create an adversarial example. So if their hypothesis is correct the non robust features now say that's a cat. So they derive an entire data set where they change every image to another image and they also change the labels accordingly. And then they train again a regular neural network on this and they look what happens on the unmodified test set. So the unmodified test set will. So imagine if you're the you're this classifier and what you get is an image X and it has robust features. That's a dog and has non robust features say cat and its label. You're asked to predict cat. Right. And then you see the next image and the next image X to the non robust features. Maybe it's derived from some other class it will say plain. But the robust the non robust features again say cat. Right. And you're asked to predict cat. So basically the constructed data set where the non robust features always agree with with the label but the robust features they don't. So naturally what you can expect is the classifier will learn to disregard the robust features because they're no longer useful. Right. But it will actually only will learn to view these features. It's different from before before we only had these features. Now we these features are still in there. Right. But they're not informative. So the classifier will naturally learn to pick up on the non robust features and classify and classify according to them so much that if we now test on the test set and we feed in an actual cat. Right. It's of course it's robust features will say cat and its non robust features will say cat and the classifier is able to accurately predict. This is a cat even though the all the images of cats it has seen during training were actually of basically of non cats of here a dog. So this is pretty cool and shows that kind of these these features that these non robust features that adversarial examples abuse since they're created by adversarial examples. They they are actually predictive and generalize to the test set. So that's pretty pretty good evidence for their hypothesis so far. Now the kind of final remaining question is how do they create what is the procedure where they can create a robust and then basically non robust version of the data set. And here is kind of where we get into the into the sort of what I find. Yeah. So here you see basically examples of so this is an original image of a ship in the CIFAR 10 data set I believe. And this is a robust sample. So these are only robust features of the ship. And this is a ship made with only non robust features you see is actually a moose. But the non robust features have been changed to ship. So the way they construct a robust version of the data set. They have a formal definition but the way they do it is as follows. So and then they say OK here is where we where we get into the details. They say imagine we have a classifier. Right. The classifier outputs features and here we call them here they call them G which is the representation. It can be larger than features. It can be a bigger class. But in essence G is the features which then goes into the into the classifier and into the labels and so on. So the neural network outputs the features inputs some X. Now what if what if I have another X let's say X prime and I just initialize this with random noise. And if I feed this and I get G prime here and I try to make the two as close as possible by changing X. So I'm going to change my X here. Basically I'm going to change my image such that the outputs the features here match each other as close as possible. What does it mean? And I do this via back propagation right. I match these and I back propagate to X. I can do that with gradient descent. What happens is that my image X will basically pick up will match the image. My X prime will match the X in all the ways that are relevant for the features. Basically I will transfer all of the features from X to X prime. But nothing else right since I start with random. Now what if my classifier and that's what they do. What if the classifier is a robust classifier. So remember we talked about we can actually robustify a classifier by doing adversarial training. What if I have a classifier like such that is robust. If I input an X and it outputs me a feature representation of X. If the classifier is robust that representation will only contain robust features. And then if I have a second image X or and I started from random noise and I match the representation of X. And by changing XR basically I will transfer all of the robust features from X. But nothing else right. Given that I start from random noise here this means random noise has no features. That's the assumption. Random noise has no features since it's random noise. And if I transfer only the robust features basically what I've done is I've have now an image that I know has no non robust features. And only robust features of X. So that's how they derive a robustified version of X. Second how do they derive a non robust version. And that's even even easier if I have a classifier. A regular classifier and I want a non robust version of X. I have X input output G output some label. What I do is I simply derive an adversarial example of X like we did before adversarial example in here out here. And that gives me some X Y2 which is different from Y right. If I have a adversarial example then basically I've transferred. I've transferred the non robust features that lead to class Y2. I've transferred the non robust features here while still maintaining the robust features from here. So if this is too abstract imagine here X is an image of a dog right dog. And I derive from it an adversarial image that now says airplane right. So the robust features will still be of a dog will still be of the original image. But the non robust features will be of the airplane class. So that's how I derive a non robust non robust version that has features of kind of one. Robust features of one class but non robust features of the other class. That's what you see up here with the moose right. The moose clearly has been started from the image of a moose and then has been has received non robust features from the ship class. And that's just your classic adversarial example procedure. So that's the that's the kind of procedure. And so what's kind of my criticism here if you look at the first part the first part where they say well in order to determine what the robust features are we actually need a classifier that's already robust. So we've seen before we have a we have a data set sorry let's go up here. They say aha here we have a data set right and we can disentangle this and then it will which color have we not used we have a data set. We only we robustify the data set to a robust data set. We train a standard neural network and that gives us good robust accuracy which is really cool because we don't do anything special during training and we still get good robust accuracy. But in order to do this procedure here this one you actually have to have a robust classifier right. You have to have this already robustified classifier which you have obtained by adversarially training the robust classifier. Basically what you're doing now is you take this adversarial training procedure which the point here is that you don't do anything different during training right. But here you take the adversarial training procedure and via training the robust classifier via changing this data set here you basically get good robust accuracy which to me is just a reflection that you've obtained the data set using this robust classifier in the first place. I mean yeah of course their their method gives a hint that I can actually this is actually due to things in the data set themselves right. But there and I mean that's really important because it surely means that it's not a point of let's say the the classifier itself but it's a point of the data set which also say OK. It also explains why these adversarial examples transfer between classifiers if you have two classifiers that are different but classify the same thing they're vulnerable to the same adversarial example which basically means it must be some property of the data set that these things learn. But to do then say we have a procedure to extract the robust features and if we only train on the robust features we become robust right as here but you obtain the robust features by using a robustified classifier which you have adversarially trained to me that's kind of kind of back door in adversarial training into this whole procedure. And yeah so that's that's kind of my first criticism my second criticism is the fact that you know I mean it's it's an interesting take on this but this whole notion this whole seeing of these features are robust these features are non robust is basically just reframing the problem of adversarial examples in terms of in terms of features. It says nothing why these features are there. It's just postulating that they're there. It says nothing why they're there. It says nothing about why the classifiers pick up on them or how they do it or how you know how this is to be mitigated without first having a robustly trained network to extract the robust features. It's very much widely or not. Things are very much widely not known about these samples it's just a reframing of the problem, I feel. And it's cool experiments I mean they, it does show some a lot of things about these adversarial examples but certainly not an explanation. I find, at least that's my opinion. Alright, so down here then they show that they make an kind of simplified version of this a theoretical setting where they can analyze this. And they basically say, okay, this is generally what happens at the fundamental level at the fundamental level, you have classes, and let's say the classes are distributed like, like this right this these are the examples in the data set and they're distributed like that right. Mean, and you have some covariance. So they're distributed like that. If I have two classes like this, such as here, right, and they're distributed like that, and I create like the separator, the linear classifier, the linear classifier will classify like this it will be like super this is the best linear classifier. Right, we can calculate this accurately. But what do I say when I say okay. I want an adversarial example adversarial examples means that I can shift my examples by a little bit but achieve a big change in output. And since, since this distance here. Right, so if I have a sample here, I need to go a long way to the boundary to achieve another output but if I go into another direction. Right, if I go down here, I only need to go a very short way. And since adversarial examples as they're specified, they say, okay, we want to go a short way and the short way is characterized by going a short way in any direction, right, this is a terrible circle in any direction, we want to go a short way. That's another example. You see that if I have this any direction property, there's actually directions where this classification boundary is very, very close. And so that's what they say this is a fundamental misalignment between the geometry of the data, which is like this, and the geometry of how we specify adversarial examples, which is, you know, kind of equal in each direction, which leads to that. And they say, okay, what if I now robust parameters so what if I adversarially train my network to be robust, it basically means that I expand my data, because I add adversarial examples right of the circle here, I actually add adversarial examples, so my, my class, my data distribution will actually more like this. And my separating hyperplane will change here. And the geometry of the adversarial examples will be much more aligned with my separating hyperplane. So this is kind of a toy example of where they say this is fundamentally what's going on. There's a misalignment between the geometry of the adversarial examples and the inherent geometry of the data. So that's kind of the theoretical analysis they do. And with that, I finish here, and I hope this was clear enough and goodbye.
 Hi there! Today we're looking at Adversarial Examples Are Not Bugs, They Are Features by Andrew Elias et al.
 So this paper is pretty interesting as a catchy title and we'll try to kind of dissect what it says.
 So first of all, in the abstract they say adversarial examples have attracted significant attention,
 but the reasons for their existence and pervasiveness remain unclear.
 So if you don't know what an adversarial example is, an adversarial example is basically the following.
 Say you have an image classifier, right? Classifier, boom, neural network, image here, and the image is of a, let's say, a cat.
 Right, this is my best attempt at a cat, bang, cat. And you feed it through the classifier and the classifier says cat.
 Now if you perturb this image, if you derive an image from it and you perturb it just very slightly, very subtly,
 so you introduce some pixels here, there, here, there, right, you change some pixels in a very targeted way,
 and you feed that new image through here, then the classifier will say dog or something really, you can make it say anything like airplane or,
 I don't know, sky or whatever you want. So these are called adversarial examples.
 And it's true, their existence and pervade, the reasons for their existence and pervasiveness remain unclear.
 They say we demonstrate that adversarial examples can be directly attributed to the presence of non-robust features.
 So they're basically, their paper is about these non-robust features and they define later what they mean exactly.
 But here they say features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans.
 And this is pretty neat. So the fundamental idea, as I understand it, and I'm going to take this away right here,
 that if you have images, let's say here of cats, and I'm going to draw another one over here, if you have an image, say, of cats,
 there is multiple features in this image and the feature is something that the classifier can pick up on and kind of learn to,
 this is a horrible cat, learn to classify images from. So features that we humans generally use are a cat has ear,
 ear, eyes, whiskers, right, and the general relationship to each other of these things.
 This is what constitutes a cat. And that's how we classify it. But also they say there are other features that are also very indicative, right?
 If you think what differentiates a cat from a dog and a dog here, let's pick fluffy ears, also eyes, yeah, not going to go further with the dog too much.
 What differentiates a cat from a dog? And we, of course, we would say, well, the head shape is different and the ears are different and the relationship to them,
 to each other are different, but it could also be, and this is a simplistic right now, right? But it's also that cats, for example, have different fur than dogs.
 And yeah, being overly simplistic here, but bear with me. So let's say in our hypothetical world, cats have fur that goes like this, left to right, right?
 Every hair is basically vertical, sorry, horizontal. If you look at it like that and dog fur, on the other hand, is always like this, right?
 This is vertical, right? Top to bottom. And so the classifier might just as well pick up on the fur direction in order to classify images, right?
 Since all cats have that type of fur and all dogs have that other type of fur, the classifier might just as well pick up on that, right?
 And to us humans, we don't really pay attention to these things because they're minute, right?
 You don't look at the directions of individual hairs to classify in an animal to cat or dog.
 You would much rather go for these kind of large features like where are the ears, how do they look and so on.
 But a classifier, there's actually you can make an argument that the classifier would more likely pick up on the fur direction, right?
 In order to in order to classify since we're using convolutional neural networks and they're generally neighborhood pixel neighborhood operators.
 It can much easier pick up on these patterns than it can on the general relationship of the of the large features.
 So if a classifier now learns that cats always have fur like this and dogs always have fur like that, what we could do is we can go over here to the dog and change its fur, right?
 Change in the image, change its fur to this direction.
 Now, to us humans, that would still very much look like a dog because the fur direction is almost imperceptible.
 But to the classifier that has only learned, hey, a cat always has this type of fur and the dog always has that type of fur.
 That new image would totally look like a cat.
 Right. So this paper argues exactly that this paper argues that in the data set, there are features and these are real features like this.
 This actually could be the case that cats fur is always like that and dogs fur is always like this.
 It could be the case and the classifier could pick up on this.
 Right. And then the adversarial examples, the reason why they exist is because the classifier has picked up on these imperceptible features.
 And so by changing the features, we can change the classifiers decision.
 And without changing the image in a large scale.
 So they they say that they make this hypothesis and they kind of they say, OK, we established a widespread existence in standard data sets.
 So they kind of give supporting evidence for their hypothesis.
 And then they say, finally, we present a simple setting, which is a theoretical setting where we can rigorously tie the phenomena
 we observe to a misalignment between the human specified notion of robustness and the inherent geometry of the data.
 All right. So it's kind of different pieces of the of this paper.
 And we're going to look at them in succession.
 So the introduction, we largely skip, except that their main claim here is specifically we claim that adversarial vulnerability is a direct result of our models, sensitivity to well generalizing features in the data.
 So that's the core point, I think, is well generalizing features, which is what we mentioned.
 These are features that actually describe the data well, but but features that are kind of imperceptibly small to humans or that don't fit our notion of robustness.
 All right. So they go on and they define more clearly what they mean here.
 Here, whenever we talk of a feature, right?
 Remember, we had the our classifier here, then we input an image and the image is called X.
 Right. And that classifier, usually, if we look at it closer, consists of multiple layers of interconnected neurons, whatever.
 And the last layer will be an output layer into different classes.
 Right.
 And so the features, when they say a feature, what they mean specifically is the last here, the last representation before it goes into the classifier.
 So the way you would classify them and here they just establish a two class setting.
 The way you would establish that is you have feature one, feature two, feature three, and you have a weight vector W1 for each feature W2, W3.
 You make the inner product and that will give you a Y hat.
 Basically, if that is high, you say it's class one.
 If that is low, you say it's class minus one.
 So the classes here are plus one and minus one, just to make things simple.
 So but you see the features are basically what comes out after these layers, what is then used to make a linear classification.
 This last thing is basically just a logistic regression.
 So you can think of the features as the output of the neural network, but before it goes into the classifier.
 So a feature basically since then, it's linearly classified.
 If the feature is high, it will give a signal for one class.
 And if a feature is low, it will give a signal for the other class, depending on, of course, if this W is negative or positive.
 All right, so they say we call a feature row useful.
 And if this thing holds here, what is this thing?
 This thing means so the expectation over the dates.
 So generally in the data set, this must hold Y times the feature.
 So why is the class?
 And remember, it's plus or minus one.
 And the feature, as we've seen, is some some number Y times a feature must be higher than some some number.
 So what does it mean when a product is high?
 It means either both are high or both are low.
 So they're correlated. That's what that means.
 So basically, this is says a feature F is useful if whenever it an example, X is of class one, if it's class class one or let's if it if Y is one plus one, then F is high.
 And whenever Y is minus one, then F is low, which means it's high in the negative direction.
 Right. So this is this is our this is intuitive.
 Right. If a feature is useful, it means it should say one thing in samples of class one, then it should say another thing in samples of class two.
 Then I can actually use the feature to make a decision when it's, you know, very correlated with the class.
 So that, you know, that makes perfect sense.
 So that's kind of when is a feature useful if it correlates with the class label?
 Yes. Cool. But the usefulness simply any feature basically that classifier will extract will be useful.
 That's an assumption we can make. Otherwise, the classifier wouldn't extract it.
 So the neural network here, that's an assumption, will only extract useful features.
 Right. Because the non-useful features, there would simply be no reason for it to extract them because they don't contribute to solving the task, because they're not correlated with an output class.
 Right. So next, they define robust, robustly useful features.
 So in addition to being useful, they're now also robust.
 What does it mean? Again, we want a correlation of why and the feature to be higher than some constant.
 But not only the feature of the image X, but the feature of the image X that has been perturbed by a small perturbation.
 So and we take the infinum here over a class of perturbations.
 Of course, this class of perturbations is exactly the adversarial perturbations.
 Basically, what this means is it says that however we try to perturb X, right, and the infinum here means that the minimum correlation,
 however we try to make the feature not correlated with Y, however much we try, we can't get it lower than some some gamma, some number, right?
 We can't we can't get it down. So whatever we try to make the feature bad for the classifier, basically, we can't.
 If this holds for a feature, if this is the case, then we call that feature a robust feature.
 Right. That feature is robustly useful if it correlates, no matter how hard we try to make it not correlate.
 And of course, a non robust features, so a useful non robust feature is a feature which is useful.
 You see here is useful. But is not gamma robust feature for any gamma.
 So it is a feature that is useful like the cat fur.
 Right. So this here, an example of this would be that the cat's eyes and ear position.
 Right. We can't just make a small perturbation for the image and make the ears be somewhere completely else.
 That's just that would require a large perturbation of the image.
 So the position of the ears and eyes are pretty robust features.
 But here the cat's fur, no matter how no matter how small we we make this this gamma, we can always kind of change the fur to make the feature not to make the feature not useful.
 Right. If we can change the cat fur into a dog fur and the dog fur into a cat fur, then the feature will become not useful anymore.
 Because we can, you know, we can we can change that arbitrarily for any image and then the classifier will have no clue.
 It can't be like, well, this fur could be of any of any class.
 Right. So the feature is not useful anymore.
 So this is a non robust feature. The technique you can say any feature that is useful but not robust is a non robust feature.
 All right. So this is kind of the definition of what robust and non robust features are.
 Yeah. Remember, maybe remember robust features like position of the ears and their shape and non robust features would be which direction are the individual hairs in the fur going.
 Right. And in our world where cat fur is going different ways than dog fur.
 So they now go into experimental evidence for their for their hypothesis.
 And here you have to understand they do two experiments which give pretty good indication that their hypothesis is actually correct.
 And what you have to understand before this is is two things.
 First of all, here you basically you just have to assume that they already they have some procedure where they can do the following where they can take an image of the training data set
 and they can decompose it into its robust and non robust features.
 Right. Don't I mean don't ask yet how they do this.
 But they can decompose it into these two parts.
 Right. So that's assumption one. They have a procedure that can actually do that.
 And then number two is what they what they do here is basically the general theme of these experiments is they they have a training data set.
 Right. This is the original training. They create a derived version of it.
 So let's put a tick here. This is a derived version of the data set.
 Then they train a regular neural network with that.
 So what you can do with a neural network if you train one.
 All right. What you usually do is you feed images X you feed images in it gives you some output Y hat and you say well but I know why is the true label.
 So I feed an image of a cat that the network says airplane.
 You say well but this should be a cat. So please make this why more to be more to be.
 Please make this why had more be like why. And then you have a loss function here.
 You say this is wrong. Please correct this.
 You back propagate and all the network in here will update to make that a bit more likely.
 That's how you train usually in our network.
 Now what you can do is if you want to become robust adversarial examples you can do what is called adversarial training which means that you have the same network here.
 But of each of the training data points you create a derived version an adversarial example to that to this X you feed the adversarial examples through the network together with the original examples.
 Then this will give you some why hat to and then you say but this should also be equal to why.
 Basically you train the classifier also on adversarial examples right.
 Since the hypothesis is if you train on an image data set then you can teach the classifier about that data set right.
 Like you do with the regular data set say well OK I can now just train on adversarial examples and my classifier will be able to better classify these correctly right.
 This usually works it's called adversarial training and it's been a kind of standard method to make your classifier robust.
 They don't do that here. They don't do this.
 They simply want to say OK we now have we have a regular training procedure right like this except for what we change is here the training data set.
 We change this to in one case for example only robust images. So we've changed all the X to be only robust and we do the regular training procedure.
 And then we evaluate that resulting classifier here this thing we evaluate that.
 How does that behave. It's kind of a new approach where you modify the date the original data set.
 So what did they do. First of all they decompose this training data set into a version that is only robust features right.
 We assume we have such a procedure. We then train a regular neural network on that right.
 We train a regular neural network on this on this data set and what we get is two things.
 First of all good standard accuracy. What does good standard accuracy mean.
 It means that we we can test it on what's called the unmodified test set.
 So the the test set the original test set of the data set the test set belonging to this training data set.
 We can test it on that and it works just fine. Right.
 So that basically means that the robust features are predictive of the of the kind of they generalize well.
 It means that if I train a classifier only on robust features that can actually classify well to to the to the test set.
 Right. So that means that's standard accuracy standard accuracy is how well do I classify the test set just an unmodified test set.
 So they also obtain good robust accuracy which means that what is robust accuracy.
 Robust accuracy means your accuracy on adversarial examples of the test set.
 And usually classifiers are vulnerable to this classifier is usually obtained good standard accuracy but bad robust accuracy.
 But if I only train my classifier on what they call robust features then I all of a sudden retain good standard accuracy.
 But I also get good robust accuracy which means that.
 It gives pretty good support to their hypothesis that the adversarial examples are abusing the fact that the classifiers learn the non robust features.
 Since if I don't have any non robust features it means my classifier can't learn any non robust features which in turn means my classifier isn't vulnerable to adversarial attacks because they would abuse the fact that the classifier has learned about the non robust features.
 So that's pretty good evidence for their hypothesis.
 Second thing they do is they now create this on this modified data set where they only have non robust features.
 Right. So the only thing they have is non robust features.
 Again they train a standard neural network.
 They train just a regular neural network on that and they also get good standard accuracy.
 So this means that also the non robust features as we seen like the cats fur direction can lead to you generalize well to the test set since in the test set also the cats will have that property.
 But you get bad robust accuracy and this gives further support to their hypothesis if you train a classifier on only non robust features.
 They are features because they generalize well but they are very vulnerable because they're non robust.
 Right. So the classifier that has learned about non robust features is vulnerable.
 They didn't do a third experiment which I find pretty cool where they take they take the training image and of course it's an unmodified training image.
 So it's robust features will basically say this is a dog.
 It's non robust features will also say this is a dog because it's a training image of a dog.
 And what they then do is they derive from this dog an adversarial example towards the cat class.
 Right. So what does it mean in their hypothesis if their hypothesis is correct.
 It now means that the robust features still say it's a dog.
 We can also see this here right.
 The kind of big shape of the image still is a dog to us humans.
 But the non robust features will say it's a cat.
 Right. This hinges on their hypothesis that adversarial examples actually abuse the non robust features.
 Right. They create an adversarial example. So if their hypothesis is correct the non robust features now say that's a cat.
 So they derive an entire data set where they change every image to another image and they also change the labels accordingly.
 And then they train again a regular neural network on this and they look what happens on the unmodified test set.
 So the unmodified test set will. So imagine if you're the you're this classifier and what you get is an image X and it has robust features.
 That's a dog and has non robust features say cat and its label.
 You're asked to predict cat. Right. And then you see the next image and the next image X to the non robust features.
 Maybe it's derived from some other class it will say plain. But the robust the non robust features again say cat.
 Right. And you're asked to predict cat. So basically the constructed data set where the non robust features always agree with with the label but the robust features they don't.
 So naturally what you can expect is the classifier will learn to disregard the robust features because they're no longer useful.
 Right. But it will actually only will learn to view these features.
 It's different from before before we only had these features. Now we these features are still in there. Right.
 But they're not informative. So the classifier will naturally learn to pick up on the non robust features and classify and classify according to them so much that if we now test on the test set and we feed in an actual cat.
 Right. It's of course it's robust features will say cat and its non robust features will say cat and the classifier is able to accurately predict.
 This is a cat even though the all the images of cats it has seen during training were actually of basically of non cats of here a dog.
 So this is pretty cool and shows that kind of these these features that these non robust features that adversarial examples abuse since they're created by adversarial examples.
 They they are actually predictive and generalize to the test set.
 So that's pretty pretty good evidence for their hypothesis so far.
 Now the kind of final remaining question is how do they create what is the procedure where they can create a robust and then basically non robust version of the data set.
 And here is kind of where we get into the into the sort of what I find.
 Yeah. So here you see basically examples of so this is an original image of a ship in the CIFAR 10 data set I believe.
 And this is a robust sample.
 So these are only robust features of the ship.
 And this is a ship made with only non robust features you see is actually a moose.
 But the non robust features have been changed to ship.
 So the way they construct a robust version of the data set.
 They have a formal definition but the way they do it is as follows.
 So and then they say OK here is where we where we get into the details.
 They say imagine we have a classifier.
 Right. The classifier outputs features and here we call them here they call them G which is the representation.
 It can be larger than features.
 It can be a bigger class.
 But in essence G is the features which then goes into the into the classifier and into the labels and so on.
 So the neural network outputs the features inputs some X.
 Now what if what if I have another X let's say X prime and I just initialize this with random noise.
 And if I feed this and I get G prime here and I try to make the two as close as possible by changing X.
 So I'm going to change my X here.
 Basically I'm going to change my image such that the outputs the features here match each other as close as possible.
 What does it mean? And I do this via back propagation right.
 I match these and I back propagate to X.
 I can do that with gradient descent.
 What happens is that my image X will basically pick up will match the image.
 My X prime will match the X in all the ways that are relevant for the features.
 Basically I will transfer all of the features from X to X prime.
 But nothing else right since I start with random.
 Now what if my classifier and that's what they do.
 What if the classifier is a robust classifier.
 So remember we talked about we can actually robustify a classifier by doing adversarial training.
 What if I have a classifier like such that is robust.
 If I input an X and it outputs me a feature representation of X.
 If the classifier is robust that representation will only contain robust features.
 And then if I have a second image X or and I started from random noise and I match the representation of X.
 And by changing XR basically I will transfer all of the robust features from X.
 But nothing else right.
 Given that I start from random noise here this means random noise has no features.
 That's the assumption. Random noise has no features since it's random noise.
 And if I transfer only the robust features basically what I've done is I've have now an image that I know has no non robust features.
 And only robust features of X.
 So that's how they derive a robustified version of X.
 Second how do they derive a non robust version.
 And that's even even easier if I have a classifier.
 A regular classifier and I want a non robust version of X.
 I have X input output G output some label.
 What I do is I simply derive an adversarial example of X like we did before adversarial example in here out here.
 And that gives me some X Y2 which is different from Y right.
 If I have a adversarial example then basically I've transferred.
 I've transferred the non robust features that lead to class Y2.
 I've transferred the non robust features here while still maintaining the robust features from here.
 So if this is too abstract imagine here X is an image of a dog right dog.
 And I derive from it an adversarial image that now says airplane right.
 So the robust features will still be of a dog will still be of the original image.
 But the non robust features will be of the airplane class.
 So that's how I derive a non robust non robust version that has features of kind of one.
 Robust features of one class but non robust features of the other class.
 That's what you see up here with the moose right.
 The moose clearly has been started from the image of a moose and then has been has received non robust features from the ship class.
 And that's just your classic adversarial example procedure.
 So that's the that's the kind of procedure.
 And so what's kind of my criticism here if you look at the first part the first part where they say well in order to determine what the robust features are we actually need a classifier that's already robust.
 So we've seen before we have a we have a data set sorry let's go up here.
 They say aha here we have a data set right and we can disentangle this and then it will which color have we not used we have a data set.
 We only we robustify the data set to a robust data set.
 We train a standard neural network and that gives us good robust accuracy which is really cool because we don't do anything special during training and we still get good robust accuracy.
 But in order to do this procedure here this one you actually have to have a robust classifier right.
 You have to have this already robustified classifier which you have obtained by adversarially training the robust classifier.
 Basically what you're doing now is you take this adversarial training procedure which the point here is that you don't do anything different during training right.
 But here you take the adversarial training procedure and via training the robust classifier via changing this data set here you basically get good robust accuracy which to me is just a reflection that you've obtained the data set using this robust classifier in the first place.
 I mean yeah of course their their method gives a hint that I can actually this is actually due to things in the data set themselves right.
 But there and I mean that's really important because it surely means that it's not a point of let's say the the classifier itself but it's a point of the data set which also say OK.
 It also explains why these adversarial examples transfer between classifiers if you have two classifiers that are different but classify the same thing they're vulnerable to the same adversarial example which basically means it must be some property of the data set that these things learn.
 But to do then say we have a procedure to extract the robust features and if we only train on the robust features we become robust right as here but you obtain the robust features by using a robustified classifier which you have adversarially trained to me that's kind of kind of back door in adversarial training into this whole procedure.
 And yeah so that's that's kind of my first criticism my second criticism is the fact that you know I mean it's it's an interesting take on this but this whole notion this whole seeing of these features are robust these features are non robust is basically just reframing the problem of adversarial examples in terms of in terms of features.
 It says nothing why these features are there.
 It's just postulating that they're there. It says nothing why they're there. It says nothing about why the classifiers pick up on them or how they do it or how you know how this is to be mitigated without first having a robustly trained network to extract the robust features.
 It's very much widely or not.
 Things are very much widely not known about these samples it's just a reframing of the problem, I feel.
 And it's cool experiments I mean they, it does show some a lot of things about these adversarial examples but certainly not an explanation.
 I find, at least that's my opinion.
 Alright, so down here then they show that they make an kind of simplified version of this a theoretical setting where they can analyze this.
 And they basically say, okay, this is generally what happens at the fundamental level at the fundamental level, you have classes, and let's say the classes are distributed like, like this right this these are the examples in the data set and they're distributed like that right.
 Mean, and you have some covariance.
 So they're distributed like that. If I have two classes like this, such as here, right, and they're distributed like that, and I create like the separator, the linear classifier, the linear classifier will classify like this it will be like super this is the best linear classifier.
 Right, we can calculate this accurately.
 But what do I say when I say okay.
 I want an adversarial example adversarial examples means that I can shift my examples by a little bit but achieve a big change in output.
 And since, since this distance here.
 Right, so if I have a sample here, I need to go a long way to the boundary to achieve another output but if I go into another direction.
 Right, if I go down here, I only need to go a very short way. And since adversarial examples as they're specified, they say, okay, we want to go a short way and the short way is characterized by going a short way in any direction, right, this is a terrible circle in any direction, we want to go a short way.
 That's another example. You see that if I have this any direction property, there's actually directions where this classification boundary is very, very close.
 And so that's what they say this is a fundamental misalignment between the geometry of the data, which is like this, and the geometry of how we specify adversarial examples, which is, you know, kind of equal in each direction, which leads to that.
 And they say, okay, what if I now robust parameters so what if I adversarially train my network to be robust, it basically means that I expand my data, because I add adversarial examples right of the circle here, I actually add adversarial examples, so my, my class, my data distribution will actually more like this.
 And my separating hyperplane will change here. And the geometry of the adversarial examples will be much more aligned with my separating hyperplane.
 So this is kind of a toy example of where they say this is fundamentally what's going on. There's a misalignment between the geometry of the adversarial examples and the inherent geometry of the data.
 So that's kind of the theoretical analysis they do. And with that, I finish here, and I hope this was clear enough and goodbye.