The Dimpled Manifold Model of Adversarial Examples in Machine Learning (Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper adversarial examples goodfellow goodfellow adversarial attacks adversarial attacks on neural networks features not bugs madry dimpled manifold why do adversarial examples exist adversarial examples explanation adversarial attacks explanation computer vision decision boundary data manifold low dimensional manifold what are adversarial examples what is deep learning
#adversarialexamples #dimpledmanifold #security

Adversarial Examples have long been a fascinating topic for many Machine Learning researchers. How can a tiny perturbation cause the neural network to change its output by so much? While many explanations have been proposed over the years, they all appear to fall short. This paper attempts to comprehensively explain the existence of adversarial examples by proposing a view of the classification landscape, which they call the Dimpled Manifold Model, which says that any classifier will adjust its decision boundary to align with the low-dimensional data manifold, and only slightly bend around the data. This potentially explains many phenomena around adversarial examples. Warning: In this video, I disagree. Remember that I'm not an authority, but simply give my own opinions.

OUTLINE:
0:00 - Intro & Overview
7:30 - The old mental image of Adversarial Examples
11:25 - The new Dimpled Manifold Hypothesis
22:55 - The Stretchy Feature Model
29:05 - Why do DNNs create Dimpled Manifolds?
38:30 - What can be explained with the new model?
1:00:40 - Experimental evidence for the Dimpled Manifold Model
1:10:25 - Is Goodfellow's claim debunked?
1:13:00 - Conclusion & Comments

Paper: https://arxiv.org/abs/2106.10151
My replication code: https://gist.github.com/yk/de8d987c4eb6a39b6d9c08f0744b1f64
Goodfellow's Talk: https://youtu.be/CIfsB_EYsVI?t=4280

Abstract:
The extreme fragility of deep neural networks when presented with tiny perturbations in their inputs was independently discovered by several research groups in 2013, but in spite of enormous effort these adversarial examples remained a baffling phenomenon with no clear explanation. In this paper we introduce a new conceptual framework (which we call the Dimpled Manifold Model) which provides a simple explanation for why adversarial examples exist, why their perturbations have such tiny norms, why these perturbations look like random noise, and why a network which was adversarially trained with incorrectly labeled images can still correctly classify test images. In the last part of the paper we describe the results of numerous experiments which strongly support this new model, and in particular our assertion that adversarial perturbations are roughly perpendicular to the low dimensional manifold which contains all the training examples.

Abstract: Adi Shamir, Odelia Melamed, Oriel BenShmuel

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today we're going to look at the dimpled manifold model of adversarial examples in machine learning by Adi Shamir, Odelia Melamed and Oriol Ben-Schmuel. This paper on a high level proposes a new way of looking at the phenomenon of adversarial examples in machine learning, specifically in deep learning, and they proposed this model called the dimpled manifold model, essentially arguing that classifiers put their decision boundaries right next to the manifold of data, while only slightly sort of curving it around the data like this. Now the data manifold being low dimensional, this results in a situation where you can cross the decision boundary really easily if you simply go perpendicular to the data manifold, which also is perpendicular to the decision boundary, and if because it's just such a small dimple there, the decision boundary is pretty close, and that's how you end up with adversarial examples that are super easy to get. So it's not a new attack, a new defense, anything like this, it's simply a mental framework of explaining why adversarial examples exist on a high level. They have some conceptual thought experiments, they have some explanations, and some real-world experiments. Now I personally don't think that this is entirely, it's not necessarily incorrect, but I don't think that this is really useful to think in this way, and I'm gonna explain why. In general my opinion of this is it doesn't really add anything, and I think it explains less than the models we already had. Yeah so that's my opinion, I'm gonna get to it. Specifically also the experiments they propose, I think that there is a big Occam's razor failure right there. But as I said we're gonna get to all of this, I'm gonna go through the paper and I want you to make up your own mind, even though I'm going to try to bias you. So yeah this is not a neutral channel in case you haven't noticed. Alright so if you like content or if you dislike it tell me in the comments, tell me what you think of the paper, whether it makes sense, whether it doesn't make sense, and so on. I'd be very interested to see what you have to say. Yeah I read the comments, so please. They say the extreme fragility of deep neural networks when presented with tiny perturbations, yeah but okay this starts out how every single adversarial examples paper always starts out saying okay deep neural networks are extremely fragile, there's this phenomenon of adversarial examples. Now if you don't know what adversarial examples are, really briefly essentially what this is, it's a phenomenon where you take an image like the thing here on the left, the neural network thinks it's a plane with a very high probability and you change it to this thing right here, which you as a human can't even tell it's different, however the neural network will think that this is now a bird with very high probability and the this is the change that you made. It's magnified for you to see, it kind of looks like random noise but it's a very particular noise that makes the neural network think it's something different and this is just it's tiny in the in its norm. So you don't see a difference. Now bird here is kind of close to plane but you can change this into anything, literally anything you want, you can change this into banana or I don't know dog or any class you want using these techniques. So it's not about being close it's really kind of a separate phenomenon. So that's adversarial examples and many frameworks have been proposed in order to explain these adversarial examples and they make a they make a nice overview right here. Many have been proposed over the last eight years that DNNs are too nonlinear, that they're too linear, that they were trained with insufficient number of training examples, that are just rare cases where they error, that images contain robust and non robust features etc. They say however none of these vague qualitative ideas seem to provide a simple intuitive explanations for the existence and bizarre properties of adversarial examples. So that is pretty harsh criticism specifically the first ones are kind of yeah but specifically this last one that images contain robust and non robust features which is sort of the leading hypothesis right now of why adversarial examples exist and what they are and then here saying none of these can none of these vague qualitative ideas seem to provide a simple intuitive explanation for the existence. Like let's see whether or not they're gonna do better okay. So also in the abstract they go on and they say okay they introduced this new conceptual framework which they call the dimpled manifold model which provides a simple explanation for why adversarial examples exist, why their perturbations have such tiny norms, why these perturbations look like random noise and why a network which was adversarially trained with incorrectly labeled images can still correctly classify test images. Now this last part if you're not familiar with the literature it might come to you a bit random this why a network which was adversarially trained with incorrectly labeled images can still correctly classify test images. This is a famous experiment from the group of Alexander Modri where also this hypothesis this one the robust and non robust feature comes from and any attempt at explaining adversarial examples after this paper has to explain why that experiment makes sense because it's kind of a non intuitive experiment and we're gonna get to that as well but just so you know that's why they write it in the abstract. Now I personally think they don't have a good like this model here doesn't have a good explanation for why that works. They're sort of hand wavy trying in any case. So they say in in the last part of the paper we describe the results of numerous experiments which strongly support this new model and in particular our assertion that adversarial perturbations are roughly perpendicular to the low dimensional manifold which contains all the training examples. Okay also remember this experiment they strongly support what in particular the assertion that adversarial perturbations are roughly perpendicular to the low dimensional manifold which contains all the training examples. Now remember this that the experiments are supposed to support this particular claim because also that is going to be important down the road. Okay so let's get into the dimpled manifold model. What is it? What do these authors propose? I'm gonna try as best as I can to say what the authors are saying in the paper. So they claim that there is an old mental image of adversarial examples and the old mental image is here. They say we think the old mental image is based on the highly misleading 2d image on the left side of figure one and that's this thing right here. So the old mental image is that there is a there is a data space right this here if you think of pic of images as data points this would be the pixel space right so this is images with two pixels right now in this conceptual framework but you have to sort of think yourself into higher dimension. So they claim the old mental image is the following you have sort of the data distributed somehow in this space the data being the all the set of natural images or images you consider which is kind of these these sub space these subgroups right here there are a bunch of images right there and there and also there and there so these are images of two different classes the red class and the blue class now they're distributed like this and what is a classifier supposed to do a classifier is supposed to put a decision boundary between them and that's what they draw in here so this would be sort of a reasonable decision boundary between the two classes right so now what do you do if you want to create an adversarial examples well necessarily you have to start at an image of a class this one maybe and you have to cross the decision boundary right you want to fool the classifier ergo necessarily by definition you have to cross the decision boundary so what do you do the the easiest way to do this is to sort of go straight towards the decision boundary which is approximately in this direction right here and then once you cross the decision boundary you are done you're on the other side you have created an adversarial example provided of course that the image still kind of looks like the original image and so they say this has this has many many problems here they say the in this mental this mental image adversarial examples are created by moving the given images along the green arrows towards some kind of centroid of the nearest training images with the opposite label in which they mean this this thing right here so we would move the images towards the other class towards images of the other class and they say as stated for example by Ian Goodfellow in his lecture at this time I'm gonna cut this in right here I've said that the same perturbation can fool many different models or the same perturbation can be applied to many different clean examples I've also said that the subspace of adversarial perturbations is only about 50 dimensional even if the input dimension is 3,000 dimensional so how is it that these subspaces intersect the reason is that the choice of the subspace directions is not completely random it's generally going to be something like pointing from one class centroid to another class centroid and if you look at that vector and visualize it as an image it might not be meaningful to a human just because humans aren't very good at imagining what class centroid look like and we're really bad at imagining differences between centroid but there is more or less this systematic effect that causes different models to learn similar linear functions just because they're trying to solve the same task okay so it really appears like Goodfellow says this thing right here however they claim now they claim this doesn't make sense so they claim that you should think about adversarial examples in a different way and this is their dimpled manifold hypothesis so what is their dimpled manifold hypothesis they say what you have to do is you have to think about the data manifold in the higher dimensional space that they hand the higher dimensional input space so in this case they consider instead of here this 2d landscape they consider the 3d landscape so this would be the pixel space right now we consider three pixel images and the data is embedded in a low dimensional manifold in this higher space so because if you think about all combinations of pixels that are possible so not all of them are natural images in fact only very few of the possible combinations of pixels are natural images or images that make sense to you as a human or are images that you could potentially generate by going out with a camera so the data you're considering lives on a very low dimensional manifold in this big space and you have to explicitly think about that now the data is the data manifold here is represented in this in this sheet in the middle and on this manifold you're going to have your different classes of data here the blue or one class and the red or the other class what this paper claims is that what classifiers do what neural networks do when they classify the training data here is they go and they lay their decision boundary instead of so in the old model you would have thought maybe something like this happened where you put your decision boundary sort of in the middle between the two classes right crossing the manifold right here so you sort of put it in the middle between the two classes and then when you have to create an adversarial example again what you would do is you would maybe start here what you would have to do is you would go straight towards the decision boundary right here okay crossing the decision boundary and then on the other side you'd have an adversarial example in this new model what they claim is the decision boundary actually doesn't look like this right here okay the decision boundary actually is very much aligned with the manifold of data as you can see right here so this mesh that they show is the decision boundary now and their claim is that that usually just aligns with the manifold of data however around the actual data around the training samples what the classifier will do is it will create these what these dimples okay and these dimples are just tiny well dimples tiny perturbations in the decision manifold such that the data is on the correct side of the decision manifold sorry of the decision boundary right so the blue points here are under or one side of the decision boundary and the red points are on the other side of the decision boundary and for the rest the decision boundary just aligns with the data the data manifold now if you want to make an adversarial example now what you have to do again you start from an image and again you walk straight towards the decision boundary however now you don't have to go like this you what you can do is you can go simply perpendicular to the data manifold and you will cross the decision boundary very quickly because the dimple you're in is kind of shallow and they give a reason why the dimples are shallow because they claim this is results from training these models and that explains some things so the difference is the difference is we started out from this to make an adversarial example we have to go towards the decision boundary okay if we sort of transfer this image into higher dimensions it looks like this in the middle again in order to make an adversarial example we have to go towards the decision boundary now in the old mental image going perpendicular to the decision boundary means walking on the data manifold because we walk from this group of data towards this group of data you can see right here that we're walking on the data manifold when we walk perpendicular to the decision boundary whereas in the new model walking perpendicular to the decision boundary coincides with also walking perpendicular to the data manifold so this is the difference right here that they that they claim so this they say there's we call this conceptual framework the dimpled manifold model and note that it makes three testable claims about the kinds of decision boundaries created by trained deep neural networks first natural images are located in a K dimensional manifold where K is much smaller than N second deep neural network decision boundaries pass very close to this image manifold and third the gradient of the classifications confidence level has a large norm and points roughly perpendicular to the image manifold alright so these are these are the claims that they're going to make to be tested and to be supported by experiments I guess so I hope I've represented enough what the authors claim right here I hope they would agree that I've represented this is accurately so now where is the problem with this in my opinion the problem isn't necessarily with what they claim right here it's it's you know I don't necessarily disagree with this mental image I don't necessarily disagree with these claims in fact that the data is on low dimensional manifold this we've this is kind of commonly agreed upon assumption right as I said not all the possible pixels combinations make good natural images and that the fact that it is then a manifold is a commonly held assumption decision boundaries pass very close to the image manifold well the fact that we can generate adversarial examples right already means that decision boundaries pass very close to the image manifold so this also is not news this this has been like in everybody's conceptual framework for the last five years at least and then third the gradient of the classifications confidence level has a large norm and points roughly perpendicular to the image manifold and this claim right here I'm pretty pretty sure there so this is not a trivial claim which yes okay this is not something that was like set around much however I'm going to claim that their model is not the only model by far that makes this happen or any something like this specifically when we go look at the experiments I'm going to show you that this doesn't necessarily support their claims it doesn't disprove them right but it also doesn't necessarily support them just because they show that okay so the other problem I have with this is that this in this thing they build up as ooh this is this is the old mental image this is how people thought about adversarial examples until now I look I just I disagree like this it's a bit of a it's a bit of a straw man almost I feel like this no one no one thought no one that is sort of in the literature of adversarial examples thought or thinks that this is an appropriate model for what is happening like we know that these distances here are very small right the distance until you cross the decision boundary and we know also like if this were true you should just be able to go to the decision boundary and then go the same distance right and then at some point you would actually arrive at a sample of a different class so you could you could actually transform images into the other class by simply going into the adversarial direction which is precisely what we don't see right we see the image still largely looks the same what gets added looks like a bit of noise okay so no no one was having this mental image because clearly this mental image is it is not appropriate for adversarial examples as well as saying look if you think of this in sort of higher dimensions and I realize I've drawn this decision boundary but this is what they describe in the text then I don't I don't see that this is the correct way of like there are many different kinds of decision boundaries that are compatible with with the decision boundary right here by the way this decision boundary I drew doesn't even separate the classes all the classes correctly what I'm saying is that also if you consider the decision boundary that for example looks like out of colors looks like this that also crosses here however it's sort of kind of flat like this but it's still a linear decision boundary right like this okay so this is above and the other part is below if you think of this if you project this down it looks the same in 2d and in 3d it's also explains that decision boundaries are very close to the data samples it's a bit different though than this dimpled manifold hypothesis right if you I think the at least in my estimation what's happening is much more that you have just a bunch of these kind of linear decision boundaries flying around right here partitioning up the space and so on and this might result in a similar situation as here but it has quite different predictions in form of what it does then what it does right here here it's sort of a flat manifold dimpling around the data whereas here it's kind of the class are separating the space into many regions always trying to sort of distinguish one class from the other and yeah so might end up bit the same but I don't think they give a fair shot at what we know so far like we that this model is not a a model that people hold in general especially the one on the left I can make an attempt at making a mental model that people hold so far maybe it's just me but I have a feeling this is a bit more so the model that I call let's call it something because they call it there something right I call mine the squishy feet the stretchy feature model okay let's contrast this with the stretchy feature model so what I want to do is I have two features and this is a coordinate system in feature space okay so there's two features this in feature space I mean sort of the the last representation before the classification layer in feature space the two classes look like this so there is the red class and there is the blue class and you can see right here there are two features and for some reason the network can classify along these two features maybe because there are other classes other data points so we can't put a decision boundary like this between the two we can classify along the two features okay so you can see there are two features right here feature one and feature two and both features are actually pretty good features for keeping these two data points apart okay now there are empty spaces as you can see right here which we're gonna get to in a second but you can you can use both features and ideally a classifier would actually use both features it would say you know if feature one is high it's there probably a red class if feature two is low it's probably the red class and the combination makes even more of the red class okay however since we are in a deep neural network which is has transformations it transforms the data along the way if you look at the same situation in input space so in the actual pixel space it looks different and this is due to not necessarily the non-linearity of things but actually it is due to the linear transformation it's actually the problem of adversarial examples at least in my estimation appears to happen in the linear layers if you think of for example like eigenvectors of matrices and the largest eigenvalues determine how far you can go in a particular direction by having a sort of a standard input delta and the same happens here by the way this is why spectral norm regularization tends to work at least a little bit against adversarial examples so what I mean is if you look at the scale of these features right they are like one two three four five of this features one two three four five if you look in the input space some of the features are going to have roughly the same scale right here and these features are going to be features that you have to change the input a lot in order to change the feature a lot what do I mean by this this is something like the shape of an of an image okay if you think of a cat the general shape of a cat you know it has it has two years pointy it has a head and and so on that's the general shape of a cat sorry that is actually the left right feature right this is the the left right feature is the shape and I have to change the input a lot in order to affect the feature right so that if they're roughly on the same scale of what I have to change to change the feature however the other the other feature in the input space has a much different scale than it has on in the feature space and this might be something like the fur structure of a cat so the fur structure of a cat like is I can change the pixels a tiny bit and I'm going to change the first structure by a lot I can change the first structure of a cat to the first structure of a dog by just changing the by just changing the pixels a little however it will be different and now it will be the first structure of a dog so how does this change now in input space in input space it's going to look something like this where one feature dimension is going to look rather the same and the other feature direction is going to be very very stretched okay now remember both of these features are good features they both can be used to read to classify the images so you can see changing the shape requires a lot of pixels changing the first structure however requires just a little pixel now if I take some image and I draw an L2 ball around it which was what we usually do when we create an adversarial example we say only we only allow small perturbations you can see that in in this direction it's a very you know you don't get very far in feature space but if you go the same distance in the in the input space into this direction in the feature space you're going to walk a lot you're going to walk like way far and this is just by definition there are going to be many features that you can use to classify images and they're going to be good features they're not going to be errors or aberrations like the first structure is a good feature to classify a cat they want to be many features in there and some of them are going to be of large magnitude and some of them are going to be of small magnitude and this is just what happens okay so I called this the the stretchy feature model and this is sort of a direct result of this paper that they cite by Alexandre Modri's group which we're gonna get to in a second right but keep those two in mind and we're gonna see how which one explains the phenomena better and which one doesn't okay so they say why deep neural networks are likely to create dimpled manifolds as decision boundaries and the the idea here is that okay we have to now explain why this even happens so if you consider the data manifold in green right here and here we have just one dimensional data and you can see it's not linearly separable right so we have to have sort of a curve decision boundary around this and why would this result in a dimpled manifold so they say look if you start off your your deep neural network training you're maybe your decision boundary is going to be somewhere like here okay not very effective what's gonna happen is let's say what you want what you want is you want to have the blue data you want to have the blue data above and the red data below the decision boundary so right now the red data is is oh that's the other way around the red above and the blue below so right now the blue are fine like the blue don't complain you do get a gradient out of the red examples pushing the entire decision boundary down there's no resistance right the blue ones they they're fine so you're gonna push down this is your next decision boundary okay same situation you're gonna push the entire decision boundary down now you're here now you're too far so you're gonna push the entire decision boundary up because now the red ones are fine the blue ones complain and this results you being sort of right on top of the data for once okay and then both gradients kick in so now the red data are gonna push such the decision boundary down the blue data are gonna push the decision boundary up which is going to result in this sort of dimples around the data otherwise the decision boundary coinciding with the data okay this is their explanation for why the why this works I hope this makes a little bit of sense now yeah so they claim that that this is happening contrast this with the mental model of having a bunch of linear half spaces which would result in something like you know a decision boundary being through here a decision boundary being through here a decision boundary being through here and through here through here which would also explain what we see but this is their claim why this decision boundary looks the way it is to me it's it's a bit it's a bit weird right like here why should the decision boundary align with the data manifold maybe it doesn't maybe they don't they don't claim that I should not complain about this but for example in between the data why does it do that they give some examples right here that decision boundary it should be rather simple right it doesn't like to curve a lot they say the new model can help to understand why the training phase of a given network typically converges to the same global optimal placement of the decision boundary regardless of its random initialization they're gonna make a claim right here why this happens to demonstrate this point consider the old model in which you sprinkle at random locations in the two-dimensional square alert as the large number of classes depicted in figure three sorry um I was confused for a second I am no longer so they're talking about this figure right here and they say look in the old model you have if you want to pass sort of simple decision boundaries through this you have to sort of pass them like some of the gray ones we see right here and they are not going to be so good okay so our goal is to pass a decision boundary of bounded complexity and this bounded complexity comes up again and again they claim of course their decision boundary is very smooth and very simple which will best separate the red and blue clusters they say there is a large number of way to do ways to do this like the green lines and most of them will be about equally bad in particular any decision to pass one side or the other of some cluster can make it harder to accommodate other clusters elsewhere along the line consequently there likely be many local minimum of roughly the same quality in the dimpled manifold model however there is likely to be a single globally best decision boundary shape since there is no conflict between our ability to go above one cluster and below a different cluster when they do not intersect so their idea here is that rather putting the decision boundaries like this what they want to do is you look at this in three dimensions and then they just kind of put a sheet over top of it and go above the blue ones and they're below the red ones in all of the three dimensions right so you go above the blue ones and below the red ones rather than this these gray things like here which are not very optimal now this one I'm not really sure what to make of this because for first of all they say it typically converges to the same global optimal placement of the decision boundary regardless of random initialization we know that this is not true right I've specifically made videos on research by Stanislav Ford who shows that if you randomly initialize a network differently what it will happen is you will reach the same accuracy but it will it will make mistakes on different samples of the test set right and there's actually a structure to how these decision boundaries are going to be different depending on your random initialization which actually would support what they claim is the old view right here second of all I have no trouble making a decision boundary here that separates red and blue right I can go something like this like this come here okay you get here right I have no trouble separating red and blue I guess this should go here so they're this this kind of this kind of bounded complexity does a lot of work here them saying who the decision boundary should be simple and so on and that's why they really insist that this decision boundary should be somehow straight but then a lot but I disagree that their decision boundaries are so simple if you have to curve around every data sample and otherwise follow the image manifold that seems to be like a rather complex decision boundary honestly because it's it's it's kind of a generative model of the data right if you follow the data manifold so I disagree that there's is so much simpler right just because it doesn't bend that much and here it like bends a lot that's also something they say like you you don't want to bend decision boundaries so much that hardens training and third of all why do they give their model the benefit of the third dimension right so they claim like oh look the old model doesn't work because if you have to place decision boundary between the data points you're gonna end up with a bad decision boundary however in order for their model to work they need the third dimension they need to pass like under and over the data in the third dimension whereas if you actually go into the third dimension you know every single lecture you have on kernelized SVMs and whatnot they show you like if you go in higher dimensions these things are actually separable like you would make if you have like RBF kernels these would become a cluster these would become a cluster and so on this is sort of the first lecture on going into higher dimensions in order to linearly classify stuff so it's not like their method can explain anything more than any other method if you give it this third dimension and the fact that they don't give the old model the third dimension but they give themselves the third dimension in order to explain it is a little bit I'm not I don't know it's this like yeah so I don't think this is any argument for for their model it just simply shows that if you have a lower dimensional manifold of data and you classify it in a higher dimension there are ways to do that right and if you like if you have relu networks and linear classifiers it's going to look like more chunky it's going to kind of divide the space into these kind of relu cells where you classify the data all of this is compatible with what they're saying not just their dimpled manifold hypothesis all right so this is yeah I don't I don't see the big explanation here so they claim what can they explain with their model explaining the mysteries of adversarial examples okay there are five things they claim they can explain with this first of all the mixture mystery right how can it be that a tiny distance away from any cat image there is also an image of a guacamole and vice versa and okay if these and if these classes are intertwined in such a fractal way how can a neural network correctly distinguish between them our answer is that all the real cat and guacamole images reside in on the tiny image manifold but below the real cat images there is a whole half space of pseudo guacamole images which are not natural images of guacamole and above the guacamole images there is a whole half space of a pseudo cat images so their idea here is that okay you have this one-dimensional data manifold here are the cats here the guacamole is if you have your dimpled manifold curving sort of around the data right here you know all of this is technically guacamole so if you go from the cat to here you reach a non-natural guacamole image just by the fact so the explanation here is that the explanation is that this this the decision boundary lines up with the data manifold except around the data where it creates a small dimple and therefore you can cross the dimple into the other region okay you this is very it's the same effect as this model right here you know I can draw this dimpled manifold I can draw it right here right if I classify the image I can draw this dimpled manifold I get the same effect however this model here explains much more it actually explains like here there is no reason if you think about a multi-class setting right if you think of this in two classes fine but if you think of this in a multi-class setting there is no reason why this region right here should be guacamole it can be any other class right if the if the idea is the decision boundary follows the data manifold and then just dimples around the data to make the data correct clout they classify the only constraint here is is that these are cats it says nothing about sorry it says nothing about why on the other side there is guacamole instead of anything else and that does not coincide with what we know about adversarial examples like this region here is a consistent region what so first of all first of all my bigger problem is why does this even generalize why does the dimpled manifold hypothesis even generalize right like if it follows the if it follows the data manifold largely except around the the training data why does it exactly generalize well to test data you have to like argue that the test data I see are quite close because otherwise it would be it would get very confused on test data which would be somewhere else on the manifold right but we know that generally neural networks classify data that's on the manifold of natural images quite well they generalize quite well however this model is sort of an anti generalization model but okay maybe you can claim that their test images are close enough to the training images such that this works but for example we know that if that this this is a consistent region what do I mean by this we know for example we can make universal adversarial perturbations which means that we can find directions that no matter from which image or from which class we start from they will always result in guacamole okay this is not explained by the dimpled manifold there is no reason why these regions on the other side should be of a consistent label in a multi-class setting we also know that adversarial perturbations are transferable which means that we can make an adversarial perturbation in one classifier and then in a different classifier even if it's trained with a different data set actually we can we can apply the same adversarial perturbation and it will most likely still be of the same like the adversarial perturbation going towards the same class there is no reason in the dimpled manifold hypothesis that explains these phenomena if you think of this of the stretchy feature model this is really easy right if I create an adversarial example I go across the decision boundary right here what do I do I change the fur without changing the shape now I change the fur by so much that you know now there is a conflict right in feature space I go up here now there is a conflict it has the fur of a dog but the shape of a cat still now I there is a conflict but neural networks in the final layer are linear which means they just weigh the different features now I just pump that fur to be so doggish right that it overpowers the shape feature of the cat neural networks are biased towards sort of structure anyway over shape already so I just I just hammer that fur and now the neural network thinks it's it's a dog and a different neural network trained on the same data will also think it's a dog because it will also have learned to classify images by shape and fur therefore therefore it will it will be vulnerable to the same attack right this is super easy to explain in this model there is no reason why this should happen in the dimpled manifold model unless you amend it by some more hand wavy things they say the direction mystery when we use an adversarial attack to modify a cat into guacamole why doesn't the perturbation look green and mushy okay so they say well in the old model you would have to walk along the image manifold from here towards the guacamole images and that should mean that your image should sort of change to look like a guacamole in our in the dimpled manifold model you go off the manifold perpendicular and that explains why the adversarial perturbation looks like a little bit like just random noise again no one thought this in the old model in fact we have a pretty good explanation why it still looks the same and that's because humans are much more receptive to this thing right here to the shape whereas neural networks also or much more consider this thing right here the fur also they consider fur and shape in different proportions than the humans do and so that's we already sort of knew this and it's in fact a better explanation the uniformity mystery you know why the decision boundary is ever present so they claim because the there's this dimple right here even you know the most far away cat image here has a close crossing to the decision boundary so there is no cat images that are kind of closer to the decision boundary but this is I think this is just a property of a high-dimensional classifier I think that here our 2d view of the world betrays us and yeah especially if we can go really far in feature space with a tiny perturbation and input space this is not not a mystery not even a mystery the vanishing gap mystery okay which is about adversarial training I think which we're gonna skip here and then there is the accuracy robustness trade-off mystery so this is if you do if you train a model adversarially which means that here look here I have my cat okay I train I have a data set of cats and dogs I train my neural network on it it's vulnerable what can I do what I can do is I can create adversarial images this is a cat right I can create adversarial images by making this into a dog okay so this is a dog because I changed the first structure a little bit this is an adversarial example now I add this so this is comes from the data set now I add this to the data set but I tell it this is a cat too right this is a cat and this is a cat if I do this with my neural network the neural network will become robust to adversarial examples to a degree not fully but to a degree this is the best method we have so far of defending against adversarial examples called adversarial training now what you do when you do this is you train the network to to sort of classify the advert to yeah classify to incorporate the adversarial ness into its decision-making process and this results usually in a degradation of the generalization performance of the network so as it becomes more robust it becomes less accurate on real data right you gain accuracy on adversarial data you decrease the accuracy in real data which makes sense intuitively but it is a strong effect which is not the same as you know I simply teach my model to do yet another class it is quite it is actually a trade-off now they try to explain this right here when we train the network we keep the images stationary and move to decision boundary by creating dimples when we create adversarial examples we keep the decision boundary stationary and move the images to the other side by allowing a large perpendicular derivative we make the training easier since we do not have to sharply bend decision boundary against around the training examples so this is when you train normally when you train without adversarial examples they say there is a large perpendicular derivative which in the like the what they mean is that the data samples are of push these dimples out that that's the large perpendicular derivative the perpendicularity is to the image manifold and that makes it easy because you don't have to bend the decision boundary a lot so you can kind of remain here and you have to kind of create these dimples again their argument is you don't want to bend this boundary a lot which makes training easy however such a large derivative also creates very close adversarial examples yeah this is their claim that now the decision boundary is pretty close because you don't bend the decision boundary by too much around the data because you do dimples any attempts to robustify a network by limiting all its directional derivatives will make the network harder to train and thus less accurate I'm not super sure how to interpret this so I might be doing this wrong right here but if you create adversarial example what you do is you essentially have this data point and you create an adversarial example this data one is yeah well these are of the same class so now that is now the the decision boundary has a sort of bend harder okay which makes it more hard to train and at some point it so it's harder to train and that's why you have less accuracy and at some point it says well actually I don't want to bend that much I'd rather make a mistake here and just bend around both of these data points and now you have a wrong classification so that's sort of their explanation of why this happens which I find a bit hand wavy you have to argue like ooh ease of training bending the decision boundary and so on in this model right here super easy okay what happens if I create cats that have cat fur and dog fur and I tell the network these both are cats well essentially I tell them I tell the network look there are two features right here the fur and the cat and you know the fur just just disregard it just don't do that don't regard the fur as a feature because it's useless now because I now have cats with cat fur and cat with dog fur so the network can't use that to classify anymore and that explains why it gets less accurate because I take away one useful feature okay so you know now the network has less useful features and that's why it gets worse this it's it's a pretty simple explanation in the stretchy feature model it has there's a lot of work to make this happen in the dimpled manifold model so lastly they try to explain and they what they came an interesting mystery in this this paper that I have cited throughout and what that is is that it's kind of the same experiment as here where we create adversarial examples and we add them to the training set except for two things first of all we don't have the original so our new data set is not going to contain the original images it's only going to contain the adversarial examples second it is going to contain the adversarial example image but the label isn't going to be the correct label quote-unquote correct from where we created but the label is actually going to be the adversarial label the wrong label okay so we're going to tell the network this is a dog please learn that this is a dog right it's a cat with dog fur and the old training images are nowhere in the data set we just do a data set with these wrongly labeled images now when we go and we apply this so we train we use this we train a network right to classify cats and dogs and now we once we've trained this network we go we take one of these samples of the original data set we classify it it's going to give us a correct classification right so it will recognize that this here is a cat even though we told it that this here is a dog now how does it do this it does this by looking at the fur you know we've we've doubled down on the fur here right so this is like we really made that fur feature super strong in these adversarial examples so it's going to look at the cat fur and even though none of the cats have the shape like this we sort of we sort of supercharged that fur feature again in this model not a problem essentially what we've done is we've created two data classes you know one up here and one down here that have the fur supercharged and now it's just going to mainly look at that fur structure and that is a useful feature right so this this what's called their features not bugs paper adversarial examples are features not bugs or other way around not bugs they are features has demonstrated with this experiment this notion that there are adversarial examples result from useful generalizing features in the data set that are simply of by definition the features that are not large enough for humans to see what they call non robust features how do they explain this they say the original people try to explain this highly surprising role by distinguishing between robust and non robust features in any given image where some of them are preserved by the adversarial change and some are not however it is not clear what makes some of the features more robust than others definition just definition like like if you have features and you order them by their size like by their how much you have to change the pixels that some features are going to be larger than other features and then some features going to be below that cutoff where you define adversarial examples budget this is definition makes them such that some of more robust it's not it's not clear our new model provides very simple alternative explanation which does not necessarily contradict the original one okay at least this which is summarized in figure four to simplify the description will use 2d vertical cut through the input space and consider only the decision boundary that separates between cats and anything else okay so they have this example right here they say look we have a decision boundary that distinguishes cats see from non cats and the green one here is the image manifold and the gray is the decision boundary okay so now what we do is we create adversarial examples in frame two right here you can see that we make the cats into non cats and we make the be the bats into bats aren't very popular lately the badgers into into cats so we make the badgers into cats and we make the cats into these whatever DS ducks okay and now we relabel those and that gives us a new data manifold so the new data manifold is this data manifold right here and we have also new labels and now they claim the resulting decision boundary in figure four as you can see right here this is the resulting decision boundary the gray one it is it is very similar to the decision boundary in the first frame and therefore we shouldn't be surprised that this new decision boundary that results from this perturbed data results in the same decision boundary as the original one okay however like why like why so their whole they have two notions notion one is that the decision boundary follows the data manifold closely except it sort of bends around the data a little and you can see this right here like this decision boundary kind of follows the data yet it just happens to be on the correct side of the data points at any given moment which okay okay however they also make the claim in different parts of their paper that bending the decision boundary and so on is not good you'd rather want to have a simple decision boundary so to me there is no reason why the decision boundary couldn't just look like this it would correctly classify this new data set right however it would not correctly classify it would not correctly classify the let's say the C that was right where was it right here or right here these data points it would not correctly classify so you see that this until now they've always had this data manifold to be sort of super duper straight and smooth and that's how they can also say well following the data manifold and not bending too much and so on those are not in conflict with each other but now that they are in conflict with each other you have to give you gonna give up one or the other and only in one of them do actually does this experiment here still make sense in the other one it doesn't and but if you give up the ooh bending too much is bad then you know you lose a bunch of explanations that you have up here so yeah like it's one in my mind it's one or the other and there's I there's still no reason I think no good reason why this like the decision boundary should align super closely with the data points like if there if there is nothing here right if this is perpendicular really to the data manifold like why would the decision boundary align so closely with the data manifold in that point I don't know okay so they ask why are DNN so sensitive and humans so insensitive to adversarial perturbations essentially their argument here is that humans project the input data onto the image manifold which is a contested claim right I don't I don't think that is a I think that is not not a widely accepted I mean it's it's certainly possible but also I'm not sure I'm not sure that humans do project they have like an internal manifold of natural images and project onto that every time they analyze an image and also the yeah how do you project right like how like both of these features are useful okay so both of the features are useful if you project an adversarial example like why do you project it onto the shape dimension and not onto the fur dimension right why there's no explanation right here we know that sort of humans are more receptive to shapes and so on but just projecting won't get you there so now they're going to into experiments and I want to highlight one particular experiment right here they have synthetic experiments they have their experiments I want to highlight this experiment right here remember they said their experiments were going to give you know strong support that and this experiment right here what they want to claim is that okay you have the data manifold here if you are if you have a data point and you make an adversarial example the question is do adversarial examples go along the image manifold or do adversarial examples go sort of perpendicular to the image manifold they they their claim again is that V this here would give support to the old view of adversarial examples and this here would support the dimpled manifold view because of course the decision boundary would be sort of following the data manifold curving around the data and then following the image manifold again so here would be sort of the other data point going below that a little bit all right so that is the view right here now what they're going to try to show you is that if you want to create an adversarial example on the manifold you have to walk much longer for much longer until you find an adversarial example then if you go off the manifold if you go yeah and they're also going to show you that if you are not constrained if you can go anywhere you want with an adversarial example then that will be very similar to when you force the adversarial example to go off the manifold and this gives a bit of proof that you know if two things behave equally they're you know probably equal so what they're going to do is they're going to try to make an adversarial attack first of all a regular one this one they're gonna say okay we're gonna make an adversarial attack let's measure how far we have to go to cross the decision boundary second they're going to say let's make the same thing but let's force the attack to be on the manifold of natural images and let's measure that and lastly they're going to mask okay let's do the same thing but force it to be off the data manifold and then they're going to measure how long these are how long the adversarial attacks are what's their their norm and they're going to find of course they're gonna want to find that these two are a about similar norms and way smaller than the one that is on the data manifold sort of giving evidence to you know if you go perpendicular to the data manifold you have to go very not very far and that's what adversarial attacks do okay yeah so how first of all how do they force the the adversarial attack to be on the manifold what they do is they do an autoencoder so they train an autoencoder so they an autoencoder is a neural network that has sort of a bottleneck layer and you try to just reconstruct the inputs data okay you tried that these two are equal however in the middle here you have a very low dimensional representation so where this is an n dimensional representation this is a k dimensional representation and a k much smaller than n if you can reconstruct the images correctly that means that you sort of have captured the representation in these low dimensions right here so what they're going to do is they train an autoencoder they take that low dimensional representation they linearize around it and that's how they have a way to project onto the image manifold by simply only moving around in this low dimensional manifold right here or always projecting onto it first of all it's a bit of a trouble because how you train the autoencoder is like for these experiment I think it's very relevant to how they this image manifold is going to look like if you train it with L2 you sort of already make some claims about what are important features and whatnot but let's disregard this right here let's say they have an accurate way of projecting onto the image manifold onto the manifold of natural data and here's what they find look let's look at image net okay no constraint PGD it this is the norm you know it's some number okay so like 0.14 now off manifold PGD is where they deliberately project off the manifold so they project on the manifold they subtract that they say you're not to do anything with the mana of the image manifold and that's 0.152 which is slightly larger than the no constraint PGD but essentially the same size now on manifold PGD okay here is a way bigger number like six times bigger number so their claim is look up up to six times more you have to go on the manifold than off the manifold and that gives credence to their claims now okay so what I've done is they have you know they have some descriptions of their experiment specifically they have descriptions of what library they used they used advert torch okay so I used advert torch to they used you know L2 PGD I use that too and they told me how much their low dimensional representation is so the K here how much that is how much the N is and so I was able to reproduce that experiment now what I've done is I have done the same thing and you can see right here this is this the panda image from image net they use an image net classifier and what they do is they do it greedy so they stop as soon as they cross the decision boundary and then they measure the norm you can see right here this is the perturbation now it's a soccer ball and here is the size 0.7772 that's the norm of the original perturbation adversarial what I now do as I project onto the manifold but I don't the difference is I don't project onto the image manifold what I do is here you see project onto K I simply project onto any K dimensional manifold so I know what K is K is 3,500 so it's a very small number compared to the input number and so what they project is actually the gradient so the gradient of the adversarial attack that you use to update your image that's what they project they have the algorithm clearly lined out so what I do is I simply take you can see right here I take a random set of of dimensions like of pixel coordinates in the gradient and I denote the first you know the first few the first K as the manifold and the last K as not the manifold this is not the image manifold there's nothing to do with the image manifold this is simply a random K dimensional subspace of the pixel space okay and now when I project onto K I simply take all the others in the gradient and I set them to zero that's I project onto a K dimensional manifold after that you normalize the gradient and so on so you proceed you proceed as you would right so here you can see the the project is used before you normalize the gradient so there's no issue with sort of the the step size you simply project onto the manifold and I have the same thing by the way projecting off the manifold where I simply take the K dimensions and set them to zero okay so now let's look what happens if I project on to the manifold oh wow before it was 0.77 and now it's 6.5 so about eight times larger and now let's look what happens if I project off the manifold it's 0.7773 instead of 0.7772 so what they're seeing right here and you know maybe okay maybe I've done it modulo I've done it wrong and I completely don't understand what's going on what they have found is simply an effect of projecting onto any lower dimensional space yet they claim that this is like in support of their hypothesis which clearly I have no clue what the data manifold is I've just projected onto a random manifold and I got the same results so I see they have other experiments where they try to kind of convince you with all the types of perturbations and so on but you know like no this these they have other experiments but this is just one that I could try quickly again maybe I've done it wrong to me this Occam's razor is strong here like Occam's razor in this work is quite a bit like there can be like there can be many hypotheses that coincide with the results you're getting and with the phenomena and it's easy to think that stuff is in favor of your hypothesis is providing support for it when there are other explanations available oh I almost forgot about Goodfellow's claim that you know they say belongs to the sort of old thinking that is now that is not a correct thinking and the claim that when you make an adversarial examples you somehow go towards the centroid of a different class and in imagination it's something like this on the on the left right here however if you think about this in this space okay let's say you start out here and you go towards the centroid of the other class right the pro where's the centroid here approximately like this what happens in feature space because of the stretchy feature because of the different scales okay what happens in feature space is it pretty much like the blue arrow here so it's that in feature space you go a long way actually this is probably I should have drawn this here to be square and this here to be super stretchy right yeah yeah I think so yeah I was I was wrong in drawing this so this here should be squares and this here actually should be super duper stretchy right so the centroid what was the centroid here is like way up here like way up here somewhere okay so this gets super stretched and you cross the boundary in this one feature right like the fur feature and yeah so I think this is it's still a correct claim you go towards the centroid of another class but because you go this in input space in the feature space this results in sort of a dramatic shift in some features and a not so dramatic shift in other features so while in the input space you go towards the centroid equally in all pixel directions you don't go towards the centroid equally in all pixel directions in the sorry in all feature directions so I think the claim that Goodfellow made is valid here still and explains like is concurrent with the stretchy feature explanation that I'm pretty sure that's also kind of what maybe I can't read his mind but maybe what he meant by that and not necessarily this picture right here not necessarily that actually the entire picture is going to change into the other class okay that was the interjection and back to the conclusion but as I said make up your own mind what do you what do you think of this go through the paper they it's it's a good paper like it's written it's written well there it has a lot of experiments has quite a lot of appendix where they give you more results and so on and it's not like again it's not like it's in it's necessarily incompatible right it's not I don't disagree with them I just think it's it's not as useful as they claim and it's kind of insufficient I don't disagree with their their main claims yeah and I think we already kind of knew a lot of those stuff and our current mental models are explaining things maybe a little a little better and yeah if you use the the squishy feature what would I call it the the stretchy feature model has a fancy name now but again is this is not mine this is just kind of a a bringing together of of what we what I think we know about adversarial examples safe to say there's going to be something that challenges this and that's going to be exciting alright thanks so much for being here listening and I'll see you next time bye bye
 Hello there! Today we're going to look at the dimpled manifold model of
 adversarial examples in machine learning by Adi Shamir, Odelia Melamed and Oriol
 Ben-Schmuel. This paper on a high level proposes a new way of looking at the
 phenomenon of adversarial examples in machine learning, specifically in deep
 learning, and they proposed this model called the dimpled manifold model,
 essentially arguing that classifiers put their decision boundaries right next to
 the manifold of data, while only slightly sort of curving it around the data like
 this. Now the data manifold being low dimensional, this results in a situation
 where you can cross the decision boundary really easily if you simply go
 perpendicular to the data manifold, which also is perpendicular to the
 decision boundary, and if because it's just such a small dimple there, the
 decision boundary is pretty close, and that's how you end up with adversarial
 examples that are super easy to get. So it's not a new attack, a new defense,
 anything like this, it's simply a mental framework of explaining why adversarial
 examples exist on a high level. They have some conceptual thought
 experiments, they have some explanations, and some real-world experiments. Now I
 personally don't think that this is entirely, it's not necessarily
 incorrect, but I don't think that this is really useful to think in this way, and
 I'm gonna explain why. In general my opinion of this is it doesn't really add
 anything, and I think it explains less than the models we already had. Yeah so
 that's my opinion, I'm gonna get to it. Specifically also the
 experiments they propose, I think that there is a big Occam's razor failure
 right there. But as I said we're gonna get to all of this, I'm gonna go through
 the paper and I want you to make up your own mind, even though I'm going to try to
 bias you. So yeah this is not a neutral channel in case you haven't
 noticed. Alright so if you like content or if you dislike it tell me in
 the comments, tell me what you think of the paper, whether it makes sense, whether
 it doesn't make sense, and so on. I'd be very interested to see what you have to
 say. Yeah I read the comments, so please. They say the extreme fragility of deep
 neural networks when presented with tiny perturbations, yeah but okay this starts
 out how every single adversarial examples paper always starts out saying
 okay deep neural networks are extremely fragile, there's this phenomenon of
 adversarial examples. Now if you don't know what adversarial examples are,
 really briefly essentially what this is, it's a phenomenon where you take an
 image like the thing here on the left, the neural network thinks it's a plane
 with a very high probability and you change it to this thing right here, which
 you as a human can't even tell it's different, however the neural network
 will think that this is now a bird with very high probability and the this is
 the change that you made. It's magnified for you to see, it kind of looks like
 random noise but it's a very particular noise that makes the neural network
 think it's something different and this is just it's tiny in the in its norm.
 So you don't see a difference. Now bird here is kind of close to plane
 but you can change this into anything, literally anything you want, you can
 change this into banana or I don't know dog or any class you want using these
 techniques. So it's not about being close it's really kind of a separate
 phenomenon. So that's adversarial examples and many frameworks have been
 proposed in order to explain these adversarial examples and they make a
 they make a nice overview right here. Many have been proposed over the last
 eight years that DNNs are too nonlinear, that they're too linear, that they
 were trained with insufficient number of training examples, that are just rare
 cases where they error, that images contain robust and non robust features
 etc. They say however none of these vague qualitative ideas seem to provide a
 simple intuitive explanations for the existence and bizarre properties of
 adversarial examples. So that is pretty harsh criticism specifically the first
 ones are kind of yeah but specifically this last one that images contain robust
 and non robust features which is sort of the leading hypothesis right now of why
 adversarial examples exist and what they are and then here saying none of
 these can none of these vague qualitative ideas seem to provide a
 simple intuitive explanation for the existence. Like let's see whether or not
 they're gonna do better okay. So also in the abstract they go on and they say
 okay they introduced this new conceptual framework which they call the dimpled
 manifold model which provides a simple explanation for why adversarial
 examples exist, why their perturbations have such tiny norms, why these
 perturbations look like random noise and why a network which was adversarially
 trained with incorrectly labeled images can still correctly classify test images.
 Now this last part if you're not familiar with the literature it might
 come to you a bit random this why a network which was adversarially trained
 with incorrectly labeled images can still correctly classify test images. This
 is a famous experiment from the group of Alexander Modri where also this
 hypothesis this one the robust and non robust feature comes from and any
 attempt at explaining adversarial examples after this paper has to explain
 why that experiment makes sense because it's kind of a non intuitive experiment
 and we're gonna get to that as well but just so you know that's why they write
 it in the abstract. Now I personally think they don't have a good like this
 model here doesn't have a good explanation for why that works. They're
 sort of hand wavy trying in any case. So they say in in the last part of the
 paper we describe the results of numerous experiments which strongly
 support this new model and in particular our assertion that adversarial
 perturbations are roughly perpendicular to the low dimensional manifold which
 contains all the training examples. Okay also remember this experiment they
 strongly support what in particular the assertion that adversarial perturbations
 are roughly perpendicular to the low dimensional manifold which contains all
 the training examples. Now remember this that the experiments are supposed to
 support this particular claim because also that is going to be important down
 the road. Okay so let's get into the dimpled manifold model. What is it? What
 do these authors propose? I'm gonna try as best as I can to say what the
 authors are saying in the paper. So they claim that there is an old mental image
 of adversarial examples and the old mental image is here. They say we
 think the old mental image is based on the highly misleading 2d image on the
 left side of figure one and that's this thing right here. So the old mental
 image is that there is a there is a data space right this here if you think of
 pic of images as data points this would be the pixel space right so this is
 images with two pixels right now in this conceptual framework but you have to
 sort of think yourself into higher dimension. So they claim the old mental
 image is the following you have sort of the data distributed somehow in this
 space the data being the all the set of natural images or images you consider
 which is kind of these these sub space these subgroups right here there are a
 bunch of images right there and there and also there and there so these are
 images of two different classes the red class and the blue class now they're
 distributed like this and what is a classifier supposed to do a classifier
 is supposed to put a decision boundary between them and that's what they draw
 in here so this would be sort of a reasonable decision boundary between the
 two classes right so now what do you do if you want to create an adversarial
 examples well necessarily you have to start at an image of a class this one
 maybe and you have to cross the decision boundary right you want to fool the
 classifier ergo necessarily by definition you have to cross the
 decision boundary so what do you do the the easiest way to do this is to sort of
 go straight towards the decision boundary which is approximately in this
 direction right here and then once you cross the decision boundary you are done
 you're on the other side you have created an adversarial example provided
 of course that the image still kind of looks like the original image and so
 they say this has this has many many problems here they say the in this
 mental this mental image adversarial examples are created by moving the given
 images along the green arrows towards some kind of centroid of the nearest
 training images with the opposite label in which they mean this this thing right
 here so we would move the images towards the other class towards images of the
 other class and they say as stated for example by Ian Goodfellow in his
 lecture at this time I'm gonna cut this in right here I've said that the same
 perturbation can fool many different models or the same perturbation can be
 applied to many different clean examples I've also said that the subspace of
 adversarial perturbations is only about 50 dimensional even if the input
 dimension is 3,000 dimensional so how is it that these subspaces intersect the
 reason is that the choice of the subspace directions is not completely
 random it's generally going to be something like pointing from one class
 centroid to another class centroid and if you look at that vector and visualize
 it as an image it might not be meaningful to a human just because humans
 aren't very good at imagining what class centroid look like and we're really bad
 at imagining differences between centroid but there is more or less this
 systematic effect that causes different models to learn similar linear
 functions just because they're trying to solve the same task
 okay so it really appears like Goodfellow says this thing right here
 however they claim now they claim this doesn't make sense so they claim that
 you should think about adversarial examples in a different way and this is
 their dimpled manifold hypothesis so what is their dimpled manifold hypothesis
 they say what you have to do is you have to think about the data manifold in the
 higher dimensional space that they hand the higher dimensional input space so in
 this case they consider instead of here this 2d landscape they consider the 3d
 landscape so this would be the pixel space right now we consider three pixel
 images and the data is embedded in a low dimensional manifold in this higher
 space so because if you think about all combinations of pixels that are possible
 so not all of them are natural images in fact only very few of the possible
 combinations of pixels are natural images or images that make sense to you
 as a human or are images that you could potentially generate by going out with a
 camera so the data you're considering lives on a very low dimensional manifold
 in this big space and you have to explicitly think about that now the data
 is the data manifold here is represented in this in this sheet in the middle and
 on this manifold you're going to have your different classes of data here the
 blue or one class and the red or the other class what this paper claims is
 that what classifiers do what neural networks do when they classify the
 training data here is they go and they lay their decision boundary instead of
 so in the old model you would have thought maybe something like this
 happened where you put your decision boundary sort of in the middle between
 the two classes right crossing the manifold right here so you sort of put
 it in the middle between the two classes and then when you have to create an
 adversarial example again what you would do is you would maybe start here what
 you would have to do is you would go straight towards the decision boundary
 right here okay crossing the decision boundary and then on the other side
 you'd have an adversarial example in this new model what they claim is the
 decision boundary actually doesn't look like this right here okay the decision
 boundary actually is very much aligned with the manifold of data as you can see
 right here so this mesh that they show is the decision boundary now and their
 claim is that that usually just aligns with the manifold of data however around
 the actual data around the training samples what the classifier will do is
 it will create these what these dimples okay and these dimples are just tiny
 well dimples tiny perturbations in the decision manifold such that the data is
 on the correct side of the decision manifold sorry of the decision boundary
 right so the blue points here are under or one side of the decision boundary and
 the red points are on the other side of the decision boundary and for the rest
 the decision boundary just aligns with the data the data manifold now if you
 want to make an adversarial example now what you have to do again you start from
 an image and again you walk straight towards the decision boundary however
 now you don't have to go like this you what you can do is you can go simply
 perpendicular to the data manifold and you will cross the decision boundary
 very quickly because the dimple you're in is kind of shallow and they give a
 reason why the dimples are shallow because they claim this is results from
 training these models and that explains some things so the difference is the
 difference is we started out from this to make an adversarial example we have
 to go towards the decision boundary okay if we sort of transfer this image into
 higher dimensions it looks like this in the middle again in order to make an
 adversarial example we have to go towards the decision boundary now in the
 old mental image going perpendicular to the decision boundary means walking on
 the data manifold because we walk from this group of data towards this group of
 data you can see right here that we're walking on the data manifold when we
 walk perpendicular to the decision boundary whereas in the new model
 walking perpendicular to the decision boundary coincides with also walking
 perpendicular to the data manifold so this is the difference right here that
 they that they claim so this they say there's we call this conceptual
 framework the dimpled manifold model and note that it makes three testable claims
 about the kinds of decision boundaries created by trained deep neural networks
 first natural images are located in a K dimensional manifold where K is much
 smaller than N second deep neural network decision boundaries pass very
 close to this image manifold and third the gradient of the classifications
 confidence level has a large norm and points roughly perpendicular to the
 image manifold alright so these are these are the claims that they're going
 to make to be tested and to be supported by experiments I guess so I hope I've
 represented enough what the authors claim right here I hope they would agree
 that I've represented this is accurately so now where is the problem with this in
 my opinion the problem isn't necessarily with what they claim right here it's
 it's you know I don't necessarily disagree with this mental image I don't
 necessarily disagree with these claims in fact that the data is on low
 dimensional manifold this we've this is kind of commonly agreed upon assumption
 right as I said not all the possible pixels combinations make good natural
 images and that the fact that it is then a manifold is a commonly held
 assumption decision boundaries pass very close to the image manifold well the
 fact that we can generate adversarial examples right already means that
 decision boundaries pass very close to the image manifold so this also is not
 news this this has been like in everybody's conceptual framework for the
 last five years at least and then third the gradient of the classifications
 confidence level has a large norm and points roughly perpendicular to the
 image manifold and this claim right here I'm pretty pretty sure there so this is
 not a trivial claim which yes okay this is not something that was like set
 around much however I'm going to claim that their model is not the only model
 by far that makes this happen or any something like this specifically when we
 go look at the experiments I'm going to show you that this doesn't necessarily
 support their claims it doesn't disprove them right but it also doesn't
 necessarily support them just because they show that okay so the other problem
 I have with this is that this in this thing they build up as ooh this is this
 is the old mental image this is how people thought about adversarial
 examples until now I look I just I disagree like this it's a bit of a it's
 a bit of a straw man almost I feel like this no one no one thought no one that
 is sort of in the literature of adversarial examples thought or thinks
 that this is an appropriate model for what is happening like we know that
 these distances here are very small right the distance until you cross the
 decision boundary and we know also like if this were true you should just be
 able to go to the decision boundary and then go the same distance right and then
 at some point you would actually arrive at a sample of a different class so you
 could you could actually transform images into the other class by simply
 going into the adversarial direction which is precisely what we don't see
 right we see the image still largely looks the same what gets added looks
 like a bit of noise okay so no no one was having this mental image because
 clearly this mental image is it is not appropriate for adversarial examples as
 well as saying look if you think of this in sort of higher dimensions and I
 realize I've drawn this decision boundary but this is what they describe
 in the text then I don't I don't see that this is the correct way of like
 there are many different kinds of decision boundaries that are compatible
 with with the decision boundary right here by the way this decision boundary I
 drew doesn't even separate the classes all the classes correctly what I'm
 saying is that also if you consider the decision boundary that for example looks
 like out of colors looks like this that also crosses here however it's sort of
 kind of flat like this but it's still a linear decision boundary right like this
 okay so this is above and the other part is below if you think of this if you
 project this down it looks the same in 2d and in 3d it's also explains that
 decision boundaries are very close to the data samples it's a bit different
 though than this dimpled manifold hypothesis right if you I think the at
 least in my estimation what's happening is much more that you have just a bunch
 of these kind of linear decision boundaries flying around right here
 partitioning up the space and so on and this might result in a similar situation
 as here but it has quite different predictions in form of what it does then
 what it does right here here it's sort of a flat manifold dimpling around the
 data whereas here it's kind of the class are separating the space into many
 regions always trying to sort of distinguish one class from the other and
 yeah so might end up bit the same but I don't think they give a fair shot at
 what we know so far like we that this model is not a a model that people hold
 in general especially the one on the left I can make an attempt at making a
 mental model that people hold so far maybe it's just me but I have a feeling
 this is a bit more so the model that I call let's call it something because
 they call it there something right I call mine the squishy feet the stretchy
 feature model okay let's contrast this with the stretchy feature model so what
 I want to do is I have two features and this is a coordinate system in feature
 space okay so there's two features this in feature space I mean sort of the the
 last representation before the classification layer in feature space
 the two classes look like this so there is the red class and there is the blue
 class and you can see right here there are two features and for some reason the
 network can classify along these two features maybe because there are other
 classes other data points so we can't put a decision boundary like this
 between the two we can classify along the two features okay so you can see
 there are two features right here feature one and feature two and both features are
 actually pretty good features for keeping these two data points apart okay
 now there are empty spaces as you can see right here which we're gonna get to
 in a second but you can you can use both features and ideally a classifier would
 actually use both features it would say you know if feature one is high it's
 there probably a red class if feature two is low it's probably the red class and the
 combination makes even more of the red class okay however since we are in a deep
 neural network which is has transformations it transforms the data
 along the way if you look at the same situation in input space so in the
 actual pixel space it looks different and this is due to not necessarily the
 non-linearity of things but actually it is due to the linear transformation it's
 actually the problem of adversarial examples at least in my estimation
 appears to happen in the linear layers if you think of for example like eigenvectors
 of matrices and the largest eigenvalues determine how far you can go in a
 particular direction by having a sort of a standard input delta and the same
 happens here by the way this is why spectral norm regularization tends to
 work at least a little bit against adversarial examples so what I mean is
 if you look at the scale of these features right they are like one two
 three four five of this features one two three four five if you look in the
 input space some of the features are going to have roughly the same scale
 right here and these features are going to be features that you have to change
 the input a lot in order to change the feature a lot what do I mean by this
 this is something like the shape of an of an image okay if you think of a cat
 the general shape of a cat you know it has it has two years pointy it has a
 head and and so on that's the general shape of a cat sorry that is actually
 the left right feature right this is the the left right feature is the shape and I
 have to change the input a lot in order to affect the feature right so that if
 they're roughly on the same scale of what I have to change to change the
 feature however the other the other feature in the input space has a much
 different scale than it has on in the feature space and this might be
 something like the fur structure of a cat so the fur structure of a cat like
 is I can change the pixels a tiny bit and I'm going to change the first
 structure by a lot I can change the first structure of a cat to the first
 structure of a dog by just changing the by just changing the pixels a little
 however it will be different and now it will be the first structure of a dog so
 how does this change now in input space in input space it's going to look
 something like this where one feature dimension is going to look rather the
 same and the other feature direction is going to be very very stretched okay now
 remember both of these features are good features they both can be used to
 read to classify the images so you can see changing the shape requires a lot of
 pixels changing the first structure however requires just a little pixel now
 if I take some image and I draw an L2 ball around it which was what we usually
 do when we create an adversarial example we say only we only allow small
 perturbations you can see that in in this direction it's a very you know you
 don't get very far in feature space but if you go the same distance in the in
 the input space into this direction in the feature space you're going to walk a
 lot you're going to walk like way far and this is just by definition there are
 going to be many features that you can use to classify images and they're
 going to be good features they're not going to be errors or aberrations like
 the first structure is a good feature to classify a cat they want to be many
 features in there and some of them are going to be of large magnitude and some
 of them are going to be of small magnitude and this is just what happens
 okay so I called this the the stretchy feature model and this is sort of a
 direct result of this paper that they cite by Alexandre Modri's group which
 we're gonna get to in a second right but keep those two in mind and we're gonna
 see how which one explains the phenomena better and which one doesn't okay so
 they say why deep neural networks are likely to create dimpled manifolds as
 decision boundaries and the the idea here is that okay we have to now explain
 why this even happens so if you consider the data manifold in green right here
 and here we have just one dimensional data and you can see it's not linearly
 separable right so we have to have sort of a curve decision boundary around this
 and why would this result in a dimpled manifold so they say look if you start
 off your your deep neural network training you're maybe your decision
 boundary is going to be somewhere like here okay not very effective what's
 gonna happen is let's say what you want what you want is you want to have the
 blue data you want to have the blue data above and the red data below the
 decision boundary so right now the red data is is oh that's the other way
 around the red above and the blue below so right now the blue are fine like the
 blue don't complain you do get a gradient out of the red examples pushing
 the entire decision boundary down there's no resistance right the blue ones
 they they're fine so you're gonna push down this is your next decision boundary
 okay same situation you're gonna push the entire decision boundary down now
 you're here now you're too far so you're gonna push the entire decision boundary
 up because now the red ones are fine the blue ones complain and this results you
 being sort of right on top of the data for once okay and then both gradients
 kick in so now the red data are gonna push such the decision boundary down the
 blue data are gonna push the decision boundary up which is going to result in
 this sort of dimples around the data otherwise the decision boundary
 coinciding with the data okay this is their explanation for why the why this
 works I hope this makes a little bit of sense now yeah so they claim that that
 this is happening contrast this with the mental model of having a bunch of
 linear half spaces which would result in something like you know a decision
 boundary being through here a decision boundary being through here a decision
 boundary being through here and through here through here which would also
 explain what we see but this is their claim why this decision boundary looks
 the way it is to me it's it's a bit it's a bit weird right like here why should
 the decision boundary align with the data manifold maybe it doesn't maybe they
 don't they don't claim that I should not complain about this but for example in
 between the data why does it do that they give some examples right here that
 decision boundary it should be rather simple right it doesn't like to curve a
 lot they say the new model can help to understand why the training phase of a
 given network typically converges to the same global optimal placement of the
 decision boundary regardless of its random initialization they're gonna make
 a claim right here why this happens to demonstrate this point consider the old
 model in which you sprinkle at random locations in the two-dimensional square
 alert as the large number of classes depicted in figure three sorry um I was
 confused for a second I am no longer so they're talking about this figure right
 here and they say look in the old model you have if you want to pass sort of
 simple decision boundaries through this you have to sort of pass them like some
 of the gray ones we see right here and they are not going to be so good okay so
 our goal is to pass a decision boundary of bounded complexity and this bounded
 complexity comes up again and again they claim of course their decision boundary
 is very smooth and very simple which will best separate the red and blue
 clusters they say there is a large number of way to do ways to do this like
 the green lines and most of them will be about equally bad in particular any
 decision to pass one side or the other of some cluster can make it harder to
 accommodate other clusters elsewhere along the line consequently there likely
 be many local minimum of roughly the same quality in the dimpled manifold
 model however there is likely to be a single globally best decision boundary
 shape since there is no conflict between our ability to go above one cluster and
 below a different cluster when they do not intersect so their idea here is that
 rather putting the decision boundaries like this what they want to do is you
 look at this in three dimensions and then they just kind of put a sheet over
 top of it and go above the blue ones and they're below the red ones in all of the
 three dimensions right so you go above the blue ones and below the red ones
 rather than this these gray things like here which are not very optimal now this
 one I'm not really sure what to make of this because for first of all they say
 it typically converges to the same global optimal placement of the decision
 boundary regardless of random initialization we know that this is not
 true right I've specifically made videos on research by Stanislav Ford who shows
 that if you randomly initialize a network differently what it will happen
 is you will reach the same accuracy but it will it will make mistakes on
 different samples of the test set right and there's actually a structure to how
 these decision boundaries are going to be different depending on your random
 initialization which actually would support what they claim is the old view
 right here second of all I have no trouble making a decision boundary here
 that separates red and blue right I can go something like this like this come
 here okay you get here right I have no trouble separating red and blue I guess
 this should go here so they're this this kind of this kind of bounded
 complexity does a lot of work here them saying who the decision boundary should
 be simple and so on and that's why they really insist that this decision
 boundary should be somehow straight but then a lot but I disagree that their
 decision boundaries are so simple if you have to curve around every data sample
 and otherwise follow the image manifold that seems to be like a rather complex
 decision boundary honestly because it's it's it's kind of a generative model of
 the data right if you follow the data manifold so I disagree that there's is
 so much simpler right just because it doesn't bend that much and here it like
 bends a lot that's also something they say like you you don't want to bend
 decision boundaries so much that hardens training and third of all why do they
 give their model the benefit of the third dimension right so they claim like
 oh look the old model doesn't work because if you have to place decision
 boundary between the data points you're gonna end up with a bad decision
 boundary however in order for their model to work they need the third
 dimension they need to pass like under and over the data in the third dimension
 whereas if you actually go into the third dimension you know every single
 lecture you have on kernelized SVMs and whatnot they show you like if you go in
 higher dimensions these things are actually separable like you would make
 if you have like RBF kernels these would become a cluster these would become a
 cluster and so on this is sort of the first lecture on going into higher
 dimensions in order to linearly classify stuff so it's not like their method can
 explain anything more than any other method if you give it this third
 dimension and the fact that they don't give the old model the third dimension
 but they give themselves the third dimension in order to explain it is a
 little bit I'm not I don't know it's this like yeah so I don't think this is
 any argument for for their model it just simply shows that if you have a lower
 dimensional manifold of data and you classify it in a higher dimension there
 are ways to do that right and if you like if you have relu networks and linear
 classifiers it's going to look like more chunky it's going to kind of divide the
 space into these kind of relu cells where you classify the data all of this
 is compatible with what they're saying not just their dimpled manifold
 hypothesis all right so this is yeah I don't I don't see the big explanation
 here so they claim what can they explain with their model explaining the
 mysteries of adversarial examples okay there are five things they claim they
 can explain with this first of all the mixture mystery right how can it be that
 a tiny distance away from any cat image there is also an image of a guacamole
 and vice versa and okay if these and if these classes are intertwined in such a
 fractal way how can a neural network correctly distinguish between them our
 answer is that all the real cat and guacamole images reside in on the tiny
 image manifold but below the real cat images there is a whole half space of
 pseudo guacamole images which are not natural images of guacamole and above
 the guacamole images there is a whole half space of a pseudo cat images so
 their idea here is that okay you have this one-dimensional data manifold here
 are the cats here the guacamole is if you have your dimpled manifold curving
 sort of around the data right here you know all of this is technically guacamole
 so if you go from the cat to here you reach a non-natural guacamole image just
 by the fact so the explanation here is that the explanation is that this
 this the decision boundary lines up with the data manifold except around the data
 where it creates a small dimple and therefore you can cross the dimple into
 the other region okay you this is very it's the same effect as this model right
 here you know I can draw this dimpled manifold I can draw it right here right
 if I classify the image I can draw this dimpled manifold I get the same effect
 however this model here explains much more it actually explains like here
 there is no reason if you think about a multi-class setting right if you think
 of this in two classes fine but if you think of this in a multi-class setting
 there is no reason why this region right here should be guacamole it can be any
 other class right if the if the idea is the decision boundary follows the data
 manifold and then just dimples around the data to make the data correct
 clout they classify the only constraint here is is that these are cats it says
 nothing about sorry it says nothing about why on the other side there is
 guacamole instead of anything else and that does not coincide with what we know
 about adversarial examples like this region here is a consistent region what
 so first of all first of all my bigger problem is why does this even generalize
 why does the dimpled manifold hypothesis even generalize right like if it
 follows the if it follows the data manifold largely except around the the
 training data why does it exactly generalize well to test data you have
 to like argue that the test data I see are quite close because otherwise it
 would be it would get very confused on test data which would be somewhere else
 on the manifold right but we know that generally neural networks classify data
 that's on the manifold of natural images quite well they generalize quite well
 however this model is sort of an anti generalization model but okay maybe you
 can claim that their test images are close enough to the training images such
 that this works but for example we know that if that this this is a consistent
 region what do I mean by this we know for example we can make universal
 adversarial perturbations which means that we can find directions that no
 matter from which image or from which class we start from they will always
 result in guacamole okay this is not explained by the dimpled manifold there
 is no reason why these regions on the other side should be of a consistent
 label in a multi-class setting we also know that adversarial perturbations are
 transferable which means that we can make an adversarial perturbation in one
 classifier and then in a different classifier even if it's trained with a
 different data set actually we can we can apply the same adversarial
 perturbation and it will most likely still be of the same like the
 adversarial perturbation going towards the same class there is no reason in the
 dimpled manifold hypothesis that explains these phenomena if you think
 of this of the stretchy feature model this is really easy right if I create an
 adversarial example I go across the decision boundary right here what do I
 do I change the fur without changing the shape now I change the fur by so much
 that you know now there is a conflict right in feature space I go up here now
 there is a conflict it has the fur of a dog but the shape of a cat still now I
 there is a conflict but neural networks in the final layer are linear which
 means they just weigh the different features now I just pump that fur to be
 so doggish right that it overpowers the shape feature of the cat neural networks
 are biased towards sort of structure anyway over shape already so I just I
 just hammer that fur and now the neural network thinks it's it's a dog and a
 different neural network trained on the same data will also think it's a dog
 because it will also have learned to classify images by shape and fur
 therefore therefore it will it will be vulnerable to the same attack right this
 is super easy to explain in this model there is no reason why this should
 happen in the dimpled manifold model unless you amend it by some more hand
 wavy things they say the direction mystery when we use an adversarial attack
 to modify a cat into guacamole why doesn't the perturbation look green and
 mushy okay so they say well in the old model you would have to walk along the
 image manifold from here towards the guacamole images and that should mean
 that your image should sort of change to look like a guacamole in our in the
 dimpled manifold model you go off the manifold perpendicular and that
 explains why the adversarial perturbation looks like a little bit
 like just random noise again no one thought this in the old model in fact
 we have a pretty good explanation why it still looks the same and that's because
 humans are much more receptive to this thing right here to the shape whereas
 neural networks also or much more consider this thing right here the fur
 also they consider fur and shape in different proportions than the humans do
 and so that's we already sort of knew this and it's in fact a better
 explanation the uniformity mystery you know why the decision boundary is ever
 present so they claim because the there's this dimple right here even you
 know the most far away cat image here has a close crossing to the decision
 boundary so there is no cat images that are kind of closer to the decision
 boundary but this is I think this is just a property of a high-dimensional
 classifier I think that here our 2d view of the world betrays us and yeah
 especially if we can go really far in feature space with a tiny perturbation
 and input space this is not not a mystery not even a mystery the vanishing
 gap mystery okay which is about adversarial training I think which we're
 gonna skip here and then there is the accuracy robustness trade-off mystery so
 this is if you do if you train a model adversarially which means that here look
 here I have my cat okay I train I have a data set of cats and dogs I train my
 neural network on it it's vulnerable what can I do what I can do is I can
 create adversarial images this is a cat right I can create adversarial images by
 making this into a dog okay so this is a dog because I changed the first
 structure a little bit this is an adversarial example now I add this so
 this is comes from the data set now I add this to the data set but I tell it
 this is a cat too right this is a cat and this is a cat if I do this with my
 neural network the neural network will become robust to adversarial examples
 to a degree not fully but to a degree this is the best method we have so far
 of defending against adversarial examples called adversarial training now
 what you do when you do this is you train the network to to sort of classify
 the advert to yeah classify to incorporate the adversarial ness into
 its decision-making process and this results usually in a degradation of the
 generalization performance of the network so as it becomes more robust it
 becomes less accurate on real data right you gain accuracy on adversarial data
 you decrease the accuracy in real data which makes sense intuitively but it is
 a strong effect which is not the same as you know I simply teach my model to do
 yet another class it is quite it is actually a trade-off now they try to
 explain this right here when we train the network we keep the images
 stationary and move to decision boundary by creating dimples when we create
 adversarial examples we keep the decision boundary stationary and move
 the images to the other side by allowing a large perpendicular derivative we make
 the training easier since we do not have to sharply bend decision boundary
 against around the training examples so this is when you train normally when you
 train without adversarial examples they say there is a large perpendicular
 derivative which in the like the what they mean is that the data samples are
 of push these dimples out that that's the large perpendicular derivative the
 perpendicularity is to the image manifold and that makes it easy because
 you don't have to bend the decision boundary a lot so you can kind of
 remain here and you have to kind of create these dimples again their
 argument is you don't want to bend this boundary a lot which makes training easy
 however such a large derivative also creates very close adversarial examples
 yeah this is their claim that now the decision boundary is pretty close
 because you don't bend the decision boundary by too much around the data
 because you do dimples any attempts to robustify a network by limiting all its
 directional derivatives will make the network harder to train and thus less
 accurate I'm not super sure how to interpret this so I might be doing this
 wrong right here but if you create adversarial example what you do is you
 essentially have this data point and you create an adversarial example this data
 one is yeah well these are of the same class so now that is now the the
 decision boundary has a sort of bend harder okay which makes it more hard to
 train and at some point it so it's harder to train and that's why you have
 less accuracy and at some point it says well actually I don't want to bend that
 much I'd rather make a mistake here and just bend around both of these data
 points and now you have a wrong classification so that's sort of their
 explanation of why this happens which I find a bit hand wavy you have to argue
 like ooh ease of training bending the decision boundary and so on in this
 model right here super easy okay what happens if I create cats that have cat
 fur and dog fur and I tell the network these both are cats well essentially I
 tell them I tell the network look there are two features right here the fur and
 the cat and you know the fur just just disregard it just don't do that don't
 regard the fur as a feature because it's useless now because I now have cats with
 cat fur and cat with dog fur so the network can't use that to classify
 anymore and that explains why it gets less accurate because I take away one
 useful feature okay so you know now the network has less useful features and
 that's why it gets worse this it's it's a pretty simple explanation in the
 stretchy feature model it has there's a lot of work to make this happen in the
 dimpled manifold model so lastly they try to explain and they what they came
 an interesting mystery in this this paper that I have cited throughout and
 what that is is that it's kind of the same experiment as here where we create
 adversarial examples and we add them to the training set except for two things
 first of all we don't have the original so our new data set is not going to
 contain the original images it's only going to contain the adversarial examples
 second it is going to contain the adversarial example image but the label
 isn't going to be the correct label quote-unquote correct from where we
 created but the label is actually going to be the adversarial label the wrong
 label okay so we're going to tell the network this is a dog please learn that
 this is a dog right it's a cat with dog fur and the old training images are
 nowhere in the data set we just do a data set with these wrongly labeled
 images now when we go and we apply this so we train we use this we train a
 network right to classify cats and dogs and now we once we've trained this
 network we go we take one of these samples of the original data set we
 classify it it's going to give us a correct classification right so it will
 recognize that this here is a cat even though we told it that this here is a
 dog now how does it do this it does this by looking at the fur you know we've
 we've doubled down on the fur here right so this is like we really made that fur
 feature super strong in these adversarial examples so it's going to
 look at the cat fur and even though none of the cats have the shape like this we
 sort of we sort of supercharged that fur feature again in this model not a
 problem essentially what we've done is we've created two data classes you know
 one up here and one down here that have the fur supercharged and now it's just
 going to mainly look at that fur structure and that is a useful feature
 right so this this what's called their features not bugs paper adversarial
 examples are features not bugs or other way around not bugs they are features
 has demonstrated with this experiment this notion that there are adversarial
 examples result from useful generalizing features in the data set
 that are simply of by definition the features that are not large enough for
 humans to see what they call non robust features how do they explain this they
 say the original people try to explain this highly surprising role by
 distinguishing between robust and non robust features in any given image where
 some of them are preserved by the adversarial change and some are not
 however it is not clear what makes some of the features more robust than others
 definition just definition like like if you have features and you order them by
 their size like by their how much you have to change the pixels that some
 features are going to be larger than other features and then some features
 going to be below that cutoff where you define adversarial examples budget this
 is definition makes them such that some of more robust it's not it's not clear
 our new model provides very simple alternative explanation which does not
 necessarily contradict the original one okay at least this which is summarized
 in figure four to simplify the description will use 2d vertical cut
 through the input space and consider only the decision boundary that
 separates between cats and anything else okay so they have this example right
 here they say look we have a decision boundary that distinguishes cats see
 from non cats and the green one here is the image manifold and the gray is the
 decision boundary okay so now what we do is we create adversarial examples in
 frame two right here you can see that we make the cats into non cats and we make
 the be the bats into bats aren't very popular lately the badgers into into
 cats so we make the badgers into cats and we make the cats into these whatever
 DS ducks okay and now we relabel those and that gives us a new data manifold so
 the new data manifold is this data manifold right here and we have also new
 labels and now they claim the resulting decision boundary in figure four as you
 can see right here this is the resulting decision boundary the gray one it is it
 is very similar to the decision boundary in the first frame and therefore we
 shouldn't be surprised that this new decision boundary that results from this
 perturbed data results in the same decision boundary as the original one
 okay however like why like why so their whole they have two notions notion one is
 that the decision boundary follows the data manifold closely except it sort of
 bends around the data a little and you can see this right here like this
 decision boundary kind of follows the data yet it just happens to be on the
 correct side of the data points at any given moment which okay okay however they
 also make the claim in different parts of their paper that bending the decision
 boundary and so on is not good you'd rather want to have a simple decision
 boundary so to me there is no reason why the decision boundary couldn't just look
 like this it would correctly classify this new data set right however it would
 not correctly classify it would not correctly classify the let's say the C
 that was right where was it right here or right here these data points it would
 not correctly classify so you see that this until now they've always had this
 data manifold to be sort of super duper straight and smooth and that's how they
 can also say well following the data manifold and not bending too much and so
 on those are not in conflict with each other but now that they are in conflict
 with each other you have to give you gonna give up one or the other and only
 in one of them do actually does this experiment here still make sense in the
 other one it doesn't and but if you give up the ooh bending too much is bad then
 you know you lose a bunch of explanations that you have up here so yeah
 like it's one in my mind it's one or the other and there's I there's still no
 reason I think no good reason why this like the decision boundary should align
 super closely with the data points like if there if there is nothing here right
 if this is perpendicular really to the data manifold like why would the
 decision boundary align so closely with the data manifold in that point I don't
 know okay so they ask why are DNN so sensitive and humans so insensitive to
 adversarial perturbations essentially their argument here is that humans
 project the input data onto the image manifold which is a contested claim
 right I don't I don't think that is a I think that is not not a widely accepted
 I mean it's it's certainly possible but also I'm not sure I'm not sure that
 humans do project they have like an internal manifold of natural images and
 project onto that every time they analyze an image and also the yeah how do
 you project right like how like both of these features are useful okay so both
 of the features are useful if you project an adversarial example like why
 do you project it onto the shape dimension and not onto the fur dimension
 right why there's no explanation right here we know that sort of humans are
 more receptive to shapes and so on but just projecting won't get you there so
 now they're going to into experiments and I want to highlight one particular
 experiment right here they have synthetic experiments they have their
 experiments I want to highlight this experiment right here remember they said
 their experiments were going to give you know strong support that and this
 experiment right here what they want to claim is that okay you have the data
 manifold here if you are if you have a data point and you make an adversarial
 example the question is do adversarial examples go along the image manifold or
 do adversarial examples go sort of perpendicular to the image manifold they
 they their claim again is that V this here would give support to the old view
 of adversarial examples and this here would support the dimpled manifold view
 because of course the decision boundary would be sort of following the data
 manifold curving around the data and then following the image manifold again
 so here would be sort of the other data point going below that a little bit all
 right so that is the view right here now what they're going to try to show you is
 that if you want to create an adversarial example on the manifold you
 have to walk much longer for much longer until you find an adversarial example
 then if you go off the manifold if you go yeah and they're also going to show
 you that if you are not constrained if you can go anywhere you want with an
 adversarial example then that will be very similar to when you force the
 adversarial example to go off the manifold and this gives a bit of proof
 that you know if two things behave equally they're you know probably equal
 so what they're going to do is they're going to try to make an adversarial
 attack first of all a regular one this one they're gonna say okay we're gonna
 make an adversarial attack let's measure how far we have to go to cross the
 decision boundary second they're going to say let's make the same thing but
 let's force the attack to be on the manifold of natural images and let's
 measure that and lastly they're going to mask okay let's do the same thing but
 force it to be off the data manifold and then they're going to measure how long
 these are how long the adversarial attacks are what's their their norm and
 they're going to find of course they're gonna want to find that these two are a
 about similar norms and way smaller than the one that is on the data manifold
 sort of giving evidence to you know if you go perpendicular to the data
 manifold you have to go very not very far and that's what adversarial attacks
 do okay yeah so how first of all how do they force the the adversarial attack to
 be on the manifold what they do is they do an autoencoder so they train an
 autoencoder so they an autoencoder is a neural network that has sort of a
 bottleneck layer and you try to just reconstruct the inputs data okay you
 tried that these two are equal however in the middle here you have a very low
 dimensional representation so where this is an n dimensional representation
 this is a k dimensional representation and a k much smaller than n if you can
 reconstruct the images correctly that means that you sort of have captured
 the representation in these low dimensions right here so what they're
 going to do is they train an autoencoder they take that low dimensional
 representation they linearize around it and that's how they have a way to
 project onto the image manifold by simply only moving around in this low
 dimensional manifold right here or always projecting onto it first of all
 it's a bit of a trouble because how you train the autoencoder is like for these
 experiment I think it's very relevant to how they this image manifold is going
 to look like if you train it with L2 you sort of already make some claims about
 what are important features and whatnot but let's disregard this right here
 let's say they have an accurate way of projecting onto the image manifold onto
 the manifold of natural data and here's what they find look let's look at image
 net okay no constraint PGD it this is the norm you know it's some number okay
 so like 0.14 now off manifold PGD is where they deliberately project off the
 manifold so they project on the manifold they subtract that they say you're not
 to do anything with the mana of the image manifold and that's 0.152 which is
 slightly larger than the no constraint PGD but essentially the same size now on
 manifold PGD okay here is a way bigger number like six times bigger number so
 their claim is look up up to six times more you have to go on the manifold than
 off the manifold and that gives credence to their claims now okay so what I've
 done is they have you know they have some descriptions of their experiment
 specifically they have descriptions of what library they used they used advert
 torch okay so I used advert torch to they used you know L2 PGD I use that too
 and they told me how much their low dimensional representation is so the K
 here how much that is how much the N is and so I was able to reproduce that
 experiment now what I've done is I have done the same thing and you can see
 right here this is this the panda image from image net they use an image net
 classifier and what they do is they do it greedy so they stop as soon as they
 cross the decision boundary and then they measure the norm you can see right
 here this is the perturbation now it's a soccer ball and here is the size 0.7772
 that's the norm of the original perturbation adversarial what I now do
 as I project onto the manifold but I don't the difference is I don't project
 onto the image manifold what I do is here you see project onto K I simply
 project onto any K dimensional manifold so I know what K is K is 3,500 so it's a
 very small number compared to the input number and so what they project is
 actually the gradient so the gradient of the adversarial attack that you use to
 update your image that's what they project they have the algorithm clearly
 lined out so what I do is I simply take you can see right here I take a random
 set of of dimensions like of pixel coordinates in the gradient and I denote
 the first you know the first few the first K as the manifold and the last K
 as not the manifold this is not the image manifold there's nothing to do with
 the image manifold this is simply a random K dimensional subspace of the
 pixel space okay and now when I project onto K I simply take all the others in
 the gradient and I set them to zero that's I project onto a K dimensional
 manifold after that you normalize the gradient and so on so you proceed you
 proceed as you would right so here you can see the the project is used before
 you normalize the gradient so there's no issue with sort of the the step size you
 simply project onto the manifold and I have the same thing by the way
 projecting off the manifold where I simply take the K dimensions and
 set them to zero okay so now let's look what happens if I project on to the
 manifold oh wow before it was 0.77 and now it's 6.5 so about eight times
 larger and now let's look what happens if I project off the manifold it's 0.7773
 instead of 0.7772 so what they're seeing right here and you know maybe
 okay maybe I've done it modulo I've done it wrong and I completely don't
 understand what's going on what they have found is simply an effect of
 projecting onto any lower dimensional space yet they claim that this is like
 in support of their hypothesis which clearly I have no clue what the data
 manifold is I've just projected onto a random manifold and I got the same
 results so I see they have other experiments where they try to kind of
 convince you with all the types of perturbations and so on but you know
 like no this these they have other experiments but this is just one that I
 could try quickly again maybe I've done it wrong to me this Occam's razor is
 strong here like Occam's razor in this work is quite a bit like there can be
 like there can be many hypotheses that coincide with the results you're getting
 and with the phenomena and it's easy to think that stuff is in favor of your
 hypothesis is providing support for it when there are other explanations
 available oh I almost forgot about Goodfellow's claim that you know they say
 belongs to the sort of old thinking that is now that is not a correct thinking
 and the claim that when you make an adversarial examples you somehow go
 towards the centroid of a different class and in imagination it's something
 like this on the on the left right here however if you think about this in this
 space okay let's say you start out here and you go towards the centroid of the
 other class right the pro where's the centroid here approximately like this
 what happens in feature space because of the stretchy feature because of the
 different scales okay what happens in feature space is it pretty much like the
 blue arrow here so it's that in feature space you go a long way actually this is
 probably I should have drawn this here to be square and this here to be super
 stretchy right yeah yeah I think so yeah I was I was wrong in drawing this so
 this here should be squares and this here actually should be super duper
 stretchy right so the centroid what was the centroid here is like way up here
 like way up here somewhere okay so this gets super stretched and you cross the
 boundary in this one feature right like the fur feature and yeah so I think this
 is it's still a correct claim you go towards the centroid of another class
 but because you go this in input space in the feature space this results in
 sort of a dramatic shift in some features and a not so dramatic shift in
 other features so while in the input space you go towards the centroid
 equally in all pixel directions you don't go towards the centroid equally in
 all pixel directions in the sorry in all feature directions so I think the claim
 that Goodfellow made is valid here still and explains like is concurrent with the
 stretchy feature explanation that I'm pretty sure that's also kind of what
 maybe I can't read his mind but maybe what he meant by that and not
 necessarily this picture right here not necessarily that actually the entire
 picture is going to change into the other class okay that was the
 interjection and back to the conclusion but as I said make up your own mind what
 do you what do you think of this go through the paper they it's it's a good
 paper like it's written it's written well there it has a lot of experiments
 has quite a lot of appendix where they give you more results and so on and it's
 not like again it's not like it's in it's necessarily incompatible right it's
 not I don't disagree with them I just think it's it's not as useful as they
 claim and it's kind of insufficient I don't disagree with their their main
 claims yeah and I think we already kind of knew a lot of those stuff and our
 current mental models are explaining things maybe a little a little better
 and yeah if you use the the squishy feature what would I call it the the
 stretchy feature model has a fancy name now but again is this is not mine this
 is just kind of a a bringing together of of what we what I think we know about
 adversarial examples safe to say there's going to be something that challenges
 this and that's going to be exciting alright thanks so much for being here
 listening and I'll see you next time bye bye