The Odds are Odd: A Statistical Test for Detecting Adversarial Examples
https://arxiv.org/abs/1902.04818

Abstract:
We investigate conditions under which test statistics exist that can reliably detect examples, which have been adversarially manipulated in a white-box attack. These statistics can be easily computed and calibrated by randomly corrupting inputs. They exploit certain anomalies that adversarial attacks introduce, in particular if they follow the paradigm of choosing perturbations optimally under p-norm constraints. Access to the log-odds is the only requirement to defend models. We justify our approach empirically, but also provide conditions under which detectability via the suggested test statistics is guaranteed to be effective. In our experiments, we show that it is even possible to correct test time predictions for adversarial attacks with high accuracy.

Authors:
Kevin Roth, Yannic Kilcher, Thomas Hofmann
 Hello and welcome. Today we're looking at the odds are odd, a statistical test for detecting adversarial examples. So shameless self-promotion here since this is me. So this is an archive and basically what we do is we're detecting adversarial examples. For those who don't know what an adversarial example is, it's basically a way of fooling a classifier in order to kind of get it to do something weird. Let's look at it. So maybe you have an image of a cat. I have no clue how a cat looks. Alright, so you have an image of a cat and you have a classifier. So the classifier takes this image as an input, kind of winds it down to some probabilities of classes and cat, dog and so on. And it then gives you an estimate of how likely each class is. So what the adversarial example does is it changes this image and it adds a noise. So this is just a very specific noise and you have kind of a multiplier here, gamma, which is super small. So the noise is almost... you can't see it with a human eye basically, it's so small. But it's able to perturb this image in a way that the probabilities will change such that all of a sudden a different class now is the highest class. So basically we're able to fool these classifiers by adding just very little bit of very very specific noise. So that's an adversarial example. These have many implications in, let's say, security applications and also in understanding how these classifier works. Alright, so our task is to explain and detect them, explain why they happen and detect when they happen. Alright, so what do we do? Basically let's just jump right into the thing here. We view a classifier as an output, so you have logits, what's called logits, L is this. This here is your neural network up to the last layer. Basically it can be like something like a convolutional neural network and so on. It gives you a feature representation. So you extract from the image X a feature representation, which is this entire thing here, and then you multiply this feature representation. So this is going to be some vector of dimension D. You multiply this by this weight matrix, which is going to be something like, okay I've drawn it in the wrong direction here. Let's draw W over here. It's going to be D by, let's say K, where K is the number of classes. Okay, still wrong. D by K, right? And output a vector of dimension K, which then is this cat, dog and so on. So these are the logits and the logits get transformed to the probabilities by running it through a softmax layer. But basically we view a classifier as having a feature representation and a weight matrix. And this here is a matrix multiplication adult product by matrix. So what we see basically is, this is kind of where the adversarial examples happen. So when we look at this weight matrix, right, again we look at the D dimensional feature vector here, and we look at the weight matrix, what it does is it has columns, right? Columns. Let's say we have four classes here, right? So it has these four columns and each of them is D dimensional. So each of them is going to be multiplied by this thing and giving a score. So the final score for a class is going to be the multiplication of a row W1, W2, W3, W4 by this feature vector. Let's call the feature vector little f. So your logit of class i is going to be the inner product of W i and f. Alright, we'll leave away biases for now. There's okay, we can introduce biases to make it a bit more complicated but it changes nothing. So your logit is going to be the inner product and whichever logit is the highest wins. So that's going to be the prediction of the classifier. So since you can, in an adversarial example, what can you change? You can change this feature vector here, this f. By changing the x you can change the output of the convolutional neural network which is the feature vector. And what you have to do in order to make a logit as high as possible, basically make one class as high as possible, is you need to make this inner product as high as possible. And what's an inner product? If you look in a classic vector representation space, if this is W i and this is f, what you want to do is you want to make f and W align as much as possible. Because the inner product is going to be basically dependent on the angle and the magnitude. So you can you can stretch f for sure but it's going to be kind of aligned with all the W's then more by stretching or more negatively in whatever way you want it. But basically you want to rotate, you want to align as much as possible the f with the W i. So you want to kind of go into this direction with f. So now not only do you want to kind of maximize f with a particular W i, what you want to do is be adversarial. The adversarial task is often framed as just it's either targeted, so find a... so i needs to be a particular other class, or it's untargeted which means just just give me a perturbation that will make the classifier be fooled. And be fooled means whatever it predicts right now it should predict something different. So what you ultimately want to do is you want this as high as possible for some i that is not the correct i and you want this other quantity W y. Let's call it W y. Let's say the classifier is 100% correct. So W y, y is the label of x. W y is whatever column here is currently predicted. So you want the sum column where i is not equal to y to have maximum inner product and so this is not no longer l i, we'll get to that, to have maximum inner product and you want this inner product with the correct class to be as small as possible, which ultimately means you want this entire quantity maximized. So it's a pretty simple idea. We'll call, let's say this is the log i minus the log y. We have slightly different notation in the paper. I think we call this z but never mind. So you basically just want to make this as large as possible. So our point is since this is not the only thing, you want to maximize this but you have a constraint. Namely your constraint is that your delta x can only be small. Your delta x can only be small because the point of an adversarial example is that the perturbation is so small you can't see it and that means that you basically don't have much wiggle room to do these perturbations, which means that we should be able to detect a pattern like this in the latent space. So this here is the latent space feature vector and if we can kind of detect a pattern in the latent space then we kind of get the adversarial example detector. So how do we do this? We measure exactly this. What we do is we measure the alignment between the original, between the currently predicted class and between all other classes. So in this graphic here you see this. It's a 10 class classifier. This is CIFAR10. We only show one, two, three, four, we only show six of the other classes but we have the full graphic in this. So this shows an adversarial example. The axis going on top of each of the images is the alignment with the adversarial class. So this has been an adversarially perturbed sample. So this shows the alignment with the adversarial class and of course you see the bright red dot, if you just focus on that, that is the adversarial example projected into this. So of course the alignment is going to be very very high with this class since the classifier actually predicts this class. The blue here is the sample that the adversarial sample was derived from which means the original image. And you already see without looking at any of the other dots that the blue is around zero here, around zero here, around zero here, here and here. But here it's very high in this axis. So the axis to the right is for each of these plots here it's one of the other classes except for the currently predicted adversarial class. So that this axis is always the same axis and while the axis to the right for each plot is a different one. And you can already see, and that's why we frame it in the green, this plot here is where the axis to the right corresponds to the original class of the classifier. So don't look yet at the other plots. What you see here is basically the blue is really high in this class right and the adversarial example procedure basically has driven it down this class and up this class which is exactly saying it has made this inner product small and this inner product large. So where do we go from here? Let's actually jump this graphic a bit and go to this one that's way down here. Alright so what we've done is we've taken the an example just out of the data set right and then we've taken an adversarial example. So say X is the example of the data set and then X hat is the adversarial example derived from this. Alright in this plot to the right X would be sitting about here. I'm gonna explain what the what the the kind of meaning is and X hat would be sitting down here right it's about one third from the top one third from the bottom let me draw this more. Alright so what this axis represents here is basically what we've done is we've gone from X to X hat in very small steps and at each step we've asked the classifier hey classifier what's the probability of the class Y so Y is the class of X right and the class of X X hat is some some different what some some other class right since it's an adversarial example we've so we've asked the classifier what's the class of X and basically no basically we've asked what what's the probability that Y is the class of X and that's represented in white right so the more white the higher the classifier thinks the class Y is probable. So the direction down is going into the direction of X hat minus X so it's going into the into the adversarial direction and then the direction across is we've taken some direction that's orthogonal to this direction and then also went into tiny steps and asked the classifier hey classifier what do you think is the probability of Y here so we've basically done this kind of grid sampling and at each point we've asked the classifier what do you think which how probable is Y and the classifier will always output some number and we plot it in white and so this direction again is always the same in this direction we basically randomize it and then aggregate it over over lots and lots of different samples and we also aggregate this entire thing over the entire over the data set so we get a comprehensive view of what adversarial examples look like in the view of the classifier and what we find is pretty interesting so when you go from the original class you basically in every direction here in every direction it kind of the original class kind of decreases smoothly right you see at the edges here it kind of gets black so the further away you go from the original example that the more the kind of shadier the classifier gets it's like yeah I'm not so sure anymore that this is the class right but if you go into the direction of here if you go into the direction of the adversarial example the kind of drop-off is first of all it's very steep so all of a sudden here you're in very dark territory which means the classifier is doesn't think why is probable at all anymore and moreover you get this kind of cone here so what we see is what we what we think is happening is that given an example there are these directions in late in in in image space basically straight directions that go to adversarial examples right and we call these cones because they they're kind of low dimensional directions in in the space where the adversarial example lies and what's really interesting is we have those plots here do we have more so what's what's quite interesting is that if you if you go come on well this is kind of okay the quality of the of the plot is not is not very very good so I'm gonna I may be able to to draw this here so if your start here and you go here what happens to the original class is you start out high you go down rapidly and you stay down even if you go super far into this direction the this class will stay down whereas let's say this is y hat y hat will start low go up and then kind of fade so here is where the adversarial example would sit sorry at about this distance that's this distance here means as you go towards the adversarial example right here the probability of the adversarial class rises and the probability of the original class drops then as you go further this is what's what's interesting kind of this probability here drops which means the classifier is kind of like yeah okay there's too much noise now I'm not so sure about this class anymore but the this this class here kind of stays low very very long even if you go into this direction so this this gives us kind of a hint that adversarial examples are characterized by specific directions that you go into that you that you can go into and kind of suppress the original class and pump the new class up which is kind of exactly what we've claimed with this inner inner product alignment right that the next experiment we've done is we've taken this adversarial example here and said well if we go outside if we go into random directions right it's just really this one direction that's problematic if we go into random directions actually we should be you know go back to the original class right since it's basically surrounded by the original class this is just one direction and this here represents all the other directions there are and how many directions are there in in pixel space like a lot so we should be able to get back to the original class but that's not the case that's we found that's not the case and we also found why so I still want to go back to this plot here if you do this if you add noise and this is the noise magnitude here what you'll see is the orange here is the adversarial class so orange will go down down down down down right as you increase the noise the blue is the source class so the blue goes up and it goes up faster you see it goes up faster than the green which is the highest other class so green is whatever class is not that was there not the source but the highest class other than that so the source class goes up quickly but before the source class can overpass the adversarial class which happens back there the highest other class has already kind of taken over so the source class is basically too weak and if you again look at this this plot here if you go with an actual color picker you see that the amount of white here and here is is not high enough it's like 0.3 or something out of one or even lower so the the kind of source class is not strong enough that by simply adding a bit of noise you can go back but we thought hey if this is correct we can actually detect we can detect this effect here this rising of the source class faster so our plan is basically we add noise a particular amount of noise just a little bit actually and then we detect which basically which class falls and which class rises and the way we do this is we we detect the this exact alignment that I've described before under noise so we form this quantity here for all classes other than y so y is the the class that's currently predicted and we look at it what happens under it under noise right so and that's where we get to this graphic here so again this axis is the adversarial class or the class that's currently predicted right this axis here is all the other classes for each plot one and when we add noise what do you see is the noise magnitude is encoded in the brightness of the dots so the darker the red dots the more noise we've added here is the original adversarial sample then as we add noise you see here here more noise more noise more noise it nothing's really happening for the for the if if if it's like one class that has nothing to do with the original class it simply kind of goes down simply kind of gets less sure about this class right but in case of the original class that the adversarial example was derived from it really rises it really kind of at the same time that it drops it rises into that direction so we're able to measure these these deltas here under noise and we're able to to devise basically statistics of what happens to these quantities under like if it's not an adversarial sample versus what happens to these quantities if it's an adversarial sample so here you see pairings of basically source class and adversarial class samples so each of these histograms is collected from that and what you can see is in blue the kind of alignment under noise of the source class sorry the alignments under noise of a non perturbed sample and in orange the alignments under noise of an adversarial sample and what's cool is that these these alignments you can see in all of these cases are very different so there is a clear signature in the adversarial sample in these noise induced alignments with the with the weight matrix rows that makes you able to basically build a detector you can say all right anything to the left is clean anything to the right is adversarial and we can do this over many different types of noises and then build basically a voting mechanism on that and thereby detect adversarial examples so we have a bunch of experiments we mostly experiment on the c410 and on the image net data set and you can see over here so this is the main kind of one of the main results the detection rates of our statistical test so as you can see we are detection rate this is on clean samples on clean samples you want the detection rate to be low on adversarial samples you want the detection rate to be high and this we achieve very large detection rates while having very low false positive rates especially on image net so it seems like the more tuned these models are the better these models are the better we are at detecting adversarial examples to it it's kind of a direct correlation to how well the models perform on accuracy in a clean setting and what we can do is now since we cannot only detect these things but we can detect these things in a fashion so if if you look at these things and you have like a sample of a particular class that's predicted right let's say this class and you go and look at it at the position of the noise induced features over each of them so let's say here here here here here here here here here right you can then clearly say well not only do I detect an adversarial example here right I look at the I look at each of the class of the classes that it could be derived from right if all if all of them say it's a clean sample then all right it's a clean sample but if one of them says it's an adversarial sample then I don't not only do I know it's an adversarial sample but I say aha this must be the source class right this is the exact effect we saw here all right we can if we detect this pattern here we can also back deduce basically aha so this must be the original class that the adversarial example was derived from so we're basically able to build a not only a detector but we're basically able to reconstruct the original class and here you see for these models let's say on CIFAR-10 we imagine that is a bit too large as of yet for our compute but on these models that have clean accuracies that are pretty high on CIFAR-10 plus this this kind of toy network here we're able to reconstruct the original class so basically this is defense against adversarial examples by by getting to almost clean accuracy back so this is a really surprising actually and kind of nice so we we do a bunch of other experiments including we defend against an attacker that's actually aware of this thing but the main the main point here is we don't say this is kind of the end-all method of defending against adversarial examples we simply want to kind of encourage the way of thinking of of these kind of noise what what if you what if you noise induce perturbations how does your network react to that can you can you detect these effects here can you detect effects like this and are these unavoidable or are there architectures are there architectures we can basically build such that adversarial examples have no chance except doing something like this which we can then easily detect all right so that was a bit of an introduction if you like it check out the entire paper and goodbye
 Hello and welcome. Today we're looking at the odds are odd, a statistical test for
 detecting adversarial examples. So shameless self-promotion here since this
 is me. So this is an archive and basically what we do is we're detecting
 adversarial examples. For those who don't know what an adversarial example is, it's
 basically a way of fooling a classifier in order to kind of get it to do
 something weird. Let's look at it. So maybe you have an image of a cat.
 I have no clue how a cat looks. Alright, so you have an image of a cat and you have a
 classifier. So the classifier takes this image as an input, kind of winds it
 down to some probabilities of classes and cat, dog and so on. And it then gives you
 an estimate of how likely each class is. So what the adversarial example does
 is it changes this image and it adds a noise. So this is just a very
 specific noise and you have kind of a multiplier here, gamma, which is super
 small. So the noise is almost... you can't see it with a human eye basically, it's
 so small. But it's able to perturb this image in a way that the
 probabilities will change such that all of a sudden a different class now is the
 highest class. So basically we're able to fool these classifiers by adding just
 very little bit of very very specific noise. So that's an adversarial example.
 These have many implications in, let's say, security applications and also in
 understanding how these classifier works. Alright, so our task is to explain and
 detect them, explain why they happen and detect when they happen.
 Alright, so what do we do? Basically let's just jump right into
 the thing here. We view a classifier as an output, so you
 have logits, what's called logits, L is
 this. This here is your neural network up to the last layer.
 Basically it can be like something like a convolutional neural network and so on.
 It gives you a feature representation. So you extract from the image X a feature
 representation, which is this entire thing here, and then you multiply this
 feature representation. So this is going to be some vector of dimension D. You
 multiply this by this weight matrix, which is going to be something like, okay
 I've drawn it in the wrong direction here. Let's draw W over here.
 It's going to be D by, let's say K, where K is the number of classes.
 Okay, still wrong. D by K, right? And output a vector of dimension K, which
 then is this cat, dog and so on. So these are the logits and the logits get
 transformed to the probabilities by running it through a softmax layer. But
 basically we view a classifier as having a feature representation and a weight
 matrix. And this here is a matrix multiplication adult product by
 matrix. So what we see basically is, this is kind of where the
 adversarial examples happen. So when we look at this weight matrix, right, again
 we look at the D dimensional feature vector here, and we look at the weight
 matrix, what it does is it has columns, right? Columns. Let's say we have four
 classes here, right? So it has these four columns and each of them is D
 dimensional. So each of them is going to be multiplied by this thing and giving a
 score. So the final score for a class is going to be the multiplication of a row
 W1, W2, W3, W4 by this feature vector. Let's call the feature vector little f.
 So your logit of class i is going to be the inner product of W i and f.
 Alright, we'll leave away biases for now. There's okay, we can introduce biases to
 make it a bit more complicated but it changes nothing. So your logit is going to
 be the inner product and whichever logit is the highest wins. So that's going
 to be the prediction of the classifier. So since you can, in an
 adversarial example, what can you change? You can change this feature vector here,
 this f. By changing the x you can change the output of the
 convolutional neural network which is the feature vector. And what you have to
 do in order to make a logit as high as possible, basically make one class
 as high as possible, is you need to make this inner product as high as possible.
 And what's an inner product? If you look in a classic vector representation
 space, if this is W i and this is f, what you want to do is you want to make f and
 W align as much as possible. Because the inner product is going to be basically
 dependent on the angle and the magnitude. So you can you can stretch f for sure
 but it's going to be kind of aligned with all the W's then more by stretching
 or more negatively in whatever way you want it. But basically you want to rotate,
 you want to align as much as possible the f with the W i. So you want to kind
 of go into this direction with f. So now not only do you want to kind of maximize
 f with a particular W i, what you want to do is be adversarial. The adversarial
 task is often framed as just it's either targeted, so find a... so i needs to be a
 particular other class, or it's untargeted which means just just give me
 a perturbation that will make the classifier be fooled. And be fooled means
 whatever it predicts right now it should predict something different. So what
 you ultimately want to do is you want this as high as possible for some i
 that is not the correct i and you want this other
 quantity W y. Let's call it W y. Let's say the classifier is 100% correct.
 So W y, y is the label of x. W y is whatever column here is
 currently predicted. So you want the sum column where i is not equal to y to
 have maximum inner product and so this is not no longer l i, we'll get to that,
 to have maximum inner product and you want this inner product
 with the correct class to be as small as possible, which ultimately means you want
 this entire quantity maximized. So it's a pretty simple idea. We'll call, let's say
 this is the log i minus the log y. We have slightly different notation in the
 paper. I think we call this z but never mind. So you basically just want to make
 this as large as possible. So our point is since this is not the only
 thing, you want to maximize this but you have a constraint. Namely your constraint
 is that your delta x can only be small. Your delta x can only be small
 because the point of an adversarial example is that the perturbation is so
 small you can't see it and that means that you basically don't have much
 wiggle room to do these perturbations, which means that we should be
 able to detect a pattern like this in the latent space. So this here is
 the latent space feature vector and if we can kind of detect a pattern in the
 latent space then we kind of get the adversarial example detector.
 So how do we do this? We measure exactly this. What we do is we measure the
 alignment between the original, between the currently predicted class and
 between all other classes. So in this graphic here you see this. It's a 10
 class classifier. This is CIFAR10. We only show one, two, three, four, we only show six of the other
 classes but we have the full graphic in this. So this shows an adversarial
 example. The axis going on top of each of the images is the alignment with the
 adversarial class. So this has been an adversarially perturbed sample. So this
 shows the alignment with the adversarial class and of course you see the bright
 red dot, if you just focus on that, that is the adversarial example
 projected into this. So of course the alignment is going to be very very
 high with this class since the classifier actually predicts this class.
 The blue here is the sample that the adversarial sample was derived from
 which means the original image. And you already see without looking at any of
 the other dots that the blue is around zero here, around zero here, around zero
 here, here and here. But here it's very high in this axis. So the axis to the
 right is for each of these plots here it's one of the other
 classes except for the currently predicted adversarial class. So that
 this axis is always the same axis and while the axis to the right for each
 plot is a different one. And you can already see, and that's why we frame it
 in the green, this plot here is where the axis to the right
 corresponds to the original class of the classifier. So don't look yet at
 the other plots. What you see here is basically the blue is really high
 in this class right and the adversarial example procedure basically has driven
 it down this class and up this class which is exactly saying it has made this
 inner product small and this inner product large. So where do we go from
 here? Let's actually jump this graphic a bit and go to this one that's way down
 here. Alright so what we've done is we've taken the an example just out of the
 data set right and then we've taken an adversarial example. So say X is the
 example of the data set and then X hat is the adversarial example derived from
 this. Alright in this plot to the right X would be sitting about here. I'm gonna
 explain what the what the the kind of meaning is and X hat would be sitting
 down here right it's about one third from the top one third from the bottom
 let me draw this more. Alright so what this axis represents here is basically
 what we've done is we've gone from X to X hat in very small steps and at each
 step we've asked the classifier hey classifier what's the probability of the
 class Y so Y is the class of X right and the class of X X hat is some some
 different what some some other class right since it's an adversarial example
 we've so we've asked the classifier what's the class of X and basically no
 basically we've asked what what's the probability that Y is the class of X and
 that's represented in white right so the more white the higher the classifier
 thinks the class Y is probable. So the direction down is going into the
 direction of X hat minus X so it's going into the into the adversarial direction
 and then the direction across is we've taken some direction that's orthogonal
 to this direction and then also went into tiny steps and asked the classifier
 hey classifier what do you think is the probability of Y here so we've basically
 done this kind of grid sampling and at each point we've asked the classifier
 what do you think which how probable is Y and the classifier will always output
 some number and we plot it in white and so this direction again is always the
 same in this direction we basically randomize it and then aggregate it over
 over lots and lots of different samples and we also aggregate this entire thing
 over the entire over the data set so we get a comprehensive view of what
 adversarial examples look like in the view of the classifier and what we find
 is pretty interesting so when you go from the original class you basically in
 every direction here in every direction it kind of the original class kind of
 decreases smoothly right you see at the edges here it kind of gets black so the
 further away you go from the original example that the more the kind of
 shadier the classifier gets it's like yeah I'm not so sure anymore that this
 is the class right but if you go into the direction of here if you go into
 the direction of the adversarial example the kind of drop-off is first of all
 it's very steep so all of a sudden here you're in very dark territory which
 means the classifier is doesn't think why is probable at all anymore and
 moreover you get this kind of cone here so what we see is what we what we think
 is happening is that given an example there are these directions in late in in
 in image space basically straight directions that go to adversarial
 examples right and we call these cones because they they're kind of low
 dimensional directions in in the space where the adversarial example lies and
 what's really interesting is we have those plots here do we have more so
 what's what's quite interesting is that if you if you go come on well this is
 kind of okay the quality of the of the plot is not is not very very good so I'm
 gonna I may be able to to draw this here so if your start here and you go here
 what happens to the original class is you start out high you go down rapidly
 and you stay down even if you go super far into this direction the this class
 will stay down whereas let's say this is y hat y hat will start low go up and
 then kind of fade so here is where the adversarial example would sit sorry at
 about this distance that's this distance here means as you go towards the
 adversarial example right here the probability of the adversarial class
 rises and the probability of the original class drops then as you go
 further this is what's what's interesting kind of this probability here
 drops which means the classifier is kind of like yeah okay there's too much noise
 now I'm not so sure about this class anymore but the this this class here
 kind of stays low very very long even if you go into this direction so this this
 gives us kind of a hint that adversarial examples are characterized by specific
 directions that you go into that you that you can go into and kind of
 suppress the original class and pump the new class up which is kind of exactly
 what we've claimed with this inner inner product alignment right that the next
 experiment we've done is we've taken this adversarial example here and said
 well if we go outside if we go into random directions right it's just really
 this one direction that's problematic if we go into random directions actually we
 should be you know go back to the original class right since it's
 basically surrounded by the original class this is just one direction and this
 here represents all the other directions there are and how many directions are
 there in in pixel space like a lot so we should be able to get back to the
 original class but that's not the case that's we found that's not the case and
 we also found why so I still want to go back to this plot here if you do this if
 you add noise and this is the noise magnitude here what you'll see is the
 orange here is the adversarial class so orange will go down down down down down
 right as you increase the noise the blue is the source class so the blue goes up
 and it goes up faster you see it goes up faster than the green which is the
 highest other class so green is whatever class is not that was there not the
 source but the highest class other than that so the source class goes up quickly
 but before the source class can overpass the adversarial class which happens back
 there the highest other class has already kind of taken over so the source
 class is basically too weak and if you again look at this this plot here if you
 go with an actual color picker you see that the amount of white here and here
 is is not high enough it's like 0.3 or something out of one or even lower so
 the the kind of source class is not strong enough that by simply adding a
 bit of noise you can go back but we thought hey if this is correct we can
 actually detect we can detect this effect here this rising of the source
 class faster so our plan is basically we add noise a particular amount of noise
 just a little bit actually and then we detect which basically which class falls
 and which class rises and the way we do this is we we detect the this exact
 alignment that I've described before under noise so we form this quantity
 here for all classes other than y so y is the the class that's currently
 predicted and we look at it what happens under it under noise right so and that's
 where we get to this graphic here so again this axis is the adversarial class
 or the class that's currently predicted right this axis here is all the other
 classes for each plot one and when we add noise what do you see is the noise
 magnitude is encoded in the brightness of the dots so the darker the red dots
 the more noise we've added here is the original adversarial sample then as we
 add noise you see here here more noise more noise more noise it nothing's
 really happening for the for the if if if it's like one class that has nothing
 to do with the original class it simply kind of goes down simply kind of gets
 less sure about this class right but in case of the original class that the
 adversarial example was derived from it really rises it really kind of at the
 same time that it drops it rises into that direction so we're able to measure
 these these deltas here under noise and we're able to to devise basically
 statistics of what happens to these quantities under like if it's not an
 adversarial sample versus what happens to these quantities if it's an adversarial
 sample so here you see pairings of basically source class and adversarial
 class samples so each of these histograms is collected from that and
 what you can see is in blue the kind of alignment under noise of the source
 class sorry the alignments under noise of a non perturbed sample and in orange
 the alignments under noise of an adversarial sample and what's cool is
 that these these alignments you can see in all of these cases are very different
 so there is a clear signature in the adversarial sample in these noise
 induced alignments with the with the weight matrix rows that makes you able
 to basically build a detector you can say all right anything to the left is
 clean anything to the right is adversarial and we can do this over many
 different types of noises and then build basically a voting mechanism on that and
 thereby detect adversarial examples so we have a bunch of experiments we mostly
 experiment on the c410 and on the image net data set and you can see over here
 so this is the main kind of one of the main results the detection rates of our
 statistical test so as you can see we are detection rate this is on clean
 samples on clean samples you want the detection rate to be low on adversarial
 samples you want the detection rate to be high and this we achieve very large
 detection rates while having very low false positive rates especially on image
 net so it seems like the more tuned these models are the better these models
 are the better we are at detecting adversarial examples to it it's kind of
 a direct correlation to how well the models perform on accuracy in a clean
 setting and what we can do is now since we cannot only detect these things but
 we can detect these things in a fashion so if if you look at these things and
 you have like a sample of a particular class that's predicted right let's say
 this class and you go and look at it at the position of the noise induced
 features over each of them so let's say here here here here here here here here
 here right you can then clearly say well not only do I detect an adversarial
 example here right I look at the I look at each of the class of the classes that
 it could be derived from right if all if all of them say it's a clean sample then
 all right it's a clean sample but if one of them says it's an adversarial sample
 then I don't not only do I know it's an adversarial sample but I say aha this
 must be the source class right this is the exact effect we saw here all right
 we can if we detect this pattern here we can also back deduce basically aha so
 this must be the original class that the adversarial example was derived from so
 we're basically able to build a not only a detector but we're basically able to
 reconstruct the original class and here you see for these models let's say on
 CIFAR-10 we imagine that is a bit too large as of yet for our compute but
 on these models that have clean accuracies that are pretty high on CIFAR-10
 plus this this kind of toy network here we're able to reconstruct the original
 class so basically this is defense against adversarial examples by by
 getting to almost clean accuracy back so this is a really surprising actually and
 kind of nice so we we do a bunch of other experiments including we defend
 against an attacker that's actually aware of this thing but the main the
 main point here is we don't say this is kind of the end-all method of defending
 against adversarial examples we simply want to kind of encourage the way of
 thinking of of these kind of noise what what if you what if you noise induce
 perturbations how does your network react to that can you can you detect
 these effects here can you detect effects like this and are these
 unavoidable or are there architectures are there architectures we can basically
 build such that adversarial examples have no chance except doing something
 like this which we can then easily detect all right so that was a bit of an
 introduction if you like it check out the entire paper and goodbye