Radioactive data: tracing through training (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper cnn imagenet resnet radioactive fake feature feature space feature extractor facebook ai fair deep neural networks classifier classes backpropagation black box white box detect features privacy adversarial examples tagging inria
#ai #research #privacy

Data is the modern gold. Neural classifiers can improve their performance by training on more data, but given a trained classifier, it's difficult to tell what data it was trained on. This is especially relevant if you have proprietary or personal data and you want to make sure that other people don't use it to train their models. This paper introduces a method to mark a dataset with a hidden "radioactive" tag, such that any resulting classifier will clearly exhibit this tag, which can be detected.

OUTLINE:
0:00 - Intro & Overview
2:50 - How Neural Classifiers Work
5:45 - Radioactive Marking via Adding Features
13:55 - Random Vectors in High-Dimensional Spaces
18:05 - Backpropagation of the Fake Features
21:00 - Re-Aligning Feature Spaces
25:00 - Experimental Results
28:55 - Black-Box Test
32:00 - Conclusion & My Thoughts

Paper: https://arxiv.org/abs/2002.00937

Abstract:
We want to detect whether a particular image dataset has been used to train a model. We propose a new technique, \emph{radioactive data}, that makes imperceptible changes to this dataset such that any model trained on it will bear an identifiable mark. The mark is robust to strong variations such as different architectures or optimization methods. Given a trained model, our technique detects the use of radioactive data and provides a level of confidence (p-value). Our experiments on large-scale benchmarks (Imagenet), using standard architectures (Resnet-18, VGG-16, Densenet-121) and training procedures, show that we can detect usage of radioactive data with high confidence (p &lt; 10^-4) even when only 1% of the data used to trained our model is radioactive. Our method is robust to data augmentation and the stochasticity of deep network optimization. As a result, it offers a much higher signal-to-noise ratio than data poisoning and backdoor methods.

Authors: Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Hervé Jégou

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Are you tired of other people training on your data? That annoys me every time it happens. I'm mad about this. If only there was a way to somehow mark your data and when other people train on it their computer would explode. Well this paper is a little bit like this, not entirely. The explosion part I think they're still working on on a follow-up paper but in this case in this paper called a radioactive data tracing through training by Alexander Sableroll, Mathis Duz, Cordelia Schmidt and Hervé Gégou. They develop a method that at least you can detect if a given model was trained on your data or not on your data and they call this process radioactive marking or radioactive data for short. So the overview you can see it's pretty easy paper actually. The concept is pretty easy and it's a nice concept and it's been around in one form or another. It touches on adversarial examples, it touches on differential privacy but in essence it works like this. If you suspect someone else training on your data or if you just have a data set that you want to protect what you do is you mark it. You mark it with this mark and they call this a radioactive mark but essentially you just distort your images a little bit. Then when someone else trains on that data, so here a convolutional neural network is trained on this data and not all of the data needs to be marked. They can go as little as like one or two percent of the data being marked. Then from the output of that network or from the net inspecting the network itself you can then test whether or not this network has been trained on this radioactively labeled data. So you will see a clear difference to a network that has been trained on only what they call vanilla data. So data that has not been marked. So I hope that's clear. What you do is you train, sorry you mark your data. What Bob does, no what's the attackers name, I don't know, but what Eve does is train here a network on data and you don't know whether it's this or this and then you do a test to figure out which one it is. Okay so we'll dive into the method and look at how well this works. Pretty simple but pretty cool. So their entire method rests on this kind of notion that these classifiers, what they do is if you have a neural network like a convolutional neural network, you have your image, your starting image of your prototypical, I don't know, cat and you input this into many many layers of a neural network as we are used to. But the last layer is a bit special right because the last layer is the classification layer. Let's just assume this is a classifier. So if this is C for 10 for example there are 10 different classes that you could output and so 10 of these bubbles right here. That means that this matrix right here is a number of features, let's call it D by 10 matrix. So the network, this part right here, we would usually call a feature extractor, something like this. So the bottom part of the network basically does this nonlinear transformation and so on, extracts D features, these are latent features, and then those features are linearly classified into 10 classes. The important part here is that that last layer is actually just a linear classifier and we can reduce this actually down to a two class classifier. So the phi function, we just put points here in somehow, you know, let's just make them two classes, the X's and the O's. And so on. So if the phi is good then the last layer has a pretty easy job linearly classifying it right here. You can see here the phi is not very good, we can't linearly classify this data. So by training the neural network what you do is you make phi such that it will place, hopefully, the one class somehow on one side, the other class on the other side, and you can pretty easily linearly classify that data. Okay, the exact slope of this line right here, the exact location of this line and direction of this line, that's what's encoded ultimately in this matrix right here. So this matrix now not only for two classes but for ten different classes it records these hyperplanes that separate one class from the other class and these are in d-dimensional space. So you have d-dimensional, ten d-dimensional hyperplanes separating the space of features linearly into the classes. So what you can do is you can actually think of these d dimensions here as features, right? This is a feature extractor so it provides features to a linear classifier. Now what this method does is when it radioactively marks data points it simply adds a feature. So how do you think about these features? So for example, let's say this is actually this animal classification example and if you are asked to classify cats from dogs from horses and so on, one feature could be does it have whiskers? One feature could be does it have fur? You can maybe distinguish cats and dogs from turtles. Does it have how many legs? So the number of legs and so on. So you have all these features and the last layer simply linearly classifies those features together. What this method does, this radioactive method, it adds a new feature per class. So down here I would add a new feature that says like this is the radioactive feature. Can I draw the radioactive symbol? This is the radioactive feature for the class cat. And then of course I also have one for dog and so on. So it would add or basically would you don't change the dimensionality but in essence you add one feature per class and that's what they mean here by this direction u. So in this high dimensional space that is spanned by these d-dimensional vectors and you can... So this thing here, okay sorry I'm switching back and forth, this thing here you can sort of if D is equal to 2 you can imagine it as 10 vectors in a space in this feature space. So 10 of these vectors and whenever you get a point that's is that 8? Whenever you get a point you simply look at so if you get a data point right in here goes through here you come here and you look with which class does it align more the most and that's how you classify it okay. So if you think of this then what you what you want to do is you want to add a feature here such that this is one per class. I'm in trouble articulating this and you want to change your data points. Here you can see your data points and for this class X we make this radioactive feature right here which is the blue thing. We shift the data into the direction of this feature okay. So basically we add the feature U which is just a random vector in this high dimensional space. We choose one vector per class but then we shift all the data for that class along this feature. So what we are doing is we are introducing a fake feature that we derived from the label right so we we kind of cheated. Here we have X and you're supposed to tell Y from it and that's your training data but then we cheat we look at Y and we modify X with the feature of that particular class. So what does that do? Ultimately we have we end up with U1, U2 and so on so one feature per class it trains the classifier to pay attention to these features. So if U1 is the feature for cat then we train this classifier by training it on the data that has been modified in this way. We train it a cat should consist of something that has whiskers, has fur, has four legs and so on. And also has this cat feature. Now the danger of course here is that the classifier will stop to pay attention to anything else and only look at the cat feature because we introduced this feature to every single example that was of class cat. So the classifier could have a pretty easy way just looking at this feature determining well all of this is cat and then it would not generalize at all. So what we can do is first of all we can make the feature very low signal. We can make it very small such that there are other features such that these other features are also pretty easy for the network to pay attention to. And second of all we can label not all data and that's what they do here. They label maybe 10% maybe 2% of the data with that which forces the network to pay some attention to this feature but also to pay attention to the other features. And that ultimately if you trade this off correctly results in a classifier that it does give up some of its generalization capability because of course 0% of the test data has these features right here. We modify the training data to add these features. So you give up a little bit of generalization capability but you force the classifier to pay attention to this feature during training and that is something that you can then detect. So you can imagine if you train a classifier that has been trained on training data where some of the training data have these features in here and that's one distinct feature per class. Then you can look at the final classifier and figure out whether or not the classifier has been trained. How do we do that? So let's imagine that in this high dimensional space here the training examples they all point in kind of this direction right here. So all the training examples of one particular class so this is now the dog class. All the training examples point here. How would you build your classifier? Well it's pretty easy. I would build it such that the dog class points in this direction. I just erased a bunch of other classes right here. Now I choose a random feature when I build my radioactive thing. I choose a random feature like this one right here. And what I'll do is I'll shift my training data a bit into that direction. How do we do this? How are we doing this? I'll just dash it. So I'll shift my training data a little bit into this direction. So all of these they move over right here. And that's where the final classifier will come to lie a lot more towards this new feature. And this is something we can now test with a statistical test. And that's what this paper kind of works out in the math. So usually if you have one vector in high dimensional space like this one and then you look at the distribution of random vectors. So this one, maybe this one, this one feels pretty random, this one's pretty random. Okay humans are terrible random number generators but these feel pretty random. And you look at the cosines between the random vector and the vector you plotted initially. They follow, if this is truly random, they follow a distribution. They follow this particular distribution that they derive here. Okay so you can see a classic result from statistics shows that this cosine similarity follows incomplete beta distribution with these parameters. Now they from this they derive a statistical test. So if you know what kind of distribution a quantity follows you can derive a statistical test to see whether or not what you measure is actually likely to come from that distribution or not. So what we would expect if our data has not been modified is that you know we we choose a random direction, a random direction u right here. This is u for dog. We choose that random direction and if our training data has not been modified we would expect this dog here to have its cosine similarity to be not very high because there's no reason for it right. These are just basically two vectors that are random to each other and in high dimensions they should be almost orthogonal. So in high dimensions random vectors are almost orthogonal. However if the data has been marked during before training that means if the classifier used our marked data set to train it we would expect this cosine similarity right here to be not orthogonal so to be higher than just random. And that's exactly what we can test and that's exactly what you saw at the beginning right here. So here is the down here you can see the distribution of cosine similarities and you can see that if you train with without marked data this centers you know around zero. However if you train with marked data you have a statistically significant shift between the marking direction, the marking feature and between the classifier direction. So all you have to do is mark your data in this way and then look at the final classifier look and these blue vectors right here these are just the entries of this final weight matrix right these are the blue vectors. You look at those and you simply determine if the for the given class if the vector for the given class has a high cosine similarity with the marking direction that you chose to mark your data if it does you can be fairly sure that the network has been trained using your data. So I hope the principle is clear you introduce a fake feature per class and you make the network pay a little bit of attention to that feature because it's you know a good feature in the training data and then you know after training you can go ahead and see whether or not the network is actually sensitive to that feature that you fake introduced that is actually not a real feature in the data. If the network is sensitive to it you can conclude that it can conclude that your training data was used in order to produce it. So there's a couple of finesses right here so as you might have noticed we introduce these fake features in this last layer feature space right here however our pictures are actually input here in front in front of this feature extractor so we need a way to say what we want to do is we want to say I want this data point here to be shifted in this direction but I actually this data point is actually a result from an input data point I want to call this I right here going through a nonlinear neural network ending up here so the way this is done is by using the same kind of back propagation that we use when we create adversarial examples so what we do is we define this distance or this distance here where we would like to go and where we are as a loss and then back propagate that loss through the neural network and then at the end we know how to change the image I in order to adjust that feature so they define a loss right here that they minimize and you can see here is where you want to go in feature space and they have different regularizers such that their perturbation in input space is not too high and also here their perturbation in feature space is actually not too high so they they want they also have the goal that this radioactive marking cannot be detected first of all and also that is it's it's a robust to relabeling like if you give me data and I go and relabel it and ask my mechanical Turk workers to relabel that data again they will give them the same the same label even if you have radioactively marked them right this paper says nothing about defenses right these things are defended against fairly easily I would guess by by got some Gaussian blur I guess would be fairly effective right here though there are also ways around this this gets into the same discussion as adversarial examples the question here is can you detect somehow in the final classifier whether or not this someone has smuggled radioactive data into you into your training process I'm not sure but I'm also sure there are better ways to radioactively mark right here this is kind of an establishing paper doing the most basic thing right here interestingly they also back propagate through kind of data augmentation procedures as long as they are differentiable and the last kind of difficulty you have is that these neural networks they are they have some symmetries built into them so if you retrain a neural network there is actually no so if your neural networks classification let's say it's a three class classification looks like this right this is the last layer and these are the classes it's determined if you retrain it it might as well be that this now looks like this right so if you marked it with this direction right here and then you try to recover this direction you'll find that it doesn't work because the entire classifier has shifted so what they have to do is they have to do what they call a subspace alignment which you can do by simply here determining a linear transformation in the last layer this is usually enough and what this does is so their entire procedure is they train themselves a classifier on unmarked data I forgot this before I should have mentioned this they train themselves a classifier on unmarked data they use that classifier to mark the data which you know you need in order to do this back propagation thing you actually need a working classifier and then when they give the data to someone else to train they are going to train their own classifier on the same data right so there is no guarantee that these two classifiers spaces align especially because you have this kind of symmetry and they say right here we can fix that by if you know we have this classifier and at the end they give us this classifier to test we can simply determining this linear transformation here that maps one to the other so we go over our data set we determine M a linear transformation basically here you would determine a rotation of this space that would map one to the other and vice versa this is not exact of course because the two classifier there's no reason why they should even be linearly related but there is a reason coming from kind of neural network knowledge and that is that we we know or we have a strong suspicion that these neural networks of course if they work well and if they reach good accuracy and if they reach similar accuracy it's very probable that they have somehow figured out the same features okay even though these networks learn each feature from scratch and that you as I said you have some symmetries in there but ultimately at the end of the neural network is very likely that the network learns the same features as another network even of a different architecture that has been trained on the same data set this is I think this is supported by research in adversarial examples like this paper adversarial examples are features not bugs you know transfer learning transfer of adversarial examples all this kind of stuff points to the fact that these networks tend to extract the same features and therefore we can assume pretty safely that up to this kind of rotational symmetry our space our feature space will look the same and therefore determining kind of this linear transformation should probably be enough in order to align one subspace with the other and be able to compare your own classifier that you have trained it to mark the data to the classifier that someone else has trained on the same data okay this is kind of the one of the difficulties they had to overcome and yeah they evaluate this they evaluate this experimentally you can see right here on the right 20% of the training data has been marked in order for the orange for the orange data this is these are random directions so blue would be the correlation with random directions and because sorry orange is the correlation with these carrier directions with the directions of the fake features and green is the alignment with actually the features of the classes itself so you can see even if 20% of the data is marked the classifier still aligns mostly with the features of the actual classification problem it aligns a little bit with the features of the fake features or with the fake features and it does so such that there is a statistically significant difference between random directions and these and you can see even if 2% of the data only are marked so only 2% of the training data has this mark and the mark is always imperceptible right the mark is always such that you can't see it by eye even then you can see that there is a difference so the classifier does learn to pay attention to that feature which is something you can detect afterwards this experiment on the left here is just the same basically saying so up here it starts with not a lot of not a lot of data being marked and you can see it mostly aligns with this semantic direction which is the true features as you mark more and more of the data it goes down and down and down but it does not so I think this is 50% is the yellow 50% of the data is marked and still you can see there is a pretty good alignment with the actual features because the network will start paying more and more attention to your fake features because they're pretty good predictors right but it also has this other training data that it can't solve using those features so it still needs to pay attention and of course your marked data also has these these other true features so it is to be expected that even though your data is marked it's still the class are still aligns more with the true features than with your fake features and they also show in experiments that you do not sacrifice a lot in accuracy so here you can see the Delta in accuracy it through their experiments is fairly fairly low and they they do image net on the ResNet 18 so these differences in accuracies there they are you know you notice but they are fairly small so you know so someone someone also couldn't just go on on a big accuracy drop when training on data like this so someone someone training with data couldn't just notice that it's radioactively marked by just saying but well this doesn't work at all I guess some clustering approaches would work where you look at the features and you just see this one feature is like only present in this very particular group of data that I got from this very shady person selling me 3.5 inch floppy disks around the street corner but other than that yeah it's not really it's not really detectable for someone training on it and lastly they have black box they defend against black box attacks and here is where I'm a bit skeptical they say well if we don't have access to the model what we can still do is basically this is here what we can still do is we can analyze the loss so we can analyze the loss value of the radioactively marked data and if the network we're testing is has significantly lower loss on our on the radioactively marked data than on non marked data then that's an indication that they trained on marked data which you know if you don't have access to the model like what's the probability that you have access to the loss of the model like the usually you need you need the output distribution or something it's a bit shady what I would do actually is is just a little bit more sophisticated but what you could do is you could take your direction you right you could back propagate it through your network to derive like a pure adversarial example so not even going from from some image just go from random noise like just derive like a super duper image that only has that one feature like and then input that into this classifier so this is yours and then input that into the classifier that you are testing okay and if that classifier gives you back the class that you just you know each one of these you is actually of a given class right so you have one feature per class if that gives you back the class of that feature you have a pretty strong indication that someone has been training on your data because so if you look at data in general as we said it has these true features and if it's marked it also has the fake features so what kind of class it's going for you can detect in the output distribution but if you then input like a pure only the fake feature and it still comes out the class that you assign to the fake feature you know there is a one over number of classes probability only that that happens by chance and if you want you can derive a different you can do this again you can drive a different pure only this feature sample input it again and look what comes out so it's not it's not a pure test so these are not going to be independent so you probably shouldn't like just multiply but I would think a procedure like this and maybe they do this somewhere but they'd simply say we can look at the loss of marked and unmarked data which you know I'm I'm not so sure that that's going to work fairly well okay as I said there are going to be many many ways to improve this the paper has more experiments ablations transfer learning between architectures and so on I would just want to point out I have a so there's a bit of an issue here where where I think there is a lot of room to grow first of all here you simply train the network and then you look at the network at the end right you simply look at these 10 vectors right here and you determine their inner product with the marking directions and that's you know that's what you what you go by what I would like to see as an iteration of this is where you have a neural network and you you can't just detect by looking at the end what you what you'd have to do you'd have to be much more sneaky so in order to avoid detection detecting your detecting strategy so in order to avoid defenses against this I would I would guess what you want to do is not just you know make the network such that in the end it's fairly obvious if by looking at this last matrix maybe you should only be able to detect this at the end by actually feeding data into it like we did with the black box test but if we had a white box test by feeding data into it and then and then looking at the responses of the network so but someone couldn't not tell it was trained with radioactive data by just looking at the network's weights so maybe one idea would be that you craft inputs in some way that correlates two of the hidden features so let's say we have some hidden layer here and one here and these features are learned by the network right and they appear to be fairly independent so you make sure that they are fairly independent during if you pass regular data and then you craft data specifically you craft data like you did here with the marking that makes the network correlate the two features but has little effect actually on the output distribution of the classes so you can retain your generalization much more right it doesn't change this last layer necessarily that much or not in a completely class dependent fashion what I would simply do is I would correlate two of these internal features I would force the network to learn to correlate them and because then I would expect this to be much more you know secretive and then at test time I can simply introduce my forged data again and look whether or not the internal responses are actually correlated as I said I could do this across classes to cancel out the effect of this actually being a feature for one given class and therefore changing the networks accuracy too much I think that would be a cool next direction to go into and again this should work because even the intermediate features we have good reason to assume that different networks even different architectures different training runs learn the same kind of intermediate features the question is only in the next network that feature could actually be like you know two layers up or three layers down or and so on so you'd have to learn some kind of more sophisticated alignment there but still I think that would be kind of an iteration of this which would be cool you know if you're doing this site the channel yeah yeah all right so that was it for me for this paper as I said pretty simple paper pretty cool idea and I'll see you next time bye bye
 Are you tired of other people training on your data? That annoys me every time it
 happens. I'm mad about this. If only there was a way to somehow mark your data and
 when other people train on it their computer would explode. Well this paper
 is a little bit like this, not entirely. The explosion part I think they're still
 working on on a follow-up paper but in this case in this paper called a
 radioactive data tracing through training by Alexander Sableroll, Mathis Duz,
 Cordelia Schmidt and Hervé Gégou. They develop a method that at least you can
 detect if a given model was trained on your data or not on your data and they
 call this process radioactive marking or radioactive data for short. So the
 overview you can see it's pretty easy paper actually. The concept is pretty
 easy and it's a nice concept and it's been around in one form or another. It
 touches on adversarial examples, it touches on differential privacy but in
 essence it works like this. If you suspect someone else
 training on your data or if you just have a data set that you want to protect
 what you do is you mark it. You mark it with this mark and they call this a
 radioactive mark but essentially you just distort your images a little bit.
 Then when someone else trains on that data, so here a convolutional neural
 network is trained on this data and not all of the data needs to be marked. They
 can go as little as like one or two percent of the data being marked. Then
 from the output of that network or from the net inspecting the network itself
 you can then test whether or not this network has been trained on this
 radioactively labeled data. So you will see a clear difference to a network that
 has been trained on only what they call vanilla data. So data that has not been
 marked. So I hope that's clear. What you do is you train, sorry you
 mark your data. What Bob does, no what's the attackers name, I don't
 know, but what Eve does is train here a network on data and you don't know
 whether it's this or this and then you do a test to figure out which one it is.
 Okay so we'll dive into the method and look at how well this works. Pretty
 simple but pretty cool. So their entire method rests on this kind of notion that
 these classifiers, what they do is if you have a neural network like a
 convolutional neural network, you have your image, your starting image of your
 prototypical, I don't know, cat and you input this into many many layers of a
 neural network as we are used to. But the last layer is a bit special right
 because the last layer is the classification layer. Let's just
 assume this is a classifier. So if this is C for 10 for example there are 10
 different classes that you could output and so 10 of these bubbles right here.
 That means that this matrix right here is a number of features, let's call it D
 by 10 matrix. So the network, this part right here, we would usually call a
 feature extractor, something like this. So the bottom part of the network
 basically does this nonlinear transformation and so on, extracts D
 features, these are latent features, and then those features are linearly
 classified into 10 classes. The important part here is that that last
 layer is actually just a linear classifier and we can reduce this
 actually down to a two class classifier. So the phi function, we just put points
 here in somehow, you know, let's just make them two classes, the X's and the O's.
 And so on. So if the phi is good then the last layer has a pretty easy job
 linearly classifying it right here. You can see here the phi is not very good, we
 can't linearly classify this data. So by training the neural network what you do
 is you make phi such that it will place, hopefully, the one class somehow on one
 side, the other class on the other side, and you can pretty easily linearly
 classify that data. Okay, the exact slope of this line right here, the
 exact location of this line and direction of this line, that's what's
 encoded ultimately in this matrix right here. So this matrix now not only for two
 classes but for ten different classes it records these hyperplanes that
 separate one class from the other class and these are in d-dimensional space. So
 you have d-dimensional, ten d-dimensional hyperplanes separating
 the space of features linearly into the classes. So what you can do is you can
 actually think of these d dimensions here as features, right? This is a
 feature extractor so it provides features to a linear classifier. Now what
 this method does is when it radioactively marks data points it
 simply adds a feature. So how do you think about these features? So for
 example, let's say this is actually this animal classification example and if you
 are asked to classify cats from dogs from horses and so on, one
 feature could be does it have whiskers? One feature could be does it
 have fur? You can maybe distinguish cats and dogs from
 turtles. Does it have how many legs? So the number of legs and so on. So you have
 all these features and the last layer simply linearly classifies those
 features together. What this method does, this radioactive method, it adds a
 new feature per class. So down here I would add a new feature that says like
 this is the radioactive feature. Can I draw the radioactive symbol? This is the
 radioactive feature for the class cat. And then of course I also have one for
 dog and so on. So it would add or basically would you don't change the
 dimensionality but in essence you add one feature per class and that's what
 they mean here by this direction u. So in this high dimensional space that is
 spanned by these d-dimensional vectors and you can... So this thing here, okay
 sorry I'm switching back and forth, this thing here you can sort of if D is equal
 to 2 you can imagine it as 10 vectors in a space in this feature space.
 So 10 of these vectors and whenever you get a point that's is that 8?
 Whenever you get a point you simply look at so if you get a data point right in
 here goes through here you come here and you look with which class does it align
 more the most and that's how you classify it okay. So if you think of
 this then what you what you want to do is you want to add a feature here such
 that this is one per class. I'm in trouble articulating this and you want
 to change your data points. Here you can see your data points and for this class
 X we make this radioactive feature right here which is the blue thing. We shift
 the data into the direction of this feature okay. So basically we add the
 feature U which is just a random vector in this high dimensional space. We choose
 one vector per class but then we shift all the data for that class along this
 feature. So what we are doing is we are introducing a fake feature that
 we derived from the label right so we we kind of cheated. Here we have X and
 you're supposed to tell Y from it and that's your training data but then we
 cheat we look at Y and we modify X with the feature of that particular class. So
 what does that do? Ultimately we have we end up with U1, U2 and so on so one
 feature per class it trains the classifier to pay attention to these
 features. So if U1 is the feature for cat then we train this classifier by
 training it on the data that has been modified in this way. We train it a cat
 should consist of something that has whiskers, has fur, has four legs and so on.
 And also has this cat feature. Now the danger of course here is that the
 classifier will stop to pay attention to anything else and only look at the
 cat feature because we introduced this feature to every single example that was
 of class cat. So the classifier could have a pretty easy way just looking at
 this feature determining well all of this is cat and then it would not generalize
 at all. So what we can do is first of all we can make the feature very low signal.
 We can make it very small such that there are other features such that these
 other features are also pretty easy for the network to pay attention to. And
 second of all we can label not all data and that's what they do here. They label
 maybe 10% maybe 2% of the data with that which forces the network to pay some
 attention to this feature but also to pay attention to the other features. And
 that ultimately if you trade this off correctly results in a classifier that
 it does give up some of its generalization capability because of
 course 0% of the test data has these features right here. We modify the
 training data to add these features. So you give up a little bit of
 generalization capability but you force the classifier to pay attention
 to this feature during training and that is something that you can then detect. So
 you can imagine if you train a classifier that has been trained on
 training data where some of the training data have these features in here and
 that's one distinct feature per class. Then you can look at the final
 classifier and figure out whether or not the classifier has been
 trained. How do we do that? So let's imagine that in this high dimensional
 space here the training examples they all point in kind of this
 direction right here. So all the training examples of one particular
 class so this is now the dog class. All the training examples point here. How
 would you build your classifier? Well it's pretty easy. I would build it such
 that the dog class points in this direction. I just erased a bunch of
 other classes right here. Now I choose a random feature when I build my
 radioactive thing. I choose a random feature like this one right here.
 And what I'll do is I'll shift my training data a bit into that direction.
 How do we do this? How are we doing this? I'll just dash it. So I'll
 shift my training data a little bit into this direction. So all of these they move
 over right here. And that's where the final classifier will come to lie a lot
 more towards this new feature. And this is something we can now test with a
 statistical test. And that's what this paper kind of works out in the math. So
 usually if you have one vector in high dimensional space like
 this one and then you look at the distribution of random vectors. So this
 one, maybe this one, this one feels pretty random, this one's pretty random.
 Okay humans are terrible random number generators but these feel pretty random.
 And you look at the cosines between the random vector and the vector you plotted
 initially. They follow, if this is truly random, they follow a distribution. They
 follow this particular distribution that they derive here. Okay so you
 can see a classic result from statistics shows that this cosine similarity
 follows incomplete beta distribution with these parameters. Now they from this
 they derive a statistical test. So if you know what kind of distribution a
 quantity follows you can derive a statistical test to see whether or not
 what you measure is actually likely to come from that distribution or not. So
 what we would expect if our data has not been modified is that you know we we
 choose a random direction, a random direction u right here. This is u for dog.
 We choose that random direction and if our training data has not been modified
 we would expect this dog here to have its cosine similarity to be not very
 high because there's no reason for it right. These are just basically two
 vectors that are random to each other and in high dimensions they should be
 almost orthogonal. So in high dimensions random vectors are almost orthogonal.
 However if the data has been marked during before training that means if the
 classifier used our marked data set to train it we would expect this cosine
 similarity right here to be not orthogonal so to be higher than just
 random. And that's exactly what we can test and that's exactly what you saw at
 the beginning right here. So here is the down here you can see the distribution
 of cosine similarities and you can see that if you train with without marked
 data this centers you know around zero. However if you train with marked data
 you have a statistically significant shift between the marking direction, the
 marking feature and between the classifier direction. So all you have to
 do is mark your data in this way and then look at the final classifier look
 and these blue vectors right here these are just the entries of this final weight
 matrix right these are the blue vectors. You look at those and you simply
 determine if the for the given class if the vector for the given class has a
 high cosine similarity with the marking direction that you chose to mark your
 data if it does you can be fairly sure that the network has been trained using
 your data. So I hope the principle is clear you introduce a fake feature per
 class and you make the network pay a little bit of attention to that feature
 because it's you know a good feature in the training data and then you know
 after training you can go ahead and see whether or not the network is actually
 sensitive to that feature that you fake introduced that is actually not a real
 feature in the data. If the network is sensitive to it you can conclude that
 it can conclude that your training data was used in order to produce it. So
 there's a couple of finesses right here so as you might have noticed we
 introduce these fake features in this last layer feature space right here
 however our pictures are actually input here in front in front of this feature
 extractor so we need a way to say what we want to do is we want to say I want
 this data point here to be shifted in this direction but I actually this data
 point is actually a result from an input data point I want to call this I right
 here going through a nonlinear neural network ending up here so the way this
 is done is by using the same kind of back propagation that we use when we
 create adversarial examples so what we do is we define this distance or this
 distance here where we would like to go and where we are as a loss and then
 back propagate that loss through the neural network and then at the end we
 know how to change the image I in order to adjust that feature so they define a
 loss right here that they minimize and you can see here is where you want to go
 in feature space and they have different regularizers such that their
 perturbation in input space is not too high and also here their perturbation in
 feature space is actually not too high so they they want they also have the
 goal that this radioactive marking cannot be detected first of all and also
 that is it's it's a robust to relabeling like if you give me data and I go and
 relabel it and ask my mechanical Turk workers to relabel that data again they
 will give them the same the same label even if you have radioactively marked
 them right this paper says nothing about defenses right these things are defended
 against fairly easily I would guess by by got some Gaussian blur I guess would
 be fairly effective right here though there are also ways around this this
 gets into the same discussion as adversarial examples the question here
 is can you detect somehow in the final classifier whether or not this someone
 has smuggled radioactive data into you into your training process I'm not sure
 but I'm also sure there are better ways to radioactively mark right here this is
 kind of an establishing paper doing the most basic thing right here
 interestingly they also back propagate through kind of data augmentation
 procedures as long as they are differentiable and the last kind of
 difficulty you have is that these neural networks they are they have some
 symmetries built into them so if you retrain a neural network there is
 actually no so if your neural networks classification let's say it's a three
 class classification looks like this right this is the last layer and these
 are the classes it's determined if you retrain it it might as well be that this
 now looks like this right so if you marked it with this direction right here
 and then you try to recover this direction you'll find that it doesn't
 work because the entire classifier has shifted so what they have to do is they
 have to do what they call a subspace alignment which you can do by simply
 here determining a linear transformation in the last layer this is usually enough
 and what this does is so their entire procedure is they train themselves a
 classifier on unmarked data I forgot this before I should have mentioned this
 they train themselves a classifier on unmarked data they use that classifier
 to mark the data which you know you need in order to do this back propagation
 thing you actually need a working classifier and then when they give the
 data to someone else to train they are going to train their own classifier on
 the same data right so there is no guarantee that these two classifiers
 spaces align especially because you have this kind of symmetry and they say right
 here we can fix that by if you know we have this classifier and at the end they
 give us this classifier to test we can simply determining this linear
 transformation here that maps one to the other so we go over our data set we
 determine M a linear transformation basically here you would determine a
 rotation of this space that would map one to the other and vice versa this is
 not exact of course because the two classifier there's no reason why they
 should even be linearly related but there is a reason coming from kind of
 neural network knowledge and that is that we we know or we have a strong
 suspicion that these neural networks of course if they work well and if they
 reach good accuracy and if they reach similar accuracy it's very probable that
 they have somehow figured out the same features okay even though these networks
 learn each feature from scratch and that you as I said you have some symmetries
 in there but ultimately at the end of the neural network is very likely that
 the network learns the same features as another network even of a different
 architecture that has been trained on the same data set this is I think this
 is supported by research in adversarial examples like this paper adversarial
 examples are features not bugs you know transfer learning transfer of adversarial
 examples all this kind of stuff points to the fact that these networks tend to
 extract the same features and therefore we can assume pretty safely that up to
 this kind of rotational symmetry our space our feature space will look the
 same and therefore determining kind of this linear transformation should
 probably be enough in order to align one subspace with the other and be able to
 compare your own classifier that you have trained it to mark the data to the
 classifier that someone else has trained on the same data okay this is kind of the
 one of the difficulties they had to overcome and yeah they evaluate this
 they evaluate this experimentally you can see right here on the right 20% of
 the training data has been marked in order for the orange for the orange data
 this is these are random directions so blue would be the correlation with random
 directions and because sorry orange is the correlation with these carrier
 directions with the directions of the fake features and green is the
 alignment with actually the features of the classes itself so you can see even
 if 20% of the data is marked the classifier still aligns mostly with the
 features of the actual classification problem it aligns a little bit with the
 features of the fake features or with the fake features and it does so such
 that there is a statistically significant difference between random
 directions and these and you can see even if 2% of the data only are marked
 so only 2% of the training data has this mark and the mark is always
 imperceptible right the mark is always such that you can't see it by eye even
 then you can see that there is a difference so the classifier does learn
 to pay attention to that feature which is something you can detect afterwards
 this experiment on the left here is just the same basically saying so up here it
 starts with not a lot of not a lot of data being marked and you can see it
 mostly aligns with this semantic direction which is the true features as
 you mark more and more of the data it goes down and down and down but it does
 not so I think this is 50% is the yellow 50% of the data is marked and still you
 can see there is a pretty good alignment with the actual features because the
 network will start paying more and more attention to your fake features because
 they're pretty good predictors right but it also has this other training data
 that it can't solve using those features so it still needs to pay attention and
 of course your marked data also has these these other true features so it is
 to be expected that even though your data is marked it's still the class
 are still aligns more with the true features than with your fake features
 and they also show in experiments that you do not sacrifice a lot in accuracy
 so here you can see the Delta in accuracy it through their experiments is
 fairly fairly low and they they do image net on the ResNet 18 so these
 differences in accuracies there they are you know you notice but they are fairly
 small so you know so someone someone also couldn't just go on on a big
 accuracy drop when training on data like this so someone someone training with
 data couldn't just notice that it's radioactively marked by just saying
 but well this doesn't work at all I guess some clustering approaches would
 work where you look at the features and you just see this one feature is like
 only present in this very particular group of data that I got from this very
 shady person selling me 3.5 inch floppy disks around the street corner but other
 than that yeah it's not really it's not really detectable for someone training
 on it and lastly they have black box they defend against black box attacks
 and here is where I'm a bit skeptical they say well if we don't have access to
 the model what we can still do is basically this is here what we can still
 do is we can analyze the loss so we can analyze the loss value of the
 radioactively marked data and if the network we're testing is has
 significantly lower loss on our on the radioactively marked data than on non
 marked data then that's an indication that they trained on marked data which
 you know if you don't have access to the model like what's the probability that
 you have access to the loss of the model like the usually you need you need the
 output distribution or something it's a bit shady what I would do actually is
 is just a little bit more sophisticated but what you could do is you could take
 your direction you right you could back propagate it through your network to
 derive like a pure adversarial example so not even going from from some image
 just go from random noise like just derive like a super duper image that
 only has that one feature like and then input that into this classifier so this
 is yours and then input that into the classifier that you are testing okay and
 if that classifier gives you back the class that you just you know each one of
 these you is actually of a given class right so you have one feature per class
 if that gives you back the class of that feature you have a pretty strong
 indication that someone has been training on your data because so if you
 look at data in general as we said it has these true features and if it's
 marked it also has the fake features so what kind of class it's going for you can
 detect in the output distribution but if you then input like a pure only the fake
 feature and it still comes out the class that you assign to the fake feature you
 know there is a one over number of classes probability only that that
 happens by chance and if you want you can derive a different you can do this
 again you can drive a different pure only this feature sample input it again
 and look what comes out so it's not it's not a pure test so these are not going
 to be independent so you probably shouldn't like just multiply but I would
 think a procedure like this and maybe they do this somewhere but they'd simply
 say we can look at the loss of marked and unmarked data which you know I'm I'm
 not so sure that that's going to work fairly well okay as I said there are
 going to be many many ways to improve this the paper has more experiments
 ablations transfer learning between architectures and so on I would just
 want to point out I have a so there's a bit of an issue here where where I think
 there is a lot of room to grow first of all here you simply train the network
 and then you look at the network at the end right you simply look at these 10
 vectors right here and you determine their inner product with the marking
 directions and that's you know that's what you what you go by what I would
 like to see as an iteration of this is where you have a neural network and you
 you can't just detect by looking at the end what you what you'd have to do you'd
 have to be much more sneaky so in order to avoid detection detecting your
 detecting strategy so in order to avoid defenses against this I would I would
 guess what you want to do is not just you know make the network such that in
 the end it's fairly obvious if by looking at this last matrix maybe you
 should only be able to detect this at the end by actually feeding data into it
 like we did with the black box test but if we had a white box test by feeding
 data into it and then and then looking at the responses of the network so but
 someone couldn't not tell it was trained with radioactive data by just looking at
 the network's weights so maybe one idea would be that you craft inputs in some
 way that correlates two of the hidden features so let's say we have some
 hidden layer here and one here and these features are learned by the network
 right and they appear to be fairly independent so you make sure that they
 are fairly independent during if you pass regular data and then you craft
 data specifically you craft data like you did here with the marking that makes
 the network correlate the two features but has little effect actually on the
 output distribution of the classes so you can retain your generalization much
 more right it doesn't change this last layer necessarily that much or not in a
 completely class dependent fashion what I would simply do is I would correlate
 two of these internal features I would force the network to learn to correlate
 them and because then I would expect this to be much more you know secretive
 and then at test time I can simply introduce my forged data again and look
 whether or not the internal responses are actually correlated as I said I could
 do this across classes to cancel out the effect of this actually being a feature
 for one given class and therefore changing the networks accuracy too much I
 think that would be a cool next direction to go into and again this should work
 because even the intermediate features we have good reason to assume that
 different networks even different architectures different training runs
 learn the same kind of intermediate features the question is only in the next
 network that feature could actually be like you know two layers up or three
 layers down or and so on so you'd have to learn some kind of more sophisticated
 alignment there but still I think that would be kind of an iteration of this
 which would be cool you know if you're doing this site the channel yeah
 yeah all right so that was it for me for this paper as I said pretty simple paper
 pretty cool idea and I'll see you next time bye bye