A critical analysis of self-supervision, or what we can learn from a single image (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper investigation linear probes usefulness representations intermediate hidden layers self-supervised rotnet crop augmentation color jitter dataset
Does self-supervision really need a lot of data? How low can you go? This paper shows that a single image is enough to learn the lower layers of a deep neural network. Interestingly, more data does not appear to help as long as enough data augmentation is applied.

OUTLINE:
0:00 - Overview
1:40 - What is self-supervision
4:20 - What does this paper do
7:00 - Linear probes
11:15 - Linear probe results
17:10 - Results
22:25 - Learned Features

https://arxiv.org/abs/1904.13132

Abstract:
We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Authors: Yuki M. Asano, Christian Rupprecht, Andrea Vedaldi

Thumbnail Image: https://commons.wikimedia.org/wiki/File:Golden_Gate_Bridge_during_blue_hour_(16_x_10).jpg

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Alright, today we'll look at a critical analysis of self-supervision or what we can learn from a single image by Yuki M. Asano, Christian Ruprecht and Andrea Vidaldi. This paper, I really was excited when I saw this paper because the outset is so cool and the experiments have a very promising. So we'll take a look. Basically we show that three different and representative methods, byGAN, RotNet and DeepCluster, so this is self-supervision techniques, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabeled images are used for training. We conclude that first the weights of the early layers of deep networks contain limited information about statistics of natural images, that second such low-level statistics can be learned through self-supervision just as well as through strong supervision and that three the low-level statistics can be captured via synthetic transformations instead of using large image data set. As I said, I was kind of excited when I saw this and what they're talking about is self-supervision. So really quickly for those who don't know, self-supervision is a technique where if you have images but no labels and you would still like to learn something from them, you can do a pre-training step for your network. So basically you have your neural network F and what you would like to do is you would like F of X to be close to Y for X and Y to in your training data set. But you can if you have a much larger data set of just X's, right here you have pairs of X and Y of just X, that are sort of similar to the X in your label data set. You can kind of get your network used to the data by doing this self-supervision. So what you would do is you would sort of come up with your own labels for the data points and one way is this, we'll take just this rot net as an example. So what you'll do is you'll input an image, so maybe an image of the number 3 in handwritten digit recognition, but you flip it to its side. So it's the number 3 right here and then you ask the network to come up with an answer. Is it upright? Is it flipped to the right? Is it flipped to the left? Or is it turned on its head? Which one is it? And of course you who did the transformation know the correct label. So this is how you come up with sort of fake labels for your data and this works surprisingly well and what this paper basically says is that you do not actually need this giant database here. It's actually sufficient sometimes if you have one single image where you do this on. Now the claim is a bit of a cheat I have to say, but we'll go into that. And further they say okay it's enough to have one single image is to learn the features of the lower layers of the neural network because they usually tend to extract low level features that you know you can learn from a single image but the higher level features you really need the supervised data set. It's not enough to have these self-supervision techniques and even if you have many many many self-supervision samples, so if you actually do have this giant data set it still doesn't help you for the higher layers. Almost causing to question this you have a giant data set of unlabeled things notion that is often presented including by me. Okay so what do they do? They take either single images or just very few images so they also have a setting where they take 10 images. For the single image they hand select it. So they hand select the following three images. So this image right here they select because it's very crowded there's a lot going on. There's people, there's objects, there is lighting and so on. There's you know houses, these lines, there's perspective. So that's why they select image A. Image B here they also experiment with image B is a drawn image as you can see there's also lots of stuff going on but they basically want to to research how does a natural image compared to a artificial image. And then in C they have this as sort of a control because there is lots of parts here where there's not much going on compared to here and most of the image there's lots of stuff going on and this image on the image number C or letter C has large areas where there's nothing going on. Okay so these are the single images. Now why I say it's a bit of a cheat is that these images are actually super large. So for ImageNet and for C410 this might be one of the samples of the C410 or ImageNet classifier. Now of course C410 is a lot smaller than ImageNet but still for ImageNet these are you know there are many pictures here not just one. So to say this is from a single image it's technically true but then if you split it into multiple images it's technically not true. So it would have been fun to see what actually happens with a single image when you downscale it. But okay so how do they investigate this? They have this five layer neural network right here so this five convolutional I'm gonna guess there's five convolutional layer after each convolutional layer I think there is some batch norm and relu and then to the next convolutional layer and so on and then at the end maybe there is or there's also some pooling here max pool at the end there is going to be some linear classifier that classifies it into either a ten or a hundred or a thousand classes whatever you want. The way they investigate this is through linear probes so-called. Now linear probes are somewhat of a technique to inspect how much each of the layers learn. So if we again draw our network right here and this is the input X right so you have the hidden representation one hidden representation two hidden representation three and here you output it to the y hat and that you compare with the y from your data set right the X is from your data set and the Y is from your data set. Now linear probe wants to investigate how useful a given hidden representation is to classify the output. So what a linear probe would do is it would take the hidden representation here and learn one single linear classifier to classify that hidden representation to come up with a y hat given h1 or something like this. So the important part here is that this is linear right this is a this is a linear classifier. You do nothing more you take the representation and instead of this entire giant neural network on top of it you simply build a linear classifier and you can build these linear probes from any layer right here. You can build a linear classifier on top of this on top of this and then you basically look how good is your linear classifier when trained on that hidden representation that the network comes up with and that's how you estimate how much information about the target or let's say no how how optimal the representation already is because at the end of the network right you do have a linear classifier. So at some point this representation must go into a form where it is now linearly well classifiable and the assumption is basically that these layers of the neural network successively make a representation that is more and more linearly classifiable and that is a strong assumption right and this paper here uses linear probes exclusively and that is a bit worrisome to me because I have my troubles with these linear probe approach because this strong assumption that more linearly classifiable is better it just rubs me in the wrong way right. We know that the information content can never increase from layer to layer about the label so any information about the label that is in H1 sorry that is in H2 must also have been present in H1 so technically if we just built the correct classifier we could predict from H1 just as well as from H2 right because we're actually doing it we're building the neural network but the fact that we cannot predict linearly as well using H1 so the fact that this classifier here performs worse than this classifier here because H1 is a less optimal representation in a linear sense it's and it's just meh and the fact that I mean yes but then to use that and to estimate oh how useful is a representation you're equating usefulness with linearly classifiable and that I disagree a representation can be extremely useful if the following layers manage to do something useful with it and that can be something completely different or it can even be the opposite of the linear classifiability right so this is kind of my problem here and they don't do a good work of convincing me otherwise so they don't employ different techniques other than these linear probes in any case when they do this linear probe you can see right here that the percent supervised performance so that's how much how much percent of supervised performance do you get oh single single image self-supervision we show that several self-supervision methods can be used to train the first few layers of a deep neural networks using a single training image such as this image a B or even C provided that sufficient data augmentation is used so what they do here is they use this self-supervision then they take the signal from the convolutional layer one the hidden representation that's h1 right here they train this linear probe on it and they see how how well does it perform after and this is after the network has been self supervised with rot net for example and then they compare that to the linear probe at layer one of the supervised network right so you take the supervised network and you do the same thing and there they find okay this rot net and all the other techniques they perform very well and especially if you only do a single image they perform better as you can see right here I mean if I interpret this correctly this one rot net one by again one deep cluster these are these top things right here and the 100 is the comparison to the supervised performance right so 100 means 100% of the performance of the supervised representation this is absolutely crazy to me and this in fact so let's just interpret it from their perspective right so you also have random so if you I guess if you randomly initialize a network then with the linear with training a linear classifier on the hidden representation one you could reach something like 60% accuracy which is impressive okay but if you do the linear probe at layer two you reach a lower accuracy now remember this is lower accuracy compared to the supervised performance right so the the there are two effects at play here the supervised performance is gonna go up because the well if you believe the assumption that the successive layers make the representation more and more linear linearly classifiable but also it could be that just at the same time the self supervised performance the performance of the self supervised representation is going down so the graph here is sort of I don't really know how to interpret it and it really goes down after that that's why they say you can learn the first layers fairly well with self supervision even from a single image but you cannot learn the upper layers and they're basically just measuring this using this linear probe method compared to the supervised performance what I would somewhat like to see is that you train let's say you train a self supervised network fine but then you freeze this layer and then you fine-tune the rest of your network on top of that representation that would actually give you an estimate of how useful is that representation if I had an you know an all-powerful function approximator which is a neural network and then of course you're probably not going to get supervised performance and by the way you'd have to compare that also to supervised with and without pre training using self supervision and then you actually get a good estimate of what how well what kind of a representation do these things learn in this case all we you know all we get out of this is this linear probe thing compared to the supervised representation and it just seems a bit uninterpretable honestly and the fact that here you can go beyond 100% you can actually be better than supervised should already tell you that the linear this linear probe thing might not be a good instrument to might not be such a good instrument especially in the lower layers the lower layers will be most inaccurate with these linear probe measurement but that's that's their finding basically they can learn the features of the lower layers as well in terms of this linear probe formulation as the supervised learning again they never compare this to fine-tuning on top of these representation or compare it to self supervision plus supervision which I would really expect all right so they say they do a lots and lots of data augmentation since of course they only have a single image they basically supercharge data augmentation and they show that this helps now I don't want to actually go into the into the very into the very details of what they're doing because they just have different methods of augmentation they just have different networks but here are the results so if this is on on image net if we use full supervision we use the entire data set and we do these linear probe evaluation we get a 20% accuracy after layer 1 36 after layer 2 and so on this goes up as we go through the layer so this kind of gives credence to the hypothesis that these layers sort of make the representation more linear then they have a bunch of scattering and random networks and K means pre training which doesn't get you a lot like but that's what they compare it to basically the self supervision to just the scattering transforms and things like that but then they get into their methods and here we'll look at for example this rod net so if you train on just one image this image a of course if you have one image then you get this many this this much of the layer one now okay so now that I see this here they have this column right here which uses the full data set what I think this is is the self supervised training using this many images so what if you do rod net self supervision on this many it could also be the performance after supervised training after pre training with this method but I think it is the performance after just after self supervision again with no fine-tuning on top and then evaluating these linear probes that's why this number is lower than this number right here but astonishingly after you do it with just one image you get a higher number and if you do it with a thousand images you get an even higher number but if you do it with many more images you do you you somehow don't get a higher number this all seems a bit it seems a bit weird honestly basically means that okay it is more important to augment the same thing over and over and over in different ways than it is to incorporate different images I mean there's ways I can believe that but I'm not sure but you basically see that after a while the performance compared to the first of all to the supervised method so yes if you look for example here up here drops dramatically and even if you have the full young now I'm convinced that this this is just self supervision using the full data set even if you have the full data set but only do self supervision your performance still suffers compared to the supervised training so that's why they claim they have these two claims you can learn the first layer representations fairly well with self supervision that's comparing this number to this number you can do so even from a single image that's comparing this number to this number right and noticing that it's almost the same these two numbers are almost the same actually one is a bit higher you can learn that fairly well but if you go down the layers you will basically suffer with your single image and with your full image soup self supervision so you need the supervised signal to learn the features of these later layers and that's all evaluated with these linear probe things yeah so that is their main claims right here and they kind of analyze image a and image B so they come to the conclusion that image a works much better because it's natural and image B is not working so well but this depends on the self supervision used and image C still apparently works quite well even though it has these large areas of nothing which all of this is a bit weird but it's definitely cool to see these results now again I would like to see something like you freeze these representations and then you actually train a neural network on top of that and look how that performs that would actually be an interesting thing though maybe they've done this and I'm just unaware right here they look at the filters that these methods have learned just from self supervision on a single image and you can see these are the types of filters that we would see using even supervised learning if you look at the filters they turn out to look pretty much like this of course I can't decide if these particular things are good or bad filters or not they do some qualitative analysis and here they have fine-tuning okay ah fine-tuning experiments the pre-trained models first two convolutions are left frozen or replaced by the scattering transform and the network is retrained using image net training set okay here we go so if you do this fully supervised you get to a 59.4 now okay this seems very low accuracy honestly for even like for image net but maybe this is their thing but if they do this on top of the on top of the these self supervised methods they do get a fairly good okay they get a fairly good accuracy right here I would have liked to have this evaluation right here be applied in the table above and not these linear probes they just seem kind of kind of wonky but you can see that it is possible to learn this to learn this using just a single image to learn the features of the lower layers now how you exactly would would put this into a training procedure how you exactly make use of this during training if you already know that it's not gonna help for the deeper layers I'm not so sure because at least you always have your own data set right so you always have at least that many images that you can self supervise train on but it's certainly interesting interesting results and with that I think I'm going to leave it at that and thanks for listening I hope you enjoyed this and bye bye
 Alright, today we'll look at a critical analysis of self-supervision or what we
 can learn from a single image by Yuki M. Asano, Christian Ruprecht and Andrea
 Vidaldi. This paper, I really was excited when I saw this paper because the
 outset is so cool and the experiments have a very promising. So we'll take a
 look. Basically we show that three different and representative methods,
 byGAN, RotNet and DeepCluster, so this is self-supervision techniques, can learn
 the first few layers of a convolutional network from a single image as well as
 using millions of images and manual labels, provided that strong data
 augmentation is used. However, for deeper layers the gap with manual supervision
 cannot be closed even if millions of unlabeled images are used for training.
 We conclude that first the weights of the early layers of deep networks
 contain limited information about statistics of natural images, that
 second such low-level statistics can be learned through self-supervision just
 as well as through strong supervision and that three the low-level statistics
 can be captured via synthetic transformations instead of using large
 image data set. As I said, I was kind of excited when I saw this and what
 they're talking about is self-supervision. So really quickly for
 those who don't know, self-supervision is a technique where if you have images but
 no labels and you would still like to learn something from them, you can do
 a pre-training step for your network. So basically you have your neural
 network F and what you would like to do is you would like F of X to be close to
 Y for X and Y to in your training data set. But you can if you have a much
 larger data set of just X's, right here you have pairs of X and Y of just X,
 that are sort of similar to the X in your label data set. You can
 kind of get your network used to the data by doing this self-supervision.
 So what you would do is you would sort of come up with your own labels for the
 data points and one way is this, we'll take just this rot net as an example. So
 what you'll do is you'll input an image, so maybe an image of the number 3 in
 handwritten digit recognition, but you flip it to its side.
 So it's the number 3 right here and then you ask the network to come up with
 an answer. Is it upright? Is it flipped to the right? Is it flipped to the left? Or
 is it turned on its head? Which one is it? And of course you who did the
 transformation know the correct label. So this is how you come up with sort
 of fake labels for your data and this works surprisingly well and what this
 paper basically says is that you do not actually need this giant database here.
 It's actually sufficient sometimes if you have one single image where you do
 this on. Now the claim is a bit of a cheat I have to say, but we'll go into
 that. And further they say okay it's enough to have one single image is to
 learn the features of the lower layers of the neural network because they
 usually tend to extract low level features that you know you can learn
 from a single image but the higher level features you really need the supervised
 data set. It's not enough to have these self-supervision techniques and
 even if you have many many many self-supervision samples, so if you
 actually do have this giant data set it still doesn't help you for the higher
 layers. Almost causing to question this you have a giant data set of unlabeled
 things notion that is often presented including by me. Okay so what do they do?
 They take either single images or just very few images so they also
 have a setting where they take 10 images. For the single image they hand select
 it. So they hand select the following three images. So this image right here
 they select because it's very crowded there's a lot going on. There's
 people, there's objects, there is lighting and so on. There's you know houses, these
 lines, there's perspective. So that's why they select image A. Image B
 here they also experiment with image B is a drawn image as you can see there's
 also lots of stuff going on but they basically want to to research how does a
 natural image compared to a artificial image. And then in C they have this as
 sort of a control because there is lots of parts here where there's not much
 going on compared to here and most of the image there's lots of stuff going on
 and this image on the image number C or letter C has large areas where there's
 nothing going on. Okay so these are the single images. Now why I say it's a bit
 of a cheat is that these images are actually super large. So for
 ImageNet and for C410 this might be one of the samples of the C410
 or ImageNet classifier. Now of course C410 is a lot smaller than ImageNet
 but still for ImageNet these are you know there are many pictures here not
 just one. So to say this is from a single image it's technically true but
 then if you split it into multiple images it's technically not true. So it
 would have been fun to see what actually happens with a single image when you
 downscale it. But okay so how do they investigate this? They have this five layer
 neural network right here so this five convolutional I'm gonna guess there's
 five convolutional layer after each convolutional layer I think there is
 some batch norm and relu and then to the next convolutional layer and so on and
 then at the end maybe there is or there's also some pooling here max pool
 at the end there is going to be some linear classifier that classifies it into
 either a ten or a hundred or a thousand classes whatever you want. The way they
 investigate this is through linear probes so-called. Now linear probes are
 somewhat of a technique to inspect how much each of the layers learn. So if we
 again draw our network right here and this is the input X right so you have
 the hidden representation one hidden representation two hidden representation
 three and here you output it to the y hat and that you compare with the y from
 your data set right the X is from your data set and the Y is from your data set.
 Now linear probe wants to investigate how useful a given hidden representation
 is to classify the output. So what a linear probe would do is it would take
 the hidden representation here and learn one single linear classifier to classify
 that hidden representation to come up with a y hat given h1 or something like
 this. So the important part here is that this is linear right this is a this is a
 linear classifier. You do nothing more you take the representation and instead
 of this entire giant neural network on top of it you simply build a linear
 classifier and you can build these linear probes from any layer right here.
 You can build a linear classifier on top of this on top of this and then you
 basically look how good is your linear classifier when trained on that
 hidden representation that the network comes up with and that's how you
 estimate how much information about the target or let's say no how how optimal
 the representation already is because at the end of the network right you do have
 a linear classifier. So at some point this representation must go into a form
 where it is now linearly well classifiable and the assumption is
 basically that these layers of the neural network successively make a
 representation that is more and more linearly classifiable and that is a
 strong assumption right and this paper here uses linear probes
 exclusively and that is a bit worrisome to me because I have my troubles with
 these linear probe approach because this strong assumption that more linearly
 classifiable is better it just rubs me in the wrong way right. We know that the
 information content can never increase from layer to layer about the
 label so any information about the label that is in H1 sorry that is in H2 must
 also have been present in H1 so technically if we just built the correct
 classifier we could predict from H1 just as well as from H2 right because we're
 actually doing it we're building the neural network but the fact that we
 cannot predict linearly as well using H1 so the fact that this classifier here
 performs worse than this classifier here because H1 is a less optimal
 representation in a linear sense it's and it's just meh and the fact that I
 mean yes but then to use that and to estimate oh how useful is a
 representation you're equating usefulness with linearly classifiable and
 that I disagree a representation can be extremely useful if the following layers
 manage to do something useful with it and that can be something completely
 different or it can even be the opposite of the linear classifiability right so
 this is kind of my problem here and they don't do a good work of convincing me
 otherwise so they don't employ different techniques other than these linear probes
 in any case when they do this linear probe you can see right here that the
 percent supervised performance so that's how much how much percent of supervised
 performance do you get
 oh single single image self-supervision we show that several self-supervision
 methods can be used to train the first few layers of a deep neural networks
 using a single training image such as this image a B or even C provided that
 sufficient data augmentation is used so what they do here is they use this
 self-supervision then they take the signal from the convolutional layer one
 the hidden representation that's h1 right here they train this linear probe
 on it and they see how how well does it perform after and this is after the
 network has been self supervised with rot net for example and then they
 compare that to the linear probe at layer one of the supervised network
 right so you take the supervised network and you do the same thing and there they
 find okay this rot net and all the other techniques they perform very well and
 especially if you only do a single image they perform better as you can see right
 here I mean if I interpret this correctly this one rot net one by again one deep
 cluster these are these top things right here and the 100 is the comparison to
 the supervised performance right so 100 means 100% of the performance of the
 supervised representation this is absolutely crazy to me and this in fact
 so let's just interpret it from their perspective right so you also have
 random so if you I guess if you randomly initialize a network then with the
 linear with training a linear classifier on the hidden representation one you
 could reach something like 60% accuracy which is impressive okay but if you do
 the linear probe at layer two you reach a lower accuracy now remember this is
 lower accuracy compared to the supervised performance right so the the
 there are two effects at play here the supervised performance is gonna go up
 because the well if you believe the assumption that the successive layers
 make the representation more and more linear linearly classifiable but also it
 could be that just at the same time the self supervised performance the
 performance of the self supervised representation is going down so the
 graph here is sort of I don't really know how to interpret it and it really
 goes down after that that's why they say you can learn the first layers fairly
 well with self supervision even from a single image but you cannot learn the
 upper layers and they're basically just measuring this using this linear probe
 method compared to the supervised performance what I would somewhat like
 to see is that you train let's say you train a self supervised network fine but
 then you freeze this layer and then you fine-tune the rest of your network on
 top of that representation that would actually give you an estimate of how
 useful is that representation if I had an you know an all-powerful function
 approximator which is a neural network and then of course you're probably not
 going to get supervised performance and by the way you'd have to compare that
 also to supervised with and without pre training using self supervision and
 then you actually get a good estimate of what how well what kind of a
 representation do these things learn in this case all we you know all we get out
 of this is this linear probe thing compared to the supervised
 representation and it just seems a bit uninterpretable honestly and the fact
 that here you can go beyond 100% you can actually be better than supervised
 should already tell you that the linear this linear probe thing might not be a
 good instrument to might not be such a good instrument especially in the lower
 layers the lower layers will be most inaccurate with these linear probe
 measurement but that's that's their finding basically they can learn the
 features of the lower layers as well in terms of this linear probe formulation
 as the supervised learning again they never compare this to fine-tuning on top
 of these representation or compare it to self supervision plus supervision which
 I would really expect all right so they say they do a lots and lots of data
 augmentation since of course they only have a single image they basically
 supercharge data augmentation and they show that this helps now I don't want to
 actually go into the into the very into the very details of what they're doing
 because they just have different methods of augmentation they just have different
 networks but here are the results so if this is on on image net if we use full
 supervision we use the entire data set and we do these linear probe evaluation
 we get a 20% accuracy after layer 1 36 after layer 2 and so on this goes
 up as we go through the layer so this kind of gives credence to the hypothesis
 that these layers sort of make the representation more linear then they
 have a bunch of scattering and random networks and K means pre training which
 doesn't get you a lot like but that's what they compare it to basically the
 self supervision to just the scattering transforms and things like that but then
 they get into their methods and here we'll look at for example this rod net
 so if you train on just one image this image a of course if you have one image
 then you get this many this this much of the layer one now okay so now that I see
 this here they have this column right here which uses the full data set what I
 think this is is the self supervised training using this many images so what
 if you do rod net self supervision on this many it could also be the
 performance after supervised training after pre training with this method but
 I think it is the performance after just after self supervision again with no
 fine-tuning on top and then evaluating these linear probes that's why this
 number is lower than this number right here but astonishingly after you do it
 with just one image you get a higher number and if you do it with a thousand
 images you get an even higher number but if you do it with many more images you
 do you you somehow don't get a higher number this all seems a bit it seems a
 bit weird honestly basically means that okay it is more important to augment the
 same thing over and over and over in different ways than it is to incorporate
 different images I mean there's ways I can believe that but I'm not sure but
 you basically see that after a while the performance compared to the first of all
 to the supervised method so yes if you look for example here up here drops
 dramatically and even if you have the full young now I'm convinced that this
 this is just self supervision using the full data set even if you have the full
 data set but only do self supervision your performance still suffers compared
 to the supervised training so that's why they claim they have these two claims
 you can learn the first layer representations fairly well with self
 supervision that's comparing this number to this number you can do so even from a
 single image that's comparing this number to this number right and noticing
 that it's almost the same these two numbers are almost the same actually one
 is a bit higher you can learn that fairly well but if you go down the layers
 you will basically suffer with your single image and with your full image
 soup self supervision so you need the supervised signal to learn the features
 of these later layers and that's all evaluated with these linear probe things
 yeah so that is their main claims right here and they kind of analyze image a
 and image B so they come to the conclusion that image a works much
 better because it's natural and image B is not working so well but this depends
 on the self supervision used and image C still apparently works quite well even
 though it has these large areas of nothing which all of this is a bit weird
 but it's definitely cool to see these results now again I would like to see
 something like you freeze these representations and then you actually
 train a neural network on top of that and look how that performs that would
 actually be an interesting thing though maybe they've done this and I'm just
 unaware right here they look at the filters that these methods have learned
 just from self supervision on a single image and you can see these are the types
 of filters that we would see using even supervised learning if you look at the
 filters they turn out to look pretty much like this of course I can't decide
 if these particular things are good or bad filters or not they do some
 qualitative analysis and here they have fine-tuning okay ah fine-tuning
 experiments the pre-trained models first two convolutions are left frozen or
 replaced by the scattering transform and the network is retrained using image
 net training set okay here we go so if you do this fully supervised you get to
 a 59.4 now okay this seems very low accuracy honestly for even like for
 image net but maybe this is their thing but if they do this on top of the on top
 of the these self supervised methods they do get a fairly good okay they get
 a fairly good accuracy right here I would have liked to have this evaluation
 right here be applied in the table above and not these linear probes they just
 seem kind of kind of wonky but you can see that it is possible to learn this to
 learn this using just a single image to learn the features of the lower layers
 now how you exactly would would put this into a training procedure how you
 exactly make use of this during training if you already know that it's not gonna
 help for the deeper layers I'm not so sure because at least you always have
 your own data set right so you always have at least that many images that you
 can self supervise train on but it's certainly interesting interesting
 results and with that I think I'm going to leave it at that and thanks for
 listening I hope you enjoyed this and bye bye