[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper ensemble pytorch lightning cifar10 github vim code cuda gpu research ml ml research how to implement live coding python self distillation born again deep ensembles cnn resnet vgg torchvision imagenet
I share my progress of implementing a research idea from scratch. I attempt to build an ensemble model out of students of label-free self-distillation without any additional data or augmentation. Turns out, it actually works, and interestingly, the more students I employ, the better the accuracy. This leads to the hypothesis that the ensemble effect is not a process of extracting more information from labels.

OUTLINE:
0:00 - Introduction
2:10 - Research Idea
4:15 - Adjusting the Codebase
25:00 - Teacher and Student Models
52:30 - Shipping to the Server
1:03:40 - Results
1:14:50 - Conclusion

Code: https://github.com/yk/PyTorch_CIFAR10

References:
My Video on SimCLRv2: https://youtu.be/2lkUNDZld-4
Born-Again Neural Networks: https://arxiv.org/abs/1805.04770
Deep Ensembles: A Loss Landscape Perspective: https://arxiv.org/abs/1912.02757

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
 Hey what's up! So I've had this relatively dumb research idea and people have been asking me for more coding videos and so on so I thought why not do a video where I take a research idea and implement it from scratch just to show how one would go or how I would go about implementing something like this. Now this was simply meant as sort of a demonstration but then at the end it actually worked and so yeah that was unexpected and my initial reaction was just to be like oh crap just hold everything you know stop video making you develop the idea write a paper about it okay and I was about doing that when I realized that you know I'm always the one complaining that research is not transparent enough and people aren't open enough and so on so I sort of thought I might do a different thing right here in that I will actually share the process of this non-finished research project so currently I am in the middle of this I have no idea whether it's going to work out or not and that's it and I think we can do open source software development you know completely in the open whereas with research we're all like super scared that people are gonna scoop us and we people just keep it keep their work hidden until they're done and then boom they put it in an archive and all I want to go to a world where we collaborate much more in research and it's much more like open source software development so here is my way here's here's my process of implementing this idea and it's fairly long so if you just want to get to the results you can just skip at the end I'll put time stamps in there's this new YouTube chapter video so that will be very helpful I guess yeah and with that being said I hope you enjoy this let me know what you think of videos like this and I'll see you next time hey what's going on today we're going to take a research idea and implement it as fast as we can so this is not really to show you the best research idea because it's not and it's probably been done before so I have no high hopes here but this is just to show that if you had like some research idea and you've actually done the literature research and figured no one has done that yet which I haven't because probably someone has done that how you could take this and like get started up initially pretty quickly and this is just the process that I would go through and I'm going to go through with you today and we're going to try to get this up and running as quickly as possible so I had this idea that looking at SimClear v2 there's a lot of things to be done still in the space of let's say self teaching self distillation and so on you know there's mean teacher and then there's whatnot and this is all usually done in the semi supervised very few label regime and so on but we know that these self supervised techniques can help you and supervised learning and then in SimClear v2 you do semi supervised in that you do self supervised and then fully supervised and then distillation like self distillation there's there's all these kinds of interleaving stuff and I thought okay what if I just take a pre-trained network that performs really well on something and I self distill it into a bunch of student models like a number like 10 or so and then I like that's my ensemble model will that perform better than the original model like this is a terrible idea and it's probably not going to work like there's 99% chance it's not going to work but let's try to test this today so I got my drink I got my carbs since it's weekend and we're going to give this a shot all right so first thing we need some sort of base to go from in research it's good to build your own stuff but a lot of times if you want to be as fast as possible you want to go as quickly as you can so here I found this repo thankfully with an MIT license so shout out to who even fun I guess for training for putting up a repo training these C for 10 models or training these PI torch vision models on C for 10 C for 10 is a small enough data set so that we can kind of work with it and these models are already pre-trained so I've cloned this repo and we're going to adjust that so there is a first of all there is a download as you can see here which in this report says it downloads these I've not done this before I have no clue how this is gonna work out in this download script here downloads the weights from box and I hope you can see this and then I guess you can load the the pre-trained weights with pre-trained equals true and yeah we'll get into all that later so the first thing we got to do is get this to run let's say so let's look at this downloads thing first so the download thing is going to have a URL it's going to use requests to get that URL and then save this into this state dicts thing now what I usually want to do is I don't I want my folder of code to only have code and not to be intermixed with data and code because this is the thing that I'm gonna ship around to various servers and so on so I'd rather have the code in one folder and then the data and like a central folder so I'm not really fine with this sort of downloading the this right here into into the folder that we have so what I'm going to do is I'm going to change that such that it downloads it into a central folder so first we already have OS so what we're going to do is we're going to get some like data path going which is going to be our home folder OS path and I guess I also I already have a C for 10 folder right here so we'll use this and then so path dot join will join that and that's going to download it's not really data is it it's more like models okay let's do this cool so data path is this is the models that is going to download all right and then it unzips the file again so here it unzips the file to the current working directory I don't want this so I'm going to change that again to the models path all right no directory to path to zip file directory to extract to I think we're fine right now so this download script is going to download the path the all the weights there now I want this to happen sort of automatically while this is in a server or while this is on a server so what I'm going to do is probably just to so if this script runs you can see it runs the main but in the other script I might just want to do this automatically so let's go to the test script right here or let's say we go to the train script this is probably the main script right here the train script so we have to somehow call this other script here probably in the main function all right so let's import this other so import C for 10 what was it called C for 10 download okay and here we're going to call that and does this does this not download it if it already exists we have to check that so a lot of this is just going to be you know beating the stuff into beating stuff into into existence so if this zip file already exists we're not going to we're not going to do anything right which leaves us open if like if the unzipping fails then we're going to be in a kind of dumb path but you know we'll risk it zip path would be that so that's if OS path exists zip path then return okay so we're good in the download script what else do we need the data set I probably already have the data set from torch vision so that's not going to be an issue okay so here we're gonna call C for 10 download dot main all right and that should do we can't really call that yet let's actually just run this download script no no such file or directory probably need to make that probably need to make that directory right okay if OS make theirs models path exist okay true yeah that should be something all right and we're downloading so this is 2.4 gigabytes which can you know be put by itself let's put that over there and while that's downloading let's check out the test script actually let's check out the test script so this simply takes in this C for 10 module and instantiates a trainer and as you can see it calls test on it so this should not be too hard I'm going to guess this C for 10 module is a lightning module as you can see right here it is we know how tensor sorry pytorch lightning works if you don't know how pytorch lightning works pretty easy you configure this module right here you configure a bunch of stuff like the data sets the training step and so on and you're good to go so I guess what we're going to do is we're going to change this train script and change it to our needs okay so let's copy that let's go with train ensemble bang so this is what we're going to change all right so first if the GPUs is a string then the other yada yada if it's two then wow that's that's that's kind of a weird engineering quirk right here okay what I want to do is make the GPU use transparent so we'll only ever use one GPU so let's call that CUDA and put that to true and then we'll say da da da da oh come on there is like a lot of stuff going on here let's so and then torch is called torch I'd hate that can I do this can I import it like twice with different names probably it's probably not very good but I'll do it okay so if CUDA is not available we'll just set the CUDA to false if th.cuda.is available okay not if it's not available then hparams.cuda equals false and then we'll set the GPUs to zero comma I guess that's what it expects if else none and that should do it for the GPUs okay so second thing that we need we're going to need we're calling fit here and there is this logs directory where the checkpoints are going to be saved I'm fine with that I just want to kind of remove the logs directory at the beginning so I'll do that and whenever we start this I'm going to remove the logs directory this is a controversial move but you know on remove tree recursively delete the directory tree yes logs good okay our download is done so what do we do next we might want to do just try to test test something and here in the test thing we might want to set the GPUs I don't have a GPU right here so none and the data directory is going to be yeah I'll put it so nope nope nope okay it doesn't find the it doesn't find the the state dicts and so on now we're going to have to fix this we're going to have to fix the fact that it doesn't load okay okay and that's probably going to be here in these models so if I look in the dense net for example which we can learn and there's this pre-trained argument and what's that going to be it's oh that's bad okay it like has a hard code at the fact as hard code at the fact that there are there is this state dicts directory okay um yeah that's terrible terrible terrible terrible so I guess this is going to be in every single one of these models and that's not good so what we're going to do is probably always late loaded without the pre-trained and then kind of loaded ourselves from the from the correct directory so what's the correct directory again we're going to set the model dear we probably can just take that from the download script like that state dicts okay and then we want the architecture I guess that's a thing we can actually put the classifier here right here that's something we can so it's going to be the classifier if you look in the state dates directory I'm gonna guess you can models see for ten state dicts we haven't unpacked it where have we not where have we unpacked it to help help oh no have we unpacked it to here we have not we have not so what is in here ah it's the C for ten models sub thing and then state dicts okay so it's always going to be the architecture plus PT so we can you know we can deal with that so it's going to be C for ten models state dicts that's fine and then it's always going to be the architecture plus a PT so let's look at one of these models to see how this is loaded we've saw we've seen this here so we simply want to load this state dict in and here it constructs the thing this is let's do proper string interpolation shall we oh device where this device come from we should check that out device is given device device device CPU where is device given okay dense net device CPU oh I guess device is always CPU and then then we map it to wherever I'm not entirely sure so here we say set device I guess we can just get the device from somewhere let's try it out okay so we're going to need this right here so we're going to OS path join models path and something that's dot PT so and here we're going to get the architecture which is the classifier cool so that's how we load something and then the device maybe we can just go torch kuda dot get device is that possible let's try nope ah okay nope no get device device maybe nope map location was given okay so we have to figure out where this device comes from honestly here no module there's this get classifier right here but just just says pre trained this device always CPU I just can't believe that I guess I'll believe it we'll always load to the CPU okay cool we can do that I guess pytorch lightning will then put it on the GPU for us cool so this is about how far I got when I tried to do this by myself and now the problems start missing keys in state dick a lot of missing stuff we can't we can't possibly load that yeah no not going to you so we can't load stuff what does it do load file name equals and then let's paste this and let's put some kind of break point here so we can check it out okay that exists now she feels like that should exist yeah that exists what's the what's the deal what's the matter here so we got model which is I guess a resin at 18 and we got this thing that we might want to load so why doesn't it work torch load load file name see that works so that's the state dict is that let's look at its keys we got a you know a bunch of stuff okay so why can't we load that well load state dict state dict and now unexpected keys in state dick missing keys so this is always prepended with model dot and here it's not okay what do we do about that I guess this is because we loaded ourselves okay cool so our model is not yes so our model has the sub path model so we need model dot model dot load state dict right look at us we made it so this is testing I guess this is this resin at 18 or whatnot so we can leave that to run for itself so we figured out how to load this stuff took us a while now let's go ahead and we know how to load the models we know how to load the weights so this is our teacher model right our teacher model is supposed to load up the weights and then and then teach the student models so here what does this training thing do we download the thing we make our GPUs to be really good okay and then we instantiate this module right here as you can see so now we're going to check out this module by the way the testing is done and as you can see there's an accuracy of 93.33 which I'm pretty happy with this is congruent with what we saw right here the resin at 18 do okay and we can I guess we can take a resin at 18 or a resin at 50 they're both fairly small right here so a lot of them are going to fit on our GPUs once we use the GPUs so let's change this module around right here to actually do the to actually do the let's say the the proper thing that we wanted to do so here we have self dot model as you can see and it's get classifier and the question is does it load it pre-trained so what we want to do is this is going to be our teacher model and this in this get classifier we want pre-trained to be false always right here we don't want any sort of we don't want to load the pre-trained instead what we want to do is we actually want to have the we want to load it ourselves right so here pretend false and now we're going from our test script we're going to take over the path they think the code that we used to load this okay all right so but a beam but a boom OS we don't have OS that common along just fine yep yep yep so now here we're going to have our self teacher model to load that state dict all right so this is it for initialization now we also need our student models of course so our student models are going to be a bunch of models models are going to be a bunch of models where what do we say so this is going to be a torch or a like a module list there's this module list torch dot n and dot module list right so I initialize that with a list and the list is going to be get me the classifier and we're just going to go for the same kind of classifiers right now to really boil it down to have the same architecture for the students and for the teachers for bar in range in range and here we probably need a flag so h params dot num students okay so these are going to be our student models so let's quickly create this num students thing right here I'll probably have to have an integer and we'll go with five students for now okay so we're creating five students all of them are not pre trained so we're going to are we going to train them from scratch or do we want actually to take over the weights we probably don't want to take over the weights let's just train them from scratch in a distillation mode I have no clue about this stuff by the way okay I guess this concludes this already concludes what we what we wanted to do so because this module list what can we do with it does anyone know I don't know by the way I'm sorry for the switching between the dark and the bright background I don't know how to fix that so pytorch and an module list it would be nice if we could give them some names right so I guess that's just an iterable right here so probably there's nothing that we can do to give them proper names or we'd have to hack around and I don't want to do that so I guess we can just check if that actually computes until here so let's check it out let's try the ensemble it doesn't data set not found or corrupted okay so what we'll have to do is we'll have to implement have to change this data directory right here so the data deer is going to be OS this whatever my C for 10 directory is no such file directory logs okay so logs doesn't exist so let's actually make it still no such file a directory logs why why doesn't it make it no such file a directory logs ah okay we need to ignore errors here and we're good okay so it computes until the point you probably you probably can't see that right I guess now you can see it let's check yeah now you can see it all right so where are we we are at the point right here in our module after we've created the teacher and the students so if we look at self technically we should be able to see right here a whole bunch of resin at 18s whole bunch so here you can see the teacher model right and I'm going to guess you can see layer 4 and here you can see the student models so the student models are going to be in a whole list of models and now we're going to train them so since they're initialized differently our hope is going to be that they're sort of going to end up at different places we're going to train them with the same like we're going to be really really stupid about this okay all right so let's be really stupid about it so what what are we gonna have to change here is our training step and our training step is actually fine we'll simply forward we'll get a loss from that and then we are going to return that and that's going to be back propped so in our optimizer wherever we initialize our optimizer we should probably give it the parameters that are not only the student model parameters right not the teacher model parameters so it should only train the student models okay and even even like that we should probably always set the teacher model in eval mode but we'll do that in the forward step right here so in the forward step we get images and labels and here it runs it just forward through the model we want to change that we actually want to have teacher predictions which we're going to have the teacher model we're going to forward this through the teacher models now the criterion I'm going to guess is a cross entropy so the predictions here are actually going to be logits right and this is this is good except that what we want to do is have a distribution of over labels so after the teacher here runs through and let's put a break point right here and actually look at it I find it's always easy if you go and just run until the point where you are at the code and then you can just look at stuff so here there's oh there's a validation sanity check okay probably don't want that and now we have the break right here and now we can look at teacher predictions dot shape so that's a batch size times 10 and if we look at it I'm going to guess there's some negative numbers in there so that's not going to be that those are going to be logits now we want them that to be a soft max over the last dimension and that's going to be of the same shape but of course now we're going to have a proper distribution so if we sum over the last dimension you should see a bunch of ones all right so the teacher predictions are going to be soft max over the last dimension and since we since we don't want to back prop through the teacher we can do this in an environment of no grad right here so we have that with not being stupid and we also set the teacher model into eval mode so I guess that does it set train no that should do it I have no idea yeah let's let's run it again we could have done that there okay so so far so good so we have the teacher predictions now what we need to do is run them through the student and use them as labels so we'll go for student in student models we'll go student forward or we simply run the images through that and that give us the logits and then we use our loss function on the logits and that not the labels but the teacher predictions right so we never actually use the labels here as you can see and that's going to be the student loss and now we have a bunch of losses and we're going to append that ah nope dot like this and our loss is simply going to be the sum of all the student losses not even the average I guess we could losses I guess we could make it the average just so if we change the number of students we'll get some kind of some sort of a better sense of the actual numbers what what what okay I think over here we're good yeah so our teacher model is not in training mode but our student models hopefully are in training mode no is this the eval pass I guess this is the eval pass this is the validation sanity check pass okay so this is going to be our loss and our accuracy now right so okay what's going to be our accuracy our accuracy is going to be we have these student losses all of them and what we are going to do is we're simply going to take the maximum prediction across the students pretty easy pretty easy but we need to collect the log it's so come on so we'll also have the log it's append the student log it's okay so we have a whole bunch of log it's right here and we'll get some predictions out of that now the question is do we want to simply take the mode or do we actually want to run a softmax over each and then take the average prediction I'm not duper super sure but we can try to do it in different different ways so right now we might just want to take the maybe the average log it and then run a softmax on top of that because I'm gonna guess the log it's our outputs of a linear layer so they might behave more in a linear fashion than if we were to average the actual probabilities that come out right maybe let's let's do this okay so we'll go we'll take these log it's they're all and we need to somehow concatenate those or stack them so how are we gonna stack them so they're 256 their batch size by number of classes so we'll just stack them at dimension zero I guess that's fine and then we are going to mean also across dimension zero so those are going to be our log it's our final log it's and then our predictions are going to be the argmax of the log it's in the last dimension yep that should be pretty straightforward I guess that's it easy as that yes the rest here should just do by itself and I'm going to go ahead and run give this another run and see where we run into problems can't really see how this could ever go wrong we'll just take everything over okay we actually got a problem 1d target tensor expected multi-target not supported so the cross entropy loss in pytorch does not support that let's let's give it a shot make this a little bigger for you and let's go for the cross P loss I can't type today so here we have the cross entropy loss and the cross entropy loss is useful when training cross because problem with the classes yada yada yada wait should be one that okay criterion expects a class index as the target okay so what we need is like a soft loss right we don't need this cross entropy loss we actually want we want to have soft targets so what do we do we want to do I think the cross entropy loss is a combination of the here of the log softmax and the NLL loss can we take the NLL loss maybe so the NLL loss right here is going to be the target that this loss expect should be a class index no okay that's not good so next let's go do we have we somehow need a soft cross entropy loss let's search for that pytorch soft cross entropy soft classes I guess people do that kind of stuff so the problem with these kind of losses is that what you do what you have to do is kind of protect yourself against against numerical instabilities right so what we want to do is find a function that does this for us I guess if we do the loft the the log softmax that should take care of it for us okay this is tensor flow okay following thread cross entropy loss I guess people just do really the log softmax and then do that and we should be fine with this okay thanks okay Frank yeah maybe maybe this has advanced since then so we can give like a last look at this and this is a bit too big I'm sorry your eyes are gonna have to suffer and we're going to look at loss functions and we're going to just look through them multi label soft margin loss hmm multi label we don't really want multi label right we want this but not with the targets okay I guess we're just gonna have to write this ourselves so ultimately what is the cross entropy that cross entropy is simply the probability of the true label times the log probability of the wrong or of the predicted label yeah as you see right here so we are going to simply multiply target times the log probability of the predicted label and then some some dot take that mean across the batch I guess yeah that should do we can implement this let's do it so this criterion right here is going to be our loss function and that's only used once so what we can do is going to be a function so we're going to take student logits and we're going to take teacher probabilities okay so how's that gonna work out we're going to do the log soft max from the student logits so and then dot does that exist log soft max functional okay we need functional and student logits of that dimension so now we have properly normalized student logit so that's going to be student log probes and then what we want to do is simply multiply the teacher probes times the student log probes and the negative of that is going to be our loss the question is do we want to sum that I guess across this dimension or mean it I guess the sum sum should do all right this is it easy as that why have we searched for so long so the criterion we can simply replace that now by our loss function cool so let's run it again yada yada yada okay so I need to check maybe we should have taken like a smaller model it sometimes pays off to you know start with a really small model small model just so you can you can do these kind of things fast so here we have dimension out of range to do okay which is where is that in forward in line 78 let's go there line 78 okay here the max is not going to be as much fun let's go there I think this is some of these things change over time in pytorch so this code might be written when you know so what we have is we have a max over the predictions and predictions oh it's already an arg max so I guess we can remove this all whether or not that agrees with labels dot data we don't need any of that dot float and we also don't need that we so the accuracy is simply going to be the mean of this no I guess so we're here can't we just do predictions equals labels yes and we want the sum of that actually we want the float first and then we want the mean yeah that seems reasonable so let's do it like this yeah float mean perfect how could this be any more easy but this right here is all of it so validation step accuracy corrects we'll just look at it once we've done it and what I do is usually just run it until it doesn't give me any mistakes anymore and then I know I sort of have succeeded okay we're pretty close I feel so it says grad can implicitly create it only for scalar outputs which probably means our loss function is not a scalar so when we return the loss right here here we have the sum of the losses divided by the length of the losses let's go here and check out what's up with that so what will do I see this loss function here will output basically one loss for each data point so what we need to do I guess is call mean on this or some when they created the criterion in the original one now we've thrown it away look at the git diff right here so I guess this reduces how does this reduce the cross entropy loss when we don't do anything cross entropy loss reduces reduction mean okay so let's reduce with the mean so if we call loss then after that we should call mean and then here I'm not so sure anymore we should divide here because the learning rate is kind of tuned to the original loss size so I guess we'll be content for now with summing up these things and over here I guess we've we've solved that right no losses yeah see our losses is going to be an entire tensor and now we just fixed that right now okay so let's try it again in the meantime what can we do we can so we already take care of our GPU we take care of the logs one thing to do when with respect to this download stuff right here is you know if you have a server or something and you let a lot of things run in parallel what you want to do is make sure they don't all download the same stuff at the same time that's pretty bad so what you want to do is ideally have some sort of lock such that they coordinate and I usually use a file lock for this so I'm gonna create that right here from import the file lock and then I simply create the lock let's go file lock and here you have to input a file so sometimes like yeah data lock I don't know you just pick some file and that's the file that these these processes are going to sync on and then once you do this you simply wrap all of it in a with lock so only one at a time can go in in this function all right so that's that that should make us safe and right here we're now training this is excellent we are training the students now we need to do that on an actual GPU so I have multiple tools to ship this to a GPU so first of all I can try to ship this to and try to ship this to a let's say to to one GPU so the way you do that is first of all I want some sort of unbuffered version of Python and then I have this do I even have this tool I do have the tool okay so I'm gonna call one of our servers and we don't know what's going on okay so cannot import name seed everything from pytorch lightning seed everything is that some kind of new thing in pytorch lightning I guess I have it apparently so here seed everything with zero why I don't need that we are running without any seed here we are being really cool okay next mistake cannot okay still same mistake of course since we don't yep next mistake we'll just go through the mistake learning rate late learning rate loggers so I guess we need to update pytorch lightning on the servers and I'll do that quickly okay I've updated pytorch lightning let's check out whether or not we can actually run something yeah we can run something so this again is now downloading this on the server while this is happening there's another thing we can do namely I have sort of sort of made a system to run stuff on servers which I like a lot honestly so I guess we can try this out how do I hidden oh with I okay so I want to first of all delete this delete this yes I guess delete this cool and this git folder is a bit annoying let's restructure because otherwise it will always ship the git folder with everything up here does it do that yeah I don't like to have the code in a top level so quickly make a sources directory move everything in there so move the c410 models into the source directory move all the Python files into the source directory no clear the logs and we're much better much better okay much better so what we what what we will do is my system requires like a file and I'm just gonna copy one from another project quickly okay so we're back and I copied that over as you can see you basically give hyper parameters and it blasts ever the hyper parameters through in a kind of a random search fashion it's not too sophisticated but we can work with it so 10 that's the file right for 10 train ensemble yes that's the file cool and here we're just going to put all of our hyper parameters and that will remove the logs file I'm okay with that but want this bang cool so what do we want we want basically we just want to try it like a bunch of times and then see like average across it right that's all maybe we want the the architecture to change so let's say the classifier is a resnet 18 or a resnet 34 or a resnet 50 just so we have a bunch of stuff to do okay and this downloaded and is training on GPU hopefully if this works then we can ship this off and we'll make this other hyper parameter that I like to use called rep which is just basically a dummy parameter and so I can just repeat the experiment a bunch of times and let's put that in here so this is really that this has this has no effect except for randomizing it a bit I guess we can try to seed stuff so whenever it says seed everything we'll just seed it with this we'll call it seed no is it here this seed everything yeah so h params dot rep sorry seed cool what this this is doing something nice you can see it so this is unbuffered Python output thanks yeah so what other classifiers do we have we can again we can try a bunch of them we can try all of them but why don't we try all of them like this then let's go into this rat file I don't know why I called it rat I just want it like some three-letter thing so yep like this and then we can just take all of that crap and delete it and delete this and those are going to be all our models so our classifier is going to consist of all of this stuff let's I know I know I know I suck at them don't tell me actually tell me I want them tips trying to learn something new like each week in them but it is hard and tend to make myself actually do it so let's go let's go with just one repetition so far if we if we are not sure we can still up the number of repetitions we don't even have the rep right now this is called seed all right so we have different classifiers and what we're going to need we also have this num students right let's go with one with five and with 20 so here we got one epoch done and we get a validation loss do we get a validation accuracy validating validating I have no idea we'll cancel this right now and we'll go ahead and just blast this onto our servers and hopefully that that's gonna work I have no idea is everything fine everything's fine go no what what cool and let me get back to you once this is finished all right we're back so I've just written some code here to extract the results of that run and something you know it's pretty interesting what came out so in these plots you'll see on the x-axis of the number of students in the ensemble remember these students are all trained from the same teacher the teacher you can see in orange that's just the single teacher for reference you can see that if you have one student model it sometimes under performs or sometimes out performs the single teacher model but then if you have more student models you can see that there is a pretty monotonic relationship so here it's the reason this fit doesn't finish here is because there's not enough space on the GPU for that many student models but you can see that the relationship here is fairly monotonic here it's a bit of a kink so the first idea like this this is really astounding because these students have all been trained from that single teacher and they have been trained for as long as the teacher has been trained so they don't have more compute than the teacher they've been trained from scratch not from some checkpoint or from the teacher weights it's simple distillation from the teacher no labels and the students are all in parallel as well so they don't see different data or even different data augmentations it's the exact same order of the exact same data points going through all of the students the exact same learning rate schedule there's no noise and so on so the first thought that came to my mind like something fishy is going on here right like this this is this seems like to like come on there's no new information here so I thought hey I the teacher the teacher model I've just grabbed them from this from this repo from this pre-trained checkpoints and these pre-trained checkpoints they are you know the checkpoints that have performed best on the validation set so this is sort of a sneaky way of how we could train on the validation set right because we annotate each data point in the training data set with this checkpoint and the checkpoint has been selected for performing especially well on the validation data set it could explain why we get a gain on the validation data set so what I did is I retrained all of the teacher models such that I just retrained them for these 100 epochs and I just took the last checkpoint the all everything's the same the hyper parameters learning rate schedule and so on this is not tuned for any particular model and it's pretty like it's pretty standard it's like it's not like 0.12589 it's like 0.01 and 100 epochs or so fairly standard parameters and I just took the last checkpoint to make it didn't even look at its performance to make sure that I didn't you know select something that was especially good on the validation data set and the results here you'll see are actually already the results of that run which the previous run it was almost the same like I was astounded how well it works and then I thought hey maybe I'm kind of you know cheating here so I redid it with the teachers that are not specifically selected and this is already the results so that's pretty cool right so then I wondered what happens if I now if I increase my training amount so I just let this run for more like what if I let the students run for more than the teacher has run again there's no new information here so you can see that the now the okay the green is now the teacher the blue is a hundred epochs and the orange is 250 epochs and you can see with that even one student will outperform the teacher but many students will outperform even more so if you give more compute there's lots of lots of headroom here to improve you'll see this here I think this last one with the blue line is just a bit of a weird a weird configuration I guess if you were to rerun that that would you know fall in line so this is pretty pretty weird right so I have a bunch of questions so first of all I've searched the literature a bit more and I came up with a number of papers that do things like this now usually when you do distillation you people stress the importance of like how to introduce noise like in the noisy student paper or that you really need these data augmentations or you know same clear V2 uses the self distillation in order to do in in order to label more data so they say it's important that we bring more unlabeled data into the process and so on so all of this it really doesn't match right here and especially this focus on we need noise during the distillation process to build these ensembles this is also you know if you know mean teacher things like this I also found a paper called born again neural networks that does something quite similar but not very simple not like the same where they distill a teacher to the student with the same architecture and then they distill the student again into another student and then that into another student and so on and then at the end they say oh we can also build an ensemble but sometimes their ensembles outperform their you know chain of distillation sometimes they don't they don't really focus on that part a lot and it's way more complicated like you distill one student after another and I also think they they have some introduction of variability in the students like like noise or different augmentations and so on so this here seems you know really really really simple now I want to know this ensemble effect it seems pretty pretty weird right so what gives so the first thing we could do is we could say what what does how does this compare to an ensemble of teacher models like if we actually were to build an ensemble like train five teacher models on you know five five different teacher models it's still the same data but reasonably they might be able to learn something more from the data if we have five teacher models they might learn different things from the data and therefore if we combine them they might kind of overlap their knowledge and sort of catch where if one doesn't generalize in one data point the other four can overrule it whereas with these student with these self ensembles there's not really a way where we can learn more from data because we can only learn from the teacher and the teacher is fixed and has seen that much data right so how does this compare so wrote some rewrote some code is it's just plumbing and I release the code it's linked but it's just plumbing don't worry don't worry there is no great thoughts in there it's just plumbing such that my students don't are not all in parallel so the ensembles are not trained in parallel anymore I train each model individually which means that at maximum I have to have two models on the same GPU one teacher and one student so I make sure that the teachers they are trained from scratch and the students they're always trained from the same teacher right so the student ensembles will be exactly the same as we have them here that means one teacher is responsible for all the students but yeah so okay I'll just show you the results right here so if we look at those results you can see that and I've done it for a bunch of models right here the blue line is the ensemble of teachers and here on the x-axis you see the number of models and now since I'm not training everything on the same GPU but I recombine later that that basically means that I have doubt the ability to train up to 10 models are actually however many I want and the only real trick in the code is that when I evaluate one of these ensembles what I do is I load a mini batch and then I basically load the first checkpoint run the forward pass load the second checkpoint run a forward pass load the third checkpoint run a forward pass I do this for all the checkpoints until I go to the next mini batch but that's just for evaluating right it just seemed easiest with the code that I had so you can see right here that the there is a significant like this is almost overlapping right here for most models there sometimes the student wins sometimes the teacher wins so the teacher ensemble wins now remember the teachers are trained on you know ten times as much data right here but it's always the same data but still they have the opportunity to learn ten times as much information from the data whereas the students they're all distilled from that same teacher without any noise any augmentate any augmentations except for the augmentations that you use during training anyway and I've done this for a hundred epochs and I've done this for 250 is this already 250 I think that was a hundred I just put that there nope okay yeah that was a hundred epochs but you'll see the 250 epoch plots they look very much the same okay they are just a bit better if you train for 250 epochs now interestingly okay here's the interesting part about the 250 epochs the student is still distilled from a teacher model that has been trained for a hundred epochs so all of this all of this makes no sense to me right the student is still distilled from the hundred epoch teacher model yet if you train the student for 250 epochs in self distillation and then build an ensemble of these students from that same teacher model and you compare that to an ensemble of teachers that have all been trained for longer for 250 epochs which you know should out it the 250 epochs generally outperforms the hundred epochs models still they are the same this is this is pretty crazy results I I think and sort of my conclusion from this is that the ensemble effect right here is not a function of learning of extracting more information from the data the ensemble effect might actually be have something to do with the function landscape itself and kind of exploring different minima of the of the same function not of the same function but exploring different functions to describe the same phenomena and I've also found a paper that explains the lost landscape of deep ensembles and I will make a video on that maybe it's out already maybe it will be out after you see this one I I haven't decided yet which which order I'm going to release things but this here I it's it's pretty interesting and we need like a name so self self ensembles are already a thing but they are always with noise and stuff like this so let's call them something like plain self ensembles but that that that sounds like a good name plain self ensembles the act of self distillation a single model into multiple models without any noise any augmentations anything just you run as if you were to train the model itself and then you build an ensemble of these models by simply averaging the log it's plain self ensembles alright so the plan from here is to check on like at least one other data set you know these these models I appreciate that I could get them pre trained but they're just the image net models and then kind of let run on C for 10 so there's no kind of guarantee that these have been you know tuned or anything that the learning rates or whatnot so I want to take like an image net model you still make sure that I don't use any like hidden information where I could cheat on the validation set but try this on at least one thing and see if that works as well if we can sort of push image net performance simply by doing this trick so that's the plan for now and I have some other ideas but I just wanted to let you know and this is sort of how research works I guess you have a dumb idea and it turns out to work and then you go on and still probably probably there is not maybe too much interesting things here maybe it doesn't work on image net because these models are just under train and this somehow made them better somehow or regularize them somehow that usually doesn't work there's so much that can go wrong still so but yeah that was it and I invite you to like check out other papers in this space if you want it's a pretty interesting space and with that I don't have much more to say yeah I hope you enjoyed this let me know what you think of like research implementation or research process videos like this I'm not sure what people expect like I can't make this into five minute video of like whoo I discovered something because then you know there's no clue of what's what's happening but maybe like an hour or so is also too long I'm not sure yeah let me know what you think and I'll see you next time bye
 Hey what's up! So I've had this relatively dumb research idea and people
 have been asking me for more coding videos and so on so I thought why not do
 a video where I take a research idea and implement it from scratch just to show
 how one would go or how I would go about implementing something like this. Now
 this was simply meant as sort of a demonstration but then at the end it
 actually worked and so yeah that was unexpected and my initial reaction was
 just to be like oh crap just hold everything you know stop video making
 you develop the idea write a paper about it okay and I was about doing that when
 I realized that you know I'm always the one complaining that research is not
 transparent enough and people aren't open enough and so on so I sort of
 thought I might do a different thing right here in that I will actually share
 the process of this non-finished research project so currently I am in
 the middle of this I have no idea whether it's going to work out or not
 and that's it and I think we can do open source software development you know
 completely in the open whereas with research we're all like super scared
 that people are gonna scoop us and we people just keep it keep their work
 hidden until they're done and then boom they put it in an archive and all I
 want to go to a world where we collaborate much more in research and
 it's much more like open source software development so here is my way here's
 here's my process of implementing this idea and it's fairly long so if you just
 want to get to the results you can just skip at the end I'll put time stamps in
 there's this new YouTube chapter video so that will be very helpful I guess yeah
 and with that being said I hope you enjoy this let me know what you think of
 videos like this and I'll see you next time hey what's going on today we're
 going to take a research idea and implement it as fast as we can so this
 is not really to show you the best research idea because it's not and it's
 probably been done before so I have no high hopes here but this is just to show
 that if you had like some research idea and you've actually done the literature
 research and figured no one has done that yet which I haven't because probably
 someone has done that how you could take this and like get started up initially
 pretty quickly and this is just the process that I would go through and I'm
 going to go through with you today and we're going to try to get this up and
 running as quickly as possible so I had this idea that looking at SimClear v2
 there's a lot of things to be done still in the space of let's say self teaching
 self distillation and so on you know there's mean teacher and then there's
 whatnot and this is all usually done in the semi supervised very few label
 regime and so on but we know that these self supervised techniques can help you
 and supervised learning and then in SimClear v2 you do semi supervised in
 that you do self supervised and then fully supervised and then distillation
 like self distillation there's there's all these kinds of interleaving stuff
 and I thought okay what if I just take a pre-trained network that performs really
 well on something and I self distill it into a bunch of student models like a
 number like 10 or so and then I like that's my ensemble model will that
 perform better than the original model like this is a terrible idea and it's
 probably not going to work like there's 99% chance it's not going to work but
 let's try to test this today so I got my drink I got my carbs since it's
 weekend and we're going to give this a shot all right so first thing we need
 some sort of base to go from in research it's good to build your own stuff but a
 lot of times if you want to be as fast as possible you want to go as quickly as
 you can so here I found this repo thankfully with an MIT license so shout
 out to who even fun I guess for training for putting up a repo training
 these C for 10 models or training these PI torch vision models on C for 10 C for
 10 is a small enough data set so that we can kind of work with it and these
 models are already pre-trained so I've cloned this repo and we're going to
 adjust that so there is a first of all there is a download as you can see here
 which in this report says it downloads these I've not done this before I have no
 clue how this is gonna work out in this download script here downloads the
 weights from box and I hope you can see this and then I guess you can load the
 the pre-trained weights with pre-trained equals true and yeah we'll get into all
 that later so the first thing we got to do is get this to run let's say so let's
 look at this downloads thing first so the download thing is going to have a
 URL it's going to use requests to get that URL and then save this into this
 state dicts thing now what I usually want to do is I don't I want my folder
 of code to only have code and not to be intermixed with data and code because
 this is the thing that I'm gonna ship around to various servers and so on so
 I'd rather have the code in one folder and then the data and like a central
 folder so I'm not really fine with this sort of downloading the this right here
 into into the folder that we have so what I'm going to do is I'm going to
 change that such that it downloads it into a central folder so first we
 already have OS so what we're going to do is we're going to get some like data
 path going which is going to be our home folder OS path and I guess I also I
 already have a C for 10 folder right here so we'll use this and then so path
 dot join will join that and that's going to download it's not really data is it
 it's more like models okay let's do this cool so data path is this is the models
 that is going to download all right and then it unzips the file again so here it
 unzips the file to the current working directory I don't want this so I'm going
 to change that again to the models path all right no directory to path to zip
 file directory to extract to I think we're fine right now so this download
 script is going to download the path the all the weights there now I want this to
 happen sort of automatically while this is in a server or while this is on a
 server so what I'm going to do is probably just to so if this script runs
 you can see it runs the main but in the other script I might just want to do
 this automatically so let's go to the test script right here or let's say we
 go to the train script this is probably the main script right here the train
 script so we have to somehow call this other script here probably in the main
 function all right so let's import this other so import C for 10 what was it
 called C for 10 download okay and here we're going to call that and does this
 does this not download it if it already exists we have to check that so a lot of
 this is just going to be you know beating the stuff into beating stuff into
 into existence so if this zip file already exists we're not going to we're
 not going to do anything right which leaves us open if like if the unzipping
 fails then we're going to be in a kind of dumb path but you know we'll risk it
 zip path would be that so that's if OS path exists zip path then return
 okay so we're good in the download script what else do we need the data set
 I probably already have the data set from torch vision so that's not going to
 be an issue okay so here we're gonna call C for 10 download dot main all
 right and that should do we can't really call that yet let's actually just run
 this download script no no such file or directory probably need to make that
 probably need to make that directory right okay if OS make theirs models path
 exist okay true yeah that should be something all right and we're
 downloading so this is 2.4 gigabytes which can you know be put by itself
 let's put that over there and while that's downloading let's check out the
 test script actually let's check out the test script so this simply takes in this
 C for 10 module and instantiates a trainer and as you can see it calls
 test on it so this should not be too hard I'm going to guess this C for 10
 module is a lightning module as you can see right here it is we know how tensor
 sorry pytorch lightning works if you don't know how pytorch lightning works
 pretty easy you configure this module right here you configure a bunch of
 stuff like the data sets the training step and so on and you're good to go so
 I guess what we're going to do is we're going to change this train script and
 change it to our needs okay so let's copy that let's go with train ensemble
 bang so this is what we're going to change all right so first if the GPUs
 is a string then the other yada yada if it's two then wow that's that's that's
 kind of a weird engineering quirk right here okay what I want to do is make the
 GPU use transparent so we'll only ever use one GPU so let's call that CUDA and
 put that to true and then we'll say da da da da oh come on there is like a lot
 of stuff going on here let's so and then torch is called torch I'd hate that can
 I do this can I import it like twice with different names probably it's
 probably not very good but I'll do it okay so if CUDA is not available we'll
 just set the CUDA to false if th.cuda.is available okay not if it's not
 available then hparams.cuda equals false and then we'll set the GPUs to zero
 comma I guess that's what it expects if else none and that should do it for the
 GPUs okay so second thing that we need we're going to need we're calling fit
 here and there is this logs directory where the checkpoints are going to be
 saved I'm fine with that I just want to kind of remove the logs directory at the
 beginning so I'll do that and whenever we start this I'm going to remove the
 logs directory this is a controversial move but you know on remove tree
 recursively delete the directory tree yes logs good okay our download is done
 so what do we do next we might want to do just try to test test something and
 here in the test thing we might want to set the GPUs I don't have a GPU right
 here so none and the data directory is going to be yeah I'll put it so nope nope nope
 okay it doesn't find the it doesn't find the the state dicts and so on now we're
 going to have to fix this we're going to have to fix the fact that it doesn't
 load okay okay and that's probably going to be here in these models so if I look
 in the dense net for example which we can learn and there's this pre-trained
 argument and what's that going to be it's oh that's bad okay it like has a
 hard code at the fact as hard code at the fact that there are there is this
 state dicts directory okay um yeah that's terrible terrible terrible terrible so I
 guess this is going to be in every single one of these models and that's not
 good so what we're going to do is probably always late loaded without the
 pre-trained and then kind of loaded ourselves from the from the correct
 directory so what's the correct directory again we're going to set the
 model dear we probably can just take that from the download script like that
 state dicts okay and then we want the architecture I guess that's a thing we
 can actually put the classifier here right here that's something we can so
 it's going to be the classifier if you look in the state dates directory I'm
 gonna guess you can models see for ten state dicts we haven't unpacked it where
 have we not where have we unpacked it to help help oh no have we unpacked it to
 here we have not we have not so what is in here ah it's the C for ten models
 sub thing and then state dicts okay so it's always going to be the architecture
 plus PT so we can you know we can deal with that so it's going to be C for ten
 models state dicts that's fine and then it's always going to be the architecture
 plus a PT so let's look at one of these models to see how this is loaded we've
 saw we've seen this here so we simply want to load this state dict in and here
 it constructs the thing this is let's do proper string interpolation shall we oh
 device where this device come from we should check that out device is given
 device device device CPU where is device given okay dense net device CPU oh I
 guess device is always CPU and then then we map it to wherever I'm not entirely
 sure so here we say set device I guess we can just get the device from
 somewhere let's try it out okay so we're going to need this right here so we're
 going to OS path join models path and something that's dot PT so and here we're
 going to get the architecture which is the classifier cool so that's how we
 load something and then the device maybe we can just go torch kuda dot get device
 is that possible let's try
 nope ah okay
 nope no get device device maybe nope map location was given okay so we have to
 figure out where this device comes from honestly here no module there's this get
 classifier right here but just just says pre trained
 this device always CPU I just can't believe that I guess I'll believe it
 we'll always load to the CPU okay cool we can do that I guess pytorch lightning
 will then put it on the GPU for us cool so this is about how far I got when I
 tried to do this by myself and now the problems start missing keys in state
 dick a lot of missing stuff we can't we can't possibly load that yeah no not
 going to you so we can't load stuff what does it do load file name equals and
 then let's paste this and let's put some kind of break point here so we can check
 it out
 okay that exists now she feels like that should exist
 yeah that exists what's the what's the deal what's the matter here so we got
 model which is I guess a resin at 18 and we got this thing that we might want to
 load so why doesn't it work torch load load file name see that works so that's
 the state dict is that let's look at its keys we got a you know a bunch of stuff
 okay so why can't we load that well load state dict state dict and now unexpected
 keys in state dick missing keys so this is always prepended with model dot and
 here it's not okay what do we do about that I guess this is because we loaded
 ourselves okay cool so our model is not yes so our model has the sub path model
 so we need model dot model dot load state dict right look at us we made it so
 this is testing I guess this is this resin at 18 or whatnot so we can leave
 that to run for itself so we figured out how to load this stuff took us a while
 now let's go ahead and we know how to load the models we know how to load the
 weights so this is our teacher model right our teacher model is supposed to
 load up the weights and then and then teach the student models so here what
 does this training thing do we download the thing we make our GPUs to be really
 good okay and then we instantiate this module right here as you can see so now
 we're going to check out this module by the way the testing is done and as you
 can see there's an accuracy of 93.33 which I'm pretty happy with this is
 congruent with what we saw right here the resin at 18 do okay and we can I
 guess we can take a resin at 18 or a resin at 50 they're both fairly small
 right here so a lot of them are going to fit on our GPUs once we use the GPUs so
 let's change this module around right here to actually do the to actually do
 the let's say the the proper thing that we wanted to do so here we have self dot
 model as you can see and it's get classifier and the question is does it
 load it pre-trained so what we want to do is this is going to be our teacher
 model and this in this get classifier we want pre-trained to be false always
 right here we don't want any sort of we don't want to load the pre-trained
 instead what we want to do is we actually want to have the we want to
 load it ourselves right so here pretend false and now we're going from our test
 script we're going to take over the path they think the code that we used to load
 this okay all right so but a beam but a boom OS we don't have OS that common
 along just fine yep yep yep so now here we're going to have our self teacher
 model to load that state dict all right so this is it for initialization now we
 also need our student models of course so our student models are going to be a
 bunch of models models are going to be a bunch of models where what do we say so
 this is going to be a torch or a like a module list there's this module list
 torch dot n and dot module list right so I initialize that with a list and the
 list is going to be get me the classifier and we're just going to go for
 the same kind of classifiers right now to really boil it down to have the same
 architecture for the students and for the teachers for bar in range in range
 and here we probably need a flag so h params dot num students okay so these
 are going to be our student models so let's quickly create this num students
 thing right here I'll probably have to have an integer and we'll go with five
 students for now okay so we're creating five students all of them are not
 pre trained so we're going to are we going to train them from scratch or do
 we want actually to take over the weights we probably don't want to take
 over the weights let's just train them from scratch in a distillation mode I
 have no clue about this stuff by the way okay I guess this concludes this
 already concludes what we what we wanted to do so because this module list what
 can we do with it does anyone know I don't know by the way I'm sorry for the
 switching between the dark and the bright background I don't know how to
 fix that so pytorch and an module list it would be nice if we could give them
 some names right so I guess that's just an iterable right here so probably
 there's nothing that we can do to give them proper names or we'd have to hack
 around and I don't want to do that so I guess we can just check if that actually
 computes until here so let's check it out let's try the ensemble it doesn't
 data set not found or corrupted okay so what we'll have to do is we'll have to
 implement have to change this data directory right here so the data deer
 is going to be OS this
 whatever my C for 10 directory is no such file directory logs okay so logs
 doesn't exist so let's actually make it
 still no such file a directory logs why why doesn't it make it no such file a
 directory logs ah okay we need to ignore errors here and we're good okay so it
 computes until the point you probably you probably can't see that right I guess
 now you can see it let's check yeah now you can see it all right so where are we
 we are at the point right here in our module after we've created the teacher
 and the students so if we look at self technically we should be able to see
 right here a whole bunch of resin at 18s whole bunch so here you can see the
 teacher model right and I'm going to guess you can see layer 4 and here you
 can see the student models so the student models are going to be in a
 whole list of models and now we're going to train them so since they're
 initialized differently our hope is going to be that they're sort of going
 to end up at different places we're going to train them with the same like
 we're going to be really really stupid about this okay all right so let's be
 really stupid about it so what what are we gonna have to change here is our
 training step and our training step is actually fine we'll simply forward we'll
 get a loss from that and then we are going to return that and that's going to
 be back propped so in our optimizer wherever we initialize our optimizer we
 should probably give it the parameters that are not only the student model
 parameters right not the teacher model parameters so it should only train the
 student models okay and even even like that we should probably always set the
 teacher model in eval mode but we'll do that in the forward step right here so
 in the forward step we get images and labels and here it runs it just forward
 through the model we want to change that we actually want to have teacher
 predictions which we're going to have the teacher model we're going to forward
 this through the teacher models now the criterion I'm going to guess is a cross
 entropy so the predictions here are actually going to be logits right and
 this is this is good except that what we want to do is have a distribution of
 over labels so after the teacher here runs through and let's put a break point
 right here and actually look at it I find it's always easy if you go and just
 run until the point where you are at the code and then you can just look at stuff
 so here there's oh there's a validation sanity check okay probably don't want
 that and now we have the break right here and now we can look at teacher
 predictions dot shape so that's a batch size times 10 and if we look at it I'm
 going to guess there's some negative numbers in there so that's not going to
 be that those are going to be logits now we want them that to be a soft max over
 the last dimension and that's going to be of the same shape but of course now
 we're going to have a proper distribution so if we sum over the last
 dimension you should see a bunch of ones all right so the teacher predictions are
 going to be soft max over the last dimension and since we since we don't
 want to back prop through the teacher we can do this in an environment of no grad
 right here so we have that with not being stupid and we also set the teacher
 model into eval mode so I guess that does it set train no that should do it
 I have no idea yeah let's let's run it again we could have done that there okay
 so so far so good so we have the teacher predictions now what we need to do is
 run them through the student and use them as labels so we'll go for student
 in student models we'll go student forward or we simply run the images
 through that and that give us the logits and then we use our loss function on the
 logits and that not the labels but the teacher predictions right so we never
 actually use the labels here as you can see and that's going to be the student
 loss and now we have a bunch of losses and we're going to append that
 ah nope dot like this and our loss is simply going to be the sum of all the
 student losses not even the average I guess we could
 losses I guess we could make it the average just so if we change the number
 of students we'll get some kind of some sort of a better sense of the
 actual numbers what what what okay I think over here we're good yeah so our
 teacher model is not in training mode but our student models hopefully are in
 training mode no is this the eval pass I guess this is the eval pass this is the
 validation sanity check pass okay so this is going to be our loss and our
 accuracy now right so okay what's going to be our accuracy our accuracy is going
 to be we have these student losses all of them and what we are going to do is
 we're simply going to take the maximum prediction across the students pretty
 easy pretty easy but we need to collect the log it's so come on so we'll also
 have the log it's append the student log it's okay so we have a whole bunch of
 log it's right here and we'll get some predictions out of that now the question
 is do we want to simply take the mode or do we actually want to run a softmax
 over each and then take the average prediction I'm not duper super sure but
 we can try to do it in different different ways so right now we might just
 want to take the maybe the average log it and then run a softmax on top of that
 because I'm gonna guess the log it's our outputs of a linear layer so they might
 behave more in a linear fashion than if we were to average the actual
 probabilities that come out right maybe let's let's do this okay so we'll go
 we'll take these log it's they're all and we need to somehow concatenate
 those or stack them so how are we gonna stack them so they're 256 their batch
 size by number of classes so we'll just stack them at dimension zero I guess
 that's fine and then we are going to mean also across dimension zero so those
 are going to be our log it's our final log it's and then our predictions are
 going to be the argmax of the log it's in the last dimension yep that should
 be pretty straightforward I guess that's it easy as that yes the rest here should
 just do by itself and I'm going to go ahead and run give this another run and
 see where we run into problems can't really see how this could ever go wrong
 we'll just take everything over okay we actually got a problem 1d target tensor
 expected multi-target not supported so the cross entropy loss in pytorch does
 not support that let's let's give it a shot make this a little bigger for you
 and let's go for the cross P loss I can't type today so here we have the
 cross entropy loss and the cross entropy loss is useful when training cross
 because problem with the classes yada yada yada wait should be one that okay
 criterion expects a class index as the target okay so what we need is like a
 soft loss right we don't need this cross entropy loss we actually want we want to
 have soft targets so what do we do we want to do I think the cross entropy
 loss is a combination of the here of the log softmax and the NLL loss can we take
 the NLL loss maybe so the NLL loss right here is going to be the target that this
 loss expect should be a class index no okay that's not good so next let's go do
 we have we somehow need a soft cross entropy loss let's search for that
 pytorch soft cross entropy soft classes I guess people do that kind of stuff so
 the problem with these kind of losses is that what you do what you have to do is
 kind of protect yourself against against numerical instabilities right so what we
 want to do is find a function that does this for us I guess if we do the loft
 the the log softmax that should take care of it for us okay this is tensor
 flow okay following thread cross entropy loss I guess people just do really the
 log softmax and then do that and we should be fine with this okay thanks
 okay Frank yeah maybe maybe this has advanced since then so we can give like
 a last look at this and this is a bit too big I'm sorry your eyes are gonna
 have to suffer and we're going to look at loss functions and we're going to
 just look through them multi label soft margin loss hmm multi label we don't
 really want multi label right we want this but not with the targets okay I
 guess we're just gonna have to write this ourselves so ultimately what is the
 cross entropy that cross entropy is simply the probability of the true label
 times the log probability of the wrong or of the predicted label yeah as you
 see right here so we are going to simply multiply target times the log
 probability of the predicted label and then some some dot take that mean across
 the batch I guess yeah that should do we can implement this let's do it so this
 criterion right here is going to be our loss function and that's only used once
 so what we can do is going to be a function so we're going to take student
 logits and we're going to take teacher probabilities okay so how's that gonna
 work out we're going to do the log soft max from the student logits so and then
 dot does that exist log soft max functional okay we need functional and
 student logits of that dimension so now we have properly normalized student
 logit so that's going to be student log probes and then what we want to do is
 simply multiply the teacher probes times the student log probes and the negative
 of that is going to be our loss the question is do we want to sum that I
 guess across this dimension or mean it I guess the sum sum should do all right
 this is it easy as that why have we searched for so long so the criterion we
 can simply replace that now by our loss function cool so let's run it again
 yada yada yada okay so I need to check maybe we should have taken like a smaller
 model it sometimes pays off to you know start with a really small model small
 model just so you can you can do these kind of things fast so here we have
 dimension out of range to do okay which is where is that in forward in line 78
 let's go there line 78 okay here the max is not going to be as much fun let's
 go there I think this is some of these things change over time in pytorch so
 this code might be written when you know so what we have is we have a max over
 the predictions and predictions oh it's already an arg max so I guess we can
 remove this all whether or not that agrees with labels dot data we don't
 need any of that dot float and we also don't need that we so the accuracy is
 simply going to be the mean of this no I guess so we're here can't we just do
 predictions equals labels yes and we want the sum of that actually we want
 the float first and then we want the mean yeah that seems reasonable so let's
 do it like this yeah float mean perfect how could this be any more easy but this
 right here is all of it so validation step accuracy corrects we'll just look
 at it once we've done it and what I do is usually just run it until it doesn't
 give me any mistakes anymore and then I know I sort of have succeeded
 okay we're pretty close I feel so it says grad can implicitly create it only
 for scalar outputs which probably means our loss function is not a scalar so
 when we return the loss right here here we have the sum of the losses divided by
 the length of the losses let's go here and check out what's up with that so
 what will do I see this loss function here will output basically one loss for
 each data point so what we need to do I guess is call mean on this or some when
 they created the criterion in the original one now we've thrown it away
 look at the git diff right here so I guess this reduces how does this reduce
 the cross entropy loss when we don't do anything cross entropy loss reduces
 reduction mean okay so let's reduce with the mean so if we call loss then after
 that we should call mean and then here I'm not so sure anymore we should divide
 here because the learning rate is kind of tuned to the original loss size so I
 guess we'll be content for now with summing up these things and over here I
 guess we've we've solved that right no losses yeah see our losses is going to be
 an entire tensor and now we just fixed that right now okay so let's try it
 again in the meantime what can we do we can so we already take care of our GPU
 we take care of the logs one thing to do when with respect to this download
 stuff right here is you know if you have a server or something and you let a lot
 of things run in parallel what you want to do is make sure they don't all
 download the same stuff at the same time that's pretty bad so what you want to do
 is ideally have some sort of lock such that they coordinate and I usually use a
 file lock for this so I'm gonna create that right here from import the file
 lock and then I simply create the lock let's go file lock and here you have to
 input a file so sometimes like yeah data lock I don't know you just pick some
 file and that's the file that these these processes are going to sync on and
 then once you do this you simply wrap all of it in a with lock so only one at
 a time can go in in this function all right so that's that that should make us
 safe and right here we're now training this is excellent we are training the
 students now we need to do that on an actual GPU so I have multiple tools to
 ship this to a GPU so first of all I can try to ship this to and try to ship this
 to a let's say to to one GPU so the way you do that is first of all I want some
 sort of unbuffered version of Python and then I have this do I even have this
 tool I do have the tool okay so I'm gonna call one of our servers and we
 don't know what's going on okay so cannot import name seed everything from
 pytorch lightning seed everything is that some kind of new thing in pytorch
 lightning I guess I have it apparently so here seed everything with zero why I
 don't need that we are running without any seed here we are being really cool
 okay next mistake cannot okay still same mistake of course since we don't yep
 next mistake we'll just go through the mistake learning rate late learning rate
 loggers so I guess we need to update pytorch lightning on the servers and I'll
 do that quickly okay I've updated pytorch lightning let's check out whether
 or not we can actually run something yeah we can run something so this again
 is now downloading this on the server while this is happening there's another
 thing we can do namely I have sort of sort of made a system to run stuff on
 servers which I like a lot honestly so I guess we can try this out how do I
 hidden oh with I okay so I want to first of all delete this delete this yes I
 guess delete this cool and this git folder is a bit annoying let's restructure
 because otherwise it will always ship the git folder with everything up here
 does it do that yeah I don't like to have the code in a top level so quickly
 make a sources directory move everything in there so move the c410 models into
 the source directory move all the Python files into the source directory no clear
 the logs and we're much better much better okay much better so what we what
 what we will do is my system requires like a file and I'm just gonna copy one
 from another project quickly okay so we're back and I copied that over as you
 can see you basically give hyper parameters and it blasts ever the hyper
 parameters through in a kind of a random search fashion it's not too
 sophisticated but we can work with it so 10 that's the file right for 10 train
 ensemble yes that's the file cool and here we're just going to put all of our
 hyper parameters and that will remove the logs file I'm okay with that but
 want this
 bang cool so what do we want we want basically we just want to try it like a
 bunch of times and then see like average across it right that's all maybe we want
 the the architecture to change so let's say the classifier is a resnet 18 or a
 resnet 34 or a resnet 50 just so we have a bunch of stuff to do okay and this
 downloaded and is training on GPU hopefully if this works then we can ship
 this off and we'll make this other hyper parameter that I like to use called rep
 which is just basically a dummy parameter and so I can just repeat the
 experiment a bunch of times and let's put that in here so this is really that
 this has this has no effect except for randomizing it a bit I guess we can try
 to seed stuff so whenever it says seed everything we'll just seed it with this
 we'll call it seed no is it here this seed everything yeah so h params dot rep
 sorry seed cool what this this is doing something nice you can see it so this is
 unbuffered Python output thanks yeah so what other classifiers do we have we can
 again we can try a bunch of them we can try all of them but why don't we try all
 of them like this then let's go into this rat file I don't know why I called
 it rat I just want it like some three-letter thing so yep like this and
 then we can just take all of that crap and delete it and delete this and those
 are going to be all our models so our classifier is going to consist of all
 of this stuff let's I know I know I know I suck at them don't tell me actually
 tell me I want them tips trying to learn something new like each week in them but
 it is hard and tend to make myself actually do it so let's go let's go
 with just one repetition so far if we if we are not sure we can still up the
 number of repetitions we don't even have the rep right now this is called seed
 all right so we have different classifiers and what we're going to
 need we also have this num students right let's go with one with five and
 with 20 so here we got one epoch done and we get a validation loss do we get a
 validation accuracy validating validating I have no idea we'll cancel
 this right now and we'll go ahead and just blast this onto our servers and
 hopefully that that's gonna work I have no idea is everything fine everything's
 fine go no what what
 cool
 and let me get back to you once this is finished all right we're back so I've
 just written some code here to extract the results of that run and something
 you know it's pretty interesting what came out so in these plots you'll see on
 the x-axis of the number of students in the ensemble remember these students are
 all trained from the same teacher the teacher you can see in orange that's
 just the single teacher for reference you can see that if you have one student
 model it sometimes under performs or sometimes out performs the single
 teacher model but then if you have more student models you can see that there is
 a pretty monotonic relationship so here it's the reason this fit doesn't finish
 here is because there's not enough space on the GPU for that many student models
 but you can see that the relationship here is fairly monotonic here it's a bit
 of a kink so the first idea like this this is really astounding because these
 students have all been trained from that single teacher and they have been
 trained for as long as the teacher has been trained so they don't have more
 compute than the teacher they've been trained from scratch not from some
 checkpoint or from the teacher weights it's simple distillation from the
 teacher no labels and the students are all in parallel as well so they don't
 see different data or even different data augmentations it's the exact same
 order of the exact same data points going through all of the students the
 exact same learning rate schedule there's no noise and so on so the first
 thought that came to my mind like something fishy is going on here right
 like this this is this seems like to like come on there's no new information
 here so I thought hey I the teacher the teacher model I've just grabbed them
 from this from this repo from this pre-trained checkpoints and these
 pre-trained checkpoints they are you know the checkpoints that have performed
 best on the validation set so this is sort of a sneaky way of how we could
 train on the validation set right because we annotate each data point in
 the training data set with this checkpoint and the checkpoint has been
 selected for performing especially well on the validation data set it could
 explain why we get a gain on the validation data set so what I did is I
 retrained all of the teacher models such that I just retrained them for these 100
 epochs and I just took the last checkpoint the all everything's the same
 the hyper parameters learning rate schedule and so on this is not tuned for
 any particular model and it's pretty like it's pretty standard it's like it's
 not like 0.12589 it's like 0.01 and 100 epochs or so fairly standard
 parameters and I just took the last checkpoint to make it didn't even look at
 its performance to make sure that I didn't you know select something that
 was especially good on the validation data set and the results here you'll see
 are actually already the results of that run which the previous run it was almost
 the same like I was astounded how well it works and then I thought hey maybe
 I'm kind of you know cheating here so I redid it with the teachers that are not
 specifically selected and this is already the results so that's pretty
 cool right so then I wondered what happens if I now if I increase my
 training amount so I just let this run for more like what if I let the students
 run for more than the teacher has run again there's no new information here so
 you can see that the now the okay the green is now the teacher the blue is a
 hundred epochs and the orange is 250 epochs and you can see with that even
 one student will outperform the teacher but many students will outperform even
 more so if you give more compute there's lots of lots of headroom here to
 improve you'll see this here I think this last one with the blue line is just
 a bit of a weird a weird configuration I guess if you were to rerun that that
 would you know fall in line so this is pretty pretty weird right so I have a
 bunch of questions so first of all I've searched the literature a bit more and I
 came up with a number of papers that do things like this now usually when you do
 distillation you people stress the importance of like how to introduce
 noise like in the noisy student paper or that you really need these data
 augmentations or you know same clear V2 uses the self distillation in order to
 do in in order to label more data so they say it's important that we bring
 more unlabeled data into the process and so on so all of this it really doesn't
 match right here and especially this focus on we need noise during the
 distillation process to build these ensembles this is also you know if you
 know mean teacher things like this I also found a paper called born again
 neural networks that does something quite similar but not very simple not
 like the same where they distill a teacher to the student with the same
 architecture and then they distill the student again into another student and
 then that into another student and so on and then at the end they say oh we can
 also build an ensemble but sometimes their ensembles outperform their you
 know chain of distillation sometimes they don't they don't really focus on
 that part a lot and it's way more complicated like you distill one
 student after another and I also think they they have some introduction of
 variability in the students like like noise or different augmentations and so
 on so this here seems you know really really really simple now I want to know
 this ensemble effect it seems pretty pretty weird right so what gives so the
 first thing we could do is we could say what what does how does this compare to
 an ensemble of teacher models like if we actually were to build an ensemble like
 train five teacher models on you know five five different teacher models it's
 still the same data but reasonably they might be able to learn something more
 from the data if we have five teacher models they might learn different things
 from the data and therefore if we combine them they might kind of overlap
 their knowledge and sort of catch where if one doesn't generalize in one data
 point the other four can overrule it whereas with these student with these
 self ensembles there's not really a way where we can learn more from data
 because we can only learn from the teacher and the teacher is fixed and has
 seen that much data right so how does this compare so wrote some rewrote some
 code is it's just plumbing and I release the code it's linked but it's just
 plumbing don't worry don't worry there is no great thoughts in there it's just
 plumbing such that my students don't are not all in parallel so the ensembles are
 not trained in parallel anymore I train each model individually which means that
 at maximum I have to have two models on the same GPU one teacher and one student
 so I make sure that the teachers they are trained from scratch and the
 students they're always trained from the same teacher right so the student
 ensembles will be exactly the same as we have them here that means one teacher is
 responsible for all the students but yeah so okay I'll just show you the
 results right here so if we look at those results you can see that and I've
 done it for a bunch of models right here the blue line is the ensemble of
 teachers and here on the x-axis you see the number of models and now since I'm
 not training everything on the same GPU but I recombine later that that
 basically means that I have doubt the ability to train up to 10 models are
 actually however many I want and the only real trick in the code is that when
 I evaluate one of these ensembles what I do is I load a mini batch and then I
 basically load the first checkpoint run the forward pass load the second
 checkpoint run a forward pass load the third checkpoint run a forward pass I do
 this for all the checkpoints until I go to the next mini batch but that's just
 for evaluating right it just seemed easiest with the code that I had so you
 can see right here that the there is a significant like this is almost
 overlapping right here for most models there sometimes the student wins
 sometimes the teacher wins so the teacher ensemble wins now remember the
 teachers are trained on you know ten times as much data right here but it's
 always the same data but still they have the opportunity to learn ten times as
 much information from the data whereas the students they're all distilled from
 that same teacher without any noise any augmentate any augmentations except for
 the augmentations that you use during training anyway and I've done this for a
 hundred epochs and I've done this for 250 is this already 250 I think that was
 a hundred I just put that there nope okay yeah that was a hundred epochs but
 you'll see the 250 epoch plots they look very much the same okay they are
 just a bit better if you train for 250 epochs now interestingly okay here's the
 interesting part about the 250 epochs the student is still distilled from a
 teacher model that has been trained for a hundred epochs so all of this all of
 this makes no sense to me right the student is still distilled from the
 hundred epoch teacher model yet if you train the student for 250 epochs in
 self distillation and then build an ensemble of these students from that
 same teacher model and you compare that to an ensemble of teachers that have all
 been trained for longer for 250 epochs which you know should out it the 250
 epochs generally outperforms the hundred epochs models still they are the same
 this is this is pretty crazy results I I think and sort of my conclusion from
 this is that the ensemble effect right here is not a function of learning of
 extracting more information from the data the ensemble effect might actually
 be have something to do with the function landscape itself and kind of
 exploring different minima of the of the same function not of the same function
 but exploring different functions to describe the same phenomena and I've
 also found a paper that explains the lost landscape of deep ensembles and I
 will make a video on that maybe it's out already maybe it will be out after you
 see this one I I haven't decided yet which which order I'm going to release
 things but this here I it's it's pretty interesting and we need like a name so
 self self ensembles are already a thing but they are always with noise and stuff
 like this so let's call them something like plain self ensembles but that that
 that sounds like a good name plain self ensembles the act of self distillation a
 single model into multiple models without any noise any augmentations
 anything just you run as if you were to train the model itself and then you
 build an ensemble of these models by simply averaging the log it's plain
 self ensembles alright so the plan from here is to check on like at least one
 other data set you know these these models I appreciate that I could get
 them pre trained but they're just the image net models and then kind of let
 run on C for 10 so there's no kind of guarantee that these have been you know
 tuned or anything that the learning rates or whatnot so I want to take like
 an image net model you still make sure that I don't use any like hidden
 information where I could cheat on the validation set but try this on at least
 one thing and see if that works as well if we can sort of push image net
 performance simply by doing this trick so that's the plan for now and I have
 some other ideas but I just wanted to let you know and this is sort of how
 research works I guess you have a dumb idea and it turns out to work and then
 you go on and still probably probably there is not maybe too much interesting
 things here maybe it doesn't work on image net because these models are just
 under train and this somehow made them better somehow or regularize them
 somehow that usually doesn't work there's so much that can go wrong still
 so but yeah that was it and I invite you to like check out other papers in this
 space if you want it's a pretty interesting space and with that I don't
 have much more to say yeah I hope you enjoyed this let me know what you think
 of like research implementation or research process videos like this I'm
 not sure what people expect like I can't make this into five minute video of like
 whoo I discovered something because then you know there's no clue of what's
 what's happening but maybe like an hour or so is also too long I'm not sure yeah
 let me know what you think and I'll see you next time bye