Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper nas nao uber openai architecture search neural architecture search inner loop inner optimization small abstract turing performance evolutionary algorithm outer loop mlp sigmoid ptb rnn cell meta-learning
Neural Architecture Search is usually prohibitively expensive in both time and resources to be useful. A search strategy has to keep evaluating new models, training them to convergence in an inner loop to find out if they are any good. This paper proposes to abstract the problem and extract the essential part of the architecture to be optimized into a smaller version and evaluates that version on specifically custom learned data points to predict its performance, which is much faster and cheaper than running the full model.

OUTLINE:
0:00 - Intro & High-Level Overview
1:00 - Neural Architecture Search
4:30 - Predicting performance via architecture encoding
7:50 - Synthetic Petri Dish
12:50 - Motivating MNIST example
18:15 - Entire Algorithm
23:00 - Producing the synthetic data
26:00 - Combination with architecture search
27:30 - PTB RNN-Cell Experiment
29:20 - Comments & Conclusion

Paper: https://arxiv.org/abs/2005.13092
Code: https://github.com/uber-research/Synthetic-Petri-Dish

Abstract:
Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available. Our hope is that this work can inspire a new research direction in studying the performance of extracted components of models in an alternative controlled setting.

Authors: Aditya Rawal, Joel Lehman, Felipe Petroski Such, Jeff Clune, Kenneth O. Stanley

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at Synthetic Petri dish, a novel surrogate model for rapid architecture search by Adi Tarawal, Joel Lehman, Felipe Petroski-Sucs, Jeff Klun and Kenneth O. Stanley. This paper on a high level, it basically says if you want to do neural architecture search, if you for example search for a better non-linearity, you should be able to extract that non-linearity instantiated in a very small network and then evaluate that very small network in order to predict the performance of a large network and therefore you can find a better non-linearity in much less time. Now the exact procedure how you do this in the small network is the topic of this paper. As always if you like content like this I encourage you to subscribe if you are not already and to share out the video so other people can experience the joy themselves. Alright, let's dive in. So they say in the abstract, neural architecture search explores a large space of architectural motives. So it basically means you want to find a neural architecture, let's say you have a multi-layer perceptron right here, a couple of layers, okay, and they're all connected by you know feet forward weights whatnot and each of these weights basically is a multiplication. So each one of these is a multiplication of X by your weight W and then there is a non-linearity. So the non-linearity could be a sigmoid. So the sigmoid would be something like 1 over 1 plus e to the negative X. Now there's a bit of an extension in a sigmoid where you can do a sigmoid that has like a temperature parameter attached or a slope parameter where you go CX. So in one case you can set C such that the sigmoid has a shape like this and then if you put C to a different value you can make this slope, you can make it like a shape, well this is terrible, like this. You know what I mean. Okay so this this C right here can potentially change the behavior of your network and you want to find a good parameter C and this is a hyper parameter. Now there are many hyper parameters like this for example how many units you have in a particular layer in a CNN it could be your filter size in a transformer could be the number of heads and so on. It could actually be not only the slope of the non-linearity but the actual non-linearity itself or famously in recurrent neural networks you have these recurrent cells and they're like okay we have an input signal and a carry signal and then the input here is like dot multiplied here and then there is like a gate with a non-linearity and then it's kind of like multiplied by the carry but then there's also a like a forget gate and whatnot there's a minus right here. It's very complicated and so people do architecture search over these kind of problems to find better architectures for particular problems. Now the problem is of course that how do you know if a if a given architecture is good? What you have to do is you'll have to go take that cell that you have dreamed up you think well I think that's a good cell and you have to train it on the full training data set this is a data set a database right this is a full training data set then you need to evaluate it on your validation data set and then you have like a number you have like okay this is 8 good and then you go back and you say okay what if I change this cell here what if I change it to a plus instead of a minus and you do the entire thing again train it for I don't know how much validate it and then this is like a 9 and you can say oh cool that's a 9 so this is a very basic architecture search and there has been a lot of development in this space so like evolutionary search and so on but they most of the time they require pretty much evaluating the entire thing on the full data so you get a good you get a good estimate of what your final performance back here is going to be. Now people have come up with methods to counter that and they say well if we can sort of encode the cell structure let's go with the let's go with the RNN cell if we could encode the cell structure in in a sort of a continuous way so you know we can encode text in a continuous way let we could also encode a cell structure because the cell structure I can write it down as an equation I can say like okay it's the forget gate of the carry times the sigmoid output of x plus the so this is the plus here and plus the sigmoid output of x multiplied by the input let's call that I something like this right this is text I can like write it down and then I can encode that into a vector much I can for example build another RNN ironically or something to to encode that or I can represent it as a computation graph like it is here and use a graph neural network to encode that into a single vector and then I have sort of an embedding space where each cell that I could build is a point in that embedding space and then I can evaluate a couple of them I can for example say okay this one here this one here this one here this one here I'm going to train them these cells I'm going to do the full training eval and so on get their scores and then I can learn basically in this latent space I can learn a predictor I can say okay here I get I got an 8 I got a 9 I got a 2 and I got a 4 so it appears to be that in this direction that the good cells are in this direction and then I can do it again I can sample or I can do gradient descent in this space since this is now a continuous space and the gradient descent on the model that gives me this space so right so this this method basically tries to take in the building plan of a cell and learn to predict the performance just by looking at it if if you're thinking of the a Turing machine right now then you like I I immediately thought of like this this halting problem because it appears to be exactly what it is so you're trying to build a machine that takes the building plan of another machine and tries to predict its performance now in a general sense we can already state that this problem is sort of the difficulty of this problem is equivalent to the difficulty of the original problem so I'm not sure but it appears to you know it appears to work if you throw lots of compute at it but of course that's a problem you need lots of compute right so either your your option one is to run all of these things and kind of iterate them in a neural sorry in an evolutionary way or your second option is to take the building plan and predict the performance from that both are not satisfactory and both use lots of compute now neuro petri dish is a or synthetic petri dish is a way to combine the two together it says can't we take the building plan here but actually run on the data on data to predict the performance so what they're saying is basically if I have this cell right here and usually this cell you know it deals with vectors of let's say size 512 and so on it will say it since this is let me draw it again up here so here you have the cell and here you have somehow the connections in there when you carry and the input and the input here okay and this is the output or the carry I have 512 embedding like size of this so this is a giant cell there's 512 the vector has 512 dimensions going in basically can't I take the exact same thing but and keep the connection pattern so I would keep the entire pattern of connection right here but I only do it for one or two so this is 512 and this is just two right just the I just reduce the dimensionality but I sort of keep the connection pattern alive if I only do that I have like a very small network right now and the same goes for if this is so a lot of times these RNNs they have multiple layers of these things so they have another exactly equal box up here and then another up here I can just reduce this to one layer and out of the regularity of these neural network things it is known that or one can make the assumption that the performance on this thing will sort of kind of be correlated to the performance of the entire thing and that's one of the things that this petri dish paper does so we try to take out what we are trying to search over namely the connection pattern we keep that as it is up here but we reduce everything else we reduce the dimensionality we reduce the number of layers and so on now they don't actually reduce the number of layers here but you can reduce the number of units and so on okay so this in essence this works whenever you can do that whenever you can keep the structure you're searching over but can reduce the rest so that's one precondition doesn't work for everything the second part here is that you don't want to use this particular training data and then this particular validation data because first of all it's a lot of training data and second of all it won't give you that good of a prediction instead of what you're trying to do and this is the second part of the idea of petri dish is you're trying to abstract the training data to get you a very small data set search that and the validation data as well as well search that if you train on this data and evaluate on this data the performance that you get will be very predictive of the performance had you trained the big model on the big data set okay in fact in this petri dish paper these little data sets they have nothing to do with the with the original train and validation data and that's I think that's one of the cool things here these things this training data and this validation data they are optimized as well by the procedure they're optimized special data points that are trained these are trained parameters such that if you train on the training on the small training data and evaluate on the small eval data the you will be able to predict this performance back here with high accuracy okay and this I think is where previous approaches have or might have failed because it's you know the the idea of scaling down your network in order to do the architecture search is probably has it has appeared to many people before that's not you know that's not really genius idea but probably they have all found that now we can't really do it it doesn't really give us accurate enough numbers but in this case the addition of adding these synthetic data sets that are much smaller but can still if you train and evaluate on them can still predict with high accuracy the full score of the full model that I think makes this idea work alright so I guess we're already through the idea and problem setting and everything without actually reading the paper so they give this example right here at the beginning that if you have a a two layer 100 wide so 100 dimensional MNIST networks it's two layers it's I think it's an MLP it's two layer MLP with a non-linearity that is this the sigmoid right here this okay now you can see it has this temperature parameter here it has this slope parameter and you you want to do neural architecture search to find the best slope parameter now usually you would just do a grid search but this is an example because this can be of course much higher dimensional things and then you don't want to do grid search anymore okay so what do we do if you look at how the 100 wide MNIST network so we can draw it right here so this is a 100 dimensional MNIST network so this is 100 and each cell each connection here first has a weight and then has the sigmoid non-linearity and the sigmoid non- linearity is parameterized by the parameter C okay and you have you have many of them right you have one here and so on and each one has a different C and each of these networks represents one blue dot here so if you let C vary so this sigmoid slope value right here that's your parameter C if you let this variant train the big network on the entire data set to convergence and then you eval on the validation data set you get the slope like the blue curve so if you see the blue curve the blue curve is if you start over here if you reduce this slope you'll gain in performance but if you reduce it too much you drop drastically okay until it's if it's if it's zero it's basically you know the the X is not the signal doesn't propagate anymore and you you have no learning occurring okay so that's the original performance now what if I only give you training data in this range right here I only showed you this particular range I can't actually zoom in that much but if I give you this and I ask you to build one of these please take the architecture and predict the performance that we saw at the beginning like one of these girdle machines or touring touring machines you would basically say well that looks to me like a line so I'm gonna predict the red thing here and even if you can you know evaluate a bunch of these it just looks like a line and you're going to predict there's probably a slope like this right this happens almost independently of which model you choose to predict right here the data of training is simply doesn't give away that the fact that there is a there is this break down here which happens in the real world so if you just give this as training data there's no way so so the the the criticism about these models is valid that they will only work where you give them training data they can at best interpolate their training data but they can't really extrapolate now here since the synthetic petri dish method which is the green thing here uses the actual not the actual non-linearity that this thing characterizes so it it instantiates the sigmoid with the parameter C that you give it just not on the large network but on a small network in fact their network is just one unit sorry one unit and then another unit so it's just a two hidden layer but just with one unit instead of 100 and of course you can't feed in M nest right here right but we said they don't feed in the data they actually feed in their synthetic data that they learn so you give them the points here and they learn the synthetic data to to evaluate to evaluate the others and then once you ask them well if if my C is right here what's the performance going to be it's going to instantiate that in its small network it is going to use the training data that it has learned from this region right here in order to train this and then it's going to evaluate this on the synthetic validation data that is also learned on the training data and it is going to come up with a performance metric it says okay this is how good it's going to be and since it is an approximation in its building plan to the entire network it will react similarly so it will get that there is this performance dip right here okay so it you can see how this sort of makes sense you are actually running an approximation to the actual program instead of just looking at the plan of the program and trying to predict it which you know halting problem says hello okay so that is the motivating example of their MNIST thing and here is the entire algorithm all right so you take MNIST training and validation data and you instantiate a bunch of really big networks this is ground truth okay you you need this you need this to learn from you instantiate a bunch of really big networks now if I draw the graph from before right we had this was the performance of the actual networks you want you this comes from here from this region right here this is the training data okay so you instantiate a bunch of these networks each one you instantiate in one of them right each one gives rise to a different non-linearity and you do the full training ground truth training and evaluation on the full training set and the full validation set and you get validation losses right for each of these and these are the points right here now you that's the training data for your neural for your neural architecture search so for your petri dish method what the petri dish does is it says it extracts the motive and the motive is the thing that you optimize over so as I said you want to keep that thing in its essence but you want to reduce everything else so it reduces it instead of from a two layer on hundred wide MLP it reduces that to a two layer single neuron wide MLP okay and it now this over here is the training data for the procedure that we're going to do now so what it would take is it would take it would take one of these values it would instantiate and we have that here would instantiate the neural network in the small form of that and now we know that if I train the full data and evaluate if I train on the full training data and evaluate on the full validation data I should get this accuracy all right so I will create and we're going to look at in a second I will create training and validation data such that if I train on this training data and then validate on this validation data I get the same validation loss as if I had trained the big network with the same you know the same C parameter on the full training data and evaluate in the full validation data okay so in this step I'm optimizing the data here the training and validation data all right and now in the second step once I have this training and validation data such like that I can basically reproduce this this graph right here then I can go and actually ask my model okay now please tell me what happens over here so what am I gonna do I'm gonna take that I'm gonna instantiate it I'm going to use my training data that I learned to train it when to use my validation data that I learned to evaluate it and it's gonna give me a number and that number is going to be like close to hopefully close to do this so this is how we can extrapolate using that method okay now there are a number of assumptions right here and you can imagine this doesn't work in any situation this works if if you you know if you basically you have to get lucky in that you have to abstract the correct things right I said you need to reduce everything else so they reduce notably you see they reduce the 100 the 100 layer width to a single neuron wide MLP and they sort of guess that that doesn't change the fundamental thing but you can also see they leave the two layer right they leave the two layer neural network and I'm can almost guarantee you that they tried this reducing this to a one layer neural network and it did not work and so you have to be sort of very careful of what quantities you abstract and what quantities you don't because okay now you might always think oh I can reduce the you know number of dimensions or channels that's also not always the case so I think that's kind of the crux of the method you have to actually engineer this down compressing of the architecture such that it's its properties are still kept and yeah but yeah in other things how do you how do you actually produce training and validation data to match these and there are a number of ways but what comes to mind is is meta learning right so because what you're doing they initialize the training and validation data at random points so these are just random at the beginning and then they optimize the data itself using gradient descent okay now see synthetic training data and they are randomly initialized okay and they use gradient descent they have it somewhere yes so they have this inner training loop okay which is many steps of inner training and then they have the outer loss which is the it's the validation loss after the inner training loop and the difference for that to the true validation loss and then they do gradient descent on this outer loss now this outer loss is a result of the inner loss and the inner loss is a result of the inner training procedure and the inner training procedure is n steps of feeding in the training data every step you feed in the training data so your computational graph is going to look like so here's your training data as train and here are your initial parameters you at random lies initialize them randomly in the first step you use the training data to produce theta one then in the second step you use your training perhaps your training data again to produce theta 2 and then you use it again to use data for me and so on each time you feed the training data in order to evolve your parameters to give you a better prediction right so the gradient since somewhere back here there's a loss the gradient here will have to flow back through all of these paths and through all of these connections to the training data this is kind of you back propagate through an optimization procedure and we have this a bunch of times here and I've looked at the code and the code is like really crazy and it looks like proper research code but it appears to be that that's actually what's happening they backprop through the optimization procedure to find this synthetic training and validation data now that's I mean that's crazy but it also kind of limits how far you can go with this because usually you can't backprop for more than a couple of steps doing this now that the model the fact that the model inner model is small helps but also this introduces very very much like these things are very brittle if you backprop through an optimization procedure like this these things tend to be very brittle and so I think there's another thing there where you have to pay careful attention alright that's it's basically it the last thing they say is that they can combine this with architecture search in that so not only can you predict good architectures what you can do is you can actually predict the which architectures are good and then you can use that prediction to get new to basically input this into your neural architecture search to inform it so instead of the neural architecture search having to evaluate all of the candidates that it produces it only has to now evaluate the very small subset of candidates that the synthetic petri dish training deems most worthy of being evaluated in this case here instead of evaluating all of the things here it would limit itself to whatever the synthetic petri dish says are the highest performing ones because if the synthetic petri dish is any good then it will you know give accurate predictions of how they're performing and then that can go in in multiple rounds so the architecture search can find new come up with new things that it thinks are better through like an evolutionary mutation algorithm the petri dish can evaluate them in the synthetic way and then suggest the like 10 candidates to evaluate on the full test set and you that way you don't have to evaluate all the like thousand candidates alright alright cool they do this for this M nest and they also do it for finding a RNN cell for the pen tree bank this is a language modeling task and the this is a benchmark for neural architecture search where you're trying to find a good RNN cell to get the perplexity really low and here you can see if they give the same amount of data to all the methods then the benchmark neural architecture search is worse than the synthetic petri dish informed architecture search now one has to say on the full date I believe the NAO gets to about here but of course if you give all of them the same data the neural the petri dish beats this method and I think still this method here uses way more compute because it always has to evaluate all the candidates and that's exactly one of these where I learn an architecture to predict the other architecture by just looking at it so it works but it doesn't work as well as actually running the architecture in an abstract fashion this also shows you the importance of selecting your experimental evaluation in a smart way like they argue they argue for very long why it makes sense to evaluate everything on reduced data such that their method here can be better and they don't have to compare to the full thing it's easier for them to work on reduced data and they argue you know it's it's it's what people usually do in practice and that's the task they focus on so you know good good good good paper writing right here yeah that's basically it to the paper there's a lot of things to be said here I think this works in very very limited settings it seems to me that it's sort of brittle with respect to how you abstract and also it it's always the case like how many how how large is this synthetic training data in their case there they like abstract this to 20 or 30 data points or something like this so it seems to me since you're optimizing this training data with gradient descent what you would mainly find are adversarial sort of adversarial examples to this architecture here so I'm going to guess that the inner optimization is very noisy and that's because if you really let your optimizer run then it will abuse every single thing it can to match that validation loss and that will usually lead to an adversarial example since you're optimizing the data itself okay so I think this suffers from that and this is we had this in the in the planning you know planning in in learned world models in reinforcement learning where if you have a really really good planner it will just abuse the mistakes that you make in approximating the true world and the same here you're going to make mistakes approximating this architecture here and the better your your optimizer is for producing this synthetic data the probably the worse the worse the result is going to match the worst that these losses are going to actually match now okay these losses will match because they're that's what you train for but the worst these two curves will match each other because now you're just finding adversarial examples for your particular training data another concern I have here is with respect to the double descent phenomenon so if you know the double descent phenomenon if here you have your number of parameters and here you have your validation loss let's say and you know that if I add parameters I can make my validation loss go down so this is assuming I have a model with p parameters and I always train it on the train data to like to convergence if I add parameters I can generalize better until a point where I add too many parameters and I start overfitting and my validation loss goes up again but the double descent phenomenon and I think I've done a video on this shows that after a certain threshold you get the interpolation threshold the validation loss goes actually down again it goes down even further here now I'm so this is a very strange phenomenon by itself but I'm sort of concerned that if you do this abstraction that this paper proposes so you read your let's say your full model is here with a large number of parameters so it is past this interpolation threshold if you now seriously reduce the number of parameters because you want to go into this petri dish you will get maybe you will cross this interpolation threshold and actually be on this side of the curve right here now of course at the same time you reduce the amount of data which would push you over here again but it is different data so I'm not sure how all of this is going to play out it appears to work in these settings right here but I I think this is it's it's sort of it's sort of applicable in some situations and it's it'd be very cool if we develop this further such that we understand when it applies and when we can use it because I feel this can be a very cool thing if we understand it better and if we can apply it throughout alright that's the end if you like this paper leave a comment if you didn't like it leave a comment and bye bye see you next time
 Hi there! Today we're looking at Synthetic Petri dish, a novel surrogate model for
 rapid architecture search by Adi Tarawal, Joel Lehman, Felipe Petroski-Sucs, Jeff
 Klun and Kenneth O. Stanley. This paper on a high level, it basically says if you
 want to do neural architecture search, if you for example search for a better
 non-linearity, you should be able to extract that non-linearity instantiated
 in a very small network and then evaluate that very small network in order
 to predict the performance of a large network and therefore you can find a
 better non-linearity in much less time. Now the exact procedure how you do this
 in the small network is the topic of this paper. As always if you like content
 like this I encourage you to subscribe if you are not already and to share out
 the video so other people can experience the joy themselves. Alright, let's dive
 in. So they say in the abstract, neural architecture search explores a large
 space of architectural motives. So it basically means you want to find
 a neural architecture, let's say you have a multi-layer perceptron right here, a
 couple of layers, okay, and they're all connected by you know feet forward
 weights whatnot and each of these weights basically is a multiplication. So
 each one of these is a multiplication of X by your weight W and then there is a
 non-linearity. So the non-linearity could be a sigmoid. So the sigmoid would be
 something like 1 over 1 plus e to the negative X. Now there's a bit of an
 extension in a sigmoid where you can do a sigmoid that has like a temperature
 parameter attached or a slope parameter where you go CX. So in one case you can
 set C such that the sigmoid has a shape like this and then if you put C to a
 different value you can make this slope, you can make it like a shape, well this
 is terrible, like this. You know what I mean. Okay so this this C right here can
 potentially change the behavior of your network and you want to find a good
 parameter C and this is a hyper parameter. Now there are many hyper
 parameters like this for example how many units you have in a particular layer
 in a CNN it could be your filter size in a transformer could be the number of
 heads and so on. It could actually be not only the slope of the non-linearity but
 the actual non-linearity itself or famously in recurrent neural networks
 you have these recurrent cells and they're like okay we have an input
 signal and a carry signal and then the input here is like dot multiplied
 here and then there is like a gate with a non-linearity and then it's kind of
 like multiplied by the carry but then there's also a like a forget gate and
 whatnot there's a minus right here. It's very complicated and so people do
 architecture search over these kind of problems to find better architectures
 for particular problems. Now the problem is of course that how do you know if a
 if a given architecture is good? What you have to do is you'll have to go take
 that cell that you have dreamed up you think well I think that's a good cell
 and you have to train it on the full training data set this is a data set a
 database right this is a full training data set then you need to evaluate it on
 your validation data set and then you have like a number you have like okay
 this is 8 good and then you go back and you say okay what if I change this cell
 here what if I change it to a plus instead of a minus and you do the entire
 thing again train it for I don't know how much validate it and then this is
 like a 9 and you can say oh cool that's a 9 so this is a very basic architecture
 search and there has been a lot of development in this space so like
 evolutionary search and so on but they most of the time they require pretty
 much evaluating the entire thing on the full data so you get a good you get a
 good estimate of what your final performance back here is going to be. Now
 people have come up with methods to counter that and they say well if we can
 sort of encode the cell structure let's go with the let's go with the RNN cell
 if we could encode the cell structure in in a sort of a continuous way so you
 know we can encode text in a continuous way let we could also encode a cell
 structure because the cell structure I can write it down as an equation I can
 say like okay it's the forget gate of the carry times the sigmoid output of x
 plus the so this is the plus here and plus the sigmoid output of x multiplied
 by the input let's call that I something like this right this is text I can like
 write it down and then I can encode that into a vector much I can for example
 build another RNN ironically or something to to encode that or I can
 represent it as a computation graph like it is here and use a graph neural
 network to encode that into a single vector and then I have sort of an
 embedding space where each cell that I could build is a point in that embedding
 space and then I can evaluate a couple of them I can for example say okay this
 one here this one here this one here this one here I'm going to train them
 these cells I'm going to do the full training eval and so on get their scores
 and then I can learn basically in this latent space I can learn a predictor I
 can say okay here I get I got an 8 I got a 9 I got a 2 and I got a 4 so it
 appears to be that in this direction that the good cells are in this direction
 and then I can do it again I can sample or I can do gradient descent in this
 space since this is now a continuous space and the gradient descent on the
 model that gives me this space so right so this this method basically tries to
 take in the building plan of a cell and learn to predict the performance just by
 looking at it if if you're thinking of the a Turing machine right now then you
 like I I immediately thought of like this this halting problem because it
 appears to be exactly what it is so you're trying to build a machine that
 takes the building plan of another machine and tries to predict its
 performance now in a general sense we can already state that this problem is
 sort of the difficulty of this problem is equivalent to the difficulty of the
 original problem so I'm not sure but it appears to you know it appears to work
 if you throw lots of compute at it but of course that's a problem you need lots
 of compute right so either your your option one is to run all of these things
 and kind of iterate them in a neural sorry in an evolutionary way or your
 second option is to take the building plan and predict the performance from
 that both are not satisfactory and both use lots of compute now neuro petri dish
 is a or synthetic petri dish is a way to combine the two together it says can't
 we take the building plan here but actually run on the data on data to
 predict the performance so what they're saying is basically if I have this cell
 right here and usually this cell you know it deals with vectors of let's say
 size 512 and so on it will say it since this is let me draw it again up here so
 here you have the cell and here you have somehow the connections in there when you
 carry and the input and the input here okay and this is the output or the carry
 I have 512 embedding like size of this so this is a giant cell there's 512 the
 vector has 512 dimensions going in basically can't I take the exact same
 thing but and keep the connection pattern so I would keep the entire
 pattern of connection right here but I only do it for one or two so this is 512
 and this is just two right just the I just reduce the dimensionality but I
 sort of keep the connection pattern alive if I only do that I have like a
 very small network right now and the same goes for if this is so a lot of
 times these RNNs they have multiple layers of these things so they have
 another exactly equal box up here and then another up here I can just reduce
 this to one layer and out of the regularity of these neural network
 things it is known that or one can make the assumption that the performance on
 this thing will sort of kind of be correlated to the performance of the
 entire thing and that's one of the things that this petri dish paper does
 so we try to take out what we are trying to search over namely the connection
 pattern we keep that as it is up here but we reduce everything else we reduce
 the dimensionality we reduce the number of layers and so on now they don't
 actually reduce the number of layers here but you can reduce the number of
 units and so on okay so this in essence this works whenever you can do that
 whenever you can keep the structure you're searching over but can reduce
 the rest so that's one precondition doesn't work for everything the second
 part here is that you don't want to use this particular training data and then
 this particular validation data because first of all it's a lot of training data
 and second of all it won't give you that good of a prediction instead of what
 you're trying to do and this is the second part of the idea of petri dish is
 you're trying to abstract the training data to get you a very small data set
 search that and the validation data as well as well search that if you train
 on this data and evaluate on this data the performance that you get will be
 very predictive of the performance had you trained the big model on the big
 data set okay in fact in this petri dish paper these little data sets they have
 nothing to do with the with the original train and validation data and that's I
 think that's one of the cool things here these things this training data and this
 validation data they are optimized as well by the procedure they're optimized
 special data points that are trained these are trained parameters such that
 if you train on the training on the small training data and evaluate on the
 small eval data the you will be able to predict this performance back here with
 high accuracy okay and this I think is where previous approaches have or might
 have failed because it's you know the the idea of scaling down your network in
 order to do the architecture search is probably has it has appeared to many
 people before that's not you know that's not really genius idea but probably they
 have all found that now we can't really do it it doesn't really give us accurate
 enough numbers but in this case the addition of adding these synthetic data
 sets that are much smaller but can still if you train and evaluate on them can
 still predict with high accuracy the full score of the full model that I think
 makes this idea work alright so I guess we're already through the idea and
 problem setting and everything without actually reading the paper so they give
 this example right here at the beginning that if you have a a two layer 100 wide
 so 100 dimensional MNIST networks it's two layers it's I think it's an MLP
 it's two layer MLP with a non-linearity that is this the sigmoid right here this
 okay now you can see it has this temperature parameter here it has this
 slope parameter and you you want to do neural architecture search to find the
 best slope parameter now usually you would just do a grid search but this is
 an example because this can be of course much higher dimensional things and
 then you don't want to do grid search anymore okay so what do we do if you look
 at how the 100 wide MNIST network so we can draw it right here so this is a 100
 dimensional MNIST network so this is 100 and each cell each connection here first
 has a weight and then has the sigmoid non-linearity and the sigmoid non-
 linearity is parameterized by the parameter C okay and you have you have
 many of them right you have one here and so on and each one has a different C and
 each of these networks represents one blue dot here so if you let C vary so
 this sigmoid slope value right here that's your parameter C if you let this
 variant train the big network on the entire data set to convergence and then
 you eval on the validation data set you get the slope like the blue curve so if
 you see the blue curve the blue curve is if you start over here if you reduce
 this slope you'll gain in performance but if you reduce it too much you drop
 drastically okay until it's if it's if it's zero it's basically you know the
 the X is not the signal doesn't propagate anymore and you you have no
 learning occurring okay so that's the original performance now what if I only
 give you training data in this range right here I only showed you this
 particular range I can't actually zoom in that much but if I give you
 this and I ask you to build one of these please take the architecture and predict
 the performance that we saw at the beginning like one of these girdle
 machines or touring touring machines you would basically say well that looks
 to me like a line so I'm gonna predict the red thing here and even if you can
 you know evaluate a bunch of these it just looks like a line and you're going
 to predict there's probably a slope like this right this happens almost
 independently of which model you choose to predict right here the data of
 training is simply doesn't give away that the fact that there is a there is
 this break down here which happens in the real world so if you just give this
 as training data there's no way so so the the the criticism about these
 models is valid that they will only work where you give them training data they
 can at best interpolate their training data but they can't really extrapolate
 now here since the synthetic petri dish method which is the green thing here
 uses the actual not the actual non-linearity that this thing
 characterizes so it it instantiates the sigmoid with the parameter C that you
 give it just not on the large network but on a small network in fact their
 network is just one unit sorry one unit and then another unit so it's just a two
 hidden layer but just with one unit instead of 100 and of course you can't
 feed in M nest right here right but we said they don't feed in the data they
 actually feed in their synthetic data that they learn so you give them the
 points here and they learn the synthetic data to to evaluate to evaluate the
 others and then once you ask them well if if my C is right here what's the
 performance going to be it's going to instantiate that in its small network it
 is going to use the training data that it has learned from this region right
 here in order to train this and then it's going to evaluate this on the
 synthetic validation data that is also learned on the training data and it is
 going to come up with a performance metric it says okay this is how good it's
 going to be and since it is an approximation in its building plan to the
 entire network it will react similarly so it will get that there is this
 performance dip right here okay so it you can see how this sort of makes sense
 you are actually running an approximation to the actual program
 instead of just looking at the plan of the program and trying to predict it
 which you know halting problem says hello okay so that is the motivating
 example of their MNIST thing and here is the entire algorithm all right so you
 take MNIST training and validation data and you instantiate a bunch of really
 big networks this is ground truth okay you you need this you need this to learn
 from you instantiate a bunch of really big networks now if I draw the graph
 from before right we had this was the performance of the actual networks you
 want you this comes from here from this region right here this is the training
 data okay so you instantiate a bunch of these networks each one you instantiate
 in one of them right each one gives rise to a different non-linearity and you do
 the full training ground truth training and evaluation on the full training set
 and the full validation set and you get validation losses right for each of
 these and these are the points right here now you that's the training data
 for your neural for your neural architecture search so for your petri
 dish method what the petri dish does is it says it extracts the motive and the
 motive is the thing that you optimize over so as I said you want to keep that
 thing in its essence but you want to reduce everything else so it reduces it
 instead of from a two layer on hundred wide MLP it reduces that to a two
 layer single neuron wide MLP okay and it now this over here is the training data
 for the procedure that we're going to do now so what it would take is it would
 take it would take one of these values it would instantiate and we have that
 here would instantiate the neural network in the small form of that and
 now we know that if I train the full data and evaluate if I train on the full
 training data and evaluate on the full validation data I should get this
 accuracy all right so I will create and we're going to look at in a second I
 will create training and validation data such that if I train on this training
 data and then validate on this validation data I get the same validation
 loss as if I had trained the big network with the same you know the same C
 parameter on the full training data and evaluate in the full validation data
 okay so in this step I'm optimizing the data here the training and validation
 data all right and now in the second step once I have this training and
 validation data such like that I can basically reproduce this this graph
 right here then I can go and actually ask my model okay now please tell me
 what happens over here so what am I gonna do I'm gonna take that I'm gonna
 instantiate it I'm going to use my training data that I learned to train it
 when to use my validation data that I learned to evaluate it and it's gonna
 give me a number and that number is going to be like close to hopefully
 close to do this so this is how we can extrapolate using that method okay now
 there are a number of assumptions right here and you can imagine this doesn't
 work in any situation this works if if you you know if you basically you have to
 get lucky in that you have to abstract the correct things right I said you need
 to reduce everything else so they reduce notably you see they reduce the 100 the
 100 layer width to a single neuron wide MLP and they sort of guess that that
 doesn't change the fundamental thing but you can also see they leave the two
 layer right they leave the two layer neural network and I'm can almost
 guarantee you that they tried this reducing this to a one layer neural
 network and it did not work and so you have to be sort of very careful of what
 quantities you abstract and what quantities you don't because okay now
 you might always think oh I can reduce the you know number of dimensions or
 channels that's also not always the case so I think that's kind of the crux of
 the method you have to actually engineer this down compressing of the
 architecture such that it's its properties are still kept and yeah but
 yeah in other things how do you how do you actually produce training and
 validation data to match these and there are a number of ways but what comes to
 mind is is meta learning right so because what you're doing they initialize
 the training and validation data at random points so these are just random
 at the beginning and then they optimize the data itself using gradient descent
 okay now see synthetic training data and they are randomly initialized okay and
 they use gradient descent they have it somewhere yes so they have this inner
 training loop okay which is many steps of inner training and then they have the
 outer loss which is the it's the validation loss after the inner training
 loop and the difference for that to the true validation loss and then they do
 gradient descent on this outer loss now this outer loss is a result of the inner
 loss and the inner loss is a result of the inner training procedure and the
 inner training procedure is n steps of feeding in the training data every step
 you feed in the training data so your computational graph is going to look
 like so here's your training data as train and here are your initial
 parameters you at random lies initialize them randomly in the first
 step you use the training data to produce theta one then in the second
 step you use your training perhaps your training data again to produce theta 2
 and then you use it again to use data for me and so on each time you feed the
 training data in order to evolve your parameters to give you a better
 prediction right so the gradient since somewhere back here there's a loss the
 gradient here will have to flow back through all of these paths and through
 all of these connections to the training data this is kind of you back propagate
 through an optimization procedure and we have this a bunch of times here and I've
 looked at the code and the code is like really crazy and it looks like proper
 research code but it appears to be that that's actually what's happening they
 backprop through the optimization procedure to find this synthetic training
 and validation data now that's I mean that's crazy but it also kind of limits
 how far you can go with this because usually you can't backprop for more
 than a couple of steps doing this now that the model the fact that the model
 inner model is small helps but also this introduces very very much like these
 things are very brittle if you backprop through an optimization procedure like
 this these things tend to be very brittle and so I think there's another
 thing there where you have to pay careful attention alright that's it's
 basically it the last thing they say is that they can combine this with
 architecture search in that so not only can you predict good architectures what
 you can do is you can actually predict the which architectures are good and
 then you can use that prediction to get new to basically input this into your
 neural architecture search to inform it so instead of the neural architecture
 search having to evaluate all of the candidates that it produces it only has
 to now evaluate the very small subset of candidates that the synthetic petri dish
 training deems most worthy of being evaluated in this case here instead of
 evaluating all of the things here it would limit itself to whatever the
 synthetic petri dish says are the highest performing ones because if the
 synthetic petri dish is any good then it will you know give accurate predictions
 of how they're performing and then that can go in in multiple rounds so the
 architecture search can find new come up with new things that it thinks are
 better through like an evolutionary mutation algorithm the petri dish can
 evaluate them in the synthetic way and then suggest the like 10 candidates to
 evaluate on the full test set and you that way you don't have to evaluate all
 the like thousand candidates alright alright cool they do this for this M
 nest and they also do it for finding a RNN cell for the pen tree bank this is
 a language modeling task and the this is a benchmark for neural architecture
 search where you're trying to find a good RNN cell to get the perplexity
 really low and here you can see if they give the same amount of data to all the
 methods then the benchmark neural architecture search is worse than the
 synthetic petri dish informed architecture search now one has to say
 on the full date I believe the NAO gets to about here but of course if you give
 all of them the same data the neural the petri dish beats this method and I think
 still this method here uses way more compute because it always has to evaluate
 all the candidates and that's exactly one of these where I learn an
 architecture to predict the other architecture by just looking at it so it
 works but it doesn't work as well as actually running the architecture in an
 abstract fashion this also shows you the importance of selecting your
 experimental evaluation in a smart way like they argue they argue for very long
 why it makes sense to evaluate everything on reduced data such that
 their method here can be better and they don't have to compare to the full thing
 it's easier for them to work on reduced data and they argue you know it's it's
 it's what people usually do in practice and that's the task they focus on so
 you know good good good good paper writing right here yeah that's basically
 it to the paper there's a lot of things to be said here I think this works in
 very very limited settings it seems to me that it's sort of brittle with
 respect to how you abstract and also it it's always the case like how many how
 how large is this synthetic training data in their case there they like
 abstract this to 20 or 30 data points or something like this so it seems to me
 since you're optimizing this training data with gradient descent what you
 would mainly find are adversarial sort of adversarial examples to this
 architecture here so I'm going to guess that the inner optimization is very
 noisy and that's because if you really let your optimizer run then it will
 abuse every single thing it can to match that validation loss and that will
 usually lead to an adversarial example since you're optimizing the data itself
 okay so I think this suffers from that and this is we had this in the in the
 planning you know planning in in learned world models in reinforcement learning
 where if you have a really really good planner it will just abuse the mistakes
 that you make in approximating the true world and the same here you're going to
 make mistakes approximating this architecture here and the better your
 your optimizer is for producing this synthetic data the probably the worse
 the worse the result is going to match the worst that these losses are going to
 actually match now okay these losses will match because they're that's what
 you train for but the worst these two curves will match each other because now
 you're just finding adversarial examples for your particular training data
 another concern I have here is with respect to the double descent phenomenon
 so if you know the double descent phenomenon if here you have your number
 of parameters and here you have your validation loss let's say and you know
 that if I add parameters I can make my validation loss go down so this is
 assuming I have a model with p parameters and I always train it on the
 train data to like to convergence if I add parameters I can generalize better
 until a point where I add too many parameters and I start overfitting and
 my validation loss goes up again but the double descent phenomenon and I think
 I've done a video on this shows that after a certain threshold you get the
 interpolation threshold the validation loss goes actually down again it goes
 down even further here now I'm so this is a very strange phenomenon by itself
 but I'm sort of concerned that if you do this abstraction that this paper
 proposes so you read your let's say your full model is here with a large number
 of parameters so it is past this interpolation threshold if you now
 seriously reduce the number of parameters because you want to go into
 this petri dish you will get maybe you will cross this interpolation threshold
 and actually be on this side of the curve right here now of course at the
 same time you reduce the amount of data which would push you over here again but
 it is different data so I'm not sure how all of this is going to play out it
 appears to work in these settings right here but I I think this is it's it's
 sort of it's sort of applicable in some situations and it's it'd be very cool if
 we develop this further such that we understand when it applies and when we
 can use it because I feel this can be a very cool thing if we understand it
 better and if we can apply it throughout alright that's the end if you like this
 paper leave a comment if you didn't like it leave a comment and bye bye see you
 next time