Conversation about Population-Based Methods (Re-upload)
machine learning ai artificial intelligence open ended learning quality diversity conference icml icml2019 tutorial population-based search goal switching serendipidy evolution interview podcast
Being interviewed by Connor Shorten of Henry AI Labs (https://www.youtube.com/channel/UCHB9VepY6kYvZjj0Bgxnpbw) on the topic of population-based methods and open-ended learning.

Tutorial: https://www.facebook.com/icml.imls/videos/481758745967365/
Book: https://www.amazon.com/dp/B00X57B4JG/
 Hi there, I've recently been interviewed by the YouTube channel Henry AI Labs by Connor Shorten and what follows is the resulting conversation we had about population-based methods and Open-ended learning things like that basically topics of the ICML tutorial that we both saw It's important to note that none of us is really an expert on the topic but we are trying to make sense of it and mainly just kind of talking about the ideas So please enjoy the conversation with Connor Shorten definitely check out the Henry AI Labs channel and Now have a good time Thanks for watching the Henry AI Labs deep learning podcast today. I'm joined with Janek Kilcher Janek works in the data analytics lab at ETH. He has a great YouTube channel I really enjoy watching his paper summary videos If you like any of the videos that I'm making you definitely also like checking out this channel I'm gonna put the link in the description at the end of the talk So Janek, thanks for doing this with me. I really appreciate it Thanks for having me. It's cool. So what we're gonna talk about is population-based search and Presentation that ICML that I really thought was interesting about emphasizing Diversity and novelty in search. So the first question I just wanted to start by generally talking about your opinion on population-based search and The differences between population-based search and my gradient descent going straight for one solution Yeah, so the the kind of main difference Is that in population-based search as the name implies you maintain kind of a large population of solutions? So you don't want to limit yourself to just one trajectory say I start here and then I run towards my goal but you kind of maintain a lot of hypotheses of what the solution could be and then you kind of want to update all of them at the same time and So there's many different variants of population-based search but they all have this this thing in common where you maintain many solutions and you kind of bet on One of them becoming a good one basically Yes, so one other thing they they present their paper where they have the robot walking And if it breaks one of its legs, for example, it can go back to the map elites table and and say okay Well, I've lost this leg, but I think maybe this solution I was I wasn't too clear on how that would really be related. So I was maybe wondering if you had more insight on that Yes, so the so the maybe the the context is yeah You want to teach a robot to walk and the robot had six legs I believe so and if you think of what's the solution to the problem a solution is kind of an Algorithm that takes the current sensor input and outputs how to move the motors, right? So and If you just have like say your gradient descent algorithm converging on the best solution of the robot Of how to move the robot. It's just going to be like, oh, these are the sensors Okay, I'm gonna move like this like this like this like this but if One leg breaks, of course, you're lost because you only know this one way of moving and now the sorry So you only know this one way of moving Basically, and that's it But in population based search if you think of the solution as a way to move you maintain many many many ways to move so you basically the objective if you can call it like this is Algorithm find me a lot of different ways to move Right with my six legs and now if one of my legs I still can evaluate all of them I still can find okay, which one's the best but if now one of them falls away I have all these other solutions that I can try Right. So then what they would do is like this life falls away. Now. They just reevaluate all of those solutions while only having five legs and the best of those like is much more likely to kind of Work then if you had just your single solution So that kind of that's the its population base because you maintain many different ways of solving the problem Yes, I was also thinking about like using the search algorithms that control neural architecture search and things like that So it's trying to think of how you might extend these ideas from the robot walking with six legs To the RNN controller designing the convolutional network, but like maybe I might have like more of a Storage constraint and more of a latency constraint and I could jump to a different solution like that I'm just wondering how you think like these ideas of population-based search translate into the neural architecture search and specifically if it really is important because like you've got I feel like in neural architecture search you have such a direct signal with the Classification accuracy like I don't see as much variance as those in the in the objective function Yeah, I really think this population based approach is they shine in So they shine in multiple different areas, but one area where they shine is definitely when the environment changes So when you know something about whatever your input changes like the robot losing a leg so in kind of neural architecture search you might You might find these methods working if you then go for let's say transfer learning So you kind of train your network on one task you want to actually do it in another task, right? And then if you maintain many solutions and you can evaluate all of them In a in this transfer setting it's much more likely that one of them is gonna be is gonna be fine So but you're right of I also believe that directly in architecture search Maybe it's not Maybe it doesn't yield that many grades results though the other of course the other Area where these methods shine and this is with respect to algorithms like novelty search Which can be implemented as a population based method is They gave this really good example of deception in a search problem So a deception would be like if you have a robot walking a maze and the robot just wants to get to the goal Right and you would program it the robot to be rewarded the closer it gets to the goal But if like there's a wall in between and you actually need to go around the wall Kind of then for a while you would need to move away from the goal in order to reach it So if you have like a pure objective driven approach, you just go straight to the goal You would always get stuck at the wall But if you then kind of do what is called a novelty search where you basically reward the robot for things It has never done before it would actually find its way around the wall So you can maintain population of solutions that all kind of explore the space And that in our neural architecture search, maybe it's of a benefit that actually You know if I I probably always benefit from like adding more layers or neurons or something like this, but maybe I actually want to prune some stuff first and then add some more stuff So I maybe want to get worse first before I can get even better, right? So so is this a reach where I can imagine that happening? But I don't know Yeah, I was thinking the changing environment I definitely think like when you deploy a model and then you're getting new data that you could frame that as a changing environment And then also I was thinking about like in the context of GAN Which is something that I think is really interesting that the discriminator classifying the GAN Sam the generator samples, it's a changing environment because of the generators updates So maybe having some kind of population based GAN or discriminator model might help it avoid that like Continual learning problem, I guess is sort of an Yeah, that could that might as might very well be There are approaches to GANs, I believe where you basically you have like many discriminators And each one kind of only has let's say has its own limited view on the data And you're trying to kind of fool a lot of them at the same time, but it's not the same thing. But yes I think that that might make sense. Yeah, I've seen that multiple generator multiple discriminator model too I think that's really interesting as well So then one other thing I was curious about is this idea of goal switching and how that might relate to the like AutoML on our existing More like heavily studied things like classification, localization, semantic segmentation Like how do you think goal switching could be important? Like one idea I had is maybe if you've got like multi-class classification And it's got like a really low false positive rate or something on like one class You might say well you've somehow learned a decision boundary on that class Or do you think that wouldn't generalize and that there's no sense in goal switching in like a multi-class classification problem? So yeah, in general, well when you think of goal switching in general How they introduced it was also in the context of like this population based search of these map elites Maybe it's kind of so what map elites the algorithm does basically is it says Okay, I have a number of dimensions that I could solve the problem on and they introduced Okay, let's take life on earth needs to whatever survive So I can either be like a super tall creature right to reach food that no one else can reach I could be a super fast creature right to kind of run away from everything Or it can be a super heavy creature so that no one can attack me And so these are kind of the dimensions that you can solve the problem of reproduction and survival And within so what map elites does it it would segment this area So let's say size and speed it would segment this into a grid And then in each grid it would kind of maintain the best solution so far that is within that grid And then what they see is when they then kind of evolve this over time and improve each each grid is that Inventions let's say inventions algorithm discoveries in one grid say for a very fast creature They would then kind of be adapted to the very let's say the very heavy creatures so like fast creature Kind of discovers or longer legs make me even faster Maybe the longer legs can be then be combined in the heavy creature to do something else So this kind of goal switching it's think of like feathers being first kind of developed or evolved for warmth For temperature regulation then being goal switched over to adapt it for flight So in the in terms of multi class classification I guess it's a bit of a different problem if you just have one classifier You can definitely make the argument that since you know you're learning maybe to classify one class really well The low false positive rate you have learned very good features for that class And if some other class kind of like the zebra is a horse with stripes and then the horse is a horse But with the feature stripes being really low you can probably classify that better or something making stuff up here But it's a bit of a different context I feel the if you have a single classifier do multi class classification But definitely the logic applies in the feature space I would say where you learn features for one class and they might become useful for another class Yeah I had this other thoughts sort of when you're discussing that is like what about like multi class multitask learning Like maybe my intermediate features get mapped to a classifier get mapped to a segmentation get mapped to again Like could goal switching improve multitask learning Yeah I would definitely say so I think that that's exactly what we're seeing when you look at for example pre training So if you think of like these wherever these newest big language models like BERT or something they're really good at tasks I don't know what it was an NLP task labeling of sentiment sentiment classification is the classic right If they evaluate on that because it's so easy but let's say BERT is really good at sentiment classification But if you were to just to train it out right on sentiment classification it's probably not going to work because there's just too little signal But then what happens is you pre train it as a language model as this masked language model and it kind of gets really good at simply comprehending language And that skill can then be kind of adapted over into the into the cement sorry into the sentiment classification realm So I think if you look at something like pre training or multitask as you say then definitely one tap what the addition of a task might give rise to certain features That then all of a sudden can be adapted by another task whereas if you just trained the latter task by itself that maybe would have been too difficult So yeah there's definitely an analogy so then what I think about is so I'm going from my pre training language model into sentiment classification And maybe I also add like question answer during document summarization named entity like this like vector of tasks that it can go do I'm then curious like when your goal switching it's like how do you then combine the features later on or do you just like take it as if I need this task I'll go to this model like yeah Well the question here is do you whether or not you implement this as a single model and kind of refer to the goal switching of features within that model Or whether you also do this now as a population based method where basically you maintain you you maintain different neural networks for different combination of these tasks Then you'd actually need a method to kind of combine and reproduce the neural networks themselves which I yeah I see that's that's going to be a bit of a difficult task Like some cross distillation or some something crazy yeah I don't know how that will work exactly Yeah I just wonder about two things it's like do for my population based search could you have like the weights be the population like different sets of weights Or would it necessarily need to be like taking apart the layers and designing new internal like cells as in the architecture search like Because if I just have the weights maybe I could treat the diversity search or goal switching as like stochastic weight averaging and just like mesh them all together when I'm finished with my goal switching at the end But if it's yeah it's definitely be if you wanted to if you yeah if you wanted to if you wanted to implement your multi task multi task tasking as a population based approach Where yeah you could def it would definitely give you an easier time if you keep the architecture of your neural networks the same and simply have different weights And then you could indeed consider something like weight averaging or or yeah I guess a more modern approach will be like distillation from the two teacher models into one child model It's actually a good metaphor for a for reproduction kind of a distillation from multiple teacher model don't know if anyone's done that yet but yeah I guess that that might be the way to do it if you also maintain different architectures for different problems that might be a bit of a yeah Yeah that's an interesting thing too if you have the goal switching and then you model distill it all into one model that is yes Well if you think of map elites right you'd simply you'd simply distill it into the appropriate I don't even know what the what the axis would be probably I can imagine okay you have like three tasks so you have three axis and then you'd mix the task maybe in accordance on how far up your of these axes you are or something like this It's not exactly map elites because your actual objectives are on the axis but I don't know Yes pretty cool so just to backtrack one step I want to talk about like diversity centric search novelty like when I was thinking about that I was like can't you just initialize it such such that it has maximum diversity like can't you just initialize the population such that they're all like uniformly spaced and then search locally from there So I just wonder what you think on that and how this is different from that So yeah in these in these diversity search algorithms basically what you're you're doing is your your only goal is or your main goal depends on the algorithm but let's say your only goal is to find diverse behaviors or diverse solutions diverse whatever I think the main problem with that is is that the search space is so extremely large That you're going to have a hard time even even defining what a kind of a uniform distribution is because it's such a high dimensional space that even if you sample uniformly it's it's almost empty like you're almost right you're not you're not getting anywhere because you have finite finite computer you need to implement an algorithm Even if you even if my computer can hold a hundred thousand different members of a population in high dimensions that is nothing right so to me yet the initialization might be definitely important But I don't think you'll you'll get around some sort of iterative procedure and going around weeding out weeding out things such that you have space for interesting things because ultimately what you want to find is something interesting In the robot maze example the novelty search basically is here is a robot you started right and then you want to do something that you haven't done yet right so if the robots crashes into a wall the first time that's a good thing you say oh cool you haven't done that yet But if it crashes into the wall the second time you're like you've done that already right so you you you basically need a measure of saying how close to behaviors are but if the robot has crashed into every wall once the only thing it can do if it wants to do something new is actually go around the wall and then you're like oh cool you've done something new But the space of behaviors often is so large that you can't simply enumerate all the all the behaviors so you I think that's the main problem why you can't just make it diverse from the beginning Yeah when I think about that I was thinking that maybe the like reward function if you're like navigating the maze it needs to be more refined so like if it crashes into the wall that needs to be like I don't know plus three some some like unique signal I feel like in order to create that kind of because like Thinking of if it's just like reward zero everywhere but one if you hit that finish line and then maybe some kind of like discounting for how long it takes you to get there is like I don't see how it could interpret that it's done a new behavior if all it has is it so to me it feels like it's all about the design of the reward space now to implement such a thing Yes absolutely so the that the definitely if you wanted to do novelty search you would need to implement a measure of how close to behaviors are so there's no way around and I think that's kind of crux of the of this method is that by specifying how close to behaviors are so what what constitutes novelty and what doesn't You already implicitly kind of telling the robot something about the nature of of the world so I think that the kind of the objective because they now say oh we don't give the robot the objecting of reaching the target we simply give it the objective of not doing the same thing twice I think the kind of objective sneaks in Like again through the specification of how of you how close are to be a risk but definitely this is just kind of a really simple example of what they want to say is that these methods really become important when you have ambitious objectives in the maze we can all agree if we just designed the reward Crashing walls bad you don't have to actually go straight to the goal you can you know but go around walls good and so on then it's easy right but in really ambitious objectives like I don't know flying reaching the moon in the in the 1960s designing general AI Curing cancer and so on we don't actually know how to design the reward right because we don't know which steps need to be fulfilled in order to to fly to the moon I guess now we do in hindsight right but we couldn't have predicted we don't know which steps need to be discovered In order to cure cancer and it's very very probable if you look at history that the fundamental discoveries that lead to us curing cancer will not directly come from cancer research that's that's their entire point right it's not like you can have a goal go straight towards it if it's like a really ambitious goal very probably The solutions will come in part from extremely non related fields and they and you kind of have to make advances everywhere and in order to solve that problem so the the the question of it's all designed it's all about designing the reward yes but we would have to know how the reward must be must look and in these really ambitious objectives we don't And that's that's where they argue well the best thing actually you can do is to just explore and you just find interesting things along the way and you kind of hope that these interesting things will come no you know the interesting things will combine to form new interesting things right but you just don't know where you're going to end up right Yeah, I guess maybe you could just keep a trip like the trajectory of states and use that as your signal of novelty. But then I think like if you've got like a robotic arm with like x degrees of freedom it's like the state space would be too infinite to really like say oh this was significantly this is a significantly different sequential procedure of states and this other thing. So then the next thing. Yes, I think this is a good transition into their pick breeder experiment. And so anyone who listens to this who hasn't watched their talk the pick breeder is like, they've got these generator neural networks with sets of weights. And they have like humans go on and they pick two of the generated images to blend together and derive a new image. And so this repeats on and on until it goes from like just like a spiral pattern into like a skull face drawing or a butterfly drawing or something like that. And they. So this idea is supposed to represent open endedness in an environment and not so it just generally I, I just found it to be really interesting. I think it's one of the things in their talk that you look at it and you're like oh it's interesting what what is going on here. But it's like the, the mutation is really guided by the human search, which is so complex I feel like I was just wondering what you thought of that pick breeder experiment. Yeah, it's really cool. And it's, it's, it's actually the basis for their entire books I've read the book, the white greatness cannot be planned I believe I've got the title. But, so that this, they actually they kind of start out with this as a motivational example of what if, what if the only goal is to do something interesting and without any objective so all you do is kind of choose slight variations on the current picture, and you see what you end up with and I thought, I thought it illustrates their points extremely well so it illustrates, for example, goal switching is that if you were done with your sequence of image manipulations you could then save it into the database and someone else could pick up from it, and then kind of continue it. And since every human finds slightly different things interesting right, you could take someone else's final result and say, ah, you know that that kind of looks weird but then you, your modifications to it will be different than that human continued breeding the picture. So what you end up is, and they show this, for example, one picture ends up being a car, and it had been adapted from an alien face where the eyes of the alien face became the wheels of the car. And so the first person might have been like, oh, this looks more and more like an alien face, I'm going to make it more like an alien face, and then the second person is like, oh, that kind of looks nice, I'm going to modify it in a different, so they basically give this example of if you have an ambitious goal like getting to a car just from these very simple picture generation networks. Then the stepping stones to get there have nothing to do with cars, and the people that did it didn't have a car in mind while going there. And the second thing is that if you try to get a car from the beginning, I believe they've done this, if you try to, you can't. Like, it's just the sequence of things that you have to go through is so complicated and convoluted that if you were to try to end up with a result, it's basically impossible. So these kind of illustrate their points very, very nicely. And I mean, it's a cool experiment in itself, but they use it kind of as a basis metaphor for them going on, jumping off. Yeah, I just think it's so interesting, this idea that it's like you can't design a car unless you don't try it, unless you just happen to come across that. It's sort of like I think about like if I was to fire up GarageBand and start trying to make a song, it's like I don't know exactly what it's going to sound like. I'm just going to kind of explore until I come across something. So then I was thinking about like with the GANs and the way that the GANs design images. So this is sort of a design I drew up that I'm curious what you think of. It's like what if the generator just tries to make some object and then a pre-chained classifier says, oh, I think it looks like this maybe. And then you send it to like a refining network. So the GAN just sort of searches for objects and then some classifiers are like, oh, I think it looks like sort of like how the pig breeders sort of like how we're like, oh, I think this looks like a skull or whatever. So I'm going to try to refine it now. Do you think that would be an interesting thing or? You'd have like a two stage process. First you do something general and then it gets classified. And then you'd have like a special generator just for the skull class and the special discriminator just for that. Yeah, I don't see why not. It might be hard. It might be hard to get the first generator to be sufficiently diverse. So you might might need some kind of discriminator signal at the even at the beginning. So yes, I mean, you're like, how do you think the pig breeder experiment could become fully automated such that there's no human in the loop? Yeah, that's that's a thought I had as well, because to me it seems that the kind of, of course, the resulting pictures, the fact that they look like human objects or recognizable objects is a result from them being being bred by humans. Like the fact that it looks like a car or a skull or something like this is is very much. But also, I guess that that could be abstracted in. We just not expect the results to be like human recognizable objects, but maybe something else. The much more deeper construction in pig breeder is the fact that the measure of interestingness is provided by the humans. Right. So the humans, they they click on a picture and then they get variants of that picture and they click on the one that they most like. This this sense of interestingness of I like this one is that's what's that's the fundamental core that's provided by the humans as an input to the system. That's what drives the entire thing. That's exactly the same as before. It's when you write when you teach the robot which two behaviors are close enough, like, oh, no, that's too close to before. That's not novel. Or yes, that's sufficiently different than before. That is novel. Right. This this sense is somehow you either need to specify it or you need to have the human in the loop to provide it. I feel it's very, very hard to capture that in an algorithm as as of today. Yeah, like something I think about is like maybe I'd have like my thousand class image net classifier and then maybe I'd have like like a style classifier, like a neural style transfer network that I've like chopped off the like some intermediate feature. I'm going to take that as my style. And so maybe I'm like classifying. I think it's like an airplane. And then I kind of like this style for it. That's sort of like my like how I would think about trying to automate that. Like, I don't know, I guess, like, I don't know if I I guess it's interesting. But I also feel like when you're doing the pick reader, you're kind of like, oh, I'm going to try it now. Now that I see this vision, I'm going to try to make it like look like that now, I suppose. Like, yeah, yeah. I think I could mold this into a skull and then you start doing. Yes, yes, they're very much so they're not they're not advocating random exploration. What they're advocating is basically if you have an ambitious goal, then you basically don't know the stepping stones. But from stepping stone to stepping stone, that's where objectives are very handy. So when you want to say I this already kind of looks like something, I want to make it more like that. I want to make it more into a skull. Right. It already has like two circles and kind of the shape. But I'm going to drive it there. That that is very that can be very objective driven. But in the grand scheme of things, you don't know. Then once you have the skull, someone else can develop that into an even new thing. So, yeah, indeed, if if you if you are in kind of a local search in this space, then an objective driven behavior like what you're saying, like I want to make it as much this as possible. That's very that's actually a thing they're advocating for. But then from their end result, yeah, you would need to then restart again, do the same thing with like something else. Huh? Yeah, it's really interesting. Just thinking about, yeah, I think about like the stepping stones and like is how would you define the space of stepping stones to such a to any kind of thing? I guess it's like you could still design some kind of maybe it's discrete or maybe you have some kind of signal you can get back from it. And I guess it's just a lot to think about. Directly, I think they give this they give this great analogy. I feel like if you have a really ambitious objective, it's like crossing a lake, but the lake is covered in fog. So you basically can't really see very far, but you can always kind of see the next stepping stones. Right. And you can then you can then try to go from stepping stone to stepping stone, but you don't know which one to take if there's like a fork. There's two ways possible. You don't know which one. Right. So all you can do is basically go the most interesting one. And they relate this to scientific research. So, yeah, if we want to accomplish some really great research goal, like artificial general intelligence, we don't like we don't know. But we can see the next stepping stones. Right. We can see, oh, from what we have right now, what interesting combination could we make that still kind of it still kind of makes that's not total garbage. Right. So in the local search, I can try to say I want to I don't know. I want to do this. I want to do multiple generators and multi stage and then this thing. Right. This this is kind of a stepping stone and maybe that will then lead to something more interesting and so on. So, yeah, that's that's kind of how they relate. I like this metaphor of the lake. Yeah. Yeah. I just like could like a meta controller try to put the stones down and then the objective is or is the space too enormous that that idea of having a meta controller guide the stepping stone placement is too big. The stepping stone placement is just like absurd in that and there's no way that that would work. That's sort of where I'm thinking with this now is like. So they actually that's that's exactly the question. Right. Of what I so I believe you need such a meta whatever because the space is too large. You somehow need a way to choose the stepping stones in the first place. Right. You somehow need a way to do this. Now, what they're saying is that if you're if your goal is really ambitious, then a meta controller that simply wants to reach the goal is bad because right because what we discussed before, you might need a lot of inventions from other fields in order to make goal happen. And if you simply go your field maximum power towards your goal, that's not going to happen. Now, if your meta controller is actually just something that wants to produce interesting things, then that's actually something they advocate for. That is exactly what their algorithms are trying to capture. They're trying to capture locally. Yeah, we want to get better at a particular thing. What those particular things are and the order of these that should be novelty driven instead of goal driven. Yeah, yeah. Yeah. The interesting component. I guess I'm sort of biased towards liking the objective design. And now I'm thinking like, OK, well, let's abstract those meta controllers one level up and have a meta meta controller and just repeat this and hierarchy makes sense. And that if you if you if you're if you're a bit cynical, that is what you will also hear out of here out of and they have to argue in the in their book a lot against that like isn't the question isn't the kind of isn't the implementation of a meta controller that just searches for novelty in itself. And that's the objective again. And then they give some good reasons why actually you don't. It is different. It's more like a constraint on your search. If you think of natural evolution, for example, it isn't really doesn't really have an objective. You think reproduction and survival is the objective of natural evolution. It doesn't really the good the good reason they give is the objective has already been fulfilled by the very first organism to ever live. Right. Why didn't it stop there? Why didn't it stop very first cell? OK, done. We've fulfilled the objective. It's more of a it's more of an actually a constrained optimization where the constraint is you need to be able to survive. That's kind of the minimum bar of to being on this planet. And then I'm saying constrained optimization, but it's it's not it's not an optimization. It's more of like a constraint constraint search. OK, yeah, I think, yeah, I guess it's just like I don't think I'm closed in this world of trying to think of these constraint problems. And I haven't really like thought more generally about just like exploration as a whole. But but anyway, so I just wanted to ask you generally like your deep learning researcher, I want to ask like what areas of deep learning are you really interested in right now? And what do you think is promising in the near future? So I'm currently working in adversarial examples. That is a really interesting topic. There's lots of questions still still open, but I'm generally interested in pretty much any anything that is not. I'm not too interested in like the newest the newest fine technique on getting the latest state of the art numbers, even though that's probably super important for practitioners. Basically, agreeing more with the authors of this tutorial of that. Let's just try to do interesting things. And to me, these these actually these these areas in terms of open ended, open ended search, open ended learning are very interesting. I think reinforcement learning still has a long way to go. I think actually NLP still has a long way to go because I don't believe it's the current models are the end of it. So I think it's really exciting time. Yeah, I love thinking about adversarial examples because it definitely flips the CNN idea on its head. And then I had one other thing about adversarial examples that I'm interested in is there is like an interview with Elon Musk and this Lex Friedman researcher where he asked him about adversarial examples on his self-driving cars. And he seems dismissive of it. He says he thinks basically you could just average different patches of like test time augmentation to overcome adversarial examples. So in your research, do you think that like the example where they add the noise mass to the panda and they're like, oh, it's a given now, if they just perturbed it like nine more times, do you think the prediction would average out to pandas? That is a very difficult question. And from experience, simply adding noise and then feeding it to the classifier, even if you average after that, usually will defend against adversarial examples to a point. But it will also degrade your classification performance. Because so maybe I understood it wrong, but my understanding is I have my input, right? I simply add noise to it and then feed it through the network. And I could do this many times, right? And then average the prediction. But usually this will help against adversarial examples, but it will also degrade the accuracy of that classifier. So it might actually make your self-driving car worse in the overall. Because how often is it going to be attacked against a adversarial example? It's going to be attacked maybe once or twice a year, maybe if it drives by some hacker's house, right? Sticker on a stop sign or something. But the rest of the time, I would actually like to retain the best possible classifier. And if I always have to add noise, then that's not possible. So the research we're doing is actually into the direction of can we retain the original accuracy while still kind of detecting these samples? I mean, you somehow have to get a trade off somewhere, but just adding noise isn't the final solution yet. I was like, so with these adversarial examples, they're only going to make misclassifications like that if it really is adversarially sought after. It's not just like the noise perturbation would be such an enormous space to find it otherwise. Yes, you really need to try. So it's very unlikely that some random thing. Of course, these networks can be confused by random noise, but I think one of the self-driving cars once drove into a big white truck because it was large and white, so it thought it was sky. But other than these failures, you really have to try to find an adversarial example. Really cool. Yannick, thanks so much for doing this. Anybody watching or listening, definitely check out Yannick's YouTube channel. He has really great paper summaries and all sorts of things. Thank you. Thanks so much for having me.
 Hi there, I've recently been interviewed by the YouTube channel Henry AI Labs
 by Connor Shorten and
 what follows is the resulting conversation we had about population-based methods and
 Open-ended learning things like that basically topics of the ICML tutorial that we both saw
 It's important to note that none of us is really an expert on the topic
 but we are trying to make sense of it and
 mainly just kind of talking about the ideas
 So please enjoy the conversation with Connor Shorten definitely check out the Henry AI Labs channel and
 Now have a good time
 Thanks for watching the Henry AI Labs deep learning podcast today. I'm joined with Janek Kilcher
 Janek works in the data analytics lab at ETH. He has a great YouTube channel
 I really enjoy watching his paper summary videos
 If you like any of the videos that I'm making you definitely also like checking out this channel
 I'm gonna put the link in the description at the end of the talk
 So Janek, thanks for doing this with me. I really appreciate it
 Thanks for having me. It's cool. So what we're gonna talk about is population-based search and
 Presentation that ICML that I really thought was interesting about
 emphasizing
 Diversity and novelty in search. So the first question I just wanted to start by generally talking about your opinion on population-based search and
 The differences between population-based search and my gradient descent going straight for one solution
 Yeah, so the the kind of main difference
 Is that in population-based search as the name implies you maintain kind of a large population of solutions?
 So you don't want to limit yourself to just one trajectory say I start here and then I run towards my goal
 but you kind of maintain a lot of hypotheses of what the solution could be and then you kind of
 want to update all of them at the same time and
 So there's many different variants of population-based search
 but they all have this this thing in common where you maintain many solutions and you kind of bet on
 One of them becoming a good one
 basically
 Yes, so one other thing they they present their paper where they have the robot walking
 And if it breaks one of its legs, for example, it can go back to the map elites table and and say okay
 Well, I've lost this leg, but I think maybe this solution
 I was I wasn't too clear on how that would really be related. So I was maybe wondering if you had more insight on that
 Yes, so the so the maybe the the context is yeah
 You want to teach a robot to walk and the robot had six legs
 I believe so and if you think of what's the solution to the problem a solution is kind of an
 Algorithm that takes the current sensor input and outputs how to move the motors, right? So and
 If you just have like say your gradient descent algorithm converging on the best solution of the robot
 Of how to move the robot. It's just going to be like, oh, these are the sensors
 Okay, I'm gonna move like this like this like this like this but if
 One leg breaks, of course, you're lost because you only know this one way of moving and now the sorry
 So you only know this one way of moving
 Basically, and that's it
 But in population based search if you think of the solution as a way to move you maintain many many
 many ways to move
 so you
 basically the
 objective if you can call it like this is
 Algorithm find me a lot of different ways to move
 Right with my six legs and now if one of my legs I still can evaluate all of them
 I still can find okay, which one's the best but if now one of them falls away
 I have all these other solutions that I can try
 Right. So then what they would do is like this life falls away. Now. They just reevaluate all of those solutions
 while only having five legs and the best of those like is much more likely to kind of
 Work then if you had just your single solution
 So that kind of that's the its population base because you maintain many different ways of solving the problem
 Yes, I was also thinking about like using the search algorithms that control neural architecture search and things like that
 So it's trying to think of how you might extend these ideas from the robot walking with six legs
 To the RNN controller designing the convolutional network, but like maybe I might have
 like more of a
 Storage constraint and more of a latency constraint and I could jump to a different solution like that
 I'm just wondering how you think like these ideas of population-based search translate into the neural architecture
 search and specifically if it really is important because like you've got I feel like in neural architecture search you have such a direct signal with the
 Classification accuracy like I don't see as much variance as those in the in the objective function
 Yeah, I really think this population based approach is they shine in
 So they shine in multiple different areas, but one area where they shine is definitely when the environment changes
 So when you know something about whatever your input changes like the robot losing a leg so in kind of neural architecture search you might
 You might find these methods working if you then go for let's say transfer learning
 So you kind of train your network on one task you want to actually do it in another task, right?
 And then if you maintain many solutions and you can evaluate all of them
 In a in this transfer setting it's much more likely that one of them is gonna be is gonna be fine
 So but you're right of I also believe that directly in architecture search
 Maybe it's not
 Maybe it doesn't yield that many grades
 results though the other of course the other
 Area where these methods shine and this is with respect to algorithms like novelty search
 Which can be implemented as a population based method is
 They gave this really good example of deception in a search problem
 So a deception would be like if you have a robot walking a maze and the robot just wants to get to the goal
 Right and you would program it the robot to be rewarded the closer it gets to the goal
 But if like there's a wall in between and you actually need to go around the wall
 Kind of then for a while you would need to move away from the goal in order to reach it
 So if you have like a pure objective driven approach, you just go straight to the goal
 You would always get stuck at the wall
 But if you then kind of do what is called a novelty search where you basically reward the robot for things
 It has never done before it would actually find its way around the wall
 So you can maintain population of solutions that all kind of explore the space
 And that in our neural architecture search, maybe it's of a benefit that actually
 You know if I I probably always benefit from like adding more layers or neurons or something
 like this, but maybe I actually want to prune some stuff first and then add some more stuff
 So I maybe want to get worse first before I can get even better, right?
 So so is this a reach where I can imagine that happening? But I don't know
 Yeah, I was thinking the changing environment
 I definitely think like when you deploy a model and then you're getting new data that you could frame that as a changing environment
 And then also I was thinking about like in the context of GAN
 Which is something that I think is really interesting that the discriminator classifying the GAN
 Sam the generator samples, it's a changing environment because of the generators updates
 So maybe having some kind of population based GAN or discriminator model might help it avoid that like
 Continual learning problem, I guess is sort of an
 Yeah, that could that might as might very well be
 There are approaches to GANs, I believe where you basically you have like many discriminators
 And each one kind of only has let's say has its own limited view on the data
 And you're trying to kind of fool a lot of them at the same time, but it's not the same thing. But yes
 I think that that might make sense. Yeah, I've seen that multiple generator multiple discriminator model too
 I think that's really interesting as well
 So then one other thing I was curious about is this idea of goal switching and how that might relate to the like AutoML on our existing
 More like heavily studied things like classification, localization, semantic segmentation
 Like how do you think goal switching could be important?
 Like one idea I had is maybe if you've got like multi-class classification
 And it's got like a really low false positive rate or something on like one class
 You might say well you've somehow learned a decision boundary on that class
 Or do you think that wouldn't generalize and that there's no sense in goal switching in like a multi-class classification problem?
 So yeah, in general, well when you think of goal switching in general
 How they introduced it was also in the context of like this population based search of these map elites
 Maybe it's kind of so what map elites the algorithm does basically is it says
 Okay, I have a number of dimensions that I could solve the problem on and they introduced
 Okay, let's take life on earth needs to whatever survive
 So I can either be like a super tall creature right to reach food that no one else can reach
 I could be a super fast creature right to kind of run away from everything
 Or it can be a super heavy creature so that no one can attack me
 And so these are kind of the dimensions that you can solve the problem of reproduction and survival
 And within so what map elites does it it would segment this area
 So let's say size and speed it would segment this into a grid
 And then in each grid it would kind of maintain the best solution so far that is within that grid
 And then what they see is when they then kind of evolve this over time and improve each each grid is that
 Inventions let's say inventions algorithm discoveries in one grid say for a very fast creature
 They would then kind of be adapted to the very let's say the very heavy creatures so like fast creature
 Kind of discovers or longer legs make me even faster
 Maybe the longer legs can be then be combined in the heavy creature to do something else
 So this kind of goal switching it's think of like feathers being first kind of developed or evolved for warmth
 For temperature regulation then being goal switched over to adapt it for flight
 So in the in terms of multi class classification I guess it's a bit of a different problem if you just have one classifier
 You can definitely make the argument that since you know you're learning maybe to classify one class really well
 The low false positive rate you have learned very good features for that class
 And if some other class kind of like the zebra is a horse with stripes and then the horse is a horse
 But with the feature stripes being really low you can probably classify that better or something making stuff up here
 But it's a bit of a different context I feel the if you have a single classifier do multi class classification
 But definitely the logic applies in the feature space I would say where you learn features for one class and they might become useful for another class
 Yeah I had this other thoughts sort of when you're discussing that is like what about like multi class multitask learning
 Like maybe my intermediate features get mapped to a classifier get mapped to a segmentation get mapped to again
 Like could goal switching improve multitask learning
 Yeah I would definitely say so I think that that's exactly what we're seeing when you look at for example pre training
 So if you think of like these wherever these newest big language models like BERT or something they're really good at tasks
 I don't know what it was an NLP task labeling of sentiment sentiment classification is the classic right
 If they evaluate on that because it's so easy but let's say BERT is really good at sentiment classification
 But if you were to just to train it out right on sentiment classification it's probably not going to work because there's just too little signal
 But then what happens is you pre train it as a language model as this masked language model and it kind of gets really good at simply comprehending language
 And that skill can then be kind of adapted over into the into the cement sorry into the sentiment classification realm
 So I think if you look at something like pre training or multitask as you say then definitely one tap what the addition of a task might give rise to certain features
 That then all of a sudden can be adapted by another task whereas if you just trained the latter task by itself that maybe would have been too difficult
 So yeah there's definitely an analogy so then what I think about is so I'm going from my pre training language model into sentiment classification
 And maybe I also add like question answer during document summarization named entity like this like vector of tasks that it can go do
 I'm then curious like when your goal switching it's like how do you then combine the features later on or do you just like take it as if I need this task I'll go to this model like yeah
 Well the question here is do you whether or not you implement this as a single model and kind of refer to the goal switching of features within that model
 Or whether you also do this now as a population based method where basically you maintain you you maintain different neural networks for different combination of these tasks
 Then you'd actually need a method to kind of combine and reproduce the neural networks themselves which I yeah I see that's that's going to be a bit of a difficult task
 Like some cross distillation or some something crazy yeah I don't know how that will work exactly
 Yeah I just wonder about two things it's like do for my population based search could you have like the weights be the population like different sets of weights
 Or would it necessarily need to be like taking apart the layers and designing new internal like cells as in the architecture search like
 Because if I just have the weights maybe I could treat the diversity search or goal switching as like stochastic weight averaging and just like mesh them all together when I'm finished with my goal switching at the end
 But if it's yeah it's definitely be if you wanted to if you yeah if you wanted to if you wanted to implement your multi task multi task tasking as a population based approach
 Where yeah you could def it would definitely give you an easier time if you keep the architecture of your neural networks the same and simply have different weights
 And then you could indeed consider something like weight averaging or or yeah I guess a more modern approach will be like distillation from the two teacher models into one child model
 It's actually a good metaphor for a for reproduction kind of a distillation from multiple teacher model don't know if anyone's done that yet but yeah I guess that that might be the way to do it if you also maintain different architectures for different problems that might be a bit of a yeah
 Yeah that's an interesting thing too if you have the goal switching and then you model distill it all into one model that is yes
 Well if you think of map elites right you'd simply you'd simply distill it into the appropriate I don't even know what the what the axis would be probably I can imagine okay you have like three tasks so you have three axis and then you'd mix the task maybe in accordance on how far up your of these axes you are or something like this
 It's not exactly map elites because your actual objectives are on the axis but I don't know
 Yes pretty cool so just to backtrack one step I want to talk about like diversity centric search novelty like when I was thinking about that I was like can't you just initialize it such such that it has maximum diversity like can't you just initialize the population such that they're all like uniformly spaced and then search locally from there
 So I just wonder what you think on that and how this is different from that
 So yeah in these in these diversity search algorithms basically what you're you're doing is your your only goal is or your main goal depends on the algorithm but let's say your only goal is to find diverse behaviors or diverse solutions diverse whatever I think the main problem with that is is that the search space is so extremely large
 That you're going to have a hard time even even defining what a kind of a uniform distribution is because it's such a high dimensional space that even if you sample uniformly it's it's almost empty like you're almost right you're not you're not getting anywhere because you have finite finite computer you need to implement an algorithm
 Even if you even if my computer can hold a hundred thousand different members of a population in high dimensions that is nothing right so to me yet the initialization might be definitely important
 But I don't think you'll you'll get around some sort of iterative procedure and going around weeding out weeding out things such that you have space for interesting things because ultimately what you want to find is something interesting
 In the robot maze example the novelty search basically is here is a robot you started right and then you want to do something that you haven't done yet right so if the robots crashes into a wall the first time that's a good thing you say oh cool you haven't done that yet
 But if it crashes into the wall the second time you're like you've done that already right so you you you basically need a measure of saying how close to behaviors are but if the robot has crashed into every wall once the only thing it can do if it wants to do something new is actually go around the wall and then you're like oh cool you've done something new
 But the space of behaviors often is so large that you can't simply enumerate all the all the behaviors so you I think that's the main problem why you can't just make it diverse from the beginning
 Yeah when I think about that I was thinking that maybe the like reward function if you're like navigating the maze it needs to be more refined so like if it crashes into the wall that needs to be like I don't know plus three some some like unique signal I feel like in order to create that kind of because like
 Thinking of if it's just like reward zero everywhere but one if you hit that finish line and then maybe some kind of like discounting for how long it takes you to get there is like I don't see how it could interpret that it's done a new behavior if all it has is it so to me it feels like it's all about the design of the reward space now to implement such a thing
 Yes absolutely so the that the definitely if you wanted to do novelty search you would need to implement a measure of how close to behaviors are so there's no way around and I think that's kind of crux of the of this method is that by specifying how close to behaviors are so what what constitutes novelty and what doesn't
 You already implicitly kind of telling the robot something about the nature of of the world so I think that the kind of the objective because they now say oh we don't give the robot the objecting of reaching the target we simply give it the objective of not doing the same thing twice I think the kind of objective sneaks in
 Like again through the specification of how of you how close are to be a risk but definitely this is just kind of a really simple example of what they want to say is that these methods really become important when you have ambitious objectives in the maze we can all agree if we just designed the reward
 Crashing walls bad you don't have to actually go straight to the goal you can you know but go around walls good and so on then it's easy right but in really ambitious objectives like I don't know flying reaching the moon in the in the 1960s designing general AI
 Curing cancer and so on we don't actually know how to design the reward right because we don't know which steps need to be fulfilled in order to to fly to the moon I guess now we do in hindsight right but we couldn't have predicted we don't know which steps need to be discovered
 In order to cure cancer and it's very very probable if you look at history that the fundamental discoveries that lead to us curing cancer will not directly come from cancer research that's that's their entire point right it's not like you can have a goal go straight towards it if it's like a really ambitious goal very probably
 The solutions will come in part from extremely non related fields and they and you kind of have to make advances everywhere and in order to solve that problem so the the the question of it's all designed it's all about designing the reward yes but we would have to know how the reward must be must look and in these really ambitious objectives we don't
 And that's that's where they argue well the best thing actually you can do is to just explore and you just find interesting things along the way and you kind of hope that these interesting things will come no you know the interesting things will combine to form new interesting things right but you just don't know where you're going to end up right
 Yeah, I guess maybe you could just keep a trip like the trajectory of states and use that as your signal of novelty. But then I think like if you've got like a robotic arm with like x degrees of freedom it's like the state space would be too infinite to really like say oh this was significantly this is a significantly different sequential procedure of states and this other thing.
 So then the next thing. Yes, I think this is a good transition into their pick breeder experiment. And so anyone who listens to this who hasn't watched their talk the pick breeder is like, they've got these generator neural networks with sets of weights.
 And they have like humans go on and they pick two of the generated images to blend together and derive a new image. And so this repeats on and on until it goes from like just like a spiral pattern into like a skull face drawing or a butterfly drawing or something like that.
 And they. So this idea is supposed to represent open endedness in an environment and not so it just generally I, I just found it to be really interesting. I think it's one of the things in their talk that you look at it and you're like oh it's interesting what what is going on here.
 But it's like the, the mutation is really guided by the human search, which is so complex I feel like I was just wondering what you thought of that pick breeder experiment.
 Yeah, it's really cool. And it's, it's, it's actually the basis for their entire books I've read the book, the white greatness cannot be planned I believe I've got the title.
 But, so that this, they actually they kind of start out with this as a motivational example of what if, what if the only goal is to do something interesting and without any objective so all you do is kind of choose slight variations on the current picture, and you see what you end up with and I thought, I thought it illustrates their points
 extremely well so it illustrates, for example, goal switching is that if you were done with your sequence of image manipulations you could then save it into the database and someone else could pick up from it, and then kind of continue it.
 And since every human finds slightly different things interesting right, you could take someone else's final result and say, ah, you know that that kind of looks weird but then you, your modifications to it will be different than that human continued breeding the picture.
 So what you end up is, and they show this, for example, one picture ends up being a car, and it had been adapted from an alien face where the eyes of the alien face became the wheels of the car.
 And so the first person might have been like, oh, this looks more and more like an alien face, I'm going to make it more like an alien face, and then the second person is like, oh, that kind of looks nice, I'm going to modify it in a different, so they basically give this example of if you have an ambitious goal like getting to a car just from these very simple picture generation networks.
 Then the stepping stones to get there have nothing to do with cars, and the people that did it didn't have a car in mind while going there.
 And the second thing is that if you try to get a car from the beginning, I believe they've done this, if you try to, you can't.
 Like, it's just the sequence of things that you have to go through is so complicated and convoluted that if you were to try to end up with a result, it's basically impossible.
 So these kind of illustrate their points very, very nicely.
 And I mean, it's a cool experiment in itself, but they use it kind of as a basis metaphor for them going on, jumping off.
 Yeah, I just think it's so interesting, this idea that it's like you can't design a car unless you don't try it, unless you just happen to come across that.
 It's sort of like I think about like if I was to fire up GarageBand and start trying to make a song, it's like I don't know exactly what it's going to sound like.
 I'm just going to kind of explore until I come across something.
 So then I was thinking about like with the GANs and the way that the GANs design images.
 So this is sort of a design I drew up that I'm curious what you think of.
 It's like what if the generator just tries to make some object and then a pre-chained classifier says, oh, I think it looks like this maybe.
 And then you send it to like a refining network.
 So the GAN just sort of searches for objects and then some classifiers are like, oh, I think it looks like sort of like how the pig breeders sort of like how we're like, oh, I think this looks like a skull or whatever.
 So I'm going to try to refine it now.
 Do you think that would be an interesting thing or?
 You'd have like a two stage process.
 First you do something general and then it gets classified.
 And then you'd have like a special generator just for the skull class and the special discriminator just for that.
 Yeah, I don't see why not.
 It might be hard.
 It might be hard to get the first generator to be sufficiently diverse.
 So you might might need some kind of discriminator signal at the even at the beginning.
 So yes, I mean, you're like, how do you think the pig breeder experiment could become fully automated such that there's no human in the loop?
 Yeah, that's that's a thought I had as well, because to me it seems that the kind of, of course, the resulting pictures,
 the fact that they look like human objects or recognizable objects is a result from them being being bred by humans.
 Like the fact that it looks like a car or a skull or something like this is is very much.
 But also, I guess that that could be abstracted in.
 We just not expect the results to be like human recognizable objects, but maybe something else.
 The much more deeper construction in pig breeder is the fact that the measure of interestingness is provided by the humans.
 Right. So the humans, they they click on a picture and then they get variants of that picture and they click on the one that they most like.
 This this sense of interestingness of I like this one is that's what's that's the fundamental core that's provided by the humans as an input to the system.
 That's what drives the entire thing. That's exactly the same as before.
 It's when you write when you teach the robot which two behaviors are close enough, like, oh, no, that's too close to before.
 That's not novel. Or yes, that's sufficiently different than before.
 That is novel. Right. This this sense is somehow you either need to specify it or you need to have the human in the loop to provide it.
 I feel it's very, very hard to capture that in an algorithm as as of today.
 Yeah, like something I think about is like maybe I'd have like my thousand class image net classifier and then maybe I'd have like like a style classifier,
 like a neural style transfer network that I've like chopped off the like some intermediate feature.
 I'm going to take that as my style. And so maybe I'm like classifying. I think it's like an airplane. And then I kind of like this style for it.
 That's sort of like my like how I would think about trying to automate that.
 Like, I don't know, I guess, like, I don't know if I I guess it's interesting.
 But I also feel like when you're doing the pick reader, you're kind of like, oh, I'm going to try it now.
 Now that I see this vision, I'm going to try to make it like look like that now, I suppose. Like, yeah, yeah.
 I think I could mold this into a skull and then you start doing.
 Yes, yes, they're very much so they're not they're not advocating random exploration.
 What they're advocating is basically if you have an ambitious goal, then you basically don't know the stepping stones.
 But from stepping stone to stepping stone, that's where objectives are very handy.
 So when you want to say I this already kind of looks like something, I want to make it more like that.
 I want to make it more into a skull. Right.
 It already has like two circles and kind of the shape.
 But I'm going to drive it there. That that is very that can be very objective driven.
 But in the grand scheme of things, you don't know. Then once you have the skull, someone else can develop that into an even new thing.
 So, yeah, indeed, if if you if you are in kind of a local search in this space, then an objective driven behavior like what you're saying, like I want to make it as much this as possible.
 That's very that's actually a thing they're advocating for. But then from their end result, yeah, you would need to then restart again, do the same thing with like something else.
 Huh? Yeah, it's really interesting.
 Just thinking about, yeah, I think about like the stepping stones and like is how would you define the space of stepping stones to such a to any kind of thing?
 I guess it's like you could still design some kind of maybe it's discrete or maybe you have some kind of signal you can get back from it.
 And I guess it's just a lot to think about.
 Directly, I think they give this they give this great analogy.
 I feel like if you have a really ambitious objective, it's like crossing a lake, but the lake is covered in fog.
 So you basically can't really see very far, but you can always kind of see the next stepping stones.
 Right. And you can then you can then try to go from stepping stone to stepping stone, but you don't know which one to take if there's like a fork.
 There's two ways possible. You don't know which one. Right. So all you can do is basically go the most interesting one.
 And they relate this to scientific research.
 So, yeah, if we want to accomplish some really great research goal, like artificial general intelligence, we don't like we don't know.
 But we can see the next stepping stones. Right. We can see, oh, from what we have right now, what interesting combination could we make that still kind of it still kind of makes that's not total garbage.
 Right. So in the local search, I can try to say I want to I don't know. I want to do this.
 I want to do multiple generators and multi stage and then this thing. Right. This this is kind of a stepping stone and maybe that will then lead to something more interesting and so on.
 So, yeah, that's that's kind of how they relate. I like this metaphor of the lake. Yeah. Yeah. I just like could like a meta controller try to put the stones down and then the objective is or is the space too enormous that that idea of having a meta controller guide the stepping stone placement is too big.
 The stepping stone placement is just like absurd in that and there's no way that that would work. That's sort of where I'm thinking with this now is like.
 So they actually that's that's exactly the question. Right. Of what I so I believe you need such a meta whatever because the space is too large.
 You somehow need a way to choose the stepping stones in the first place. Right. You somehow need a way to do this.
 Now, what they're saying is that if you're if your goal is really ambitious, then a meta controller that simply wants to reach the goal is bad because right because what we discussed before, you might need a lot of inventions from other fields in order to make goal happen.
 And if you simply go your field maximum power towards your goal, that's not going to happen. Now, if your meta controller is actually just something that wants to produce interesting things, then that's actually something they advocate for.
 That is exactly what their algorithms are trying to capture. They're trying to capture locally. Yeah, we want to get better at a particular thing.
 What those particular things are and the order of these that should be novelty driven instead of goal driven.
 Yeah, yeah. Yeah. The interesting component. I guess I'm sort of biased towards liking the objective design.
 And now I'm thinking like, OK, well, let's abstract those meta controllers one level up and have a meta meta controller and just repeat this and hierarchy makes sense.
 And that if you if you if you're if you're a bit cynical, that is what you will also hear out of here out of and they have to argue in the in their book a lot against that like isn't the question isn't the kind of isn't the implementation of a meta controller that just searches for novelty in itself.
 And that's the objective again. And then they give some good reasons why actually you don't.
 It is different. It's more like a constraint on your search. If you think of natural evolution, for example, it isn't really doesn't really have an objective.
 You think reproduction and survival is the objective of natural evolution. It doesn't really the good the good reason they give is the objective has already been fulfilled by the very first organism to ever live.
 Right. Why didn't it stop there? Why didn't it stop very first cell? OK, done. We've fulfilled the objective.
 It's more of a it's more of an actually a constrained optimization where the constraint is you need to be able to survive.
 That's kind of the minimum bar of to being on this planet. And then I'm saying constrained optimization, but it's it's not it's not an optimization.
 It's more of like a constraint constraint search.
 OK, yeah, I think, yeah, I guess it's just like I don't think I'm closed in this world of trying to think of these constraint problems. And I haven't really like thought more generally about just like exploration as a whole.
 But but anyway, so I just wanted to ask you generally like your deep learning researcher, I want to ask like what areas of deep learning are you really interested in right now?
 And what do you think is promising in the near future? So I'm currently working in adversarial examples.
 That is a really interesting topic. There's lots of questions still still open, but I'm generally interested in pretty much any anything that is not.
 I'm not too interested in like the newest the newest fine technique on getting the latest state of the art numbers, even though that's probably super important for practitioners.
 Basically, agreeing more with the authors of this tutorial of that.
 Let's just try to do interesting things. And to me, these these actually these these areas in terms of open ended, open ended search, open ended learning are very interesting.
 I think reinforcement learning still has a long way to go.
 I think actually NLP still has a long way to go because I don't believe it's the current models are the end of it.
 So I think it's really exciting time.
 Yeah, I love thinking about adversarial examples because it definitely flips the CNN idea on its head.
 And then I had one other thing about adversarial examples that I'm interested in is there is like an interview with Elon Musk and this Lex Friedman researcher where he asked him about adversarial examples on his self-driving cars.
 And he seems dismissive of it. He says he thinks basically you could just average different patches of like test time augmentation to overcome adversarial examples.
 So in your research, do you think that like the example where they add the noise mass to the panda and they're like, oh, it's a given now, if they just perturbed it like nine more times, do you think the prediction would average out to pandas?
 That is a very difficult question. And from experience, simply adding noise and then feeding it to the classifier, even if you average after that, usually will defend against adversarial examples to a point.
 But it will also degrade your classification performance.
 Because so maybe I understood it wrong, but my understanding is I have my input, right? I simply add noise to it and then feed it through the network.
 And I could do this many times, right? And then average the prediction.
 But usually this will help against adversarial examples, but it will also degrade the accuracy of that classifier. So it might actually make your self-driving car worse in the overall.
 Because how often is it going to be attacked against a adversarial example? It's going to be attacked maybe once or twice a year, maybe if it drives by some hacker's house, right?
 Sticker on a stop sign or something. But the rest of the time, I would actually like to retain the best possible classifier.
 And if I always have to add noise, then that's not possible. So the research we're doing is actually into the direction of can we retain the original accuracy while still kind of detecting these samples?
 I mean, you somehow have to get a trade off somewhere, but just adding noise isn't the final solution yet.
 I was like, so with these adversarial examples, they're only going to make misclassifications like that if it really is adversarially sought after.
 It's not just like the noise perturbation would be such an enormous space to find it otherwise.
 Yes, you really need to try. So it's very unlikely that some random thing.
 Of course, these networks can be confused by random noise, but I think one of the self-driving cars once drove into a big white truck because it was large and white, so it thought it was sky.
 But other than these failures, you really have to try to find an adversarial example.
 Really cool. Yannick, thanks so much for doing this. Anybody watching or listening, definitely check out Yannick's YouTube channel.
 He has really great paper summaries and all sorts of things. Thank you.
 Thanks so much for having me.