Reinforcement Learning with Augmented Data (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl reinforcement learning sac ppo deep rl deep reinforcement learning dreamer curl pixel pretraining deepmind openai berkeley
This ONE SIMPLE TRICK can take a vanilla RL algorithm to achieve state-of-the-art. What is it? Simply augment your training data before feeding it to the learner! This can be dropped into any RL pipeline and promises big improvements across the board.

Paper: https://arxiv.org/abs/2004.14990
Code: https://www.github.com/MishaLaskin/rad

Abstract:
Learning from visual observations is a fundamental yet challenging problem in reinforcement learning (RL). Although algorithmic advancements combined with convolutional neural networks have proved to be a recipe for success, current methods are still lacking on two fronts: (a) sample efficiency of learning and (b) generalization to new environments. To this end, we present RAD: Reinforcement Learning with Augmented Data, a simple plug-and-play module that can enhance any RL algorithm. We show that data augmentations such as random crop, color jitter, patch cutout, and random convolutions can enable simple RL algorithms to match and even outperform complex state-of-the-art methods across common benchmarks in terms of data-efficiency, generalization, and wall-clock speed. We find that data diversity alone can make agents focus on meaningful information from high-dimensional observations without any changes to the reinforcement learning method. On the DeepMind Control Suite, we show that RAD is state-of-the-art in terms of data-efficiency and performance across 15 environments. We further demonstrate that RAD can significantly improve the test-time generalization on several OpenAI ProcGen benchmarks. Finally, our customized data augmentation modules enable faster wall-clock speed compared to competing RL techniques. Our RAD module and training code are available at this https URL.

Authors: Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there. Today we're going to take a short look at reinforcement learning with augmented data. This paper is by Michael Laskin, Kimmin Lee, and others from UC Berkeley and NYU. So the reason why this is a short look is because I believe the statements made in the paper are quite short and small, but they are quite grandiose. So we'll dive into it. The paper basically combines two things reinforcement learning and data augmentation. Now reinforcement learning, we've talked about a number of times, it's basically where an agent is in a world and has to learn to solve an optimization problem by repeatedly interacting with the world. You can see here, for example, this is the walker task, where this walker thing, it has two feet and basically needs to stand upright and walk for a number of steps. The further you go, the better. So by repeatedly trying this and getting better and better at it, that is reinforcement learning. The second part is the data augmentation. Now data augmentation is a pretty standard practice in supervised learning. What does it mean? So if you have a supervised learning task, for example, an image classification task, here is a picture of a cat, and the label is cat, then you can feed this through your neural network to arrive at a loss. But you only have so many pictures. You have a database and maybe you have, I don't know, 1 million images. Usually what people do is they go, let's say a number of times, like 20 or 50 times through that database to basically have the model learn each image multiple times. But what turns out to be more successful is if you do data augmentation, that means you have an in between layer right here that takes this image and some modifies it in some small way. This could be for example, it blocks out part of the image. So it simply blocks out the square here. And then you feed that through the the model. And then the next time the image comes up, it does something different. For example, it randomly crops the image to only the top right part here. And then the next time it does a bit of a color jitter. And then the next time it goes to grayscale and so on. So supervised learning has found data augmentation to be quite beneficial because not only do you make the model learn what this picture is, but you also make the model kind of learn some small variations of that picture where you can be pretty sure they would not change the label. So you would not feed the model false information that generally makes it more robust to test time discrepancies. So this paper has basically claims. If you want to do reinforcement learning, if you do simply do data augmentation with the input data to that reinforcement learning, it works much, much, much better. Now, of course, we can expect since in supervised learning, this is a general trick that it would do something for reinforcement learning as well. But this paper basically claims that this one plugin like here, so this is basically you plug this into your pipeline in reinforcement learning, this is basically as much of a gain as pretty much the last five years of research on on reinforcement learning on these things. So let's dive into it. This paper proposes just what I said, just plug in the data augmentation and then do reinforcement learning on the augmented data. They use these data augmentations. So crop we've already discussed, it's a random crop, grayscale means that the picture goes to gray, black and white with a certain probability. Cut out means that there's a little patch missing, like I said, cut out color the same but in a random color. Flip means you flip the image horizontally or vertically, according to a random probability. Rotate is the same but you instead of flip you rotate it. Random conv means you randomly convolve it with a filter. In this case, some red or blue or yellow filters. And color jitter means that you kind of jitter around the colors in a sort of in a sort of way that doesn't mess up the image too much. So you basically just kind of change the colors on the image, but the overall image still looks the same. The only thing you have to you have to pay attention to is that so in your reinforcement learning pipeline, usually if you have a walker like this, what you want to do is you have your network here and then you have you know, your policy and your value function. If you don't know what these are, we'll have we have I've treated them many times in reinforcement learning videos. What you want to do is you simply don't want to take this one current observation in here. But sometimes you want to take kind of the stacked of the last few frames so that the model kind of gets an idea what happened during let's say the last one second, right, so we can it can determine in this walker, for example, it's not only important where the legs are, which are up here right now. It is also important their momentum how they're moving right and you can you can you can determine that from the last few frames. So sometimes it's beneficial to feed the last few frames. And they say the important thing here is that these augmentations are applied consistently across the stacked frames. So basically you select on an augmentation and on the scale of that augmentation and then you apply it to these stacked frames all the same. And then in the next forward pass, you have a different set of stacked frames, then you can decide on a on a different augmentation. So that's basically the only difference between the supervised setting and this setting is that you have to consistently apply the augmentation. And you have to consistently apply this here and during training. So they formulate the classic proximal policy optimization here, which is an actor critic method. And the only time you have to really pay attention is when you plug the observation into these models here, right here, then it needs to be the same augmentation. Sorry, the same observation. So that means the observation augmented with the same data with the same augmentation procedures. All right, getting it together. Cool. So when you do this, when you do that, they say when applying our ad, which is the random random data augmentation to SAC, which is soft actor critic, right? Our data augmentations are applied to the observation past the Q and pi. So sorry, this is the thing up here. This is soft actor critic, which is a state of the art of policy algorithm for continuous control problems. And also you have to pay attention that when you feed the observations, they're the same observations, like here and here. And then proximal policy optimization is the one is a state of the art on policy algorithm for learning a continuous or discrete control policy. Okay, so as I said, they simply drop this in there. And then it turns out they outperform or match performance of many, many baselines. Here you can see curl, I've made a video on curl, which is another way of augmenting or pre training a for reinforcement learning, then state of the art things like planet or dreamer, I've made a video on dreamer as well. And then pixel SAC and state SAC is sort of a cheating algorithm because it has access to the state whereas all the other methods only have access to the to the pixels. And you can see that the data augmentation method, which is basically just plain RL plane pure SAC plus the plus the data augmentation outperforms in many times all of these other baselines. Now, here is a criticism of me. In order they never investigate, they simply say wow, this reaches the same performance or outperforms these other methods. Now, so it's the state of the art algorithm. It's important to note here that this is on the DM control 100k and 500k benchmarks, which means that there's a limit on the number of I believe frames from these control tasks that you get. So you either get 100k or you get 500k frames. So the difficulty is learning from limited data. It's not state of the art reinforcement learning method. Overall, it is the state of the art on this particular task on learning from limited data. Now, while I can believe that the augmentation would help here, I it is completely unclear whether or not the augmentation gives the same benefits as like something like dreamer, or whether the benefits from dreamer and the benefits from data augmentation are completely orthogonal. So in this paper, given that the claim is so simple that they make, I wouldn't I would expect like an investigation, what happens if I do dreamer plus the data augmentation? Maybe they've done it somewhere, and I just haven't seen it. But it just seems like they, they put this on the base basic RL algorithm, and then they claim, well, look here, it works well, but they never show that. So it could be that dreamer all this architecture, what it simply does is basically recover these gains that you could get by data augmentation, or it could be that it actually does something different, but just reaches the same amount of gain, right, it just reaches the same amount in improvement. And by combining them, you could improve it further. So not not just to get like a better number, but combining the two would actually give a lot of hints as to whether or not this augmentation works in line with the other methods, or whether the other methods are really doing something meaningfully different or not. But this is just not done here. And so they go into the they go into a question of which data augmentations contribute the most, and they get to the point where they say random crop is extremely effective. So they have this table here where they just basically combine two augmentations. And do you see, so for example, this thing here means that you apply grayscale, and then the rotate augmentation. And that gets you to whatever 300 points in this Walker, if you apply crop, and then crop, it gets you to 920 points and beats everything else. So they they say, okay, crop is the most the most effective. And I have I have the sneaking suspicion that these augmentations are so effective, simply because of how we set up these tasks, right, these reinforcement learning tasks, they don't tend to be a real world, they tend to be somewhat simulated. And as you can see here, the image is pretty clear. So you can pretty clearly see that here is the thing, there's no natural background or whatnot, it's procedurally generated, right, there are these stars that could confuse the model a bit. But still, it is so easy visually, this task, that I'm going to guess the whole reason why these image augmentations help is simply because of the way these reinforcement learning tasks right now are set up. And I'm would guess that if we had reinforcement learning in something like the real world, that the image augmentation methods would help in about the way they help unsupervised tasks in in the same data, for example, image net. So that is my sneaking suspicion. And this paper, I want to say it sort of over claims, it's how how absolutely great this works. Of course, it works great on these things. But I think there needs to be an investigation of where, why. So here they have some attention maps on where the algorithm focuses. And you can see when there's no data augmentation, it sort of focuses on good points. But when you do crop, it focuses on this ridge here, which makes sense, right, because that's the thing that needs to be kind of vertical in order for the walker to be stable. And in if you do other things, then you can see it, it doesn't really focus, it focuses on different things. So the crop method seems to make the model focus on the most important part of the image. And as the same with the cheetah task here, so if you don't do augmentation, and some of the augmentation, you can see that it actually focuses on some of these background stars, whereas in the cropped version, it focuses on not on the stars, but actually on the cheetah as a whole, which probably makes sense. Now, again, I have a bit of a I have a bit of a worry with these kinds of experiments, because we already know that crop will give you a much better score, right? So who's to say that if we could train this thing here to the same score, it wouldn't be paying attention to the same part. What they're trying to make clear here is that it is dependent on the particular type of data augmentation that the model gets a better grip on the input. But it is not really a valid comparison if we know that the crop agent performs a better score. And it could simply be that that's the reason why the attention is better, right? That that it is actually solving the problem better. So I mean, of course, this the fact that it's working better is due to the fact that you have crop augmented the data, but the fact that is focusing on the correct parts is not a property of the crop augmentation, but the property of the fact that it reaches a higher score. That was a long winded complaint, but I hope you get what I mean here. The last thing they do is they investigate generalization performance. So improving generalization on this open AI proc gen. Now, as I understand it, this is a reinforcement learning task or suite of tasks where you have procedurally generated levels. So you can sort of train on a bunch of levels, and then test the generalization to new levels that you haven't seen before. So there's a jumper here and star pilot. So they seem like this like a jump and run game or big fish. I don't even know what you have to do in big fish. But you can see that the levels seen here, this is one example, and unseen. So in this example, the background is very different. And I'm going to guess in the jumper thing, not only is the background but also the kind of generated level how you have to jump is quite different. So they investigate whether or not a agent trained on only the seen ones can generalize to the unseen ones. And this table presents the results. And as you can see, the RAD with the crop or with other things outperform the pixel paste based PPOs. Now, there is some nuance to this table here. First of all, you can see that these this crop thing is now only the winner in one of these three tasks, right in the in the big fish thing. In there is another augmentation technique here that wins over at star pilot. But you can see the difference is not that high. And in the dnd jumper with 200 levels, so this is 100 or 200 levels, the the original method is even the best. So here again, I believe this is evidence that that it is very much an interaction of these augmentations with the way the task is set up and not the augmentations themselves or the fact that you're augmenting. For example, if we look at this big fish, we've seen, okay, the what seems to change here is mainly the background, where as in the jumper example, the entire level structure seems to change. So then the the augmentation all of a sudden is not super effective anymore actually hurts, right? So I'm just not I'm just not super convinced by the claims you're making here. And one of the claims I find is, in particular, rad with random crop achieves, no wait, this point down here. Oh, yeah, achieves 55.8% gain over pixel based PPO. Okay. trained with 100 training levels outperforms the pixel based PPO with 200 training levels on both big fish and star pilot environment. This shows that data augmentation can be more effective in learning generalizable representations compared to simply increasing the number of training environments. I this statement. So again, how, like, why do you compare two different things if you don't? Like, if you don't show that maybe they're orthogonal. In fact, they are probably orthogonal because even on the 200 levels you you gain over the pixel based PPO, right? So why the comparison and then second of all, so here we see on the 100 levels, this method is better than the pixel based PPO. And then they claim that okay, they are even better on 100 levels than the pixel based PPO on 200 levels. And why that is true. If you know, if if if a is bigger than b, then probably a is going to be bigger than b plus some epsilon. And right, and and that doesn't, I just think that doesn't really warrant their statement where they say, Oh, look, this is even better. So as if the 100 levels of additional training were the standard measure of more data, like if there is going to be if you're better at the beginning, there's going to be a certain amount of data where you're still better than the other method with more data. And I don't find this super duper surprising, but they make a big claim here out of this. Alright, so in conclusion, I hope I'm not too harsh on this paper, it is a cool paper. And of course, it is cool findings. But I have a big suspicion that the augmentation here works so well simply because of how we set up these RL tasks, because they're visually quite, let's say easy. And therefore, these augmentations that are also our sort of easy abstractions of when an image is visually similar, because all of these things, right, to us as humans, we say, probably doesn't change anything if we just rotate the image. And we this is our prejudice. And we built this prejudice into these simulators for the RL tasks. So they will match up extremely well with these augmentations. And that's the reason I believe these things work and maybe not as much the the fact that you're augmenting. Okay, well, if you like this video, I invite you to check out the paper, subscribe to this channel, tell all your friends about it and leave a like in the comment. Thank you very much and bye bye.
 Hi there. Today we're going to take a short look at reinforcement learning with augmented
 data. This paper is by Michael Laskin, Kimmin Lee, and others from UC Berkeley and NYU.
 So the reason why this is a short look is because I believe the statements made in the
 paper are quite short and small, but they are quite grandiose. So we'll dive into it.
 The paper basically combines two things reinforcement learning and data augmentation. Now reinforcement
 learning, we've talked about a number of times, it's basically where an agent is in a world
 and has to learn to solve an optimization problem by repeatedly interacting with the
 world. You can see here, for example, this is the walker task, where this walker thing,
 it has two feet and basically needs to stand upright and walk for a number of steps. The
 further you go, the better. So by repeatedly trying this and getting better and better
 at it, that is reinforcement learning. The second part is the data augmentation. Now
 data augmentation is a pretty standard practice in supervised learning. What does it mean?
 So if you have a supervised learning task, for example, an image classification task,
 here is a picture of a cat, and the label is cat, then you can feed this through your
 neural network to arrive at a loss. But you only have so many pictures. You have a database
 and maybe you have, I don't know, 1 million images. Usually what people do is they go,
 let's say a number of times, like 20 or 50 times through that database to basically have
 the model learn each image multiple times. But what turns out to be more successful is
 if you do data augmentation, that means you have an in between layer right here that takes
 this image and some modifies it in some small way. This could be for example, it blocks
 out part of the image. So it simply blocks out the square here. And then you feed that
 through the the model. And then the next time the image comes up, it does something different.
 For example, it randomly crops the image to only the top right part here. And then the
 next time it does a bit of a color jitter. And then the next time it goes to grayscale
 and so on. So supervised learning has found data augmentation to be quite beneficial because
 not only do you make the model learn what this picture is, but you also make the model
 kind of learn some small variations of that picture where you can be pretty sure they
 would not change the label. So you would not feed the model false information that generally
 makes it more robust to test time discrepancies. So this paper has basically claims. If you
 want to do reinforcement learning, if you do simply do data augmentation with the input
 data to that reinforcement learning, it works much, much, much better. Now, of course, we
 can expect since in supervised learning, this is a general trick that it would do something
 for reinforcement learning as well. But this paper basically claims that this one plugin
 like here, so this is basically you plug this into your pipeline in reinforcement learning,
 this is basically as much of a gain as pretty much the last five years of research on on
 reinforcement learning on these things. So let's dive into it. This paper proposes just
 what I said, just plug in the data augmentation and then do reinforcement learning on the
 augmented data. They use these data augmentations. So crop we've already discussed, it's a random
 crop, grayscale means that the picture goes to gray, black and white with a certain probability.
 Cut out means that there's a little patch missing, like I said, cut out color the same
 but in a random color. Flip means you flip the image horizontally or vertically, according
 to a random probability. Rotate is the same but you instead of flip you rotate it. Random
 conv means you randomly convolve it with a filter. In this case, some red or blue or
 yellow filters. And color jitter means that you kind of jitter around the colors in a
 sort of in a sort of way that doesn't mess up the image too much. So you basically just
 kind of change the colors on the image, but the overall image still looks the same. The
 only thing you have to you have to pay attention to is that so in your reinforcement learning
 pipeline, usually if you have a walker like this, what you want to do is you have your
 network here and then you have you know, your policy and your value function. If you don't
 know what these are, we'll have we have I've treated them many times in reinforcement learning
 videos. What you want to do is you simply don't want to take this one current observation
 in here. But sometimes you want to take kind of the stacked of the last few frames so that
 the model kind of gets an idea what happened during let's say the last one second, right,
 so we can it can determine in this walker, for example, it's not only important where
 the legs are, which are up here right now. It is also important their momentum how they're
 moving right and you can you can you can determine that from the last few frames. So sometimes
 it's beneficial to feed the last few frames. And they say the important thing here is that
 these augmentations are applied consistently across the stacked frames. So basically you
 select on an augmentation and on the scale of that augmentation and then you apply it
 to these stacked frames all the same. And then in the next forward pass, you have a
 different set of stacked frames, then you can decide on a on a different augmentation.
 So that's basically the only difference between the supervised setting and this setting is
 that you have to consistently apply the augmentation. And you have to consistently apply this here
 and during training. So they formulate the classic proximal policy optimization here,
 which is an actor critic method. And the only time you have to really pay attention is when
 you plug the observation into these models here, right here, then it needs to be the
 same augmentation. Sorry, the same observation. So that means the observation augmented with
 the same data with the same augmentation procedures. All right, getting it together. Cool. So when
 you do this, when you do that, they say when applying our ad, which is the random random
 data augmentation to SAC, which is soft actor critic, right? Our data augmentations are
 applied to the observation past the Q and pi. So sorry, this is the thing up here. This
 is soft actor critic, which is a state of the art of policy algorithm for continuous
 control problems. And also you have to pay attention that when you feed the observations,
 they're the same observations, like here and here. And then proximal policy optimization
 is the one is a state of the art on policy algorithm for learning a continuous or discrete
 control policy. Okay, so as I said, they simply drop this in there. And then it turns out
 they outperform or match performance of many, many baselines. Here you can see curl, I've
 made a video on curl, which is another way of augmenting or pre training a for reinforcement
 learning, then state of the art things like planet or dreamer, I've made a video on dreamer
 as well. And then pixel SAC and state SAC is sort of a cheating algorithm because it
 has access to the state whereas all the other methods only have access to the to the pixels.
 And you can see that the data augmentation method, which is basically just plain RL plane
 pure SAC plus the plus the data augmentation outperforms in many times all of these other
 baselines. Now, here is a criticism of me. In order they never investigate, they simply
 say wow, this reaches the same performance or outperforms these other methods. Now, so
 it's the state of the art algorithm. It's important to note here that this is on the
 DM control 100k and 500k benchmarks, which means that there's a limit on the number of
 I believe frames from these control tasks that you get. So you either get 100k or you
 get 500k frames. So the difficulty is learning from limited data. It's not state of the art
 reinforcement learning method. Overall, it is the state of the art on this particular
 task on learning from limited data. Now, while I can believe that the augmentation would
 help here, I it is completely unclear whether or not the augmentation gives the same benefits
 as like something like dreamer, or whether the benefits from dreamer and the benefits
 from data augmentation are completely orthogonal. So in this paper, given that the claim is
 so simple that they make, I wouldn't I would expect like an investigation, what happens
 if I do dreamer plus the data augmentation? Maybe they've done it somewhere, and I just
 haven't seen it. But it just seems like they, they put this on the base basic RL algorithm,
 and then they claim, well, look here, it works well, but they never show that. So it could
 be that dreamer all this architecture, what it simply does is basically recover these
 gains that you could get by data augmentation, or it could be that it actually does something
 different, but just reaches the same amount of gain, right, it just reaches the same amount
 in improvement. And by combining them, you could improve it further. So not not just
 to get like a better number, but combining the two would actually give a lot of hints
 as to whether or not this augmentation works in line with the other methods, or whether
 the other methods are really doing something meaningfully different or not. But this is
 just not done here. And so they go into the they go into a question of which data augmentations
 contribute the most, and they get to the point where they say random crop is extremely effective.
 So they have this table here where they just basically combine two augmentations. And do
 you see, so for example, this thing here means that you apply grayscale, and then the rotate
 augmentation. And that gets you to whatever 300 points in this Walker, if you apply crop,
 and then crop, it gets you to 920 points and beats everything else. So they they say, okay,
 crop is the most the most effective. And I have I have the sneaking suspicion that these
 augmentations are so effective, simply because of how we set up these tasks, right, these
 reinforcement learning tasks, they don't tend to be a real world, they tend to be somewhat
 simulated. And as you can see here, the image is pretty clear. So you can pretty clearly
 see that here is the thing, there's no natural background or whatnot, it's procedurally generated,
 right, there are these stars that could confuse the model a bit. But still, it is so easy
 visually, this task, that I'm going to guess the whole reason why these image augmentations
 help is simply because of the way these reinforcement learning tasks right now are set up. And I'm
 would guess that if we had reinforcement learning in something like the real world, that the
 image augmentation methods would help in about the way they help unsupervised tasks in in
 the same data, for example, image net. So that is my sneaking suspicion. And this paper,
 I want to say it sort of over claims, it's how how absolutely great this works. Of course,
 it works great on these things. But I think there needs to be an investigation of where,
 why. So here they have some attention maps on where the algorithm focuses. And you can
 see when there's no data augmentation, it sort of focuses on good points. But when you
 do crop, it focuses on this ridge here, which makes sense, right, because that's the thing
 that needs to be kind of vertical in order for the walker to be stable. And in if you
 do other things, then you can see it, it doesn't really focus, it focuses on different things.
 So the crop method seems to make the model focus on the most important part of the image.
 And as the same with the cheetah task here, so if you don't do augmentation, and some
 of the augmentation, you can see that it actually focuses on some of these background stars,
 whereas in the cropped version, it focuses on not on the stars, but actually on the cheetah
 as a whole, which probably makes sense. Now, again, I have a bit of a I have a bit of a
 worry with these kinds of experiments, because we already know that crop will give you a
 much better score, right? So who's to say that if we could train this thing here to
 the same score, it wouldn't be paying attention to the same part. What they're trying to make
 clear here is that it is dependent on the particular type of data augmentation that
 the model gets a better grip on the input. But it is not really a valid comparison if
 we know that the crop agent performs a better score. And it could simply be that that's
 the reason why the attention is better, right? That that it is actually solving the problem
 better. So I mean, of course, this the fact that it's working better is due to the fact
 that you have crop augmented the data, but the fact that is focusing on the correct parts
 is not a property of the crop augmentation, but the property of the fact that it reaches
 a higher score. That was a long winded complaint, but I hope you get what I mean here. The last
 thing they do is they investigate generalization performance. So improving generalization on
 this open AI proc gen. Now, as I understand it, this is a reinforcement learning task
 or suite of tasks where you have procedurally generated levels. So you can sort of train
 on a bunch of levels, and then test the generalization to new levels that you haven't seen before.
 So there's a jumper here and star pilot. So they seem like this like a jump and run game
 or big fish. I don't even know what you have to do in big fish. But you can see that the
 levels seen here, this is one example, and unseen. So in this example, the background
 is very different. And I'm going to guess in the jumper thing, not only is the background
 but also the kind of generated level how you have to jump is quite different. So they investigate
 whether or not a agent trained on only the seen ones can generalize to the unseen ones.
 And this table presents the results. And as you can see, the RAD with the crop or with
 other things outperform the pixel paste based PPOs. Now, there is some nuance to this table
 here. First of all, you can see that these this crop thing is now only the winner in
 one of these three tasks, right in the in the big fish thing. In there is another augmentation
 technique here that wins over at star pilot. But you can see the difference is not that
 high. And in the dnd jumper with 200 levels, so this is 100 or 200 levels, the the original
 method is even the best. So here again, I believe this is evidence that that it is very
 much an interaction of these augmentations with the way the task is set up and not the
 augmentations themselves or the fact that you're augmenting. For example, if we look
 at this big fish, we've seen, okay, the what seems to change here is mainly the background,
 where as in the jumper example, the entire level structure seems to change. So
 then the the augmentation all of a sudden is not super effective anymore actually hurts,
 right? So I'm just not I'm just not super convinced by the claims you're making here.
 And one of the claims I find is, in particular, rad with random crop achieves, no wait, this
 point down here. Oh, yeah, achieves 55.8% gain over pixel based PPO. Okay. trained with
 100 training levels outperforms the pixel based PPO with 200 training levels on both
 big fish and star pilot environment. This shows that data augmentation can be more effective
 in learning generalizable representations compared to simply increasing the number of
 training environments. I this statement. So again, how, like, why do you compare two different
 things if you don't? Like, if you don't show that maybe they're orthogonal. In fact, they
 are probably orthogonal because even on the 200 levels you you gain over the pixel based
 PPO, right? So why the comparison and then second of all, so here we see on the 100 levels,
 this method is better than the pixel based PPO. And then they claim that okay, they are
 even better on 100 levels than the pixel based PPO on 200 levels. And why that is true. If
 you know, if if if a is bigger than b, then probably a is going to be bigger than b plus
 some epsilon. And right, and and that doesn't, I just think that doesn't really warrant their
 statement where they say, Oh, look, this is even better. So as if the 100 levels of additional
 training were the standard measure of more data, like if there is going to be if you're
 better at the beginning, there's going to be a certain amount of data where you're still
 better than the other method with more data. And I don't find this super duper surprising,
 but they make a big claim here out of this. Alright, so in conclusion, I hope I'm not
 too harsh on this paper, it is a cool paper. And of course, it is cool findings. But I
 have a big suspicion that the augmentation here works so well simply because of how we
 set up these RL tasks, because they're visually quite, let's say easy. And therefore, these
 augmentations that are also our sort of easy abstractions of when an image is visually
 similar, because all of these things, right, to us as humans, we say, probably doesn't
 change anything if we just rotate the image. And we this is our prejudice. And we built
 this prejudice into these simulators for the RL tasks. So they will match up extremely
 well with these augmentations. And that's the reason I believe these things work and
 maybe not as much the the fact that you're augmenting. Okay, well, if you like this video,
 I invite you to check out the paper, subscribe to this channel, tell all your friends about
 it and leave a like in the comment. Thank you very much and bye bye.