CURL: Contrastive Unsupervised Representations for Reinforcement Learning
deep learning machine learning rl reinforcement learning unsupervised contrast contrastive encoder self-supervised deep rl representation representation learning query key
Contrastive Learning has been an established method in NLP and Image classification. The authors show that with relatively minor adjustments, CL can be used to augment and improve RL dramatically.

Paper: https://arxiv.org/abs/2004.04136
Code: https://github.com/MishaLaskin/curl

Abstract:
We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 2.8x and 1.6x performance gains respectively at the 100K interaction steps benchmark. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency and performance of methods that use state-based features.

Authors: Aravind Srinivas, Michael Laskin, Pieter Abbeel

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're going to look at CURL, Contrastive Unsupervised Representations for Reinforcement Learning, by Aravind Srinivas, Michael Laskin and Pieter Abbeel. So this is a general framework for unsupervised representation learning for RL. So let's untangle the title a little bit. It is FOR reinforcement learning, which if you don't know what reinforcement learning is, I've done a bunch of videos on RL frameworks. So it's for general reinforcement learning. That means it can be paired with almost any RL algorithm out there. So we're not going to dive into specific RL algorithms today. It is unsupervised, which means it doesn't need any sort of labels, and it also doesn't need a reward signal for RL, which is pretty cool because usually the entire RL pipelines rely on some sort of a reward or auxiliary reward signal. Now there is a training objective here, but it doesn't have to do with the RL reward. And then it is learning representations, which means it learns intermediate representations of the input data that is useful. And in the end it is contrastive, and that is the secret sauce in here. The training objective is what's called contrastive learning, and that's what we're going to spend most of our time on today, exploring what that means. So here's the general framework. You can see it down here. Sorry about that. So you can see that reinforcement learning is just a box, which is we don't care about the RL algorithm you use, that's just what comes at the end. What comes at the beginning, oh, here is the observation. So the observation in an RL algorithm is kind of fundamental. Now if someone explains RL to you, reinforcement learning, usually what they'll say is there is some kind of actor and there is some kind of environment. And the environment will give you an observation, observation O, which is some sort of, let's say here is an image. So in this RL framework specifically, the examples they give are of image-based reinforcement learning. Let's say the Atari game where you have this little spaceship here and there are meteorites up here, and you need to shoot them. So there is a little shot here. You need to shoot those meteorites. So this is the observation O. And then as an age, as an actor, you have to come up with some sort of action. And the actions here can be something like move to the left, move to the right, press the button that does the shooting. So you have to come up with an action somehow given this observation. And then the environment will give you back a reward along with the next observation, like the next frame of the game. And you're going to have to come up with another action in response to that. And the environment is going to give you back another reward and the next observation and so on. So what you want to do is you want to find a mapping from observation to action, such that your reward is going to be as high as possible. This is the fundamental problem of RL. And usually what people do is they take this mapping here from observation to action to be some sort of function, some sort of function that is parameterized maybe. Nowadays, of course, it's often a neural network. But you're trying to learn, given the input observation, what output action you need to do. And you can think of the same here. So you have this input observation up here. And down here, after the reinforcement learning, the output is going to be an action. And so this function we talked about up here is usually implemented. It's usually implemented like this. You put the observation into the RL framework. And then the RL framework learns this f of theta function to give you an action. Now here you can see the pipeline is a bit different. We don't want to shove the observation in directly, right? We don't want the observation directly. But what we put into the RL framework is this queue thing. Now the queue is supposed to be a representation of the observation and a useful representation. So if we think of this game here, of this Atari game up here, what could be a useful representation if I had to craft one by hand? How would I construct a useful representation? Keep in mind the goal is to have a representation of the observation that is more useful to the RL algorithm than just the pure pixels of the image. So if I had to craft a representation, let's say it's a vector. Let's say our representations need to be vectors. What I would do is I would probably take the x and y coordinates of the little spaceship, x and y, and put it in the vector. That's pretty useful. Then I would probably take the x and y coordinates of the meteorites that are around. Let's say there's a maximum of two, so x, y, x, y here. I would probably take the angle where my spaceship is pointing to. That should be pretty useful because if I shoot, I want to know where I shoot. So theta here, and then probably the x and y coordinates of the red shot that I fired, if there is one. I'm also going to put that into my representation. So x and y, and maybe delta x, delta y. Something like this. You can see if I had to handcraft something, I can pretty much guarantee that if I put in this representation right here into the RL algorithm, if I put this in here, it would turn out guaranteed, it would turn out to be a better RL agent that learns faster than if I put in the original observation, which is the pixel image of the game. Because, of course, in order to play the game correctly, in order to play the game to win, you need to extract this information. You need to get, ah, there's something like a spaceship, there's something like meteorites. This is all things that the RL algorithm doesn't know per se, and would have to learn from the pixels. But if I already give it the information that is useful, it can learn much faster. So you can see if I handcraft a good representation, it's pretty easy for the RL algorithm to improve. Now we want to come up with a framework that automatically comes up with a good representation. So it alleviates the RL algorithm here, the reinforcement learning. It alleviates that from having to learn a good representation. It already is burdened with learning what a good action is in any given situation. We want to alleviate it of the burden to also extract useful information from the observation space. So how do we do this? This Q here is supposed to be exactly that. It's supposed to be a good representation, but not one that we handcrafted, but used with a technique that can be employed pretty much everywhere. The goal, sorry, the secret sauce here is this contrastive loss thing. Okay, this bombed. Contrastive learning is this kind of magic thing that will make us good representations. What is contrastive learning? In this case, I'm going to explain it. In this case, for image-based reinforcement learning, but just for image-based neural networks, how can we come up with a contrastive loss? So you see there's a two pipeline thing going on here. This and this, and then one of them is going to be the good encoding. So let's check it out. Let's say we have this image that we had before. Draw it again. This little spaceship. This and this. And shot. We want to do this. What we need to do is we need to produce three different things from it. We need to produce an anchor, what's called an anchor. We need to produce a positive sample. And we need to produce negative samples. Let's just go with one negative sample for now. The goal is to come up with a task where we produce our own labels. Since we're training an encoder, and the encoder is a neural network that is parameterized, we need some sort of loss function. The goal is to come up with a method where we can create our own labels to a task, but we construct the task in a way such that the neural network has no choice and we can create something meaningful, even though we made the task up ourselves. I hope this was kind of clear. How are we going to do this? Our method of choice here is going to be random cropping. Random cropping means that I take an image and I crop a piece from it. A smaller piece from the image. I take a view inside the image. In case of the anchor, I'm going to draw the same picture here. Bear with me, I'm going to draw the same picture here a couple of times. This is all supposed to be the same picture. With the negative sample, I'm just going to leave it empty for now. Ta-da! Two meteorites. Two meteorites. Shot. Shot. For the anchor, we're going to center crop. We're going to take the center image. The assumption is that if I center crop, I won't lose too much of the image. I can actually make the crop bigger, such that almost everything of the image is somewhat contained in this. This is going to be my anchor. The positive sample is going to be a random crop of the same image. I'm just randomly going to select a same size section from that image. Let's say this is up right here. The negative sample is going to be a random crop from a different image. A different image might be from the same game, but there is a meteorite here and there is no shot. I don't shoot. I'm going to take a random crop from this. Let's say I'm going to take a random crop here. Let's put a meteorite here as well, just for fun. These are going to be our three samples. Now the question is going to be if I give the anchor to the neural network. I give you the anchor, but I'm also going to give you this and this thing. I'm not going to give any of this. I'm just going to give whatever I cropped. Just these things. I ask the neural network, I give you the anchor. Which one of these two crops comes from the same image? As a human you look at this and if you just see the center crop, you see down here there is this tip of this thing and then there is the shot. In relation to the shot there is a meteor here. Then you look at the second one and you say I don't see the spaceship, but there is the same relation here from the shot to the meteor. I can kind of see the meteor up here. This also fits with that. The spaceship must be down here somewhere. Then I go over here and I try to do the same thing. Here is the meteor. In the original image it might be over here somewhere. That's possible. I don't see it. That's possible, but then there should be a shot somewhere here. There should be a shot somewhere here. I'm pretty sure because there is one over here and I don't see it. I am fairly sure that this image here is the positive sample, while this image here is the negative sample. This is the task that you ask of the neural network. Give it the anchor and you ask which one of these two comes from the same image. This is called contrastive learning. It is a bit more complicated in that of course what you do is you encode these things using neural networks. Each of the things you encode. The anchor you are going to encode all of these things using a neural network. Then this is what's going to become the query. These are becoming the keys. Key 1 or key 2. Then you are going to feed always two of them into a bilinear product. A bilinear product is simply an inner product in a perturbed space that you can learn. You are going to have these two here. These go into Q, W, K, 1. Then these two here, sorry, this and this go into Q, W, K, 2. Now W here is a learnable parameter. You have some freedom. Then you basically take whichever one of those two is highest. This might be this high and this might only be this high. Then you say, aha, cool, this one is higher so this one must be the positive. You train the W specifically to make the positive ones higher and the negative ones lower. This is a supervised learning task. These things here are going to be the logits. They are inner products but you basically then pick the one that is highest in a softmax way. They put this in the paper. If we go down here, the objective that they use to do the contrastive learning is this one. As you can see, it's a softmax like in multiclass classification. The inner product, the bilinear product with the positive samples over the bilinear product with the positive samples plus the bilinear product with all of the negative samples. You are going to come up with more than one negative sample. The only thing left that we don't have here is that the encoding, how you are going to come from the image space to this space here, is going to be slightly different depending on whether you are talking on the anchor or on what are called the keys, the things you compare to. This is out of a stability criterion. Maybe you know something like double Q-learning or things like this. Sometimes when you train with your own thing, in Q-learning you are trying to come up with an actor and a critic. It's not the same thing, but you are using the same neural network twice in your setup. Then you compare the outputs to each other, which leads to instability. In our case, we took it three times here, or multiple times. Especially for the same objective here, we have twice something that was encoded by the same neural network and is on the two sides of this bilinear product. If we were to use the same neural network, that tends to be somewhat unstable. We have different neural networks, one that will encode the query, which is this FQ, and one which will encode the keys, sorry, FK. We don't want to learn two neural networks. That's why there's a bit of a compromise, where we say it is the same neural network, but basically this one is the one we learn. Every now and then we transfer over the parameters to that one. In fact, each step we transfer over the parameters and do an exponentially moving average with the parameters of this momentum encoder from the step before. The momentum encoder parameters are a moving average of the parameters of the query encoder. You get the best of both worlds. You don't have to learn a second neural network, but your second neural network is not the same as your first neural network. It kind of lags behind, but it is also performing almost as well. I don't know if that makes sense, but it is the best I can explain it. To recap, you take your observation, you encode it as a query, sorry, you crop here for your anchor, that gets your query, and then you random crop for your keys into positive and negative samples. Random crop from the same observation or from different observations. These become your positive and negative samples. Then you push these through your encoders for the query and for the keys respectively. You end up with the queue, which is the encoded anchor, and the k's, which are the encoded positive and negative samples. Then you learn, you update this encoder here using the contrastive loss. At the same time, you feed the queue into the reinforcement learning algorithm, and you learn your reinforcement learning algorithm. Instead of having the observation directly as an input here, you now have the queue here as an input. The reinforcement learning works exactly the same, except having the pixel input O, you now have the representation input Q. You don't have to worry about anything else in terms of the reinforcement learning algorithm. It works exactly the same. This whole thing here can run either in parallel, or you can think of it before, you can think of it off-policy, on-policy. It is sort of modular how you fit this in. It simply comes up with good representation. That is basically the deal here. You hope that the whole procedure of this contrastive learning then gives you good representation of this anchor thing here. If you encode that to the queue, you hope that this representation now is a good representation as a basis for the RL algorithm. It turns out, at least in their experiments, it is. Here you see the same thing. You can do something more where in RL you usually deal with a stack of observations, not just a single observation. For example, in Atari, people always concatenate something like the four last frames. Their point is, if we have this stack here, if we do this data augmentation, these crops, we need to do them consistently. We need to crop every single image at the same point for the query. Also, if we do a random crop, let's say a random crop down here, we need to do this same random crop for all of the stack of images here. That is the additional thing they introduce with respect to RL that deals with stacked timeframes. It's the same diagram as above here. They explain the RL algorithms they use and exactly their thing. Here you can see that the anchor is a crop, and the positive sample is a random crop from the same image. This would be up here somewhere. The anchor is cropped from the middle. Then the negative would be a random crop from a different image or a different stack of images. They have a pseudocode here. It's pretty simple. We'll just go through it quickly. You start off with FQ and FK. These are the encoders for the query and keys. You start them off the same. Then you go through your data loader. You do this random augmentation of your query and your keys. I'm not even sure if the random augmentation needs to be a center crop for the anchor, but it's just two different crops from the same image. I guess it's a thing you could choose. I don't know what exactly is the best thing. Then I forward the query through the FQ and I forward the keys through the FK. It's important to detach this so I don't want to train the FK. I only want to train the FQ. Then I do the bilinear product here with the W. These are the bilinear product. Then I put all of this into a cross entropy loss. In the end I update my FQ and my W and I do this exponentially moving average for my key encoder. They test on two different things. They test on the DeepMind control tasks. They always test 100k time steps. Their big point is data efficiency. They claim they can learn useful representations with not much data. The task is here, how good are you at 100k time steps? You don't optimize until the end. You get 100k time steps and then the question is how good are you? The curl here outperforms all of the baselines handily in the DeepMind control tasks. It also outperforms a lot of the baselines in the Atari tasks. If you look at the results, it doesn't outperform everything. For example, the red is curl and the dashed grey is stateSAC. StateSAC has access to the state. Curl only works from pixels. If I had to craft a representation, stateSAC has access to that. You see that in many of the tasks, the curl comes close or performs equally well to stateSAC. That's pretty impressive. Especially if you look at pixelSAC, which is the same algorithm but does not have access to the state, it often fails terribly. That is pretty interesting to see. Even to me, it's pretty interesting to see that this kind of self-labeled algorithm comes up with such useful representations. I hope I have explained this satisfactorily. Check out the paper for more experiments, ablation studies and general reading. I wish you a good day.
 Hi there! Today we're going to look at CURL, Contrastive Unsupervised Representations for Reinforcement Learning,
 by Aravind Srinivas, Michael Laskin and Pieter Abbeel.
 So this is a general framework for unsupervised representation learning for RL.
 So let's untangle the title a little bit.
 It is FOR reinforcement learning, which if you don't know what reinforcement learning is,
 I've done a bunch of videos on RL frameworks.
 So it's for general reinforcement learning.
 That means it can be paired with almost any RL algorithm out there.
 So we're not going to dive into specific RL algorithms today.
 It is unsupervised, which means it doesn't need any sort of labels,
 and it also doesn't need a reward signal for RL,
 which is pretty cool because usually the entire RL pipelines rely on some sort of a reward or auxiliary reward signal.
 Now there is a training objective here, but it doesn't have to do with the RL reward.
 And then it is learning representations, which means it learns intermediate representations of the input data that is useful.
 And in the end it is contrastive, and that is the secret sauce in here.
 The training objective is what's called contrastive learning,
 and that's what we're going to spend most of our time on today, exploring what that means.
 So here's the general framework. You can see it down here.
 Sorry about that.
 So you can see that reinforcement learning is just a box, which is we don't care about the RL algorithm you use,
 that's just what comes at the end.
 What comes at the beginning, oh, here is the observation.
 So the observation in an RL algorithm is kind of fundamental.
 Now if someone explains RL to you, reinforcement learning,
 usually what they'll say is there is some kind of actor and there is some kind of environment.
 And the environment will give you an observation, observation O, which is some sort of, let's say here is an image.
 So in this RL framework specifically, the examples they give are of image-based reinforcement learning.
 Let's say the Atari game where you have this little spaceship here and there are meteorites up here,
 and you need to shoot them. So there is a little shot here.
 You need to shoot those meteorites.
 So this is the observation O.
 And then as an age, as an actor, you have to come up with some sort of action.
 And the actions here can be something like move to the left, move to the right,
 press the button that does the shooting.
 So you have to come up with an action somehow given this observation.
 And then the environment will give you back a reward along with the next observation,
 like the next frame of the game.
 And you're going to have to come up with another action in response to that.
 And the environment is going to give you back another reward and the next observation and so on.
 So what you want to do is you want to find a mapping from observation to action,
 such that your reward is going to be as high as possible.
 This is the fundamental problem of RL.
 And usually what people do is they take this mapping here from observation to action
 to be some sort of function, some sort of function that is parameterized maybe.
 Nowadays, of course, it's often a neural network.
 But you're trying to learn, given the input observation, what output action you need to do.
 And you can think of the same here.
 So you have this input observation up here.
 And down here, after the reinforcement learning, the output is going to be an action.
 And so this function we talked about up here is usually implemented.
 It's usually implemented like this. You put the observation into the RL framework.
 And then the RL framework learns this f of theta function to give you an action.
 Now here you can see the pipeline is a bit different.
 We don't want to shove the observation in directly, right?
 We don't want the observation directly.
 But what we put into the RL framework is this queue thing.
 Now the queue is supposed to be a representation of the observation
 and a useful representation.
 So if we think of this game here, of this Atari game up here,
 what could be a useful representation if I had to craft one by hand?
 How would I construct a useful representation?
 Keep in mind the goal is to have a representation of the observation
 that is more useful to the RL algorithm than just the pure pixels of the image.
 So if I had to craft a representation, let's say it's a vector.
 Let's say our representations need to be vectors.
 What I would do is I would probably take the x and y coordinates of the little spaceship,
 x and y, and put it in the vector. That's pretty useful.
 Then I would probably take the x and y coordinates of the meteorites that are around.
 Let's say there's a maximum of two, so x, y, x, y here.
 I would probably take the angle where my spaceship is pointing to.
 That should be pretty useful because if I shoot, I want to know where I shoot.
 So theta here, and then probably the x and y coordinates of the red shot that I fired, if there is one.
 I'm also going to put that into my representation.
 So x and y, and maybe delta x, delta y.
 Something like this.
 You can see if I had to handcraft something,
 I can pretty much guarantee that if I put in this representation right here into the RL algorithm,
 if I put this in here, it would turn out guaranteed,
 it would turn out to be a better RL agent that learns faster than if I put in the original observation,
 which is the pixel image of the game.
 Because, of course, in order to play the game correctly, in order to play the game to win,
 you need to extract this information.
 You need to get, ah, there's something like a spaceship, there's something like meteorites.
 This is all things that the RL algorithm doesn't know per se, and would have to learn from the pixels.
 But if I already give it the information that is useful, it can learn much faster.
 So you can see if I handcraft a good representation, it's pretty easy for the RL algorithm to improve.
 Now we want to come up with a framework that automatically comes up with a good representation.
 So it alleviates the RL algorithm here, the reinforcement learning.
 It alleviates that from having to learn a good representation.
 It already is burdened with learning what a good action is in any given situation.
 We want to alleviate it of the burden to also extract useful information from the observation space.
 So how do we do this?
 This Q here is supposed to be exactly that.
 It's supposed to be a good representation, but not one that we handcrafted,
 but used with a technique that can be employed pretty much everywhere.
 The goal, sorry, the secret sauce here is this contrastive loss thing.
 Okay, this bombed.
 Contrastive learning is this kind of magic thing that will make us good representations.
 What is contrastive learning?
 In this case, I'm going to explain it.
 In this case, for image-based reinforcement learning, but just for image-based neural networks,
 how can we come up with a contrastive loss?
 So you see there's a two pipeline thing going on here.
 This and this, and then one of them is going to be the good encoding.
 So let's check it out.
 Let's say we have this image that we had before.
 Draw it again.
 This little spaceship.
 This and this.
 And shot.
 We want to do this.
 What we need to do is we need to produce three different things from it.
 We need to produce an anchor, what's called an anchor.
 We need to produce a positive sample.
 And we need to produce negative samples.
 Let's just go with one negative sample for now.
 The goal is to come up with a task where we produce our own labels.
 Since we're training an encoder, and the encoder is a neural network that is parameterized,
 we need some sort of loss function.
 The goal is to come up with a method where we can create our own labels to a task,
 but we construct the task in a way such that the neural network has no choice
 and we can create something meaningful, even though we made the task up ourselves.
 I hope this was kind of clear.
 How are we going to do this?
 Our method of choice here is going to be random cropping.
 Random cropping means that I take an image and I crop a piece from it.
 A smaller piece from the image.
 I take a view inside the image.
 In case of the anchor, I'm going to draw the same picture here.
 Bear with me, I'm going to draw the same picture here a couple of times.
 This is all supposed to be the same picture.
 With the negative sample, I'm just going to leave it empty for now.
 Ta-da! Two meteorites. Two meteorites.
 Shot. Shot.
 For the anchor, we're going to center crop.
 We're going to take the center image.
 The assumption is that if I center crop, I won't lose too much of the image.
 I can actually make the crop bigger, such that almost everything of the image
 is somewhat contained in this.
 This is going to be my anchor.
 The positive sample is going to be a random crop of the same image.
 I'm just randomly going to select a same size section from that image.
 Let's say this is up right here.
 The negative sample is going to be a random crop from a different image.
 A different image might be from the same game,
 but there is a meteorite here and there is no shot.
 I don't shoot.
 I'm going to take a random crop from this.
 Let's say I'm going to take a random crop here.
 Let's put a meteorite here as well, just for fun.
 These are going to be our three samples.
 Now the question is going to be if I give the anchor to the neural network.
 I give you the anchor, but I'm also going to give you this and this thing.
 I'm not going to give any of this.
 I'm just going to give whatever I cropped.
 Just these things.
 I ask the neural network, I give you the anchor.
 Which one of these two crops comes from the same image?
 As a human you look at this and if you just see the center crop,
 you see down here there is this tip of this thing and then there is the shot.
 In relation to the shot there is a meteor here.
 Then you look at the second one and you say I don't see the spaceship,
 but there is the same relation here from the shot to the meteor.
 I can kind of see the meteor up here.
 This also fits with that.
 The spaceship must be down here somewhere.
 Then I go over here and I try to do the same thing.
 Here is the meteor.
 In the original image it might be over here somewhere.
 That's possible. I don't see it.
 That's possible, but then there should be a shot somewhere here.
 There should be a shot somewhere here.
 I'm pretty sure because there is one over here and I don't see it.
 I am fairly sure that this image here is the positive sample,
 while this image here is the negative sample.
 This is the task that you ask of the neural network.
 Give it the anchor and you ask which one of these two comes from the same image.
 This is called contrastive learning.
 It is a bit more complicated in that of course what you do is you encode these things using neural networks.
 Each of the things you encode.
 The anchor you are going to encode all of these things using a neural network.
 Then this is what's going to become the query.
 These are becoming the keys. Key 1 or key 2.
 Then you are going to feed always two of them into a bilinear product.
 A bilinear product is simply an inner product in a perturbed space that you can learn.
 You are going to have these two here.
 These go into Q, W, K, 1.
 Then these two here, sorry, this and this go into Q, W, K, 2.
 Now W here is a learnable parameter.
 You have some freedom.
 Then you basically take whichever one of those two is highest.
 This might be this high and this might only be this high.
 Then you say, aha, cool, this one is higher so this one must be the positive.
 You train the W specifically to make the positive ones higher and the negative ones lower.
 This is a supervised learning task.
 These things here are going to be the logits.
 They are inner products but you basically then pick the one that is highest in a softmax way.
 They put this in the paper.
 If we go down here, the objective that they use to do the contrastive learning is this one.
 As you can see, it's a softmax like in multiclass classification.
 The inner product, the bilinear product with the positive samples
 over the bilinear product with the positive samples plus the bilinear product with all of the negative samples.
 You are going to come up with more than one negative sample.
 The only thing left that we don't have here is that the encoding,
 how you are going to come from the image space to this space here,
 is going to be slightly different depending on whether you are talking on the anchor
 or on what are called the keys, the things you compare to.
 This is out of a stability criterion.
 Maybe you know something like double Q-learning or things like this.
 Sometimes when you train with your own thing,
 in Q-learning you are trying to come up with an actor and a critic.
 It's not the same thing, but you are using the same neural network twice in your setup.
 Then you compare the outputs to each other, which leads to instability.
 In our case, we took it three times here, or multiple times.
 Especially for the same objective here, we have twice something that was encoded by the same neural network
 and is on the two sides of this bilinear product.
 If we were to use the same neural network, that tends to be somewhat unstable.
 We have different neural networks, one that will encode the query, which is this FQ,
 and one which will encode the keys, sorry, FK.
 We don't want to learn two neural networks.
 That's why there's a bit of a compromise, where we say it is the same neural network,
 but basically this one is the one we learn.
 Every now and then we transfer over the parameters to that one.
 In fact, each step we transfer over the parameters and do an exponentially moving average
 with the parameters of this momentum encoder from the step before.
 The momentum encoder parameters are a moving average of the parameters of the query encoder.
 You get the best of both worlds.
 You don't have to learn a second neural network, but your second neural network
 is not the same as your first neural network.
 It kind of lags behind, but it is also performing almost as well.
 I don't know if that makes sense, but it is the best I can explain it.
 To recap, you take your observation, you encode it as a query, sorry,
 you crop here for your anchor, that gets your query,
 and then you random crop for your keys into positive and negative samples.
 Random crop from the same observation or from different observations.
 These become your positive and negative samples.
 Then you push these through your encoders for the query and for the keys respectively.
 You end up with the queue, which is the encoded anchor,
 and the k's, which are the encoded positive and negative samples.
 Then you learn, you update this encoder here using the contrastive loss.
 At the same time, you feed the queue into the reinforcement learning algorithm,
 and you learn your reinforcement learning algorithm.
 Instead of having the observation directly as an input here,
 you now have the queue here as an input.
 The reinforcement learning works exactly the same,
 except having the pixel input O, you now have the representation input Q.
 You don't have to worry about anything else in terms of the reinforcement learning algorithm.
 It works exactly the same.
 This whole thing here can run either in parallel, or you can think of it before,
 you can think of it off-policy, on-policy.
 It is sort of modular how you fit this in.
 It simply comes up with good representation.
 That is basically the deal here.
 You hope that the whole procedure of this contrastive learning then gives you good representation of this anchor thing here.
 If you encode that to the queue, you hope that this representation now is a good representation as a basis for the RL algorithm.
 It turns out, at least in their experiments, it is.
 Here you see the same thing.
 You can do something more where in RL you usually deal with a stack of observations,
 not just a single observation.
 For example, in Atari, people always concatenate something like the four last frames.
 Their point is, if we have this stack here, if we do this data augmentation, these crops,
 we need to do them consistently.
 We need to crop every single image at the same point for the query.
 Also, if we do a random crop, let's say a random crop down here,
 we need to do this same random crop for all of the stack of images here.
 That is the additional thing they introduce with respect to RL that deals with stacked timeframes.
 It's the same diagram as above here.
 They explain the RL algorithms they use and exactly their thing.
 Here you can see that the anchor is a crop, and the positive sample is a random crop from the same image.
 This would be up here somewhere.
 The anchor is cropped from the middle.
 Then the negative would be a random crop from a different image or a different stack of images.
 They have a pseudocode here.
 It's pretty simple. We'll just go through it quickly.
 You start off with FQ and FK. These are the encoders for the query and keys.
 You start them off the same.
 Then you go through your data loader.
 You do this random augmentation of your query and your keys.
 I'm not even sure if the random augmentation needs to be a center crop for the anchor,
 but it's just two different crops from the same image.
 I guess it's a thing you could choose. I don't know what exactly is the best thing.
 Then I forward the query through the FQ and I forward the keys through the FK.
 It's important to detach this so I don't want to train the FK.
 I only want to train the FQ.
 Then I do the bilinear product here with the W.
 These are the bilinear product.
 Then I put all of this into a cross entropy loss.
 In the end I update my FQ and my W and I do this exponentially moving average for my key encoder.
 They test on two different things.
 They test on the DeepMind control tasks.
 They always test 100k time steps.
 Their big point is data efficiency.
 They claim they can learn useful representations with not much data.
 The task is here, how good are you at 100k time steps?
 You don't optimize until the end.
 You get 100k time steps and then the question is how good are you?
 The curl here outperforms all of the baselines handily in the DeepMind control tasks.
 It also outperforms a lot of the baselines in the Atari tasks.
 If you look at the results, it doesn't outperform everything.
 For example, the red is curl and the dashed grey is stateSAC.
 StateSAC has access to the state.
 Curl only works from pixels.
 If I had to craft a representation, stateSAC has access to that.
 You see that in many of the tasks, the curl comes close or performs equally well to stateSAC.
 That's pretty impressive.
 Especially if you look at pixelSAC, which is the same algorithm but does not have access to the state,
 it often fails terribly.
 That is pretty interesting to see.
 Even to me, it's pretty interesting to see that this kind of self-labeled algorithm comes up with such useful representations.
 I hope I have explained this satisfactorily.
 Check out the paper for more experiments, ablation studies and general reading.
 I wish you a good day.