Datasets for Data-Driven Reinforcement Learning
deep learning machine learning reinforcement learning deep rl off-policy on-policy replay buffer dataset benchmark berkeley rail offline online
Offline Reinforcement Learning has come more and more into focus recently in domains where classic on-policy RL algorithms are infeasible to train, such as safety-critical tasks or learning from expert demonstrations. This paper presents an extensive benchmark for evaluating offline RL algorithms in a variety of settings.

Paper: https://arxiv.org/abs/2004.07219
Code: https://github.com/rail-berkeley/offline_rl

Abstract:
The offline reinforcement learning (RL) problem, also referred to as batch RL, refers to the setting where a policy must be learned from a dataset of previously collected data, without additional online data collection. In supervised learning, large datasets and complex deep neural networks have fueled impressive progress, but in contrast, conventional RL algorithms must collect large amounts of on-policy data and have had little success leveraging previously collected datasets. As a result, existing RL benchmarks are not well-suited for the offline setting, making progress in this area difficult to measure. To design a benchmark tailored to offline RL, we start by outlining key properties of datasets relevant to applications of offline RL. Based on these properties, we design a set of benchmark tasks and datasets that evaluate offline RL algorithms under these conditions. Examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multi-objective datasets, where an agent can perform different tasks in the same environment, and datasets consisting of a heterogeneous mix of high-quality and low-quality trajectories. By designing the benchmark tasks and datasets to reflect properties of real-world offline RL problems, our benchmark will focus research effort on methods that drive substantial improvements not just on simulated benchmarks, but ultimately on the kinds of real-world problems where offline RL will have the largest impact.

Authors: Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at datasets for data-driven reinforcement learning by Justin Fu, Aviral Kumar, Ofer Natschum, George Tucker and Sergei Levine. So this is what you would call a dataset paper or a benchmark paper. And the main point or the main area of the paper was called offline reinforcement learning. So offline reinforcement learning, usually in reinforcement learning you have this task, right? You have the agent and you have the environment. And the agent gets some sort of observation and has to come up with an action in response to that observation. And then it gets back a reward and another observation. And again, it has to come up with an action. And the goal is to maximize the rewards over time that the agent gets while interacting with the environment. So usually this is organized in what are called episodes, which basically means if you have some sort of environment, right, and here is the agent and here is the goal, right, the goal is a inverted triangle. And there are a bunch of walls right here, right? So it looks kind of a maze that the agent has to navigate. Then one episode could be the agent moving around until it either finds the target or hits a wall or just kind of goes around and around. And then at some point you say, all right, that's enough, game over. And usually in reinforcement learning, you perform many of these episodes and then you learn from them. So you perform episodes and each episode gets into usually some sort of replay buffer, right? Let's call this replay buffer. And you do this many times and at the same time that you're doing this, you're using the things that you stored here in order to learn, right? So the agent learns from these things, right? So it acts with the environment in this loop, in this fashion. Then once it has done an episode, it puts it into the replay buffer and then it learns from the actions it has performed. This is what is usually called online reinforcement learning, right? So this loop is online. Online means because the agent learns from its own actions, right? Now in contrast to this, there is offline reinforcement learning. So in offline reinforcement learning, the agent has to learn from someone else's actions, right? So this connection here is severed. Instead you have other agents. Let's call these agent one, agent two, multiple agents, agent three. They all have their own interaction with the environment, right? Environment environment interactions and they feed their experience into they perform these episodes. They feed their experience into the replay buffer. And then the agent just has to learn from that. So whatever happened here, this was previous, right? And now the agent has to learn how to maximize its reward just from the experience that is in the replay buffer from these other agents. This is what's called offline reinforcement learning means the agent learns from someone else's actions. Basically the power of reinforcement learning of course comes from the fact that you learn from your own actions. It means that for example, if you already have some successful trajectories here, right? You found the target. You can try to replicate that because you know which actions you performed. And if you don't, you know, change anything, you're probably going to find the target again just by randomness. All right, because you've done it already once and so on. So you kind of know all the intrinsics of your own algorithm that led you to reach the target. Now this is an entirely different case with all of these other agents. You have no clue how they were acting, why they were acting, right? You just know, okay, they did a series of actions and that gave them some kind of reward. And you have no idea what their reasoning was or anything. All you really can learn from is their sequence of actions. Now why is that problematic, right? So if all of the agents, for example, if this is an actual platform and this is really steep here, this is all of here is really steep cliffs, right? And you can actually fall off. But the agents, they're humans, right? So they don't want to fall off. So what they're going to do is they're just going to take steps that are maybe like this or maybe like this, but they're humans, they're smart. They're never going to fall off here, right? Why is this a problem? If you're not trying to learn from this experience and your policy by some chance, because you might have some entropy in there or something, you do know what happens if you make a move like this. And you also know what happens if you make a move like this, right? Already two humans have done these moves. But what happens if you make a move like this? You just don't know, right? In classic reinforcement learning, you would get a negative reward and you could learn from that to not do this action anymore. But in this case, you simply don't have any data to tell you what happens when you go off there. So you see that there's a problem if you are not able to learn from your own experience, but you have to learn from something or someone else's experience. The distribution of experience that you have available to you might be not fully specific of the environment. It might be very different from what you would do. And it might be very not conducive to what you want to do with it. So the task of offline reinforcement learning is harder than online reinforcement learning. But it also has many, many applications. Sometimes it's just not possible to do online reinforcement learning. When for example, in medical field, right? Think of the medical field where you want a robot to perform a surgery. You can't just do reinforcement learning with our online techniques because they're just going to try a bunch of things and see what works. Maybe you want that, I don't want that. So necessarily, you're going to be left with, let's have this robot learn from human experts. So that's a task for offline reinforcement learning. There are many more tasks. For example, if you think of search engine, you will have many, many, many logs from human searching things, and you simply store them, you simply have them in a buffer. Now you want to maybe train a reinforcement learning agent that serves the best possible ads or something like this. You want to do this in a way that you can use all of that data, even though that data wasn't collected by that particular agent. The crucial difference to supervised learning again, is that you have this interactive structure, this multi-step interactive structure. Because in a supervised learning, you also have this buffer here. In supervised learning, you simply have your labeled data set. But the difference is in supervised learning, you always know what the right action is currently, because you have the labels. In offline reinforcement learning, you don't know. You might be here, and there are three actions available. All you know is that the demonstrator, these actors here, one of them has done this, and then this, and then this, and then got a two. You have no clue what happens if you do this, and then this, and then this. All you know is that this action here might eventually lead to a two. You also can't try it out, because you can't try out this path, because you don't get a reward here. You have to find, and this is the task here, you'll have to find some other example or stitch together. They make a good example here. This paper basically proposes a benchmark for offline RL algorithms. What they do is they have a bunch of data sets. They have a bunch of these replay buffers around for different tasks, a collection of this, that they collected with various techniques. There is human demonstration, there is other agents, and so on. They have that, and you're supposed to take one of them, learn something, learn an agent, and then evaluate it on an environment. They propose which ones are suitable for this. They give you the data, and they give you the environment to evaluate it on. In the end, you'll get a score, and you can compare your offline RL algorithm with others. They also provide some benchmark implementations for algorithms that already do this. They show that they don't really work well. One of the tasks is this maze here. In this maze, the task is you are somewhere, let's say here, and you need to go somewhere, let's say here, and you need to find your way. The demonstrations you have, the data in your replay buffer, is such that this is the same task, but never the same start and end points like you are tasked to. You might have one in your replay buffer, you might have one trajectory, one episode that went like this from one to two. And you'll be able to see the reward of that. And you might have one trajectory that was from two to three, like this. Both of these things actually give you really high reward. If you were an agent, and you had to learn, and now the task is please go from one to three, what you could do is you could simply say, I know the green thing gave a pretty high reward, and the yellow thing gave a pretty high reward. I know the green thing started at one, and I know the yellow thing ended at three, and I know they both have this common location. So what I might do just is I might go to that common location, and then go on on the different path, right? So you have to somehow stitch together experience from other agents in order to make your task work. This is a very explicit example, of course, what we want to do is we want to do this in a more implicit deep learning way, ideally, and not manually stitch together other trajectories. Though I'm pretty sure that would not be so dumb, right? I'm pretty sure there's a lot of data augmentation you could do during training simply by stitching together other trajectories, right? So from this trajectory, you could actually, not only could you make other gold conditioned ways, for example, from here to here, or from here to here, you could make from here to here anywhere where you have shared points, you could train a policy that goes there and then goes further or something like this. I'm pretty sure there's already an algorithm that does things like this, but I'm just thinking aloud here. Alright, so this is one of the tasks and you see that the that that you will have to learn a policy to go as fast as possible from any point to any other point. And you're all you're given is a database of experience that already exists from some other agent, but never will probably never the exact route that you need to learn right now. Alright, so the goal is how fast or how efficiently can you do this? This is one task in this data set. The next task is very similar is this grid world here where there is this red square, red triangle, that's your agent. And then there is the green square, that's your goal or vice versa. And so you're basically tasked to not hit the walls here and to go about your way finding the target. There are more elaborate things like this mojo co environment here, or the ant maze where you have this little ant with you know, the spider legs. So this is no longer you can just move in either direction, you have to actually control the legs. And there's also this arm, this robotic arm. So you see there is a wide diversity of tasks. And also, there is a wide diversity of how the replay buffer was constructed. So in some cases, the replay buffer is actually constructed by a human performing in this environment. So in this hand manipulation task, you'll have demonstrations from humans. You see it's not particularly many samples here. It's 5000 samples, which I guess are is a chopped up version of I'm not really sure how the human things were constructed. But you can clearly guess that the degrees of freedom that you have in a robotic hand is much, much higher than you could learn just from these 5000 samples if you were to, you know, an online or algorithm that just does random exploration will need much more than these 5000 samples. And the 5000 samples won't be I ID distributed with all the degrees of freedom, it will just be here's what a human does, right. And so you can think of algorithms like inverse reinforcement learning or something like this. But here in inverse reinforcement learning, usually you assume that the expert the expert is kind of trying to achieve the same reward as you do. But this is not necessarily the case here. You have a given reward structure, but you are tasked to simply learn from these demonstrations. You can see it's also possible that there is this is constructed by a policy. And that usually means that they so either it's it's constructed by let's say a reinforcement learning algorithm that was trained in an online fashion, but maybe not as well. But also I think they have behavior cloning policy that they got from human demonstration, I think so that there are many ways. Also sometimes you have a planner which is, can you imagine it's it's a it's an algorithm that wasn't machine learned. So I know almost unthinkable, but in these in these kind of mazes, you can actually do planning algorithms that can can sort of so I know this is crazy and crazy talk, the niche topic but there exists things like a star search where where where you can construct the kind of shortest path through these mazes and things like this. So yeah, that's I know, I know that that is that is very niche. But you can construct policies like this. And then you can use those as your replay buffer filling. And you can already see that this also will be a massively different distribution of data than you would get with an online RL algorithm, right. So in conclusion, they do test other they do test other algorithms on this. In conclusion, they say that most offline RL algorithms nowadays, they don't work well on these on these data sets. The only data sets where they do work well is where the replay buffer was generated by some sort of like here, by some sort of policy by some sort of reinforcement learning policy. So what they would do is they would train an online policy and the experience generated by that online policy while it learns will make up the replay buffer. And if you use that replay buffer for offline learning, then they say it tends to work okay. But if you have other methods of collecting the data that are very different from this offline, sorry, from an from a reinforcement learning collection approach, then it tends not to work as well. Alright, so if you are interested in offline RL, please check out this paper, all their code is available right here. Note that the link in the paper doesn't seem to work. The true link is here. I'll also put it in the description. And with that, I wish you a good day. Bye!
 Hi there, today we're looking at datasets for data-driven reinforcement learning by
 Justin Fu, Aviral Kumar, Ofer Natschum, George Tucker and Sergei Levine.
 So this is what you would call a dataset paper or a benchmark paper.
 And the main point or the main area of the paper was called offline reinforcement learning.
 So offline reinforcement learning, usually in reinforcement learning you have this task,
 right?
 You have the agent and you have the environment.
 And the agent gets some sort of observation and has to come up with an action in response
 to that observation.
 And then it gets back a reward and another observation.
 And again, it has to come up with an action.
 And the goal is to maximize the rewards over time that the agent gets while interacting
 with the environment.
 So usually this is organized in what are called episodes, which basically means if you have
 some sort of environment, right, and here is the agent and here is the goal, right,
 the goal is a inverted triangle.
 And there are a bunch of walls right here, right?
 So it looks kind of a maze that the agent has to navigate.
 Then one episode could be the agent moving around until it either finds the target or
 hits a wall or just kind of goes around and around.
 And then at some point you say, all right, that's enough, game over.
 And usually in reinforcement learning, you perform many of these episodes and then you
 learn from them.
 So you perform episodes and each episode gets into usually some sort of replay buffer, right?
 Let's call this replay buffer.
 And you do this many times and at the same time that you're doing this, you're using
 the things that you stored here in order to learn, right?
 So the agent learns from these things, right?
 So it acts with the environment in this loop, in this fashion.
 Then once it has done an episode, it puts it into the replay buffer and then it learns
 from the actions it has performed.
 This is what is usually called online reinforcement learning, right?
 So this loop is online.
 Online means because the agent learns from its own actions, right?
 Now in contrast to this, there is offline reinforcement learning.
 So in offline reinforcement learning, the agent has to learn from someone else's actions,
 right?
 So this connection here is severed.
 Instead you have other agents.
 Let's call these agent one, agent two, multiple agents, agent three.
 They all have their own interaction with the environment, right?
 Environment environment interactions and they feed their experience into they perform these
 episodes.
 They feed their experience into the replay buffer.
 And then the agent just has to learn from that.
 So whatever happened here, this was previous, right?
 And now the agent has to learn how to maximize its reward just from the experience that is
 in the replay buffer from these other agents.
 This is what's called offline reinforcement learning means the agent learns from someone
 else's actions.
 Basically the power of reinforcement learning of course comes from the fact that you learn
 from your own actions.
 It means that for example, if you already have some successful trajectories here, right?
 You found the target.
 You can try to replicate that because you know which actions you performed.
 And if you don't, you know, change anything, you're probably going to find the target again
 just by randomness.
 All right, because you've done it already once and so on.
 So you kind of know all the intrinsics of your own algorithm that led you to reach the
 target.
 Now this is an entirely different case with all of these other agents.
 You have no clue how they were acting, why they were acting, right?
 You just know, okay, they did a series of actions and that gave them some kind of reward.
 And you have no idea what their reasoning was or anything.
 All you really can learn from is their sequence of actions.
 Now why is that problematic, right?
 So if all of the agents, for example, if this is an actual platform and this is really steep
 here, this is all of here is really steep cliffs, right?
 And you can actually fall off.
 But the agents, they're humans, right?
 So they don't want to fall off.
 So what they're going to do is they're just going to take steps that are maybe like this
 or maybe like this, but they're humans, they're smart.
 They're never going to fall off here, right?
 Why is this a problem?
 If you're not trying to learn from this experience and your policy by some chance, because you
 might have some entropy in there or something, you do know what happens if you make a move
 like this.
 And you also know what happens if you make a move like this, right?
 Already two humans have done these moves.
 But what happens if you make a move like this?
 You just don't know, right?
 In classic reinforcement learning, you would get a negative reward and you could learn
 from that to not do this action anymore.
 But in this case, you simply don't have any data to tell you what happens when you go
 off there.
 So you see that there's a problem if you are not able to learn from your own experience,
 but you have to learn from something or someone else's experience.
 The distribution of experience that you have available to you might be not fully specific
 of the environment.
 It might be very different from what you would do.
 And it might be very not conducive to what you want to do with it.
 So the task of offline reinforcement learning is harder than online reinforcement learning.
 But it also has many, many applications.
 Sometimes it's just not possible to do online reinforcement learning.
 When for example, in medical field, right?
 Think of the medical field where you want a robot to perform a surgery.
 You can't just do reinforcement learning with our online techniques because they're just
 going to try a bunch of things and see what works.
 Maybe you want that, I don't want that.
 So necessarily, you're going to be left with, let's have this robot learn from human experts.
 So that's a task for offline reinforcement learning.
 There are many more tasks.
 For example, if you think of search engine, you will have many, many, many logs from human
 searching things, and you simply store them, you simply have them in a buffer.
 Now you want to maybe train a reinforcement learning agent that serves the best possible
 ads or something like this.
 You want to do this in a way that you can use all of that data, even though that data
 wasn't collected by that particular agent.
 The crucial difference to supervised learning again, is that you have this interactive structure,
 this multi-step interactive structure.
 Because in a supervised learning, you also have this buffer here.
 In supervised learning, you simply have your labeled data set.
 But the difference is in supervised learning, you always know what the right action is currently,
 because you have the labels.
 In offline reinforcement learning, you don't know.
 You might be here, and there are three actions available.
 All you know is that the demonstrator, these actors here, one of them has done this, and
 then this, and then this, and then got a two.
 You have no clue what happens if you do this, and then this, and then this.
 All you know is that this action here might eventually lead to a two.
 You also can't try it out, because you can't try out this path, because you don't get a
 reward here.
 You have to find, and this is the task here, you'll have to find some other example or
 stitch together.
 They make a good example here.
 This paper basically proposes a benchmark for offline RL algorithms.
 What they do is they have a bunch of data sets.
 They have a bunch of these replay buffers around for different tasks, a collection of
 this, that they collected with various techniques.
 There is human demonstration, there is other agents, and so on.
 They have that, and you're supposed to take one of them, learn something, learn an agent,
 and then evaluate it on an environment.
 They propose which ones are suitable for this.
 They give you the data, and they give you the environment to evaluate it on.
 In the end, you'll get a score, and you can compare your offline RL algorithm with others.
 They also provide some benchmark implementations for algorithms that already do this.
 They show that they don't really work well.
 One of the tasks is this maze here.
 In this maze, the task is you are somewhere, let's say here, and you need to go somewhere,
 let's say here, and you need to find your way.
 The demonstrations you have, the data in your replay buffer, is such that this is the same
 task, but never the same start and end points like you are tasked to.
 You might have one in your replay buffer, you might have one trajectory, one episode
 that went like this from one to two.
 And you'll be able to see the reward of that.
 And you might have one trajectory that was from two to three, like this.
 Both of these things actually give you really high reward.
 If you were an agent, and you had to learn, and now the task is please go from one to
 three, what you could do is you could simply say, I know the green thing gave a pretty
 high reward, and the yellow thing gave a pretty high reward.
 I know the green thing started at one, and I know the yellow thing ended at three, and
 I know they both have this common location.
 So what I might do just is I might go to that common location, and then go on on the different
 path, right?
 So you have to somehow stitch together experience from other agents in order to make your task
 work.
 This is a very explicit example, of course, what we want to do is we want to do this in
 a more implicit deep learning way, ideally, and not manually stitch together other trajectories.
 Though I'm pretty sure that would not be so dumb, right?
 I'm pretty sure there's a lot of data augmentation you could do during training simply by stitching
 together other trajectories, right?
 So from this trajectory, you could actually, not only could you make other gold conditioned
 ways, for example, from here to here, or from here to here, you could make from here to
 here anywhere where you have shared points, you could train a policy that goes there and
 then goes further or something like this.
 I'm pretty sure there's already an algorithm that does things like this, but I'm just thinking
 aloud here.
 Alright, so this is one of the tasks and you see that the that that you will have to learn
 a policy to go as fast as possible from any point to any other point.
 And you're all you're given is a database of experience that already exists from some
 other agent, but never will probably never the exact route that you need to learn right
 now.
 Alright, so the goal is how fast or how efficiently can you do this?
 This is one task in this data set.
 The next task is very similar is this grid world here where there is this red square,
 red triangle, that's your agent.
 And then there is the green square, that's your goal or vice versa.
 And so you're basically tasked to not hit the walls here and to go about your way finding
 the target.
 There are more elaborate things like this mojo co environment here, or the ant maze
 where you have this little ant with you know, the spider legs.
 So this is no longer you can just move in either direction, you have to actually control
 the legs.
 And there's also this arm, this robotic arm.
 So you see there is a wide diversity of tasks.
 And also, there is a wide diversity of how the replay buffer was constructed.
 So in some cases, the replay buffer is actually constructed by a human performing in this
 environment.
 So in this hand manipulation task, you'll have demonstrations from humans.
 You see it's not particularly many samples here.
 It's 5000 samples, which I guess are is a chopped up version of I'm not really sure
 how the human things were constructed.
 But you can clearly guess that the degrees of freedom that you have in a robotic hand
 is much, much higher than you could learn just from these 5000 samples if you were to,
 you know, an online or algorithm that just does random exploration will need much more
 than these 5000 samples.
 And the 5000 samples won't be I ID distributed with all the degrees of freedom, it will just
 be here's what a human does, right.
 And so you can think of algorithms like inverse reinforcement learning or something like this.
 But here in inverse reinforcement learning, usually you assume that the expert the expert
 is kind of trying to achieve the same reward as you do.
 But this is not necessarily the case here.
 You have a given reward structure, but you are tasked to simply learn from these demonstrations.
 You can see it's also possible that there is this is constructed by a policy.
 And that usually means that they so either it's it's constructed by let's say a reinforcement
 learning algorithm that was trained in an online fashion, but maybe not as well.
 But also I think they have behavior cloning policy that they got from human demonstration,
 I think so that there are many ways.
 Also sometimes you have a planner which is, can you imagine it's it's a it's an algorithm
 that wasn't machine learned.
 So I know almost unthinkable, but in these in these kind of mazes, you can actually do
 planning algorithms that can can sort of so I know this is crazy and crazy talk, the niche
 topic but there exists things like a star search where where where you can construct
 the kind of shortest path through these mazes and things like this.
 So yeah, that's I know, I know that that is that is very niche.
 But you can construct policies like this.
 And then you can use those as your replay buffer filling.
 And you can already see that this also will be a massively different distribution of data
 than you would get with an online RL algorithm, right.
 So in conclusion, they do test other they do test other algorithms on this.
 In conclusion, they say that most offline RL algorithms nowadays, they don't work well
 on these on these data sets.
 The only data sets where they do work well is where the replay buffer was generated by
 some sort of like here, by some sort of policy by some sort of reinforcement learning policy.
 So what they would do is they would train an online policy and the experience generated
 by that online policy while it learns will make up the replay buffer.
 And if you use that replay buffer for offline learning, then they say it tends to work okay.
 But if you have other methods of collecting the data that are very different from this
 offline, sorry, from an from a reinforcement learning collection approach, then it tends
 not to work as well.
 Alright, so if you are interested in offline RL, please check out this paper, all their
 code is available right here.
 Note that the link in the paper doesn't seem to work.
 The true link is here.
 I'll also put it in the description.
 And with that, I wish you a good day.
 Bye!