Investigating Human Priors for Playing Video Games (Paper & Demo)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl reinforcement learning deep rl human prior objects game video game key visuals enemy ladder gravity ablation
Why are humans so good at video games? Maybe it's because a lot of games are designed with humans in mind. What happens if we change that? This paper removes the influence of human priors from a game and ends up with a pretty fun experience.

Paper: https://arxiv.org/abs/1802.10217
Website: https://rach0012.github.io/humanRL_website/
Code: https://github.com/rach0012/humanRL_prior_games

Abstract:
What makes humans so good at solving seemingly complex video games? Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors on human performance. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play. Videos and the game manipulations are available at this https URL

Authors: Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L. Griffiths, Alexei A. Efros

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hey there, what's going on today? We're looking at investigating human priors for playing video games by Rachid Duby, Pulkit Agrawal, Deepak Patak, Tom Griffiths and Alexei Aeferos. So there is a paper to go with this, but I actually don't want to get into the paper too much in order to not reveal too much of what's coming. But basically they're trying to investigate what makes video games work for humans. So what do humans pay attention to? What priors bring humans into a video game? And the fun thing is they've created these games where they ablate these individual priors. And we are going to play them. So the original game right here, as you can see, is kind of this Montezuma's Revenge type of game. So you only need the arrow keys. If you go to a bad blob like this, then you die. And you can jump on them and you can use the ladders and the spikes. They'll hurt you if you jump on them. And also if you fall down between the platforms. So what you've got to do is basically get the key, then go to the door over here and bada boom. Cool. So let's try out. So they basically ablate different things here. Mask semantics means that you don't know what the objects are anymore. So you might go over here and you might be like, what's this green thing? Can I jump on it? Oh. So we're probably a bit biased because we've seen the game before. So we know that these are the pink ones are the bad ones. And that's the key. So we should probably get it. But you can imagine that it is a bit harder, but you could still solve it right. Reverse semantics is very interesting if you play it for the first time, because all of a sudden now there's the coins. Oh, and the fire. But I think humans could probably still figure it out with like some minimal trial and error. I do this ice cream cone. You realize, OK, now it gets interesting because right now we've always had sort of we know that there's an object and there's no object on the platforms. But now these are masked. So basically you don't know what's like a relevant object and what isn't. So I know that there is like a bad thing here and a bad thing down to the left. So I'm going to guess these light. These light pink things are the bad things. Yeah. Yeah. These are the ladders. Cool. Bad thing right here. We are rocking this. OK. Key. Where's that? That's the key and the door. So it gets harder because you kind of have to remember the colors right. I know that the light pink ones are the bad squares. Still, still solvable. So let's jump over these on the left because these get really actually it's going to here. So masked affordances. What they're saying is that, OK, you can kind of from the way something looks, you can tell what you can do with it. For example, the platforms you can jump on them and the background is sort of empty space. So you know that there's nothing much happening there. So they trying to take that away by simply retexturing all the objects here such that you don't know how you can interact with them. And it does get significantly harder because OK, so these these green ones are these green ones are the platforms right here. So I can still see that that must be the latter. Right. You can imagine if you were playing this again for the first time that this is significantly more difficult, but you still see the key and the green ones being the platforms. We got this. Now it gets harder. Masked visual similarity. So this is where they say maybe as we did so far, maybe you as a human can kind of make out that things that are visually similar to each other can you can do the same things with. Like we said, the green ones are probably the platforms. So they took it away. Gee. OK, so that must be OK. Can't go here. Fell down. Let's try again. This is a platform. Is this one here? Yes. These are platforms. Ah, that was easy. Too easy. Too easy running into that bad blob there. OK, the ladders are still like this, but then OK. Yeah, this gets harder as as you can. Gee. OK, I'm too dumb to remember from before. I'm like the ideal subject because I don't remember. How did this work? I'm going to solve this just so you know, even if this video gets to 50 minutes, I'm going to make it through this. OK, here we can. OK, see my short term memory is so bad. OK, we got the key. Now just get over to the door. Doors over there. Yeah. OK, now let's wipe the short term memory again. Here changed ladder interaction where they basically say, OK, one of the things that you could know from the real world is how these objects work in the real world. So there's not really any pink blobs with evil faces. There might be spikes. Yes. But ladder is something, you know, that works. So if you want to go up here, that doesn't work. So you kind of have to figure. OK. So you have to go kind of left and right to go up the ladders. And so that one I actually tried before and I figured that out pretty quickly. I think humans are able to figure that out fairly quickly because you kind of on the ladder, right? You can actually go down easily. You're on the ladder and then it kind of doesn't work. And then you kind of try to wiggle because there's two of them. I don't think that's that's necessarily super hard. And now it feels a bit like, you know, the Super Mario maker thing where where people just try to make levels as hard as possible and trick you with trick blocks and invisible stuff. This is hard. So the direction of gravity. So now the left key jumps right here, this key. So this is like this is extremely, extremely hard because I have to like think about every move I make before I do it. And OK, no, no, no, this is so unintuitive for real. Yeah, got it. Got it. OK. Really? Try this out. This is crazy. Yeah. Yeah. OK, so the last thing is we combine all of it, I guess, except the changed gravity and changed interaction. So now all the priors, all the visual object priors removed. This is this is King's Discipline right here. OK, so we figured out where the blue. Cool. So where's the next? This is the next platform. Where's the OK. There must be like a yeah, take that. OK, but we know the next. So we can't really generalize from this because we know the next bad blob isn't going to be the same color. Right. OK. The white done this. I know there's a bad thing here, but we'd have to figure this out. So basically kind of the point of the paper, I think, is to say that this is what you're doing to our. No spikes. This is what you're doing to our algorithms. If you're in the most case, so they simply have to go and to basically try every single thing and remember what worked and what didn't work. Now, of course, the algorithms can also exhibit like can also use the visual similarity. That was the key. Yeah. Yeah. Let's go to the door door door. No, there is a there's like a bad thing here. Right. No spikes. OK. So either we build these priors into the algorithms if we want to get them to human level or we have some sort of learning these priors before we let the people go onto a paper or I don't know. Or we just take it that algorithms have to figure all of this out by themselves. So they ablate these things right here. You can see the masked object identity makes kind of the biggest difference in terms of time, number of deaths, the number of states explored. Reverse semantics. I believe these are humans that are trying it for the first time and they're just like, oh, an ice cream. So it can also hurt. Right. The algorithm wouldn't be super impressed by it looking like an ice cream. But the human is very much and the crazy thing here, you can see exploration, the original game and then exploration in the no object prior game, especially if you play this for the first time. This is just mad. Like no freaking way. I would actually like love to see video games like this coming out. This would be the worst selling video game of all times where dynamically it just removes these kind of priors. But it's a I think it's a really fun way to investigate what humans learn and what they already bring into the game. So here they have another game and they do this same thing on an RL agent. And you see here the RL agent just don't care about any of these things except visual similarity. So visual similarity helps the RL agent to generalize across the game. So if you see a bad blob, the next bad blob will look similar. And that's sort of kind of an invariance that we know they can exploit since they're using convolutional neural networks and so on. But I think it is really drawing attention to the importance of priors prior knowledge in reinforcement learning and human knowledge. So in this game right here, where you have these hidden rewards that the human doesn't see, right? But if they kind of touch it, they're kind of coins and the human performs way worse than the RL agent because the RL agent will actually try those things out. And the human having the prior that the black thing is like they don't see the yellow boxes that the black thing is just empty space. They won't even explore that. So maybe, you know, that is something to think about with respect to building RL agents. All right. I don't want to go into the paper too much. It's a very cool paper, but we're here to play games and I invite you to read the paper. Check out the website. Try these games for yourself. They're a lot of fun, especially if you try them first time. And bye bye.
 Hey there, what's going on today? We're looking at investigating human priors for playing video games by Rachid Duby, Pulkit Agrawal, Deepak Patak, Tom Griffiths and Alexei Aeferos.
 So there is a paper to go with this, but I actually don't want to get into the paper too much in order to not reveal too much of what's coming.
 But basically they're trying to investigate what makes video games work for humans. So what do humans pay attention to?
 What priors bring humans into a video game? And the fun thing is they've created these games where they ablate these individual priors.
 And we are going to play them. So the original game right here, as you can see, is kind of this Montezuma's Revenge type of game.
 So you only need the arrow keys. If you go to a bad blob like this, then you die.
 And you can jump on them and you can use the ladders and the spikes. They'll hurt you if you jump on them.
 And also if you fall down between the platforms. So what you've got to do is basically get the key, then go to the door over here and bada boom.
 Cool. So let's try out. So they basically ablate different things here.
 Mask semantics means that you don't know what the objects are anymore.
 So you might go over here and you might be like, what's this green thing? Can I jump on it? Oh.
 So we're probably a bit biased because we've seen the game before.
 So we know that these are the pink ones are the bad ones. And that's the key. So we should probably get it.
 But you can imagine that it is a bit harder, but you could still solve it right.
 Reverse semantics is very interesting if you play it for the first time, because all of a sudden now there's the coins. Oh, and the fire.
 But I think humans could probably still figure it out with like some minimal trial and error.
 I do this ice cream cone. You realize, OK, now it gets interesting because right now we've always had sort of we know that there's an object and there's no object on the platforms.
 But now these are masked. So basically you don't know what's like a relevant object and what isn't.
 So I know that there is like a bad thing here and a bad thing down to the left. So I'm going to guess these light.
 These light pink things are the bad things. Yeah. Yeah. These are the ladders. Cool. Bad thing right here.
 We are rocking this. OK. Key. Where's that? That's the key and the door.
 So it gets harder because you kind of have to remember the colors right. I know that the light pink ones are the bad squares.
 Still, still solvable. So let's jump over these on the left because these get really actually it's going to here.
 So masked affordances. What they're saying is that, OK, you can kind of from the way something looks, you can tell what you can do with it.
 For example, the platforms you can jump on them and the background is sort of empty space. So you know that there's nothing much happening there.
 So they trying to take that away by simply retexturing all the objects here such that you don't know how you can interact with them.
 And it does get significantly harder because OK, so these these green ones are these green ones are the platforms right here.
 So I can still see that that must be the latter. Right.
 You can imagine if you were playing this again for the first time that this is significantly more difficult, but you still see the key and the green ones being the platforms.
 We got this. Now it gets harder. Masked visual similarity.
 So this is where they say maybe as we did so far, maybe you as a human can kind of make out that things that are visually similar to each other can you can do the same things with.
 Like we said, the green ones are probably the platforms. So they took it away.
 Gee. OK, so that must be OK. Can't go here.
 Fell down. Let's try again. This is a platform. Is this one here? Yes. These are platforms.
 Ah, that was easy. Too easy. Too easy running into that bad blob there.
 OK, the ladders are still like this, but then OK.
 Yeah, this gets harder as as you can. Gee. OK, I'm too dumb to remember from before.
 I'm like the ideal subject because I don't remember.
 How did this work?
 I'm going to solve this just so you know, even if this video gets to 50 minutes, I'm going to make it through this.
 OK, here we can. OK, see my short term memory is so bad.
 OK, we got the key. Now just get over to the door. Doors over there. Yeah.
 OK, now let's wipe the short term memory again.
 Here changed ladder interaction where they basically say, OK, one of the things that you could know from the real world is how these objects work in the real world.
 So there's not really any pink blobs with evil faces. There might be spikes. Yes. But ladder is something, you know, that works.
 So if you want to go up here, that doesn't work. So you kind of have to figure. OK.
 So you have to go kind of left and right to go up the ladders. And so that one I actually tried before and I figured that out pretty quickly.
 I think humans are able to figure that out fairly quickly because you kind of on the ladder, right?
 You can actually go down easily. You're on the ladder and then it kind of doesn't work.
 And then you kind of try to wiggle because there's two of them. I don't think that's that's necessarily super hard.
 And now it feels a bit like, you know, the Super Mario maker thing where where people just try to make levels as hard as possible and trick you with trick blocks and invisible stuff.
 This is hard. So the direction of gravity. So now the left key jumps right here, this key.
 So this is like this is extremely, extremely hard because I have to like think about every move I make before I do it.
 And OK, no, no, no, this is so unintuitive for real.
 Yeah, got it. Got it. OK. Really? Try this out. This is crazy. Yeah. Yeah.
 OK, so the last thing is we combine all of it, I guess, except the changed gravity and changed interaction.
 So now all the priors, all the visual object priors removed.
 This is this is King's Discipline right here. OK, so we figured out where the blue. Cool.
 So where's the next? This is the next platform. Where's the OK.
 There must be like a yeah, take that. OK, but we know the next.
 So we can't really generalize from this because we know the next bad blob isn't going to be the same color.
 Right. OK. The white done this.
 I know there's a bad thing here, but we'd have to figure this out.
 So basically kind of the point of the paper, I think, is to say that this is what you're doing to our.
 No spikes. This is what you're doing to our algorithms.
 If you're in the most case, so they simply have to go and to basically try every single thing and remember what worked and what didn't work.
 Now, of course, the algorithms can also exhibit like can also use the visual similarity.
 That was the key. Yeah. Yeah. Let's go to the door door door. No, there is a there's like a bad thing here.
 Right. No spikes.
 OK.
 So either we build these priors into the algorithms if we want to get them to human level or we have some sort of learning these priors before we let the people go onto a paper or I don't know.
 Or we just take it that algorithms have to figure all of this out by themselves.
 So they ablate these things right here. You can see the masked object identity makes kind of the biggest difference in terms of time, number of deaths, the number of states explored.
 Reverse semantics. I believe these are humans that are trying it for the first time and they're just like, oh, an ice cream.
 So it can also hurt. Right. The algorithm wouldn't be super impressed by it looking like an ice cream.
 But the human is very much and the crazy thing here, you can see exploration, the original game and then exploration in the no object prior game, especially if you play this for the first time.
 This is just mad. Like no freaking way. I would actually like love to see video games like this coming out.
 This would be the worst selling video game of all times where dynamically it just removes these kind of priors. But it's a I think it's a really fun way to investigate what humans learn and what they already bring into the game.
 So here they have another game and they do this same thing on an RL agent. And you see here the RL agent just don't care about any of these things except visual similarity.
 So visual similarity helps the RL agent to generalize across the game. So if you see a bad blob, the next bad blob will look similar.
 And that's sort of kind of an invariance that we know they can exploit since they're using convolutional neural networks and so on.
 But I think it is really drawing attention to the importance of priors prior knowledge in reinforcement learning and human knowledge.
 So in this game right here, where you have these hidden rewards that the human doesn't see, right? But if they kind of touch it, they're kind of coins and the human performs way worse than the RL agent because the RL agent will actually try those things out.
 And the human having the prior that the black thing is like they don't see the yellow boxes that the black thing is just empty space.
 They won't even explore that. So maybe, you know, that is something to think about with respect to building RL agents.
 All right. I don't want to go into the paper too much. It's a very cool paper, but we're here to play games and I invite you to read the paper.
 Check out the website. Try these games for yourself. They're a lot of fun, especially if you try them first time.
 And bye bye.