LeDeepChef 👨‍🍳 Deep Reinforcement Learning Agent for Families of Text-Based Games
ml machine learning reinforcement learning recipe text-based games text games natural language processing nlp actor critic GRU embedding pretraining artificial intelligence ai competition microsoft
The AI cook is here! This agent learns to play a text-based game where the goal is to prepare a meal according to a recipe. Challenges? Many! The number of possible actions is huge, ingredients change and can include ones never seen before, you need to navigate rooms, use tools, manage an inventory and sequence everything correctly and all of this from a noisy textual description that the game engine throws at you. This paper mixes supervised explicit training with reinforcement learning in order to solve this task.

Abstract:
While Reinforcement Learning (RL) approaches lead to significant achievements in a variety of areas in recent history, natural language tasks remained mostly unaffected, due to the compositional and combinatorial nature that makes them notoriously hard to optimize. With the emerging field of Text-Based Games (TBGs), researchers try to bridge this gap. Inspired by the success of RL algorithms on Atari games, the idea is to develop new methods in a restricted game world and then gradually move to more complex environments. Previous work in the area of TBGs has mainly focused on solving individual games. We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme. In this work, we present our deep RL agent--LeDeepChef--that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions. The agent participated in Microsoft Research's "First TextWorld Problems: A Language and Reinforcement Learning Challenge" and outperformed all but one competitor on the final test set. The games from the challenge all share the same theme, namely cooking in a modern house environment, but differ significantly in the arrangement of the rooms, the presented objects, and the specific goal (recipe to cook). To build an agent that achieves high scores across a whole family of games, we use an actor-critic framework and prune the action-space by using ideas from hierarchical reinforcement learning and a specialized module trained on a recipe database.

Authors: Leonard Adolphs, Thomas Hofmann

https://arxiv.org/abs/1909.01646
 Hi there. Today we're looking at Le Deep Chef, deep reinforcement learning agent for families of text-based games by Leonard Adolfs and Thomas Hoffmann. So this is a paper about engineering an agent for a particular family of tasks. This is different from reinforcement learning agents that for example are just good at one game, let's say Pong or whatnot and even I guess even things like Starcraft. Though this kind of depends on what you mean by game. So what are we talking about here? The following is a text-based games where the goal is to cook recipes. So let's just jump in and see what goes on. The game starts by telling you, you are hungry. Let's cook a delicious meal and so on. So the objective is basically always the same. It's find the cookbook, read the recipe that's in it, then collect all the things that are in the recipe, prepare them in certain ways that are also specified by the recipe and then at the end you have a meal and then you can eat the meal and that will give you points. But since it's a text-based games and the input doesn't come structured but it comes in natural text. So the game tells you for example kitchen. So basically you're in the kitchen. You are now in the kitchen. I guess you better just go and list everything you see here. You hear a noise, you spin around. So you see that the kind of input you get from the game is very playful, has a lot of descriptive elements. Sometimes it's like you see a closed oven. You make out a table. Then you can see on the counter you can make out a sliced fried red hot pepper and so on. So it's very much not trivial to kind of parse this in a traditional way. If you were to go about this by simply writing an algorithm extracting things it's very hard because for example you might see that there's an oven but it's a closed oven. You make out a table. So this is kind of a synonym for you see a table but you see like there is a table. You can make out a sliced fried red hot pepper and here it's important not only do you need to realize that there is a red hot pepper but also that its state is sliced and fried. This is important because you need all ingredients in a certain state. Right? You examine here you examine the stove so there is a stove. Right? So all these things you need to kind of understand. So if you now look there is a recipe book in here. Or no there isn't a recipe. You can examine recipe. I guess there is a recipe book in that room. If there is a recipe book then you can examine the recipe and that's the command. So the arrows here always indicate that that's a user command. And these you have to type. That's like the next thing that your agent needs to do. You can't select from a predefined set of actions. You actually need to type in the things you want to do. Right? And these are a lot. Like there are a lot of possibilities of what you could type in. Even if you restrict it to kind of what you know the game accepts there are still so many actions. It's way different than for example Atari games. They always have eight actions. Like there's eight buttons you could possibly press and that's it. And here there are like combinatorically many things you can do. Like you can prepare and take and all the ingredients. You don't know which ingredients come. So here you examine the recipe. Let's look at a recipe. It says you open the recipe. Start reading. Recipe number one. Here are the ingredients. Red hot pepper. Here for right now that's just one ingredient. Then there are directions. So what do you need to do? Slice the red hot pepper. Fry the red hot pepper and prepare the meal. Those are the directions of the recipe. You also have this inventory command which tells you which you're carrying. Next difficulty. The inventory is finite. So you can't carry everything. At some points you have to drop things that are unnecessary. You can't just take everything. Here you see the command take red hot pepper. That only works if there's a red hot pepper in the room. And here says you take the red hot pepper from the counter. Your score has just gone up by one point. And then if you type inventory it says you're carrying a sliced fried red hot pepper. Again here it says the state of the ingredient. So the ingredient is the red hot pepper and the state is sliced and fried. And then you can prepare meal and then you can eat meal and then it says your score has just gone up by one point. And these are the scores you collect. So there are a lot of difficulties that are actually not shown in this example. For example there are different rooms. You may have noticed here you're in the kitchen. But there could be other rooms and you start in a random room. You also need to navigate through the rooms. Close the doors to the rooms could be closed and then you need to open them and so on. You can only for example if this pepper here weren't already sliced and fried you need to find... You can only slice it if there is a knife in the room. You can only fry it if there is a frying pan or an oven or a stove in the room. So and then you'd have to notice that there is a knife. If there is no knife you need to take the red hot pepper bring it to a new room with a knife and then slice it. So this is vastly difficult game. The last difficulty is actually that in the test set there will be ingredients that you haven't seen during training. So also that there. Your agent needs to generalize. That's why it says a family of text-based games. Because the objective always the same to kind of cook the recipe. But the things you have to do and the things that appear and so on those are those change basically from episode to episode. And the test set will be different than the training set or kind of there will be unseen data. Alright so how does this paper go about solving this problem? This paper basically does the following and we are going here from high level to low level. On the highest level it's a reinforcement learning agent and that is sort of how you would imagine an RL agent to work. So here at the end you have a policy and the policy predicts an action. If you don't know what a kind of a policy and an action things are in RL these are basic RL concept and we'll kind of skip them here and I'll assume everyone knows what they are. But essentially a policy specifies which action you take next given the current game state. So the policy is made up, scores different actions. So at each step there are k actions available. And these k actions I foresaid there are almost infinitely many actions that you could take. The first difficulty and that's the thing that actually comes in here is to reduce all of the possible actions that you can't even list to just k commands. So we'll go into that later how this is done. But basically one of the main contributions of this paper is how do you even specify what is reasonable, what would be reasonable to do in the current situation. And then the policy over here only has to decide among those reasonable actions, not among all actions. But given that you have k reasonable commands you see here command one command, these are embedded and then fed into GRUs which are recurrent neural networks. So for each of these commands you'll get a 32 dimensional vector. This 32 dimensional vector is here C1 through Ck. Each are combined with an encoding of the current state. So these 32 dimensional vector are combined with encoding of the current state which is 256 dimensional and then fed into a neural network that will output a probability distribution over these actions. This is pretty classic in deep reinforcement learning. So you have action encoding and the state encoding and the policy decides on that. The state encoding you'll see here it's the same everywhere of course because the current game state is the current game state. This comes from this model up here. What this does is over here you have the what you would call the state the current observation. The current observation is composed of many things. Specifically the following eight things. The first one is actually called observation which is I would call all of this the current observation from an RL perspective. But the first is actually observation. It's whatever you saw the big text you saw before. Like you were in the kitchen it looks like this it smells like this you turn around and so on. This would be the observation. It's what the game engine says at the current time step. This is just a piece of text. Second missing items. Third unnecessary items. Now these things you might wonder okay how do I know what what items are missing and unnecessary. These things come from another model that this paper trains and we'll get into that later. But basically they have a method of specifying which items are still missing which are unnecessary and they list those here. Then description which is the output of the last look command. So in each room you can look you can type look and then it'll give you a description of the room and what's in there. The previous commands this is often used in RL either explicitly or implicitly through a recurrent network in order to give the agent an idea what what happened in the in the previous steps or what it did so that it doesn't repeat actions unnecessarily or so it learns to not repeat actions unnecessarily. Required utilities. Again this is a model that's kind of trained to predict what utilities are required to perform some actions. So as I said before if you want to slice the red hot pepper you need a knife. If you want to fry it you need a stove. Discovered locations. As I said there are different rooms you actually don't know what rooms there are before you actually go in in there. So before you go through a door you reach another room. So the list of previously discovered and visited locations is there and then the name of the current location it is also there. So these are eight things that make up the current observation. These eight things are just strings of text and these eight things are each one as you can see here these are that the eight things from observation to location each one are embedded and fed also into an RNN. So for each of these eight things you'll obtain a 32 dimensional vector and these are all concatenated to make up one big 256 dimensional vector. So this 256 dimensional vector will contain all the necessary information about the current room what's in there what what items are you still missing what items do you have in your inventory which ones are unnecessary and so on. So if you train this correctly this 256 dimensional vector will describe the current game state as it is relevant to your agent like everything about it every relevant information that's in here will be encoded in this vector. Now this vector isn't the final state encoding yet what you'll have is you feed this into an RNN that takes as input the last time steps you have to imagine the last time step already there was observation blah blah blah this entire thing was I'm just copying I'm just copying this box over here so this entire thing was already done last step and already fed into an RNN so this this is an RNN that actually goes over time and the last whatever the output here is it will be fed to the next step and this is a trick often done in reinforcement learning as well that you actually have a recurrent neural network over the time steps so each time step you have a certain observation you encode it and so on you get a description of that and then you feed this into an RNN what the RNN can learn to do is it can learn to react to different not only to the current observation but to the current observation conditioned on the history of previous observations so it can learn before I was in this room now I'm in this new room so I actually haven't you know taken all the items from this room yet because I just came into this room and so on so the the kind of component where you are able to look at the past and what happened in the past is in captured by this RNN here so it's fairly complicated architecture but this here this state encoding that is conditioned on the also on the history then goes into this into here that's it that's the vector that goes in here is combined with each action so all of these actions here these K actions and this is all fed through a neural network and that will give you the policy this is a fairly complicated thing but if you look at it it's not it's not too it's not too difficult actually so what you'll do is you will take your observations here this is all observation it will be encoded and combined with the history in order to give you this in order to give you an encoding of the current state on the other hand you'll take all of the possible commands that you could perform right now encode each one separately right into an embedding and then you combine each one of those with this encoding you specified previously that you and and from that you make your decision which action to take next and the action here is the one that's output is the action you take next sampled from this policy the last thing you need is a value network and this is just important for reinforcement learning which tells you from this state here so I'm getting weird with colors here from this state here which is the same as this one so you'd simply transfer this over from this state how valuable is that what's my value of the state and the value is if I'm in this state and I act as I normally act what are all my future rewards going to be combined so it basically gives you a value of this state you can think of this in for example terms of chess if you had this in chess and then this here is it would be a description of the chessboard this HT and the value would be how valuable is this position for you so if you're very much ahead and material and position and so on this value would be very high if you're behind this value would be very low and this is in a real network simply trying to predict that value so with all of this you now have a never good basis to do reinforcement learning you have a policy you have a value network and from that you can train an RL agent and this is done classically in an actor critic way where you do advantage learning here the advantage and the policy you train weighted by the advantage then the value network you train to be close to their reward and then you have an entropy penalty if you don't know what these things are the video will get bit too long if I were to go over these reinforcement learning concepts but these are very standard in reinforcement learning so you can train these you can basically train what it does is you can train these neural networks in absence of label training data because you don't know what the best action is in each step right there's no one telling you you just have a reward you just sometimes you get a point and you don't know which actions led to that so these things will actually allow you to train these neural networks by using just the reward without knowing which exact actions were right and wrong and that's the core of reinforcement learning obviously alright so the the core one of the core ingredients actually is this recipe manager and the recipe manager is a sub model that does the following so here it takes as an input the cookbook here and it also takes as an input the inventory and it outputs something like this and this this is a this is a table representation of what it outputs it will output all the ingredients that you need for the recipe whether or not this input that this ingredient is currently missing from your inventory and action to perform so which actions still need to be performed so let's look at the following let's look at this example the recipe tells you you need the ingredients are a carrot a red hot pepper and a white onion and the inventory says you care you're carrying a white onion and a carrot right so down here you see aha we we do actually have we do actually have a carrot so it's not missing the carrot isn't missing you have it in your inventory the red hot pepper is missing we don't have it in the inventory but we need it for the recipe the white onion we need for the recipe but it's not missing then it also is for each of the ingredients is supposed to tell you this recipe model which of the what you still need to perform on it so here it says slice the carrot roast the carrot and you simply have a carrot it doesn't say slice the roast that means it's not sliced and roasted so the recipe is supposed to output you still need to slice and roast the carrot here for example for the white onion says fry the white onion and as you can see in the inventory it says you're carrying a fried white onion so for the white onion you see we don't need to do anything anymore so that the recipe model is basically trying to to make this table here and this table you can see as an intermediary step in order to do all the other things and the difference here to a pure RL method and this is important the difference is that this representation this intermediate table representation is done explicitly so the recipe model really produces a table like this and not just in other RL methods people go about and make this recipe model output some sort of you know let's say a 200 dimensional vector that's supposed to encompass all of this information and that doesn't appear to work as well like often that if you simply train this end-to-end that will not pick up on the important information because the training signal tends to be way too weak you have to imagine you already have this really really big model construction here and you're trying to learn it you're trying to learn it from a tiny reward signal that you get at the end right this is very noisy signal now if if you're now trying to say well the inputs to these things right this command here and we also saw the inputs to these these depend on this recipe model also now are whatever giant neural network construction here and we'll all train this end-to-end and these will actually not be text these will actually be some sort of latent vectors that will often fail because you're now just trying to extract information from too noisy of a reward signal so the authors here do actually pretty neat separation of that and they train this recipe model with actually an augmented data set so they go to freebase and get more food items and then they construct a data set that resembles this and train it in a supervised way to output tables tables like this so this is is pretty smart and I think it's a good lesson if you ever attempt something like this that really really important information such as this one if you can train it in a supervised way as a kind of a pre-processing step to your RL procedure that's extremely helpful here you can you can see how this is then used so by combining this table that was output from the recipe model and your inventory and the output of this look command you can then generate these commands so before we said it's important to reduce the everything you could do which is infinite things to everything that is reasonable to do currently and this model here does that so given this given that and given the description of what's currently in the room you can now generate these commands and for example take knife if you have to slice something because you see a knife is in the room and you could conceivably take the knife right you can construct these commands but also since you know right since you know what's since you know what's in your inventory and since you know which things are still missing you can generate commands like take the white onion or drop the water because you don't need the water right so um the the offers also group these things here in this what they call high-level commands which take all required items from here simply means take everything that's in the room that is not in the inventory but you need it so these things which for an RL agent it makes sense to group these things together because it doesn't make sense to have them as two separate things if you need both take both if you don't need any what if you have a new entry drop all of these things so that makes sense that's a small optimization that apparently brought some gains but the kind of the the overarching message here is that once you have a once you have this information from the recipe model you can then use it in many useful ways in order to make life for your RL agent easier alright so that kind of is the entire model that's very it's quite convoluted but basically you start with this here this recipe manager you decide you output this table down here which ingredients are in the recipe are they still missing and which actions we need to perform you then combine it with this information here the information about the current room and your inventory in order to come up with a set of commands that are conceivable to do here you combine these commands with some commands that are always available so commands that are always available are things like look inventory prepare meal you have that right you add that if the recipe manager does not output any missing and the agents location is the kitchen so you can add these other items and also we're not even gonna get into that you add navigational items because there are doors in these rooms and you need to navigate around so they actually train another model to here you see to detect to detect directions that you could move into and open doors for every closed door in the room so that's another challenge that the agent needs to overcome they have to build an entire model to predict which doors are there and are they closed do you need to open them so these commands if there are doors and if you can move through them these commands are also added to this set of commands that are reasonable so now we have a set of commands that are reasonable over here then you describe the room here you put both into this embedding and then finally your policy outputs an action that's that that's the entire process very convoluted very big very astonishing that this works with our L but in order to need to get it to work you actually need to do this supervised training and the experimental evidence here is quite solid in that they compare to baseline systems that that use classic techniques and they do some ablation over over their individual parts and they get second place I think in a competition about these text-based games so that's pretty good and that was it for me and check it out and bye bye
 Hi there. Today we're looking at Le Deep Chef, deep reinforcement learning agent
 for families of text-based games by Leonard Adolfs and Thomas Hoffmann. So
 this is a paper about engineering an agent for a particular family of tasks.
 This is different from reinforcement learning agents that for example are
 just good at one game, let's say Pong or whatnot and even I guess even things
 like Starcraft. Though this kind of depends on what you mean by game. So what
 are we talking about here? The following is a text-based games where the goal is
 to cook recipes. So let's just jump in and see what goes on. The game
 starts by telling you, you are hungry. Let's cook a delicious meal and so on.
 So the objective is basically always the same. It's find the cookbook, read the
 recipe that's in it, then collect all the things that are in the recipe, prepare
 them in certain ways that are also specified by the recipe and then at the
 end you have a meal and then you can eat the meal and that will give you points.
 But since it's a text-based games and the input doesn't come structured but it
 comes in natural text. So the game tells you for example kitchen. So basically
 you're in the kitchen. You are now in the kitchen. I guess you better just go and
 list everything you see here. You hear a noise, you spin around. So you see that
 the kind of input you get from the game is very playful, has a lot of descriptive
 elements. Sometimes it's like you see a closed oven. You make out a table. Then
 you can see on the counter you can make out a sliced fried red hot pepper and so
 on. So it's very much not trivial to kind of parse this in a traditional way.
 If you were to go about this by simply writing an algorithm extracting things
 it's very hard because for example you might see that there's an oven but it's
 a closed oven. You make out a table. So this is kind of a synonym for you see a
 table but you see like there is a table. You can make out a sliced fried red hot
 pepper and here it's important not only do you need to realize that there is a
 red hot pepper but also that its state is sliced and fried. This is important
 because you need all ingredients in a certain state. Right? You examine here you
 examine the stove so there is a stove. Right? So all these things you need to
 kind of understand. So if you now look there is a recipe book in here.
 Or no there isn't a recipe. You can examine recipe. I guess there is a recipe
 book in that room. If there is a recipe book then you can examine the recipe and
 that's the command. So the arrows here always indicate that that's a user
 command. And these you have to type. That's like the next thing that
 your agent needs to do. You can't select from a predefined set of actions.
 You actually need to type in the things you want to do. Right? And these are a
 lot. Like there are a lot of possibilities of what you could type in.
 Even if you restrict it to kind of what you know the game accepts there are
 still so many actions. It's way different than for example Atari games.
 They always have eight actions. Like there's eight buttons you could
 possibly press and that's it. And here there are like combinatorically many
 things you can do. Like you can prepare and take and all the ingredients. You
 don't know which ingredients come. So here you examine the recipe.
 Let's look at a recipe. It says you open the recipe. Start reading. Recipe number
 one. Here are the ingredients. Red hot pepper. Here for right now that's just one
 ingredient. Then there are directions. So what do you need to do? Slice the red
 hot pepper. Fry the red hot pepper and prepare the meal. Those are
 the directions of the recipe. You also have this inventory command which
 tells you which you're carrying. Next difficulty. The inventory is finite. So
 you can't carry everything. At some points you have to drop things that are
 unnecessary. You can't just take everything. Here you see the command take
 red hot pepper. That only works if there's a red hot pepper in the room. And
 here says you take the red hot pepper from the counter. Your score has just gone
 up by one point. And then if you type inventory it says you're carrying a
 sliced fried red hot pepper. Again here it says the state of the ingredient.
 So the ingredient is the red hot pepper and the state is sliced and fried. And
 then you can prepare meal and then you can eat meal and then it says your
 score has just gone up by one point. And these are the scores you collect. So
 there are a lot of difficulties that are actually not shown in this example. For
 example there are different rooms. You may have noticed here you're in the
 kitchen. But there could be other rooms and you start in a random room. You also
 need to navigate through the rooms. Close the doors to the rooms could be
 closed and then you need to open them and so on. You can only for example if
 this pepper here weren't already sliced and fried you need to find...
 You can only slice it if there is a knife in the room. You can only fry
 it if there is a frying pan or an oven or a stove in the room.
 So and then you'd have to notice that there is a knife. If there is no knife
 you need to take the red hot pepper bring it to a new room with a knife and
 then slice it. So this is vastly difficult game. The last difficulty is
 actually that in the test set there will be ingredients that you haven't seen
 during training. So also that there. Your agent needs to generalize. That's why it
 says a family of text-based games. Because the objective always the same to
 kind of cook the recipe. But the things you have to do and the things that
 appear and so on those are those change basically from episode to episode. And
 the test set will be different than the training set or kind of there will be
 unseen data. Alright so how does this paper go about solving this problem?
 This paper basically does the following and we are going here from high level to
 low level. On the highest level it's a reinforcement learning agent and that is
 sort of how you would imagine an RL agent to work. So here at the end you have
 a policy and the policy predicts an action. If you don't know what a kind of
 a policy and an action things are in RL these are basic RL concept and we'll
 kind of skip them here and I'll assume everyone knows what they are. But
 essentially a policy specifies which action you take next given the current
 game state. So the policy is made up, scores different actions. So at each step
 there are k actions available. And these k actions I foresaid there are almost
 infinitely many actions that you could take. The first difficulty and that's the
 thing that actually comes in here is to reduce all of the possible actions that
 you can't even list to just k commands. So we'll go into that later how this is
 done. But basically one of the main contributions of this paper is how do
 you even specify what is reasonable, what would be reasonable to do in the current
 situation. And then the policy over here only has to decide among those reasonable
 actions, not among all actions. But given that you have k reasonable commands
 you see here command one command, these are embedded and then fed into GRUs which are
 recurrent neural networks. So for each of these commands you'll get a 32
 dimensional vector. This 32 dimensional vector is here C1 through Ck. Each are
 combined with an encoding of the current state. So these 32 dimensional
 vector are combined with encoding of the current state which is 256 dimensional
 and then fed into a neural network that will output a probability distribution
 over these actions. This is pretty classic in deep reinforcement learning.
 So you have action encoding and the state encoding and the policy decides on that.
 The state encoding you'll see here it's the same everywhere of course because
 the current game state is the current game state. This comes from this model up
 here. What this does is over here you have the what you would call the state
 the current observation. The current observation is composed of many
 things. Specifically the following eight things. The first one is actually
 called observation which is I would call all of this the current observation
 from an RL perspective. But the first is actually observation. It's whatever you
 saw the big text you saw before. Like you were in the kitchen it looks like this
 it smells like this you turn around and so on. This would be the observation.
 It's what the game engine says at the current time step. This is just a piece of
 text. Second missing items. Third unnecessary items. Now these things you
 might wonder okay how do I know what what items are missing and unnecessary.
 These things come from another model that this paper trains and we'll get
 into that later. But basically they have a method of specifying which items are
 still missing which are unnecessary and they list those here. Then description
 which is the output of the last look command. So in each room you can look you
 can type look and then it'll give you a description of the room and what's in
 there. The previous commands this is often used in RL either explicitly or
 implicitly through a recurrent network in order to give the agent an idea what
 what happened in the in the previous steps or what it did so that it doesn't
 repeat actions unnecessarily or so it learns to not repeat actions
 unnecessarily. Required utilities. Again this is a model that's kind of trained
 to predict what utilities are required to perform some actions. So as I said
 before if you want to slice the red hot pepper you need a knife. If you want to
 fry it you need a stove. Discovered locations. As I said there are different
 rooms you actually don't know what rooms there are before you actually go in in
 there. So before you go through a door you reach another room. So the list of
 previously discovered and visited locations is there and then the name of
 the current location it is also there. So these are eight things that make up the
 current observation. These eight things are just strings of text and these eight
 things are each one as you can see here these are that the eight things from
 observation to location each one are embedded and fed also into an RNN. So for
 each of these eight things you'll obtain a 32 dimensional vector and these are
 all concatenated to make up one big 256 dimensional vector. So this 256
 dimensional vector will contain all the necessary information about the current
 room what's in there what what items are you still missing what items do you have
 in your inventory which ones are unnecessary and so on. So if you train
 this correctly this 256 dimensional vector will describe the current game
 state as it is relevant to your agent like everything about it every
 relevant information that's in here will be encoded in this vector. Now this
 vector isn't the final state encoding yet what you'll have is you feed this into
 an RNN that takes as input the last time steps you have to imagine the last time
 step already there was observation blah blah blah this entire thing was I'm just
 copying I'm just copying this box over here so this entire thing was already
 done last step and already fed into an RNN so this this is an RNN that actually
 goes over time and the last whatever the output here is it will be fed to the
 next step and this is a trick often done in reinforcement learning as well that
 you actually have a recurrent neural network over the time steps so each
 time step you have a certain observation you encode it and so on you get a
 description of that and then you feed this into an RNN what the RNN can learn
 to do is it can learn to react to different not only to the current
 observation but to the current observation conditioned on the history
 of previous observations so it can learn before I was in this room now I'm in this
 new room so I actually haven't you know taken all the items from this room yet
 because I just came into this room and so on so the the kind of component where
 you are able to look at the past and what happened in the past is in captured
 by this RNN here so it's fairly complicated architecture but this here
 this state encoding that is conditioned on the also on the history then goes into
 this into here that's it that's the vector that goes in here is combined
 with each action so all of these actions here these K actions and this is all fed
 through a neural network and that will give you the policy this is a fairly
 complicated thing but if you look at it it's not it's not too it's not too
 difficult actually so what you'll do is you will take your observations here this
 is all observation it will be encoded and combined with the history in order
 to give you this in order to give you an encoding of the current state on the
 other hand you'll take all of the possible commands that you could
 perform right now encode each one separately right into an embedding and
 then you combine each one of those with this encoding you specified previously
 that you and and from that you make your decision which action to take next and
 the action here is the one that's output is the action you take next sampled from
 this policy the last thing you need is a value network and this is just important
 for reinforcement learning which tells you from this state here so I'm getting
 weird with colors here from this state here which is the same as this one so
 you'd simply transfer this over from this state how valuable is that what's
 my value of the state and the value is if I'm in this state and I act as I
 normally act what are all my future rewards going to be combined so it
 basically gives you a value of this state you can think of this in for
 example terms of chess if you had this in chess and then this here is it would
 be a description of the chessboard this HT and the value would be how valuable
 is this position for you so if you're very much ahead and material and
 position and so on this value would be very high if you're behind this value
 would be very low and this is in a real network simply trying to predict that
 value so with all of this you now have a never good basis to do reinforcement
 learning you have a policy you have a value network and from that you can
 train an RL agent and this is done classically in an actor critic way where
 you do advantage learning here the advantage and the policy you train
 weighted by the advantage then the value network you train to be close to their
 reward and then you have an entropy penalty if you don't know what these
 things are the video will get bit too long if I were to go over these
 reinforcement learning concepts but these are very standard in reinforcement
 learning so you can train these you can basically train what it does is you can
 train these neural networks in absence of label training data because you don't
 know what the best action is in each step right there's no one telling you
 you just have a reward you just sometimes you get a point and you don't
 know which actions led to that so these things will actually allow you to train
 these neural networks by using just the reward without knowing which exact
 actions were right and wrong and that's the core of reinforcement learning
 obviously alright so the the core one of the core ingredients actually is this
 recipe manager and the recipe manager is a sub model that does the following so
 here it takes as an input the cookbook here and it also takes as an input the
 inventory and it outputs something like this and this this is a this is a table
 representation of what it outputs it will output all the ingredients that you
 need for the recipe whether or not this input that this ingredient is currently
 missing from your inventory and action to perform so which actions still need
 to be performed so let's look at the following let's look at this example the
 recipe tells you you need the ingredients are a carrot a red hot pepper
 and a white onion and the inventory says you care you're carrying a white onion
 and a carrot right so down here you see aha we we do actually have we do
 actually have a carrot so it's not missing the carrot isn't missing you
 have it in your inventory the red hot pepper is missing we don't have it in
 the inventory but we need it for the recipe the white onion we need for the
 recipe but it's not missing then it also is for each of the ingredients is
 supposed to tell you this recipe model which of the what you still need to
 perform on it so here it says slice the carrot roast the carrot and you simply
 have a carrot it doesn't say slice the roast that means it's not sliced and
 roasted so the recipe is supposed to output you still need to slice and roast
 the carrot here for example for the white onion says fry the white onion and
 as you can see in the inventory it says you're carrying a fried white onion so
 for the white onion you see we don't need to do anything anymore so that the
 recipe model is basically trying to to make this table here and this table you
 can see as an intermediary step in order to do all the other things and the
 difference here to a pure RL method and this is important the difference is that
 this representation this intermediate table representation is done explicitly
 so the recipe model really produces a table like this and not just in other RL
 methods people go about and make this recipe model output some sort of you
 know let's say a 200 dimensional vector that's supposed to encompass all of this
 information and that doesn't appear to work as well like often that if you
 simply train this end-to-end that will not pick up on the important information
 because the training signal tends to be way too weak you have to imagine you
 already have this really really big model construction here and you're
 trying to learn it you're trying to learn it from a tiny reward signal that
 you get at the end right this is very noisy signal now if if you're now trying
 to say well the inputs to these things right this command here and we also saw
 the inputs to these these depend on this recipe model also now are whatever
 giant neural network construction here and we'll all train this end-to-end and
 these will actually not be text these will actually be some sort of latent
 vectors that will often fail because you're now just trying to extract
 information from too noisy of a reward signal so the authors here do actually
 pretty neat separation of that and they train this recipe model with actually an
 augmented data set so they go to freebase and get more food items and
 then they construct a data set that resembles this and train it in a
 supervised way to output tables tables like this so this is is pretty smart and
 I think it's a good lesson if you ever attempt something like this that really
 really important information such as this one if you can train it in a
 supervised way as a kind of a pre-processing step to your RL
 procedure that's extremely helpful here you can you can see how this is then
 used so by combining this table that was output from the recipe model and your
 inventory and the output of this look command you can then generate these
 commands so before we said it's important to reduce the everything you could do
 which is infinite things to everything that is reasonable to do currently and
 this model here does that so given this given that and given the description of
 what's currently in the room you can now generate these commands and for example
 take knife if you have to slice something because you see a knife is in
 the room and you could conceivably take the knife right you can construct these
 commands but also since you know right since you know what's since you know
 what's in your inventory and since you know which things are still missing you
 can generate commands like take the white onion or drop the water because
 you don't need the water right so um the the offers also group these things here
 in this what they call high-level commands which take all required items
 from here simply means take everything that's in the room that is not in the
 inventory but you need it so these things which for an RL agent it makes
 sense to group these things together because it doesn't make sense to have
 them as two separate things if you need both take both if you don't need any
 what if you have a new entry drop all of these things so that makes sense that's
 a small optimization that apparently brought some gains but the kind of the
 the overarching message here is that once you have a once you have this
 information from the recipe model you can then use it in many useful ways in
 order to make life for your RL agent easier alright so that kind of is the
 entire model that's very it's quite convoluted but basically you start with
 this here this recipe manager you decide you output this table down here which
 ingredients are in the recipe are they still missing and which actions we need
 to perform you then combine it with this information here the information about
 the current room and your inventory in order to come up with a set of commands
 that are conceivable to do here you combine these commands with some
 commands that are always available so commands that are always available are
 things like look inventory prepare meal you have that right you add that if the
 recipe manager does not output any missing and the agents location is the
 kitchen so you can add these other items and also we're not even gonna get into
 that you add navigational items because there are doors in these rooms and you
 need to navigate around so they actually train another model to here you see to
 detect to detect directions that you could move into and open doors for every
 closed door in the room so that's another challenge that the agent needs to
 overcome they have to build an entire model to predict which doors are there
 and are they closed do you need to open them so these commands if there are
 doors and if you can move through them these commands are also added to this
 set of commands that are reasonable so now we have a set of commands that are
 reasonable over here then you describe the room here you put both into this
 embedding and then finally your policy outputs an action that's that that's the
 entire process very convoluted very big very astonishing that this works with our
 L but in order to need to get it to work you actually need to do this supervised
 training and the experimental evidence here is quite solid in that they compare
 to baseline systems that that use classic techniques and they do some
 ablation over over their individual parts and they get second place I think
 in a competition about these text-based games so that's pretty good and that was
 it for me and check it out and bye bye