What are the prior efforts that develop RL environments with language-informed tasks?
There is significant interest in RL for training
policies conditioned on natural language. This
includes multiple efforts at developing RL envi-
ronments with language-informed tasks, mainly
via grounded language learning in 2D or 3D envi-
ronments or using text games. Oftentimes, using
synthetic language (Narasimhan et al., 2015; John-
son et al., 2017; Chevalier-Boisvert et al., 2019;
Cao et al., 2020; Côté et al., 2018; Urbanek et al.,
2019; Hermann et al., 2017; Co-Reyes et al., 2019;
Hausknecht et al., 2020; Jiang et al., 2020). Al-
though synthetic language allows studying the prob-
lem of learning high-level concepts, many of the
complexities of natural language may be stripped
away, and such approaches run the risk of reducing
the language learning challenge to reverse engi-
neering the hand-crafted generation process. In
contrast, lilGym is based on semantically-diverse
human-written language grounded in a visual en-
vironment, and requires both the ability of reason-
ing over highly-compositional language including
sets and spatial relations, and precise alignment
between statements and states
TextWorld: A Learning Environment for Text-based Games
The limits of my language mean the limits of my world
[{"first":"Marc-Alexandre","last":"Cot'e","middle":[]},{"first":"'Akos","last":"K'ad'ar","middle":[]},{"first":"Xingdi","last":"Yuan","middle":[]},{"first":"Ben","last":"Kybartas","middle":[]},{"first":"Tavian","last":"Barnes","middle":[]},{"first":"Emery","last":"Fine","middle":[]},{"first":"James","last":"Moore","middle":[]},{"first":"Ruo","last":"Tao","middle":["Yu"]},{"first":"Matthew","last":"Hausknecht","middle":[]},{"first":"Layla","last":"Asri","middle":["El"]},{"first":"Mahmoud","last":"Adada","middle":[]},{"first":"Wendy","last":"Tay","middle":[]},{"first":"Adam","last":"Trischler","middle":[]}]
Learning to Speak and Act in a Fantasy Text Adventure Game
We introduce a large-scale crowdsourced text adventure game as a research platform for studying grounded dialogue. In it, agents can perceive, emote, and act whilst conducting dialogue with other agents. Models and humans can both act as characters within the game. We describe the results of training state-of-the-art generative and retrieval models in this setting. We show that in addition to using past dialogue, these models are able to effectively use the state of the underlying world to condition their predictions. In particular, we show that grounding on the details of the local environment, including location descriptions, and the objects (and their affordances) and characters (and their previous actions) present within it allows better predictions of agent behavior and dialogue. We analyze the ingredients necessary for successful grounding in this setting, and how each of these factors relate to agents that can talk and act successfully.
[{"first":"Jack","last":"Urbanek","middle":[]},{"first":"Angela","last":"Fan","middle":[]},{"first":"Siddharth","last":"Karamcheti","middle":[]},{"first":"Saachi","last":"Jain","middle":[]},{"first":"Samuel","last":"Humeau","middle":[]},{"first":"Emily","last":"Dinan","middle":[]},{"first":"Tim","last":"Rocktäschel","middle":[]},{"first":"Douwe","last":"Kiela","middle":[]},{"first":"Arthur","last":"Szlam","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
WordCraft: An Environment for Benchmarking Commonsense Agents
The ability to quickly solve a wide range of real-world tasks requires a commonsense understanding of the world. Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge. This is partly due to the lack of lightweight simulation environments that sufficiently reflect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment. To better enable research on agents making use of commonsense knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2. This lightweight environment is fast to run and built upon entities and relations inspired by real-world semantics. We evaluate several representation learning methods on this new benchmark and propose a new method for integrating knowledge graphs with an RL agent.
[{"first":"Minqi","last":"Jiang","middle":[]},{"first":"Jelena","last":"Luketina","middle":[]},{"first":"Nantas","last":"Nardelli","middle":[]},{"first":"Pasquale","last":"Minervini","middle":[]},{"first":"Philip","last":"Torr","middle":["H.","S."]},{"first":"Shimon","last":"Whiteson","middle":[]},{"first":"Tim","last":"Rocktaschel","middle":[]}]
Grounded Language Learning in a Simulated 3D World
We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.
[{"first":"Karl","last":"Hermann","middle":["Moritz"]},{"first":"Felix","last":"Hill","middle":[]},{"first":"Simon","last":"Green","middle":[]},{"first":"Fumin","last":"Wang","middle":[]},{"first":"Ryan","last":"Faulkner","middle":[]},{"first":"Hubert","last":"Soyer","middle":[]},{"first":"David","last":"Szepesvari","middle":[]},{"first":"Wojciech","last":"Czarnecki","middle":["Marian"]},{"first":"Max","last":"Jaderberg","middle":[]},{"first":"Denis","last":"Teplyashin","middle":[]},{"first":"Marcus","last":"Wainwright","middle":[]},{"first":"Chris","last":"Apps","middle":[]},{"first":"Demis","last":"Hassabis","middle":[]},{"first":"Phil","last":"Blunsom","middle":[]}]
Interactive Fiction Games: A Colossal Adventure
A hallmark of human intelligence is the ability to understand and communicate with language. Interactive Fiction games are fully text-based simulation environments where a player issues text commands to effect change in the environment and progress through the story. We argue that IF games are an excellent testbed for studying language-based autonomous agents. In particular, IF games combine challenges of combinatorial action spaces, language understanding, and commonsense reasoning. To facilitate rapid development of language-based agents, we introduce Jericho, a learning environment for man-made IF games and conduct a comprehensive study of text-agents across a rich set of games, highlighting directions in which agents can improve.
[{"first":"Matthew","last":"Hausknecht","middle":[]},{"first":"Prithviraj","last":"Ammanabrolu","middle":[]},{"first":"Marc-Alexandre","last":"Cot'e","middle":[]},{"first":"Xingdi","last":"Yuan","middle":[]}]
TextWorld: A Learning Environment for Text-based Games
Text-based games are complex, interactive simulations in which text describes the game state and players make progress by entering text commands. They are fertile ground for language-focused machine learning research. In addition to language understanding, successful play requires skills like long-term memory and planning, exploration (trial and error), and common sense.
TextWorld: A Learning Environment for Text-based Games
Consider Zork [Infocom, 1980], one of the genre's most famous examples. Figure 1 depicts Zork's opening scene along with two player commands and the corresponding system responses. As illustrated, the game uses natural language to describe the state of the world, to accept actions from the player, and to report subsequent changes in the environment. Through sequential decision making, the player works toward goals which may or may not be specified explicitly. In the nomenclature of reinforcement learning (RL), language is the action space and also the observation space [Narasimhan et al., 2015]. In text-based games, the observation and action spaces are both combinatorial and compositional -major challenges for reinforcement learning. Furthermore, text-based games are partially observable since descriptive text does not communicate complete information about the underlying game state or may do so ambiguously. As a consequence of Figure 1: Intro to Zork these (and other) challenges, hand-authored games like Zork are beyond the capabilities of current learning algorithms [Narasimhan et al., 2015, Haroush et al., 2018.
TextWorld: A Learning Environment for Text-based Games
To help agents progress toward mastering text games in a controlled and scientific manner, we introduce the TextWorld learning environment. TextWorld is a sandbox environment [Wright, 1996, Sukhbaatar et al., 2015 in which games of varying complexity emerge from a set of underlying world mechanics. In this setting, simpler games can act as stepping stones toward more complex games. Like the Arcade Learning Environment (ALE, [Bellemare et al., 2013]), Gym [Brockman et al., 2016], and CommAI [Baroni et al., 2017], TextWorld enables interactive play-through of a curated set of games. Unlike previous text-based environments, including TextPlayer and PyFiction, TextWorld's sandbox functionality enables users to handcraft games or to construct games automatically through a suite of generative mechanisms.
TextWorld: A Learning Environment for Text-based Games
Specifically, TextWorld features a logic engine that automatically builds game worlds, populates them with objects and obstacles, and generates quests that define a goal state and how to reach it. It automatically generates text descriptions of underlying game states using an extensible vocabulary and a context-free grammar (CFG). Common-sense rules encoded in the logic and grammar govern generated worlds and the quests within them, to make these human-interpretable and consistent with existing games: e.g., keys open locked doors and can be carried or stored in containers; food items can be combined, cooked, and eaten. Furthermore, because the vocabulary contains synonyms for most nouns, verbs, and adjectives, different surface forms can be applied automatically to abstract types to add variety and complexity: e.g., the <container> object may manifest as a chest or cabinet; the <move> action may manifest as walk or go.
TextWorld: A Learning Environment for Text-based Games
TextWorld's generative nature has several downstream implications for learning. First, it means there exists a known and structured representation of the partially observable game state. This enables exact state-tracking [Henderson et al., 2014] and the corresponding assignment of intermediate rewards in training (if desired). Second, agents can be trained on a potentially infinite set of related text games rather than a finite collection as in previous learning environments. By controlling parameters of the generative process for training and test games, TextWorld can be used to investigate curriculum learning, generalization, and transfer learning in RL. Tunable parameters include the length of quests, the size of environments, the number of abstract action and object types, the number of synonyms for each type, complexity of the descriptive grammar, and more.
TextWorld: A Learning Environment for Text-based Games
A powerful feature of language which motivates our interest in text games is that it abstracts away complex physical processes. For instance, through text an agent could learn and use the concept that opening doors provides access to connected rooms without going through the (literal) motions of turning knobs in 3D space and time. This level of abstraction can be useful for studying functions of control, planning, etc. in isolation and in tandem with the function of language itself.
TextWorld: A Learning Environment for Text-based Games
The aim of this paper is to introduce TextWorld to the research community. Its primary contributions are:
TextWorld: A Learning Environment for Text-based Games
• A survey of the machine-learning challenges of and approaches to text-based games, including a curated list of hand-authored games with corresponding analysis;
TextWorld: A Learning Environment for Text-based Games
• A detailed description of the TextWorld framework, its features, and how to use it;
TextWorld: A Learning Environment for Text-based Games
• An initial set of simple text games to be used as RL benchmarks;
TextWorld: A Learning Environment for Text-based Games
• Evaluation of several baseline algorithms on both benchmark and hand-authored games.
TextWorld: A Learning Environment for Text-based Games
Subsequent works will more deeply investigate novel approaches to RL for text games. Our hope is that TextWorld becomes a living resource, with contributors developing new benchmarks and algorithms to push the state of the art forward.
TextWorld: A Learning Environment for Text-based Games
The remainder of this paper is organized as follows. In Section 2 we introduce text-based games, formalize them as RL problems and highlight their challenges. In Section 3 we delve into details of the TextWorld framework, how it generates and interacts with text games, and how it may be used to train RL agents. Section 4 describes related frameworks and existing approaches to solving text games, while Section 5 describes some of TextWorld's benchmark tasks and our experimental results. We discuss limitations of the framework and future work in Section 6 before concluding.
TextWorld: A Learning Environment for Text-based Games
Text-based games are sequential decision-making problems that can be described naturally by the Reinforcement Learning (RL) formalism. In this section, we define some of the terminology found in text-based games, formalize the text-based environment as an RL problem, discuss challenges faced by RL agents in such environments, and show how these challenges motivate the need for a framework like TextWorld. In the following, an "agent" is a model that takes text information as input and outputs text commands to progress through a game.
TextWorld: A Learning Environment for Text-based Games
Text-based games are turn-based games usually played through a command line terminal. At each turn, several lines of text describe the state of the game, and the player may enter a text command to change this state in some desirable way (i.e., to move towards a goal). A game's built-in parser or interpreter deciphers player commands and maps them to state changes (events in the game). The genre became popular in the early 1980s especially with the release of Zork [Infocom, 1980], which featured rich storytelling and an advanced command parser.
TextWorld: A Learning Environment for Text-based Games
Text-based games can be classified according to how the player issues commands (see Figure 2): in parser-based games, the player types text commands character by character; in choice-based games, the player chooses from a given list of command options; and in hypertext-based games, the player clicks on one of several links present in the description. The work in this paper focuses on parser-based games.
TextWorld: A Learning Environment for Text-based Games
Figure 2: Types of text-based games. Image from [He et al., 2015].
TextWorld: A Learning Environment for Text-based Games
In text-based game terminology, each discrete in-game location is called a room (e.g., "Kitchen", "In front of the white house", and so on). A game may contain one or several rooms connected in some topology forming a map. To explore the environment, the player issues navigational commands: go followed by a cardinal direction (north, northeast, etc.) or an orthogonal direction (up, down). Maps vary greatly from one game to another. Room exits and entrances do not always match (e.g., go north and then go south may not return you to where you started). Also, some navigational commands may not have a reciprocal (e.g., in Zork [Infocom, 1980], a trapdoor locks from behind preventing the player from going back).
TextWorld: A Learning Environment for Text-based Games
Most of the time, rooms contain objects the player can interact with (e.g., take sandwich from table, eat sandwich, open door, etc.). Objects are defined by their attributes, which also determine how the player can interact with them (through object affordances). For instance, an object could be portable (e.g., a lamp) or portable and edible (e.g., a sandwich).
TextWorld: A Learning Environment for Text-based Games
One of the biggest challenges when playing parser-based games is figuring out what are the commands that will be understood by the parser (i.e., that are intended by the game's author). Also, depending on the game, the result of some commands may be stochastic (e.g., go up succeeds 75% of the time, but 25% of the time results in the player falling down to a previous room). A detailed list of puzzles and challenges traditionally found in text-based games can be found in Appendix A.
TextWorld: A Learning Environment for Text-based Games
Fundamentally, text-based games can be seen as partially observable Markov decision processes (POMDP) [Kaelbling et al., 1998] where the environment state is never observed directly. To act optimally, an agent must keep track of all observations, i.e., textual feedback received after entering commands. Although the observation can be augmented with feedback from commands like look and inventory, which describe the agent's surroundings and possessions, this information is still limited to the current room.
TextWorld: A Learning Environment for Text-based Games
Formally, a text-based game is a discrete-time POMDP defined by (S, T, A, Ω, O, R, γ), where S is the set of environment states, T is the set of conditional transition probabilities between states, A is the set of words that are used to compose text commands, Ω is the set of observations, O is a set of conditional observation probabilities, R : S × A → R is the reward function, and γ ∈ [0, 1] is the discount factor.
TextWorld: A Learning Environment for Text-based Games
The environment state at turn t in the game is s t ∈ S. It contains the complete internal information of the game, like the position and state of every entity (rooms, objects, player, etc.), much of which is hidden from the agent. When an agent issues a command c t (defined next), the environment transitions to state s t+1 with probability T (s t+1 |s t , c t ).
TextWorld: A Learning Environment for Text-based Games
At each turn t, the agent issues a text command c t of at least one word. In parserbased games, the interpreter can accept any sequence of characters (of any length) but will only recognize a tiny subset thereof. Furthermore, only a fraction of recognized commands will actually change the state of the world. The resulting action space is enormous and intractable for existing RL algorithms. We make the following two simplifying assumptions:
TextWorld: A Learning Environment for Text-based Games
• Word-level Commands are sequences of at most L words taken from a fixed vocabulary V .
TextWorld: A Learning Environment for Text-based Games
• Syntax Commands have the following structure: verb[noun phrase [adverb phrase]], where [. . . ] indicates that the sub-string is optional. In this context, a noun phrase is a string identifying an object (e.g., "the big wooden chest"). Similarly, an adverb phrase provides additional context for the command (e.g., "with the red key"). To simplify the syntax further, determiners are omitted. 1
TextWorld: A Learning Environment for Text-based Games
The agent's action space is some vocabulary V plus a special token <return> that indicates the end of a command. Each action a i t ∈ A is a token, where t is the turn in the game and i indicates the ith token in the command c t . A command is a sequence of n ≤ L tokens c t = [a 1 t , . . . , a n t ] that respects the syntax previously defined and ends with a n t = <return>.
TextWorld: A Learning Environment for Text-based Games
1 Typical text-based game interpreters disregard determiners.
TextWorld: A Learning Environment for Text-based Games
The agent's policy is a mapping between its states and actions. In TextWorld, the agent's policy π θ , where θ are the policy's parameters, maps a state s t and words generated in the command so far to the next word to generate: a i t = π θ (s t , a 0 t , ..., a i−1 t ).
TextWorld: A Learning Environment for Text-based Games
The text information perceived by the agent at a given turn t in the game is the agent's observation, o t ∈ Ω, which depends on the environment state and the previous command with probability O(o t |s t , c t−1 ). In other words, the function O selects from the environment state what information to show to the agent given the command entered. For instance, if the agent tries to open a chest, the observation returned by the environment might show that the chest is locked.
TextWorld: A Learning Environment for Text-based Games
Based on its actions, the agent receives reward signals r t = R(s t , a t ). The agent's goal is to maximize the expected discounted sum of rewards received E [ t γ t r t ].
TextWorld: A Learning Environment for Text-based Games
Most text-based games (including those in our curated list) have a scoring mechanism whereby players receive points for completing (sub)quests and reaching new locations. When available, this score can be used as a reward signal. Otherwise, one could define reward signals by assigning a positive reward if the agent completes the game. Intermediate rewards might also be inferred from the interpreter's feedback. Note that this feedback usually only contains information about the results of individual commands (e.g., "I don't know this verb!", "This chest is locked!") rather than about overall progress.
TextWorld: A Learning Environment for Text-based Games
Complex environments make training RL agents challenging for several reasons. Here, we list some conventional challenges known in the RL literature that are also prevalent in text-based games.
TextWorld: A Learning Environment for Text-based Games
Partial Observability As mentioned, states of text-based games are partially observable. Only the local information such as the current room description and the player's inventory is made available. Moreover, taking into account only the latest observation, it may be impossible to differentiate certain states based on observations. For instance, seeing a blue locked chest in a room, it is important for the agent to know whether or not it collected or saw a blue key in the past. The environment might give the same feedback for two different commands (e.g., "taken" might be the feedback for take blue key or take red apple). Furthermore, important information about the environment might not be apparent in the observation (e.g., whether a chest is locked or not, what it contains, etc.). Observations may also be time-sensitive (e.g., the agent only gets a reward when examining clues for the first time).
TextWorld: A Learning Environment for Text-based Games
With large state spaces, tabular methods for solving RL problems are no longer practical [Sutton and Barto, 2018]. Finding good approximate solutions is still an active area of research. In text-based games, the state space is combinatorial and enormous; the number of possible states increases exponentially with the number of rooms and objects.
TextWorld: A Learning Environment for Text-based Games
As with large state spaces, reasoning in an environment with a large number of actions necessitates finding good approximate solution methods to replace the tabular ones [Sutton and Barto, 2018]. The text-based game setting is especially challenging since the action space is large and sparse; the space of all word strings is much larger than the space of admissible commands (i.e., commands that actually change the underlying state s t ). In addition, the outcome or even the validity of some commands might depend on a specific event or how much time has passed (e.g., the tide rises and blocks the player in [Bates, 1987]).
TextWorld: A Learning Environment for Text-based Games
Balancing exploration of the environment and the exploitation of known information is a fundamental issue in RL [McFarlane, 2003]. Exploration is at the core of text-based games as they cannot be solved by learning a purely reactive controller. Instead, a strategy that promotes directed exploration must be used; the agent must deliberately explore the environment, collecting information about objects and persons encountered along the way (e.g., you never know what is in a box without opening it first). Such information hints about the goal/purpose of the game, what dangers are present, and provides clues that might become handy later in the game for solving puzzles. We expect that agents, like humans, will benefit from exploration driven by curiosity.
TextWorld: A Learning Environment for Text-based Games
Knowing which actions were responsible for obtaining a certain reward, especially when rewards are sparse, is another fundamental issue in RL [Sutton and Barto, 2018]. Sparse rewards are inherent to text-based games in which the agent must generate a sequence of actions before observing a change in the environment state or getting a reward signal. For instance, activating a switch might have no immediate effect although it is essential for completing the game. Most text-based games feature sparse rewards, on the order of a single positive reward every 10-20 steps when following an optimal state trajectory.
TextWorld: A Learning Environment for Text-based Games
By their very nature, text-based games bring additional challenges related to natural language understanding.
TextWorld: A Learning Environment for Text-based Games
Observation Modality Observations consist in the environment's textual feedback to the previous command. This means that the observation space is unbounded, consisting of arbitrary-length sequences of characters. To simplify things, we assume that an observation is made of spaceseparated words that may or may not be found in an English dictionary. One drawback of looking only at words is that we may lose some information provided by the spacing (e.g., ASCII art in Infidel [Berlyn, 1983] or a sonar map in Seastalker [Galley and Lawrence, 1984]).
TextWorld: A Learning Environment for Text-based Games
Text-based games process player input using a parser. The parser varies from game to game in terms of the actions it recognizes. For example, nearly all games recognize actions like get, take and go, but only some games recognize verbs like tickle, swim, dig and bribe. Part of the challenge of playing a parser-based text game is understanding which verbs and objects are recognized by the parser. Making this task more difficult, failure messages vary from game to game when the parser receives an invalid or unrecognized command.
TextWorld: A Learning Environment for Text-based Games
To succeed at text-based games, it is necessary to understand how to interact with everyday objects. For example, if the description of a location includes a tree, it is likely that a player could climb or chop the tree, but not eat or drive it. The problem of identifying which verbs are applicable to a given object is called affordance extraction and learning agents must solve it efficiently to make progress in text-based games without exhaustive search.
TextWorld: A Learning Environment for Text-based Games
Language Acquisition Some objects and actions may be named with invented words. Also, modifier words affect how some objects can be interacted with (this is related to affordance extraction). The meaning of these words must be learned on-the-fly while interacting with the environment. Text-based games also use linguistic coreference, since it is more pleasant to humans, which can complicate the task for learning machines.
TextWorld: A Learning Environment for Text-based Games
Solving a single text-based game often corresponds to tackling most of the above challenges at once, which makes it very difficult for existing algorithms. What would be useful is a way of testing and debugging RL agents in simpler settings (e.g., one room with two objects where the goal is to eat the edible one). This is the main purpose of TextWorld's generative functionality (described in Section 3.2). It can be used to focus on desired subsets of the challenges listed above.
TextWorld: A Learning Environment for Text-based Games
First, it is possible to control the size of the state space (e.g., the number of rooms, number of objects, and how many commands are required in order to reach the goal optimally). At the moment, TextWorld has deterministic transitions between states.
TextWorld: A Learning Environment for Text-based Games
It is also possible to control the partial observability of the state by augmenting the agent's observations. The environment can provide the agent with a list of objects present in-game or even provide all information about the current game state. For instance, instead of generating the observation that there is a blue chest, the environment could state that the chest is locked and that inside the chest there is a red apple. In this setting, the agent does not need to explore to determine the layout of the world and the objects it contains.
TextWorld: A Learning Environment for Text-based Games
TextWorld enables one to generate a large number of games and control their shared characteristics (map, objects, goals, etc.). This is useful for focusing, e.g., on language acquisition and affordance extraction: the agent can be trained on games with a fixed number of rooms and object types but with different object names. By interacting with objects, the agent should learn which have a similar function and generalize from one game instance to another.
TextWorld: A Learning Environment for Text-based Games
There are several ways to ease the language generation task. It is possible to restrict the agent's vocabulary to in-game words only or to restrict the verbs that the agent can generate to those understood by the parser. It is also possible to use a simplified grammar where object names are replaced with symbolic tokens (e.g., "You see container1 and container2."). Language generation can be circumvented completely by converting every generated game into a choice-based game. In this case, commands c t are the agent's actions, i.e., the agent's output becomes an index into the set of admissible commands (see Section 3.3.1) rather a sequence of words.
TextWorld: A Learning Environment for Text-based Games
Finally, instead of earning rewards only at the end of a game if the agent is successful, one can also provide intermediate rewards during training based on environment state transitions and the ground truth winning policy (see Section 3.1.1).
TextWorld: A Learning Environment for Text-based Games
TextWorld 2 is a Python framework for training and testing RL agents on text-based games. It enables generation from a game distribution parameterized by the map size, the number of objects, quest length and complexity, richness of text descriptions, and more. The framework's architecture is shown in Figure 3. We detail the Engine component in Section 3.1, which covers how the internal state of generated games is represented and how the transition function is defined. The Generator component is described in Section 3.2, which explains the process of automatic game generation. TextWorld can also be used to play existing text-based games (see a curated list of games in Section 5.1) but provides more limited information from the internal states of such games.
TextWorld: A Learning Environment for Text-based Games
Game generation in TextWorld relies on a simple inference engine that ensures game validity at every step in the generation process. A game is said to be valid if it is possible to reach the end goal from the initial state. Although we could have used an existing problem-solver for this purpose, we did not require the full power of logical programming languages. TextWorld's  Figure 3: Overview of the framework. The two main components (in blue) of the proposed pipeline: the game generator and the game engine, which handles interactive play. Inform 7 and Git-Glulx are third-party libraries (in green) and the agent (in red) should be provided by the user. Given some knowledge base, sampled game definitions are first converted to Inform 7 code and compiled into a Glulx executable file. Then, agents interact with the game by communicating with the Git-Glulx interpreter via TextWorld. inference engine implements simple algorithms specific to our environments, such as a one-step forward and backward chaining with or without fact creation (more on this in Section 3.2). In future work our aim is to integrate the TextWorld framework with well established frameworks such as GDL [Genesereth et al., 2005] or STRIPS [Fikes and Nilsson, 1971].
TextWorld: A Learning Environment for Text-based Games
To better explain the framework, let's consider the following simple text-based environment. There is a kitchen with a table and a closed fridge in which there is an apple. A visual representation can be seen in Figure 4. The player is represented by the small avatar and the letter P. Objects of the container type are represented by a chest, supporters (or surfaces) are represented by a table and food-type items are represented by an apple symbol. The anchor symbol next to certain objects means that they are fixed in place (i.e., cannot be taken).
TextWorld: A Learning Environment for Text-based Games
TextWorld's generated text-based games can be represented internally as a Markov Decision Process (MDP) [Puterman, 1994] defined by (S, A, T, R, γ), where S is the set of environment states, A is the set of actions (with A st those available in state s t ∈ S), T (s t , a, s t+1 ) = P(s t+1 |s t , a) is the state transition function that depends on current state s t and action taken a ∈ A st , R : S × A → R is the reward function, and γ ∈ [0, 1] is the discount factor. Text-based games are episodic since they stop when the player reaches one of the winning (goal) states G ⊂ S.
TextWorld: A Learning Environment for Text-based Games
Note that the MDP part of the POMDP, defined in Section 2.2, is semantically equivalent to the one described in this section. The sole exception is the action space; in the POMDP, we assume there exists an underlying function that maps text strings (generated by the agent) to the game actions defined in this section. This is the role of the game's interpreter.
TextWorld: A Learning Environment for Text-based Games
Environment states (S) Game states are defined in terms of logical predicates. Each predicate p(v 1 , . . . , v m ) consists of a symbol p drawn from the alphabet of predicates Σ followed by an mtuple of variables. These predicates define the relations between the entities (objects, player, room, etc.) present in the game. A logical atom is a predicate whose variables are all are bound, i.e., free variables (placeholders) have been substituted by concrete entities.
TextWorld: A Learning Environment for Text-based Games
A game state s ∈ S consists in a multiset of logical atoms representing the facts that are currently true about the world, also known as "resources" in linear logic. Taking as an example Figure 4, the state depicted there can be represented as
TextWorld: A Learning Environment for Text-based Games
where the symbol ⊗ is the linear logic multiplicative conjunction operator.
TextWorld: A Learning Environment for Text-based Games
The set of winning states G is composed of any state s for which all the winning conditions (a set of facts) hold. The winning conditions are determined during the game generation process (Section 3.2.2). For instance, a winning condition could be as simple as in(apple, I), i.e., the apple being in the player's inventory.
TextWorld: A Learning Environment for Text-based Games
The state transition function is defined using linear logic [Russell and Norvig, 2016, Ch. 8] and is inspired in part by Ceptre [Martens, 2015], a linear logic programming language. Logical rules are stored in a knowledge base and define what is possible in the game. For the working example, the relevant rules are
TextWorld: A Learning Environment for Text-based Games
The uppercase italic letters represent variables for objects of a certain type (F : food item, S: supporter, C: container and R: room). The entities P and I represent the player and its inventory. The symbol (lolli) is the linear implication operator. The interpretation of the linear implication is such that it consumes the resources on the left-hand-side (henceforth LHS) and generates resources on the right-hand-side (henceforth RHS). The notation $ is a shorthand meaning a predicate is implicitly carried over to the right-hand side. Given a state and the set of rules, we can perform forward chaining. Applying a rule to state s, whose LHS is satisfied by s, leads to a conclusion, which is a new collection of atoms generated by the RHS of the selected rule.
TextWorld: A Learning Environment for Text-based Games
Applying all possible rules for all conclusions leads to a proof-tree with triplets (s, a, s ), where s is an assumption and rule a leads to the conclusion s . Adding control to forward chaining, by only exploring paths where unseen states are introduced, leads to an algorithm that upon termination discovers all s ∈ S in the MDP. Merging the duplicate states in the proof-tree provides the underlying (S, A, T ) of the MDP.
TextWorld: A Learning Environment for Text-based Games
An action is one of the rules defined in the knowledge base for which all free variables have been "grounded", i.e., substituted by bound variables appropriately. Actions available in the current state, A st , can be obtained by performing a single step of forward-chaining given facts true in s t . In other words, the inference engine is used to retrieve all possible substitutions for every rule that can be applied to s t . In the initial state of the working example, the available actions are close(fridge) ::
TextWorld: A Learning Environment for Text-based Games
In the general case, games generated with TextWorld only provide a positive reward when reaching a winning state s ∈ G. The goal is to maximize the expected discounted sum of rewards received
TextWorld: A Learning Environment for Text-based Games
Tracking the state of the player allows us to determine a winning policy (not necessarily optimal) for any game state. A prerequisite is that a winning policy exists from the player's initial position (this is guaranteed for generated games). If so, then by monitoring state changes we can update the winning policy accordingly: if the agent performs the action dictated by the current winning policy, it progresses to the next desired state and we simply shift the policy forward one time-step; if the agent goes off the winning trajectory we add reciprocal actions to the policy to undo or correct this negative progress; and if the agent changes its state in a way that does not affect the quest, the winning policy does not change.
TextWorld: A Learning Environment for Text-based Games
In addition to the final reward, TextWorld can provide an intermediate reward which is tied to the winning policy. After each command, if the winning policy increases in length, meaning that as a result of the last action, additional commands are required to solve the game, then we assign a negative reward. If the winning policy shortens, meaning the last action brought the agent closer to the goal, we assign a positive reward. Otherwise, the reward is 0.
TextWorld: A Learning Environment for Text-based Games
TextWorld can be used as a sandbox environment in which predefined dynamics govern emergent games. With this sandbox, it is possible to generate a combinatorial (not to say infinite) set of games from which an agent could learn the underlying dynamics. Since we control the generation process, we can construct games where the knowledge to be learned is interpretable to humans.
TextWorld: A Learning Environment for Text-based Games
TextWorld's game generator takes as input a high-level specification of a game and outputs the corresponding executable game source code in the Inform 7 language (Appendix C). The game specification assigns values to parameters such as the number of rooms, the number of objects, the length of the quest, the winning conditions, and options for the text generation (e.g., theme, co-references, adjectives, and so on).
TextWorld: A Learning Environment for Text-based Games
TextWorld generates maps through a simple procedure based on the Random Walk algorithm [Pearson, 1905]. This enables us to generate a wide variety of room configurations, which in turn makes the set of possible games very large. The map generation process is parameterized by the number of rooms, the grid size of the world, and whether room connections should have doors or not. The grid size influences the compactness of the room configuration, where smaller grids mean more compact maps with potentially more loops.
TextWorld: A Learning Environment for Text-based Games
Once the map is generated, objects are added to the world uniformly across the rooms. Some objects are portable (as opposed to fixed in place), which means they can be nested on or in other objects that have the supporter or container attribute, or placed on the floor of a room or in the player's inventory.
TextWorld: A Learning Environment for Text-based Games
In TextWorld, the purpose of any game is to reach a winning state. The term quest will be used to represent a sequence of actions the player must perform to win the game. Note that this sequence does not have to be optimal or unique; many trajectories could lead to a winning state.
TextWorld: A Learning Environment for Text-based Games
We define quest generation as the process of determining interesting sequences of actions from which to derive winning conditions. As discussed in Section 3.1, the inference engine in TextWord can perform forward-chaining to construct a tree of all possible action sequences given an environment. However, not all paths are interesting (from a human or RL perspective). For this reason, we impose a dependency constraint on the actions and reject paths containing cycles. The dependency relation is defined as follows: action a t depends on action a t−1 if and only if the RHS of a t−1 generates the resource(s) required by the LHS of a t . The winning condition of a given quest is the set of resources generated by the RHS of the last action.
TextWorld: A Learning Environment for Text-based Games
We can generate quests by modifying the forward chaining algorithm to apply these constraints, calling the resulting process forward quest generation. An example quest is depicted in Figure 5. First, the player opens the door and moves south to enter the kitchen. When in the kitchen, the player opens the fridge, takes the apple, and finally eats it.
TextWorld: A Learning Environment for Text-based Games
The end goal often defines the nature of a quest and yields significant rewards for the agent. Thus, it is desirable to specify the end goal in quest generation. The forward quest generation algorithm indirectly allows this specification, by generating all possible quests from an initial condition then searching for those that satisfy the ending constraint. However, as the number of states and the length of the desired quest increases, this approach becomes intractable. To remedy this, TextWorld also supports backward chaining. Backward chaining simply reverses forward chaining, starting from a specified end state rather than an initial state. The same dependency between subsequent actions and cycle rejection apply. Step 3, and the apple placed within it. In the last step the fridge is closed. This becomes the game's starting state.
TextWorld: A Learning Environment for Text-based Games
Extending the World during Quest Generation TextWorld's generator extends forward and backward chaining with a fact creation step, which may occur before sampling a subsequent (or previous) action. Through fact creation, the generative process can add missing objects and facts to the world as needed, which yields more diverse quests and games. Figure 6 shows a simple example of backward quest generation.
TextWorld: A Learning Environment for Text-based Games
The Text Generation module takes logical elements of the game state and renders them into coherent text. This serves as the observation provided to the agent. The engine generates object names, room descriptions, and quest instructions in constrained natural language using a contextfree grammar (CFG) [Chomsky, 1956]. The ease of authoring such grammars has led to their adoption in various natural language generation (NLG) systems for games [Ryan et al., 2016].
TextWorld: A Learning Environment for Text-based Games
The module is essentially a set of grammars, each generating particular aspects: e.g., there are separate grammars for creating object names, room descriptions, and instructions, respectively. Observable text is generated by iterating over all elements of the observation and filling templates with the appropriate information. As an example for object description: For a red box, the grammar may return "There is a [object-noun] here. It is [object-adjective]"., which is filled to create "There is a box here. It is red." Some basic maintenance also ensures fluency, e.g., using "an" vs. "a" when a noun begins with a vowel.
TextWorld: A Learning Environment for Text-based Games
Using a context-free grammar gives a degree of textual variation, allowing the same world and quest to be represented a number of ways while also ensuring strict control over the results. While our current grammars function like a text templating system, CFGs offer the possibility of deeper, recursive text generation should it be desired. Our grammars can be extended and modified easily with additional production rules, enabling the generation of simpler or more complex sentence structures that may act as a level of game difficulty.
TextWorld: A Learning Environment for Text-based Games
Object Names Object names are assigned to each term in the game. Names are randomly picked from the CFG and uniquely assigned to objects. The object's type is used to derive the start symbol sent to query the CFG for a name. An object name can be decomposed into two parts: adjective and noun. The adjective is optional, and may be used to create more complex object names, and correspondingly more complex descriptions. Object name generation is in general straightforward, and consists of selecting a random adjective and noun and combining them. So, e.g., given the nouns "box" and "cup", as well as the adjectives "dusty" and "red", there are four possible object names, "dusty box", "red box", "dusty cup" and "red cup". Adjectives are also used as hints to the player; for example, a key's adjective will always match the adjective of what it opens, e.g., a "red key" will open the "red chest". 
TextWorld: A Learning Environment for Text-based Games
The description of a room is the concatenation of the room-level description of every object it contains, shown typically when entering the room or upon using the look command. The room-level description of an object contains information the player should be aware of upon entering the room (e.g., "There is a chest here. It is open and you can see some gold coins in it."). The room's description also mentions its possible exits (e.g., "There is a path leading north."). It is updated dynamically based on changes to the states of objects in the room, for example listing whether a container is open, closed, or locked, and which objects it contains.
TextWorld: A Learning Environment for Text-based Games
We use instructions to explain to the player what to do in a game. An instruction is a piece of text describing a particular action or several different actions. For example, "Retrieve the blue key" could be used to represent the action take blue key, whereas "Take the red key from the locked chest" may represent the sequence of actions unlock chest / open chest / take red key. In TextWorld, instructions may optionally describe every action of a quest (easier), only the final action (harder), or they may force the player to figure out what to do from scratch (goal identification; hardest). Likewise, the ability to combine actions into a single instruction can also be toggled; identifying a sequence of actions from an instruction rather than a single action is an additional challenge.
TextWorld: A Learning Environment for Text-based Games
Text Generation Options TextWorld offers some control over different aspects of the text generation. Objects with similar attributes/states can be grouped together when describing a room (e.g., "In here, you see two red containers: a box and a chest."). Objects mentioned in an instruction can be referred to using one or several of their attributes (e.g., "Take the red edible thing."). Use of coreference (e.g., "There is a chest. It is open. In it, you see nothing interesting.") is also optional.
TextWorld: A Learning Environment for Text-based Games
TextWorld also offers the choice between two themed grammars: house and basic. The house theme describes the world as if the game takes place in a modern house. The second theme uses a simple grammar with almost no linguistic variation (e.g., no adjectives, no multi-word names). In this case, objects with the same attributes use a shared, prototypical prefix for their names followed by a number (e.g., stand42). The basic grammar cuts down the vocabulary and the language complexity to ease the training of neural generative models. These house and basic themes can be seen applied to the same underlying game in Figure 7.  Figure 7 using TextWorld's integrated game-state viewer. The winning sequence of commands for this game: go south / go south / take tiny grape from chipped shelf / go west / put tiny grape on dusty bench.
TextWorld: A Learning Environment for Text-based Games
Most basically, TextWorld can be used to play any text-based games that are interpretable either by Z-machine (via the Frotz interpreter) or by Glulx (via a custom git-glulx interpreter). The framework handles launching and interacting with the necessary game processes. It provides a simple API for game interaction inspired by that of OpenAI's Gym [Brockman et al., 2016]. As an example, playing a text-based game requires just a few lines of Python code: import textworld env = textworld.start("zork1.z5") game_state = env.reset() # Reset/initialize the game. reward, done = 0, False while not done: # Ask the agent for a command. command = agent.act(game_state, reward, done) # Send the command to the game and get the new state. game_state, reward, done = env.step(command)
TextWorld: A Learning Environment for Text-based Games
The game_state object contains useful information 3 such as:
TextWorld: A Learning Environment for Text-based Games
Feedback The interpreter's response to the previous command, i.e., any text displayed on screen.
TextWorld: A Learning Environment for Text-based Games
Description The description of the current room, i.e., the output of the look command.
TextWorld: A Learning Environment for Text-based Games
Inventory The player's inventory, i.e., the output of the inventory command.
TextWorld: A Learning Environment for Text-based Games
The name of the current room.
TextWorld: A Learning Environment for Text-based Games
Score The current score.
TextWorld: A Learning Environment for Text-based Games
Game state information is fairly limited when it comes to existing games, whereas games generated with TextWorld can provide much more information if desired. Additional information includes:
TextWorld: A Learning Environment for Text-based Games
Objective Text describing what the player must do to win the game.
TextWorld: A Learning Environment for Text-based Games
Admissible Commands A list of commands that are guaranteed (i) to be understood by the interpreter and (ii) to affect the game state or to return information of the game state. Using this information in any way corresponds to playing a choice-based game rather than a parser-based one.
TextWorld: A Learning Environment for Text-based Games
Intermediate Reward A numerical value representing how useful the last command was for solving the game (as described in Section 3.1.1).
TextWorld: A Learning Environment for Text-based Games
Winning Policy A list of commands that guarantees winning the game starting from the current game state.
TextWorld: A Learning Environment for Text-based Games
They test their approach on 2 environments -Homeworld and Fantasyworld -using the Evennia toolkit 4 . Homeworld is a small environment with 4 rooms, a vocabulary of 84 words, and 4 quests. Quests are also specified through text; for example, "Not you are sleepy now but you are hungry now" (which indicates that the player should obtain food but should not get into bed). Fantasyworld is much larger, with a vocabulary size of 1340, and has stochastic state transitions. The LSTM-DQN completes 100% of the quests in Homeworld and 96% of quests in Fantasyworld. The authors also perform transfer-learning experiments by defining Homeworld2, which is made by shuffling the rooms and paths from Homeworld. The LSTM-DQN is trained on Homeworld and the LSTM component is transferred to Homeworld2. The transferred agent learns faster than one without training on Homeworld. 
TextWorld: A Learning Environment for Text-based Games
The DRRN model converges on both games when trained with the DQN algorithm with experience replay and Boltzmann exploration. It achieves optimal cumulative-reward on Saving John and a suboptimal but stable policy on Machine of Death. The authors test the DRRN trained on Machine of Death on state-action pairs paraphrased by participants. They show high correlation between the original and paraphrase Q(s, a, θ).
TextWorld: A Learning Environment for Text-based Games
Note that neither the LSTM-DQN nor the DRRN conditions on previous actions or observations. This means that neither has the capacity to deal with partial observability.
TextWorld: A Learning Environment for Text-based Games
Related work has been done to reduce the action space for parser-based games. Haroush et al. [2018] introduce the Action Elimination Network to estimate the probability of an action failing in a given scene. To achieve this, feedback from the game engine is also stored in the replay buffer in addition to the standard <observation, action, reward> triplets. The elimination module is trained with the stored quadruplets and assigns a score to each element in the large set of actions. During -greedy exploration, at the greedy step the agent is only allowed to consider the top-k actions, while during exploration, random actions are rejected with a predefined probability if their score is below a threshold. Fulda et al. [2017] tried to accomplish something similar by training word embeddings to be aware of verb-noun affordances. From that embedding, they manually select a group of verb-noun pairs for which they assume the vector emb(noun) − emb(verb) encodes the affordance relation. Using the average of such vectors gives them an "affordance vector" that can be used to project the embedding of new nouns to a region of the embedding space where relevant verbs should be found. Kostka et al. [2017] build an agent specifically targeting the domain of classic text-based games. They pre-train an LSTM language model on fantasy books that is used in their model to extract important keywords from scene descriptions. Furthermore, they collect a list of possible commands from game solutions and semi-automatically extract a large number of commands from online tutorials and decompiled game sources. Their system follows a modular design where each typical phase of text-adventure gameplay is modeled by a separate algorithm: command generation, battle mode, inventory management, exploration, restart. Their model generates commands by finding keywords in the scene text and cross-referencing the extracted command corpus to find plausible commands.
TextWorld: A Learning Environment for Text-based Games
In the fields of AI in general and RL in particular, games have played a major role in advancing the state of the art. The well-known Arcade Learning Environment (ALE) [Bellemare et al., 2013], which provides an interface to Atari 2600 games, led to human-level videogame play by deep RL algorithms [Mnih et al., 2015]. One limitation of ALE, however, is that it does not facilitate research in generalization, metalearning, and transfer learning because the individual games are too distinct from one other. Indeed, most research using ALE focuses on training a separate agent (with the same architecture) for each game [Machado et al., 2017].
TextWorld: A Learning Environment for Text-based Games
ALE-style game collections exist in contrast to sandbox environments, in which games emerge from a set of underlying world mechanics. Perhaps the best known such environment is SimCity [Wright, 1996]. Sandboxes generate a series of games that share components, and thereby naturally overcome some limitations of collections. Sandbox environments also allow for the programmatic construction of game variants whose difficulty can be tuned to form a curriculum.
TextWorld: A Learning Environment for Text-based Games
The MazeBase environment [Sukhbaatar et al., 2015], possibly the sandbox framework most similar to TextWorld, enables researchers to generate two-dimensional grid-based games. Each grid point may contain a certain object type, such as an obstacle, a pushable block, a switch, or the goal. Maps are provided in an egocentric text-based representation to the player. One of the main motivations of MazeBase is to foster research in learning algorithms that reuse knowledge and policies acquired in related environments and games. Quests and the language in TextWorld are significantly more complex than in MazeBase.
TextWorld: A Learning Environment for Text-based Games
The CommAI framework [Baroni et al., 2017] emphasizes the ability to generate curricula [Bengio et al., 2009], so that agents may learn incrementally from environments of increasing complexity. Interaction with the environment takes place at the lower level of bits rather than simplified natural language as in TextWorld.
TextWorld: A Learning Environment for Text-based Games
Recently, multimodal visuo-linguistic environments were introduced to study grounded language learning through RL. Chaplot et al. [2017] customized the VizDoom platform [Kempka et al., 2016] for language grounding experiments: objects with certain properties are placed in a room and the agent is instrcuted which object(s) to find. To perform similar experiments, Hermann et al. [2017] add a language instruction component to the DeepMind Lab 3D navigation environment [Beattie et al., 2016].
TextWorld: A Learning Environment for Text-based Games
In this section, we describe benchmark games that can be used through TextWorld to evaluate RL agents and algorithms. This set is preliminary; we plan to develop increasingly complex benchmarks in future work.
TextWorld: A Learning Environment for Text-based Games
Following Fulda et al. [2017], we compiled a list of 50 hand-authored text games to use as an evaluation set. Games designed for human players require, and can be used to measure, general capabilities like common-sense reasoning, planning, and language understanding. We manually analyzed all games in the set to ensure they are valid, with scores and interesting quests. From the original 50 proposed by Fulda et al. [2017], we replaced 20 which were either parodies of text-based games or did not provide scores. The information we collected for the games can be found in Appendix B.
TextWorld: A Learning Environment for Text-based Games
We evaluate three baselines on the proposed list. BYU 5 and Golovin 6 are agents developed by Fulda et al. [2017] and Kostka et al. [2017], respectively, and are described in Section 4.1. Both models were submitted to the Text-Based Adventure AI Competition [Atkinson et al., 2018] which consisted in playing 20 hidden games. The third is the Simple baseline, which consists in sampling uniformly a command from a predefined set 7 at every step.
TextWorld: A Learning Environment for Text-based Games
Each agent has 1000 steps to get a high score. If the agent loses the game, the game is reset and play resumes until the step limit is reached. If the agent wins (which never happens in practice), the game stops and the evaluation is done. The procedure proposed here is slightly different from that in Fulda et al. [2017], where they allow agents to play each game 1000 times for a maximum of 1000 steps each (i.e., a theoretical limit of a million interactions). Our main motivation for having a small "time" budget is that we are interested in measuring the generalization capabilities of agents. When using this evaluation, we assume the agents are already trained (on other similar games) or encompass some prior knowledge (which is the case for the three baselines). Figure 9 reports the normalized score (i.e., maximum score achieved divided by the game's max possible score) for each baseline agent on the curated list. Unsurprisingly, agents achieve a rather low score on a few games and zero on many. Using the information we gathered during our analysis, we can make educated guesses as to why some baselines perform well on certain games. For instance, in Advent the player starts with 36 points, which explains why all three baselines have the same score. As another example, the Detective game can be solved with mostly navigational commands. This explains why the Simple agent performs relatively well, since the commands it samples from are mostly navigational.
TextWorld: A Learning Environment for Text-based Games
This benchmark is inspired by the task proposed in Parisotto and Salakhutdinov [2017], where the agent spawns in a randomly generated maze and must find a specific object. A colored "indicator" object near the agent's starting position determines which object the agent must retrieve. The agent earns a positive reward for retrieving the correct object or a negative reward for an incorrect object. There is a limited number of turns.
TextWorld: A Learning Environment for Text-based Games
We adapted this task, which takes place in a 3D environment, to TextWorld. In our setting, the maze is a randomly generated map (see Section 3.2.1) of rooms. We randomly place the agent and two objects on the map. Then, we randomly select which object the agent should recover and mention it in the welcome message (our indicator). In navigating to and obtaining the desired object, the agent may have to complete other tasks like finding keys and unlocking doors.
TextWorld: A Learning Environment for Text-based Games
The aim of this benchmark is to assess skills of affordance extraction (agents should determine verbnoun pairs that change the environment state); efficient navigation (agents should avoid revisiting irrelevant rooms); and memory(agents should remember which object to retrieve).
TextWorld: A Learning Environment for Text-based Games
We define the difficulty levels for this benchmark as follows:
TextWorld: A Learning Environment for Text-based Games
• 1 to 10: Mode: easy, #rooms = 5, quest length linearly increasing from 1 to 5;
TextWorld: A Learning Environment for Text-based Games
• 11 to 20: Mode: medium, #rooms = 10, quest length linearly increasing from 2 to 10;
TextWorld: A Learning Environment for Text-based Games
• 21 to 30: Mode: hard, #rooms = 20, quest length linearly increasing from 3 to 20;
TextWorld: A Learning Environment for Text-based Games
where the modes are defined as
TextWorld: A Learning Environment for Text-based Games
• Easy: Rooms are all empty except where the two objects are placed. Also, connections between rooms have no door;
TextWorld: A Learning Environment for Text-based Games
• Medium: Rooms may be connected by closed doors. Container objects are added, and might need to be opened to find the object;
TextWorld: A Learning Environment for Text-based Games
• Hard: Locked doors and containers are added which may need to be unlocked (and opened) to reach the object. Note that beyond the predefined difficulty levels, this benchmark can be simplified by letting the agent directly tap into the game state information (e.g., feedback, description, inventory and objective) or using a simpler grammar.
TextWorld: A Learning Environment for Text-based Games
One of our desiderata in building TextWorld is the ability to generate unseen games, so that we can train and evaluate an agent's performance on different game sets. To support our claim of this need, we test two state-of-the-art agents on a set of games generated by TextWorld, where the agents only see each game once and therefore cannot memorize them.
TextWorld: A Learning Environment for Text-based Games
Specifically, for each difficulty level described above, we generate 100 games. We run each agent on these games for a maximum of 1000 steps. For each game, when the agent picks up either the right object or the wrong one, it receives +1 or -1 score, respectively, and the game terminates immediately; if the agent exhausts all 1000 steps without finding any object, it receives 0 score.
TextWorld: A Learning Environment for Text-based Games
Evaluation results are provided in Table 1, where we compare a choice-based 8 random agent (i.e., at each game-step the baseline model randomly selects one command from the list of admissible commands), the BYU agent and the Golovin agent. We report the average score for different difficulty levels and the average number of steps it took to finish the games (either win, lose, or exhaust).
TextWorld: A Learning Environment for Text-based Games
Complex Quest Generation We define a quest as a sequence of actions where each action depends of the outcomes of its predecessor (Section 3.2.2). This limits us to simple quests where each action is based on only the immediately previous action. Quests are rarely so straightforward in text adventure games, often being composed of multiple sub-quests. One method for generating more complex quests would be by treating a quest as a directed graph of dependent actions rather than a linear chain.
TextWorld: A Learning Environment for Text-based Games
In the current implementation, there is no support for triggering state changes without corresponding user actions. This would enable doors that automatically lock after a certain number of player steps or traps that trigger periodically.
TextWorld: A Learning Environment for Text-based Games
NPCs are characters that are not controlled by the player Figure 10: A generated Treasure Hunter game with difficulty 20. The green rectangle identifies the quest object, a passkey, which can be found on a rack in the washroom. The player, P, is currently two rooms away in the basement separated by two closed doors.
TextWorld: A Learning Environment for Text-based Games
and are capable of performing autonomous or reactive actions that alter the game state. NPC interaction is quite common in several text-based games, but is not currently supported by the game generator.
TextWorld: A Learning Environment for Text-based Games
Some text-based games allow multiple users to interact with the world at the same time. Users may need to cooperate with each other or to compete against each other, depending on the game. The framework currently supports only single-agent games.
TextWorld: A Learning Environment for Text-based Games
Text Generation Using a CFG for text generation makes it difficult to ensure continuity between sentences.
TextWorld: A Learning Environment for Text-based Games
We introduced TextWorld, a sandbox learning environment for the training and evaluation of RL agents on text-based games. After surveying the machine-learning challenges of text-based games, we cast them in the formalism of reinforcement learning. We described how the generative mechanisms of TextWorld can be used to work up towards the full complexity of hand-authored games and introduced a preliminary set of benchmark games for this purpose. We evaluated several baseline agents on this set and a curated list of hand-authored games that we analyzed.
TextWorld: A Learning Environment for Text-based Games
In future work, we will develop more complex benchmark tasks and investigate novel approaches to representation learning, language understanding and generation, and RL for text-based games.
TextWorld: A Learning Environment for Text-based Games
We thank Adam Ferguson and Marion Zepf for many helpful discussions on game analysis. We thank Alessandro Sordoni, Greg Yang, Philip Bachman, Ricky Loynd, Samira Ebrahimi Kahou, and Yoshua Bengio for helpful suggestions and comments on research perspectives.
TextWorld: A Learning Environment for Text-based Games
Language Language presents a number of difficulties Parser complexity (what a parser can and cannot handle), distracting in-game descriptions, sarcasm (e.g., If the game responds "Yeah right, that's totally going to work" when you enter a command, a solid understanding of sarcasm would prevent you from repeating the command), and fictional languages.
TextWorld: A Learning Environment for Text-based Games
Maze One of the oldest text adventure obstacles, the most basic maze is a grid of rooms with similar names and/or descriptions. The player has to find the right path through the maze to reach a goal. Many text-based games take care to innovate/complicate mazes. Some innovations include:
TextWorld: A Learning Environment for Text-based Games
• Mazes whose solution is determined stochastically, either on a play-through by playthrough basis or on a turn by turn basis.
TextWorld: A Learning Environment for Text-based Games
• Mazes whose configuration changes due to a player action, as in Zork III [Infocom, 1982], where the player can move some walls in the maze in order to solve it.
TextWorld: A Learning Environment for Text-based Games
• Mazes that are not solvable by navigation: In Photopia, for example, the player is wearing a spacesuit while navigating the maze. If the player removes the suit, it is revealed that the player has a pair of wings. The player is then able to fly over the maze.
TextWorld: A Learning Environment for Text-based Games
Clue Hunting Detective games require the player to hunt for clues that will tell them where to go next. In Sherlock, the game begins when the player receives a coded poem that directs them to Westminster Abbey, where they have to solve a number of riddle puzzles in order to find out where to find five different gems, each of which provides the player with further clues. A point of interest here is whether or not the player actually needs to get the hints in order to win the game, or whether the player could win the game if they already knew the contents of the hints. Sherlock [Bates, 1987] is an example of the former -some hints trigger crucial game state changes only once the player examines them. Other games mix state-changing hints with purely informative ones. For example, in Inhumane [Plotkin, 1985], a clue gives you directions through a maze which is still solvable if you don't find the clue. However, the end of the maze yields another clue. When you read this second clue, an important portion of the map becomes available to the player.
TextWorld: A Learning Environment for Text-based Games
Treasure Hunting Some treasures double as clues (as is the case with the gems in Sherlock) while other treasures are necessary in order to unlock further treasures. In Infidel, for example, you are given the opportunity to take some gem clusters early in the game. Towards the end of the game, these clusters are used to unlock a treasure chest. [Berlyn, 1983] Trivia Many games base their puzzles off of trivia/world knowledge, which the player might need to know in order to solve a puzzle. Many puzzles in the Hitchhiker's Guide to the Galaxy game are easier to solve if you are familiar with the source material. For example, if you've read the book, you'd already know that a Babel Fish is able to decode any language in the universe. In the game, then, you'll know what to look for when confronted with an alien language [Adams and Meretzky, 1984]. In The Enterprise Incidents, the player must solve a word puzzle whose solution, the word firefly, is not mentioned elsewhere in the game. [Desilets, 2002] In Goldilocks is a FOX, knowledge of fairy tales is required to solve puzzles involving magic beans, bears, and porridge. [Guest, 2002] Sherlock contains intricate riddle poems whose solutions are allusions to famous historical figures buried in Westminster Abbey. The player is also required decode a riddle poem and realize that references to "the conquest" and "the fire" refer to the Battle of Hastings and the Great Fire of London, respectively. Finally, the player is expected to know the dates for these two dates in order to subtract one from the other in order to solve a puzzle. [Bates, 1987] A number of modern textbased games (Curses, OMNIQUEST, All Quiet on the Library Front, Inhumane, and the later Zork games, to name a few) play off of established text adventure tropes, clichés, and catchphrases (common passwords across games include "XYZZY" and "Plugh" from Cave Adventure [Crowther and Woods, 1976], and magic spells learned in the Enchanter series carry over from game to game, although they have to be 'learned' in-game in order to be used).
TextWorld: A Learning Environment for Text-based Games
Hunger, thirst, fatigue are the most common survival elements in text adventures. Players might have to keep space open in their inventory for food and drink, and will be prompted from time to time that they are getting hungry or thirsty, or that they might need to rest. Similar obstacles that pop up are: hypothermia (when swimming in Zork III [Infocom, 1982]), the classic lamp from Zork [Infocom, 1980] (which reappears and is innovated on in the sequels, where it can go out if you neglect to shake it or turn it back on), and torches from Infidel (which you are required to regularly dip in oil and light) [Berlyn, 1983]. One complication of self-maintenance is whether the action is one time or continuous (in Infidel, for example, you need to gather food and drink, but once they're gathered, you have an unlimited amount).
TextWorld: A Learning Environment for Text-based Games
Combat Combat can come in a variety of forms. At its most basic level, a player will be confronted with an enemy and can thwart it by typing kill enemy. To complicate things further, the player might not be equipped to kill the enemy with their bare hands, and must first acquire a weapon before kill enemy will return "enemy killed". One level higher, the player might have to attack enemy and dodge enemy or aim at X (where X = a weak spot you learn about elsewhere in the game) before killing the enemy. Higher levels include stochastic combat (where attacks may or may not land), optional combat (where the only solution to the fight is to avoid it) and puzzle combat. In Reverberations, for example, the player encounters an enemy in the cosmetics section of a department store. To win the fight, the player must spray the enemy with some nearby perfume. [Glasser, 1996] Time In Wishbringer, the player has a limited amount of time at the start of the game to deliver a note. The player must use this time to collect items and information that will be inaccessible once they've delivered the note. [Moriarty, 1985] In Sherlock, the player is given 48 hours of in-game time to solve the mystery. Certain locations and events are only accessible at certain times. [Bates, 1987] Mechanical Puzzle Some examples can be found in Infidel and Inhumane, where the player has to move objects around a room, leave objects in rooms, and prop open doors with wooden beams in order to avoid being caught in traps. [Berlyn, 1983] [Plotkin, 1985] Persistence In some games, Anchorhead for example, you must repeat talk to commands in order to get more information out of an NPC. [Gentry, 1998] Stochasticity Some examples of stochasticity include randomly generated maze configurations, randomly decided combat action outcomes, and randomly selected code-words for puzzles. Text-based game players deal with stochasticity with the UNDO action or by saving often and especially right before taking any kind of risky action.
TextWorld: A Learning Environment for Text-based Games
The Cruelty Scale was designed by text-based game author Andrew Plotkin as a broad method of determining the difficulty of text-based games. The main element considered by the scale is whether or not the game can be made impossible to win, and how obvious (or not) this is made to the player. The scale has five ranks: Merciful games are impossible to die or get stuck in, Polite games warn you before you do something that can kill or trap you or otherwise render the game unwinnable. Tough games can kill the player or be rendered unwinnable, but they'll warn the player before this happens. Nasty games can be rendered unwinnable without warning, but there will be an indication that the game is now unwinnable after the fact. Cruel games can be rendered unwinnable, and there's no warning beforehand and no indication afterwards.
TextWorld: A Learning Environment for Text-based Games
Length Text-based games vary wildly in length, from the solvable-in-twenty-moves inform demo Acorncourt to the 1000+ moves required to complete Anchorhead, text-based games have many elements that impact their length. These include: Number of possible commands, number of rooms, length of descriptive text, (Reverberations, for example, is solvable with a relatively low number of commands (roughly 50), but features long descriptive passages for almost every room and command) side quests and multiple endings, and game difficulty.
TextWorld: A Learning Environment for Text-based Games
However, text-based game review sites and discussion boards often divide games up into short demo type games, medium uncomplicated games, and long difficult games. 
TextWorld: A Learning Environment for Text-based Games
Below are observations for some of the games we analyzed.
TextWorld: A Learning Environment for Text-based Games
The Acorn Court Written as an Inform demo. Very short game with only one room. Contains one multi-part spatial puzzle.
TextWorld: A Learning Environment for Text-based Games
Adventure First IF. Difficult, stochastically generated mazes, introduces made up words. We used the recompiled with Inform 6 (release 9) version which starts with an initial score of 36 points.
TextWorld: A Learning Environment for Text-based Games
Anchorhead Heavily text based. Requires a combination of world knowledge and knowledge gleaned from in-game texts. When speaking with townspeople, you have to occasionally be persistent. Awareness of Lovecraftian cliches is helpful.
TextWorld: A Learning Environment for Text-based Games
Balances Another Inform demo, features made up words and spells that the player must remember. Rare words used in the context of magic and fantasy are also used.
TextWorld: A Learning Environment for Text-based Games
Deephome Medium length game with an in game library the player can consult in order to figure out the solutions to puzzles.
TextWorld: A Learning Environment for Text-based Games
Dragon Adventure Short game written to introduce children to text based games.
TextWorld: A Learning Environment for Text-based Games
Detective Poorly written game. Navigation isn't very logical (you can walk east into a room whose only exit is to the north). Gameplay is very simple -you only need to use navigation commands to reach the end of the game.
TextWorld: A Learning Environment for Text-based Games
Enchanter Long game with a complex spell-casting system that requires player to memorize and record spells.
TextWorld: A Learning Environment for Text-based Games
Goldilocks is a FOX! Requires trivia knowledge (you need to know about Jack and the Beanstalk in order to know what to do when someone offers you magic beans). Casual language with lots of pop cultural references.
TextWorld: A Learning Environment for Text-based Games
Hitchhiker's Guide to the Galaxy Incredibly difficult to play. Several stochastic obstacles (A maze without a set solution, An object from a subset of objects is randomly made into a crucial object). Features many irrational objects and events, including objects showing up in your inventory without warning. Leans on world knowledge a bit as well, as when the navigation controls change to Port, Starboard, Aft, and Bow/Fore. Some puzzles are solvable if you are familiar with the books. Language is also sarcastic and, at times, deliberately misleads the player.
TextWorld: A Learning Environment for Text-based Games
Infidel Many spatial puzzles. Draws on world knowledge (player must decode hieroglyphics and pictograms). Instruction manual contains the navigation instructions for finding the pyramid. Contains some potential one-way paths (you need to tie a rope to an object in one room in order to re-enter it).
TextWorld: A Learning Environment for Text-based Games
Inhumane Parody of infidel but with a more straightforward, simpler map. Has a meta game where you try to nearly solve puzzles and then kill yourself. Some familiarity with Infidel makes playing it easier. Contains some clue-hunting that doubles as room unlocking, you can also find directions through a maze in the game. Contains a gag ending triggered by entering s instead of south or vice versa as your final command. There are mazes, but there are also some unconnected rooms that have identical names (T-Intersections appear in different areas).
TextWorld: A Learning Environment for Text-based Games
All Quiet on the Library Front Relatively simple treasure hunt with minimal side-quests and Easter eggs. Contains some text-based game in-jokes that do not impact the gameplay, but might be distracting.
TextWorld: A Learning Environment for Text-based Games
Lost pig Game text is written in "cave-person speak". Otherwise a simple pig hunt with some magic spell based puzzles.
Learning to Speak and Act in a Fantasy Text Adventure Game
There has been remarkable progress in language modeling (Jozefowicz et al., 2016;Devlin et al., 2018;Radford et al., 2019) and building dialogue agents (Dinan et al., 2019a). Nevertheless, the current state of the art uses only the statistical regularities of language data, without explicit understanding of the world that the language describes. This work is built on the hypothesis that dialogue agents embodied in a rich and cohesive (but tractable) world can more easily be trained to use language effectively than those only exposed to standard large-scale text-only corpora.
Learning to Speak and Act in a Fantasy Text Adventure Game
To that end, we introduce the LIGHT 1 research platform. LIGHT is a multi-player fantasy text adventure world designed for studying situated dialogue, and allows interactions between humans, 1 Learning in Interactive Games with Humans and Text. models as situated agents, and the world itself. It consists of a large crowdsourced game world (663 locations, 3462 objects and 1755 characters) described entirely in natural language. Within that game world, we collect a large set (11k episodes) of character-driven human-human crowdworker interactions involving actions, emotes, and dialogue, with the aim of training models to engage humans in a similar fashion. Our complete framework is made publicly available in ParlAI (http: //parl.ai/projects/light).
Learning to Speak and Act in a Fantasy Text Adventure Game
We use the collected dataset to investigate how a model can both speak and act grounded in perception of its environment and dialogue from other speakers. This is done by evaluating state-of-theart models on our task and evaluating the effects of providing additional grounding. In particular, we adapt the BERT contextual language model (Devlin et al., 2018) to the task of dialogue in two ways: as a bi-ranker, which is fast and practical as a retrieval model, and as a cross-ranker which is slower at inference time but allows more feature cross-correlation between context and response. Both models outperform existing methods. Our ablation analysis shows the importance of each part of the grounding (location, objects, characters, other's actions, self-actions) in terms of the ability to both understand and use language. While models that use grounding show clear improvements, our best performing models are still unable to perform at human level, making our setup a suitable challenge for future research.
Learning to Speak and Act in a Fantasy Text Adventure Game
LIGHT is a large-scale, configurable text adventure environment for research on learning grounded language and actions. It features both humans and models as agents situated (symbolically) within a multi-player fantasy MUD (multiuser dungeon)-like (Dieterle, 2009) environment. The environment is moderated by a simple game engine which passes dialogue and emote turns between characters and allows actions to cause transitions of the world state. It is in this unimodal (text-only), environment that we consider the agents to be situated.
Learning to Speak and Act in a Fantasy Text Adventure Game
To facilitate natural human-sourced (fantasy) situations described by natural language, almost the entire environment is crowdsourced, including locations, objects and their affordances, characters and their personalities, and most importantly character interactions: dialogues and actions. These components are collected through a series of annotation tasks that we will now describe. These tasks are designed so that they can be combinatorially recombined. Data quality was maintained by requiring annotators to take a test (see Appendix D). Overall statistics of the collected elements are given in Table 1. This environment can then be used to both train agents, and to evaluate them in situ via their online interactions.
Learning to Speak and Act in a Fantasy Text Adventure Game
Locations We first crowdsourced a set of 663 game location settings from a base set of 37 categories (countryside, forest, inside/outside castle, shore, graveyard, bazaar, . . . -full list in Appendix H) which were selected by us to pro- vide both inspiration and cohesion to annotators. Workers were provided a category and asked to create a description, backstory, names of connected locations, and contained objects and characters. See Table 2a for an example. Many descriptions are quite detailed, and there are clear semantics between entities (e.g. alligators being in swamps, cacti in a desert). As all remaining tasks build upon the locations created in this first step, we selected 6 location categories (underwater aquapolis, frozen tundra, supernatural, magical realm, city in the clouds, and netherworld) designed to be distinct from the others to provide an isolated set of locations, characters, and objects for testing. These will be used to build what we refer to as an unseen test set.
Learning to Speak and Act in a Fantasy Text Adventure Game
Each location is collected independently, with the eventual aim that they can be glued together as desired to randomize world generation. In this work, we consider actions and dialogues within a single location, so building a world map is not necessary. However, we will show that the environment has considerable influence on the dialogue, actions and grounded learning of models.
Learning to Speak and Act in a Fantasy Text Adventure Game
Objects We crowdsourced 3462 objects, each with a textual description, and a set of affordances (whether it is a container, can be picked up, has a surface, is a weapon, is wearable, is food, is a drink). See Table 2c for examples. As before, we sourced this list of objects to annotate from the ones annotated for the locations and characters.
Learning to Speak and Act in a Fantasy Text Adventure Game
Characters We crowdsourced 1755 game characters from animals to trolls and orcs to humans of various types (wizards, knights, village clerk). See Table 2b for detailed examples. Each character has a textual description, a persona (defined as a set of 3-5 profile sentences describing their traits, mod-eled after the Persona-Chat dataset (Zhang et al., 2018)), and a set of objects that are currently being carried, wielded, or worn. We sourced this list of characters to annotate from the ones provided in the location creation task.
Learning to Speak and Act in a Fantasy Text Adventure Game
There are a set of actions in the game consisting of physical manipulations, and a set of emotes that display feelings to other characters, in line with existing MUDs.
Learning to Speak and Act in a Fantasy Text Adventure Game
Physical actions include get, drop, put, give, steal, wear, remove, eat, drink, hug and hit, each taking either one or two arguments, e.g. put robes in closet. Every action has an explicit unambiguous effect on the underlying game state, and can only be executed if constraints are met, e.g. if the agent is holding the robes in the latter example.
Learning to Speak and Act in a Fantasy Text Adventure Game
These constraints are what indirectly provide an agent with object affordances, as the list of possible actions provides all ways the agent can interact with their environment (Gibson, 1977).
Learning to Speak and Act in a Fantasy Text Adventure Game
Emotes include applaud, blush, cringe, cry, dance, frown . . . , sulk, wave, wink (22 in total) and have no effect on the game state other than to notify nearby characters of the emote, which can have effects on their behavior. See Appendix E for further detailed descriptions.
Learning to Speak and Act in a Fantasy Text Adventure Game
Interaction Now that we have a fully realized underlying environment, we can attempt to learn and evaluate agents that can act and speak within it. For this, we collect a human-human dataset of episodic interactions within the environment.
Learning to Speak and Act in a Fantasy Text Adventure Game
For each dialogue, we place two characters in a random location (either two characters that were already assigned to it, or else randomly assigned characters), complete with the objects assigned to the location and to those characters. Each character has access to their persona, the location description, and the objects present, and the interaction episode begins. The two characters take turns within the episode, and can execute one action (physical action or emote) and produce one dialogue utterance on each turn. We crowdsourced 10,777 dialogues. Examples are given in Figure 1 and Appendix Figures 10-16.
Learning to Speak and Act in a Fantasy Text Adventure Game
We provide two distinct test sets. The seen test set consists of dialogues set in the same world (set of locations) as the training set, thus also consists of characters, objects, and personas that can appear in the training data. In contrast, the unseen test set is com-
Learning to Speak and Act in a Fantasy Text Adventure Game
Description: Two-and-a-half walls of the finest, whitest stone stand here, weathered by the passing of countless seasons. There is no roof, nor sign that there ever was one. All indications are that the work was abruptly abandoned. There is no door, nor markings on the walls. Nor is there any indication that any coffin has lain here... yet.
Learning to Speak and Act in a Fantasy Text Adventure Game
Bright white stone was all the fad for funerary architecture, once upon a time. It's difficult to understand why someone would abandon such a large and expensive undertaking. If they didn't have the money to finish it, they could have sold the stone, surely -or the mausoleum itself. Maybe they just haven't needed it yet? A bit odd, though, given how old it is. Maybe the gravedigger remembers... if he's sober.
Learning to Speak and Act in a Fantasy Text Adventure Game
Dead Tree, south, following a dirt trail behind the mausoleum Fresh Grave, west, walking carefully between fallen headstones
Learning to Speak and Act in a Fantasy Text Adventure Game
Characters: gravedigger, thief, peasant, mouse, bat
Learning to Speak and Act in a Fantasy Text Adventure Game
Objects: wall, carving, leaf, dirt (a) Example room created from the room collection and labelling tasks.
Learning to Speak and Act in a Fantasy Text Adventure Game
Persona: I live alone in a tent in the woods. I steal food I am low paid labor in this town. I do a job that many from the townspeople and coal from the people shun because of my contact with death. blacksmith. The village police can not find me I am very lonely and wish I had someone to put me in jail.
Learning to Speak and Act in a Fantasy Text Adventure Game
to talk to who isn't dead.
Learning to Speak and Act in a Fantasy Text Adventure Game
Description: The thief is a sneaky fellow who takes from the You might want to talk to the gravedigger, specially people and does so in a way that disturbs the if your looking for a friend, he might be odd but you livelihood of the others.
Learning to Speak and Act in a Fantasy Text Adventure Game
will find a friend in him.
Learning to Speak and Act in a Fantasy Text Adventure Game
Carrying: meat, potatoes, coal shovel
Learning to Speak and Act in a Fantasy Text Adventure Game
Wearing: dark tunic, cloak nothing annotated
Learning to Speak and Act in a Fantasy Text Adventure Game
Wielding: knife nothing annotated (b) Example characters annotated via character collection tasks.
Learning to Speak and Act in a Fantasy Text Adventure Game
shovel The shovel is made of metal and silver. It is quite sturdy and appears new. gettable, wieldable wall The wall is pure white, the richest of which you have ever seen. none (c) Example objects annotated via object collection tasks prised of dialogues collected on the unseen set of locations (described in 3). The unseen test set allows for evaluation of generalization capability to unseen topics in a similar domain and as we shall see, provides a more challenging test for current techniques.
Learning to Speak and Act in a Fantasy Text Adventure Game
We consider a variety of models that can predict actions, emotes and dialogue, and explore the importance of grounding upon the location, objects, and other characters within the setting. For all models, we represent context as a large text sequence with a special token preceding each input type (persona, setting, self emote, partner emote, etc.). We work with two model classes: ranking models that output the maximal scoring response from a set of potential candidate responses and generative models that decode word by word.
Learning to Speak and Act in a Fantasy Text Adventure Game
Ranking Candidates Each of the three tasks has a different method for determining candidates. Dialogue candidates are the ground truth and 19 randomly chosen candidates. Action candidates are usually the list of all possible actions, however in a no-affordance ablation we provide all well-formed actions over the current environment (which may include things that can't be executed like "wear paint can"). Emote candidates are the 22 possible emotes.
Learning to Speak and Act in a Fantasy Text Adventure Game
We report a Random baseline (selecting a random candidate from the candidates) and an Information Retrieval (IR) baseline that uses word overlap with TF/IDF weighting. We use Starspace (Wu et al., 2018a) which learns a bag-of-words embedding for context and candidates to maximize the inner product of the true label using a ranking loss. Lastly, we use fastText  to classify which emote should be predicted next as there are only 22 classes. Finally, we compare the performance of our best models to human performance on each of the prediction tasks.
Learning to Speak and Act in a Fantasy Text Adventure Game
Transformer Memory Network We use the transformer memory-based ranking model from Dinan et al. (2019b). It uses a transformer (Vaswani et al., 2017) to produce separate representations (memory slots) for each sentence from the grounding information (setting, persona, objects). It then performs attention given the dialogue context over the memories to produce a context embedding, which is used to score candidates via the dot product with the transformerbased representation of the candidate. At training time, other samples in the batch are used as negative candidates. For emote prediction, we train by ranking against the full set of possible emotes as there are only 22 distinct classes.
Learning to Speak and Act in a Fantasy Text Adventure Game
BERT Bi-Ranker and Cross-Ranker We adapt the BERT pretrained language model (Devlin et al., 2018) to the tasks of dialogue and action prediction. We explore two architectures for leveraging BERT. First, we use the BERT-based Bi-Ranker to produce a vector representation for the context and a separate representation for each can-didate utterance. This representation is obtained by passing the first output of BERT's 12 layers through an additional linear layer, resulting in an embedding of dimension 768. It then scores candidates via the dot product between these embeddings and is trained using a ranking loss.
Learning to Speak and Act in a Fantasy Text Adventure Game
Second, the BERT-based Cross-Ranker instead concatenates the context with each candidate utterance, similar to Wolf et al. (2019). Then, each candidate is scored by computing a softmax over all candidates. Unlike the BERT-based Bi-Ranker, the concatenation of the context with each individual candidate allows the model to attend to the context when encoding each candidate, building a context-dependent representation of each candidate. In contrast, the Bi-Ranker can use self-attention to build the candidate and context representations, but cannot modify their representation based upon the context. However, the Cross-Encoder is far more computationally expensive (∼11,000 slower than the Bi-Ranker for dialogue retrieval) as each concatenated representation must be recomputed, while the Bi-Ranker can cache the candidates for reuse (see Appendix B).  Generative Models Similarly to the ranking setting, we use the Transformer Memory Network from Dinan et al. (2019b) to encode the context features (such as dialogue, persona, and setting). However, to predict an action, emote, or dialogue sequence, we use a Transformer architecture to decode while attending to the encoder output. For the task of action generation, the set of candidates for ranking models to rank the true action sequence against is constrained by the set of valid actions. For example, the character cannot pick up book if there is no book. In the generative model, we compute the log likelihood for the set of possible candidates and normalize to constrain the output space to valid actions to improve the results.
Learning to Speak and Act in a Fantasy Text Adventure Game
We implement models using PyTorch in ParlAI . Ranking Transformer models are pretrained on Reddit data (Mazaré et al., 2018) and fine-tuned. We use the BERT (Devlin et al., 2018) implementation provided by Hugging Face 2 with pre-trained weights, then adapted to our Bi-Ranker and Cross-Ranker setups. Generative models are pretrained on the Toronto Books Corpus and fine-tuned except for emote prediction which does not leverage pretraining. We apply byte-pair encoding (Sennrich et al., 2016) to reduce the vocabulary size for generative models. We decode using beam search with beam size 5. 2 https://github.com/huggingface/pytorch-pretrained-BERT
Learning to Speak and Act in a Fantasy Text Adventure Game
Automatic To evaluate our models, we calculate percentage accuracy for action and emote prediction. For dialogue, we report Recall@1/20 for ranking the ground truth among 19 other randomly chosen candidates for ranking models and perplexity and unigram F1 for generative models.
Learning to Speak and Act in a Fantasy Text Adventure Game
Human We present humans with the same ranking task and report R@1/20 to estimate their performance on this task. We report the human accuracy and one standard deviation of error.
Learning to Speak and Act in a Fantasy Text Adventure Game
Quality control was particularly difficult for human evaluations, as the task of ranking one of 20 candidates is fairly tedious and easy to fake. In order to mitigate these problems, during the evaluation we provide annotated examples on the training in addition to examples on the test set. We only keep the annotations of evaluators who had high accuracy on the training examples to filter low-accuracy evaluators. The training accuracy bar was selected due to the difficulty of the separate tasks as evaluated by our own success rates. Our methods for human evaluation are described in more detail in Appendix F along with how many turns were evaluated.
Learning to Speak and Act in a Fantasy Text Adventure Game
The ranking models are compared in Table 4 on the seen and unseen test sets, and ablations are shown for both the BERT-based Bi-Ranker and    Generative Transformer in Tables 5 and 6.
Learning to Speak and Act in a Fantasy Text Adventure Game
The IR baseline shows non-random performance, but is outperformed by Starspace which is a stronger baseline. We also tried FastText on the emote tasks as classification is appropriate over the 22 possible emotes, and it did better than Starspace. Transformer architectures prove significantly stronger at all tasks, with BERT pretraining proving important for best results as used in the Bi-Ranker and Cross-Ranker architectures. The latter, which can create a context dependent representation of each label candidate, is better at actions and emotes. Human performance is still above all these models, leaving space for future improvements in these tasks. We conducted Wilcoxon signed-rank tests and found comparisons between Starspace and the differing transformers for dialogue on test seen in Table 4 are all significantly different at the p<0.01 level. The generative Transformer model did not work as well as the retrieval models using these metrics.
Learning to Speak and Act in a Fantasy Text Adventure Game
The six new unseen test settings are a slightly easier task in absolute numbers (Table 4, right), with improved scores for humans and some models. We observe that BERT-based models exhibit good transfer ability relative to other models, but the gap between their performance and human performance increases from the seen test set to the unseen one. Specifically, there is a 21 point gap on the unseen dialogue test set compared to an 11 point gap on the seen test set, making this a significant challenge for future methods.
Learning to Speak and Act in a Fantasy Text Adventure Game
To illustrate the coverage of entities and actions in the LIGHT world, and the inter-connectedness between them learnable from our data, we trained a simple Starspace embedding model with no prebuilt embeddings (so, on our data alone, thus precluding BERT) on all three tasks and show embeddings in Table 3. There is clearly a vast variety of learnable concepts and rich structure between characters, locations, objects, actions and the language describing them. We also show additional t-SNE plots and heatmaps showcasing these relationships in Appendix G.
Learning to Speak and Act in a Fantasy Text Adventure Game
Persona: I am a part of a group of travelers. I go from town to town selling food to the locals. I grew up poor, but my travels have paid off well.
Learning to Speak and Act in a Fantasy Text Adventure Game
Setting 1: Fishmonger's stall, Port Setting 2: Dunes, Desert A small booth near the edge of the port, it's protected by a A massive hilly landscape that is nothing but sand and a few piece of old, sun-bleached sailcloth. Baskets of freshly-rocks. As you walk this area, you can find some human and caught fish, bivalves, and eels sit in the shade in stained animal remains along with broken down wood wagons. wooden troughs of water. A small, aggressive-looking dog is chained to one   
Learning to Speak and Act in a Fantasy Text Adventure Game
Effect of Various Environment Features We provide a large quantity of information about the environment to each of our models -not only dialogue, but the description of the setting, the character's persona, present objects with descriptions, and more. We analyze the usefulness of the additional grounding information in Tables 5 and 6. For the dialogue task, having access to all of the environmental information provides the best performance for both retrieval and generative models. Training on dialogue alone substantially decreases performance, while each experiment that adds additional grounding information such as the past actions, persona or the setting description, improves the score. Providing object descriptions as a feature leads to the least improvement. As there are both a large quantity of objects that can be present and objects tend to have long descriptions, it can be challenging for the model to associate such information to a dialogue, action, or emote prediction task. The persona features were found to be impactful, which makes sense as they shape the things the character says (and does).
Learning to Speak and Act in a Fantasy Text Adventure Game
Action sequence and emote prediction are much improved when using the dialogue history compared to using only past action history. Other features generally have lesser impact in this case, but still give some improvements. Including all features appears challenging for the model, perhaps because of the large input to attend over, resulting in improved results for some ablations.
Learning to Speak and Act in a Fantasy Text Adventure Game
For the action prediction task, we found that affordance information provides comparable impact to model performance as the rest of the features. A model given only actions and emotes as features and the set of possible actions as candidates has similar perfomance to a model given the best features for the task (dialogue, action, emote, persona, and setting) with a candidate set including actions that would be normally filtered by affordances (such as "wield paint"). This suggests the context or quantity we provided the models in these experiments is not sufficient to be able to implicitly predict affordances at a high enough accuracy to not be distracted by impossible choices.
Learning to Speak and Act in a Fantasy Text Adventure Game
Most importantly, for all tasks training on the available dialogue data is necessary for good performance. Providing only the action and emote as context results in the worst performance, even on action and emote prediction tasks. Moreover, using dialogue and actions simultaneously improves results almost everywhere. The integrated environment in which agents can both act and speak to other agents provides relevant information that can be used across all tasks.
Learning to Speak and Act in a Fantasy Text Adventure Game
Context affects predicted utterances We investigate the effect of the environmental context on the predictions by modifying the context and examining the changes in predicted dialogue, ac-  tion, and emotes using the BERT-based Bi-Ranker. The input dialogue and speaker has a strong effect on the predicted action, as shown in Table 9, ranking over all training set actions. For example, a partner asking for an item results in a predicted action dependent on the asker to retrieve it, despite our dataset not being explicitly instructional.
Learning to Speak and Act in a Fantasy Text Adventure Game
A similar effect is observed for emote prediction. Modifying the dialogue and emote input produces a variety of different predicted emotes in Table 8. Further, keeping the context otherwise fixed but modifying the partner name from mermaid to orc results in a different predicted emote -the mermaid stating I will battle him leads to a stare while the orc receives a nod.
Learning to Speak and Act in a Fantasy Text Adventure Game
Finally, for dialogue prediction we find the model produces different outputs that are more appropriate for a given setting, even if the dialogue and characters are the same, see Table 7. With the same text about food, the model retrieved dialogue that was setting appropriate. In the fishmonger's stall, it asked if the human agent was a customer shopping for fish, but in the desert dunes it suggested we might be looking in the wrong place.
Learning to Speak and Act in a Fantasy Text Adventure Game
We introduced a large-scale crowdsourced fantasy text adventure game research platform where agents-both models and humans-can act and speak in a rich and diverse environment of locations, objects, and other characters. We analyzed a variety of models and their ability to leverage the grounding information present in the environment. We hope that this work can enable future research in grounded language learning and further the ability of agents to model a holistic world, complete with other agents within it.
WordCraft: An Environment for Benchmarking Commonsense Agents
Recent progress in Reinforcement Learning (RL), while impressive (Silver et al., 2016;Vinyals et al., 2019;Berner et al., 2019), has mostly focused on tabula-rasa learning, rather than exploitation of prior world knowledge. For example, while Go is an extremely difficult game to master, its rules and thus environment dynamics are simple and not heavily reliant on knowledge about concepts that can be encountered in the real world. Focusing on tabula-rasa learning confines state-of-the-art approaches to simulation environments that can be solved without the transfer of commonsense, world, or domain knowledge.
WordCraft: An Environment for Benchmarking Commonsense Agents
Many real-world applications (e.g. personal assistants and household robots) require agents that can learn fast and generalise well to novel situations, which is likely not possible * Equal contribution 1 University College London, London, UK 2 University of Oxford, Oxford, UK. Correspondence to: Minqi Jiang <minqi. jiang.19@ucl.ac.uk>. Accepted to the 1 st Workshop on Language in Reinforcement Learning, ICML 2020, Copyright 2020 by the author(s).   without the ability to reason with commonsense and general knowledge about the world. Consider, for example, an agent tasked with performing common household chores that has never seen a dirty ashtray. When presented with this new object, the agent needs to know that a reasonable set of actions involving this object include cleaning the ashtray, but not feeding it to the cat.
WordCraft: An Environment for Benchmarking Commonsense Agents
Humans encode a large amount of commonsense knowledge in written language. For example, Wikipedia encodes such knowledge implicitly through corpus statistics, and at times explicitly in writing. Outside of RL, learning to represent and utilise prior knowledge has improved considerably over recent years. For example, a common approach is to pre-train neural language models and fine-tune them on downstream tasks (Devlin et al., 2019;Raffel et al., 2019). Recent works highlight that such pre-trained models capture many aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019;Hewitt & Manning, 2019;Reif et al., 2019). Another popular approach is to represent commonsense knowledge explicitly as Knowledge Graphs (KGs), as done by ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017 stream RL tasks has not received the same level of attention, partly due to a scarcity of simulation environments that directly benefit from transferring such prior knowledge. Robotics environments such as MuJoCo require knowledge not readily expressed in natural language, such as an intuitive understanding of physics (Todorov et al., 2012;Yu et al., 2019). Rule-based game environments such as Go or Chess (Silver et al., 2017;, while very hard to master, are based on simple transition dynamics and largely disconnected from entities and concepts encountered in the real world. On the other hand, commonsense knowledge could be helpful in environments such as StarCraft II (Vinyals et al., 2019;Berner et al., 2019), which contain many complex interactions with a large number of entities (many of which have real-world analogues). However, such environments are costly to run, and learning to ground knowledge from external sources remains challenging, as natural language annotations of states and actions in such games are not readily available.
WordCraft: An Environment for Benchmarking Commonsense Agents
To study agents with commonsense knowledge, we present WordCraft, a fast RL environment based on the popular game Little Alchemy 2. In this game, the player is tasked with crafting over 700 different entities by combining previously discovered entities. For example, combining "water" and "earth" creates "mud". Learning policies that generalize to unseen entities and combinations requires commonsense knowledge about the world. The environment runs quickly (around 8,000 steps per second on a single machine), enabling fast experimentation cycles. As the entities in the game correspond to words, they can be easily grounded with external knowledge sources. In addition, we introduce a method for conditioning agents trained in WordCraft on KGs encoding prior knowledge, such as ConceptNet and other commonsense KGs.
WordCraft: An Environment for Benchmarking Commonsense Agents
We formulate the RL problem as a Markov Decision Process (MDP) (Sutton & Barto, 2018) defined by the tuple (S, A, T, R, γ), where s ∈ S is a state, a ∈ A is an action, T (s, a) → s is the transition function, R(s, a) → R is the reward function, and γ, the discount factor. The goal is to find a policy π that maximises the expected sum of future discounted rewards: ∞ k=0 γ k r k+1 . We represent knowledge about entities and the relations among them as a knowledge graph (KG): a collection of s, p, o triples, each encoding a relationship of type p (predicate) between the subject s and the object o of the triple. In order to work with incomplete knowledge graphs and infer missing relationships, we use the ComplEx link prediction model (Trouillon et al., 2016) (see Appendix A for details).
WordCraft: An Environment for Benchmarking Commonsense Agents
In order to benchmark the ability of RL agents to make use of commonsense knowledge, we present WordCraft, based on Little Alchemy 2 1 , a simple association game: Starting from a set of four basic items, the player must create as many different items as possible. Each non-starter item can be created by combining two other items. For example, combining "moon" and "butterfly" yields "moth", and combining "human" and "medusa" yields "statue". There are 700 total items (including rare and compound words) and 3,417 permissible item combinations (referred to as valid recipes throughout this paper) (IGN, 2020). Efficiently solving this game without trying every possible combination of items requires using knowledge about relations between common concepts.
WordCraft: An Environment for Benchmarking Commonsense Agents
WordCraft is a simplified version of Little Alchemy 2, sharing the same entities and valid recipes: First, the interface is text-based rather than graphical. Second, instead of a single open-ended task, WordCraft consists of a large number of simpler tasks. Each task is created by randomly sampling a goal entity, valid constituent entities, and distractor entities. The agent must choose which entities to combine in order to create the goal entity. The task difficulty can be adjusted by increasing the number of distractors or increasing the number of intermediate entities that must be created in order to reach the goal entity (we refer to this variant as depth-n, with n − 1 intermediate entities). The training and testing split is created by selecting a set of either goal entities or valid recipes that are not used during training. The relatively large number of entities and valid recipes enables the study of generalization to unseen goals.
WordCraft: An Environment for Benchmarking Commonsense Agents
The environment is deterministic and fully-observable. At each time step, the state s consists of the goal entity x g , a set of k currently selectable entities x table = {x i } k i=0 , and the currently selected entity x s,1 . The action a corresponds to choosing one of the available entities as the second selected entity x s,2 ∈ x table . When no entity is currently selected, x s,1 is set to a placeholder empty entity. If (x s,1 , x s,2 ) corresponds to a valid recipe, the corresponding resulting entity is added to the table x table and x s,1 is set to the empty entity. The episode terminates when the newly created entity is x g or after a maximum number of steps is reached. We keep the maximum number of steps low to discourage bruteforce solutions (O(k 2 ) for depth-1 tasks). The reward can be sparse (with r = 1 or 0 received only at the end of the episode) or shaped (with a penalty for choosing a wrong pair of items or creating irrelevant entities and a reward for creating intermediate ingredients).
WordCraft: An Environment for Benchmarking Commonsense Agents
Compared to other environments that combine language and RL (Janner et al., 2018;Chevalier-Boisvert et al., 2019;Côté et al., 2018;Das et al., 2018), WordCraft contains a much larger number of objects, with object relationships that reflect the structure of the real world. For example, the underlying KG of TextWorld (which is perhaps the closest comparison), contains 99 different objects and 10 types of relations (Zelinka et al., 2019). Furthermore, the environment does not contain movement-related actions. The state space is limited to combinations of entity words, and the action space, to table positions, yet generalization remains challenging. This allows allowing us to study commonsense reasoning in isolation, without requiring that the agent learn to ground states to text or deal with a combinatorially large action space (as seen in text games).
WordCraft: An Environment for Benchmarking Commonsense Agents
We propose an agent architecture that makes use of information from an external KG to guide the agent's policy. At a high level, the model consists of a self-attention-based actor-critic network and an external KG link prediction model. Given that the recipes in WordCraft are based on real-world semantics among common entities, conditioning on commonsense knowledge present in a KG should enable agents to learn more efficiently by constraining their search space to policies biased toward interactions with underlying commonsense semantics.
WordCraft: An Environment for Benchmarking Commonsense Agents
In order to handle a variable number of selectable entities at each time step and ensure selection decisions are invariant to the table position of entities, our policy network makes use of the scaled dot-product attention mechanism introduced in (Vaswani et al., 2017). The self-attention policy network encodes the features x g and current selection features x s,1:2 into attention weights over the set of selectable entities. These attention weights α i act as the logits to the action distribution over the set of selectable entities e i at each time step:
WordCraft: An Environment for Benchmarking Commonsense Agents
, where x table are learned representations of selectable entities.
WordCraft: An Environment for Benchmarking Commonsense Agents
In order to predict value estimates, we first apply a linear transform W to x table and encode the state as an attentionweighted sum of these linearly transformed entity features. This encoding is passed through a fully-connected layer to yield value estimates V (x) = MLP(α W (x table )).
WordCraft: An Environment for Benchmarking Commonsense Agents
We assume access to a link prediction model L trained on an external knowledge graph that contains information relevant to the RL task. While our model makes use of ComplEx for link prediction (Trouillon et al., 2016), in general this choice can be substituted by any other link prediction model. We chose ComplEx for its reliable performance in practice.
WordCraft: An Environment for Benchmarking Commonsense Agents
The link prediction model is trained on a set of subjectobject-predicate relation triplets of the form (s, p, o) to predict higher likelihood scores for true relation triplets. In the context of WordCraft, each recipe {e 1 , e 2 } − → e 3 reduces to four relevant relation triplets: (e 1 , combinesWith, e 2 ), (e 2 , combinesWith, e 1 ), (e 1 , componentOf, e 3 ), and (e 2 , componentOf, e 2 ). These triplets, when aggregated across all recipes, forms the recipe graph. In our experiments, we train ComplEx on a subgraph of the recipe graph containing all nodes to ensure full entity coverage.
WordCraft: An Environment for Benchmarking Commonsense Agents
At each time step, we compute relation score vectors u and v between the goal and selected entities and each e i on the table and e s,j in the selection: u i = L(e i , combinesWith, e s,j ), v i = L(e i , componentOf, e g ). These scores are then mixed component-wise with the attention weights, before passing the mixed weights into the softmax policy head, thereby guiding the agent toward relevant predictions from the KG model. Details on learning the mixing coefficients are provided in Appendix B.1.
WordCraft: An Environment for Benchmarking Commonsense Agents
Our experiments focus on zero-shot generalization performance. We split the set of all valid recipes into train (80%) and test (20%) sets. Train tasks are generated such that task goals do not involve any recipes in the test set. The model is trained using the TorchBeast implementation of IMPALA (Espeholt et al., 2018;Küttler et al., 2019).
WordCraft: An Environment for Benchmarking Commonsense Agents
To demonstrate how well the task captures real-world relationships between entities, we represent entities as GloVe embeddings (Pennington et al., 2014) when evaluating the model in Section 4.1. We expect GloVe to help generalization if semantically similar entities have similar uses in this environment. We assess the zero-shot performance of our agent model without a link prediction module on depth-1 tasks with 1 and 8 distractors.
WordCraft: An Environment for Benchmarking Commonsense Agents
We also collect a human baseline at the same difficulty settings of WordCraft. This human baseline serves as an estimate of the zero-shot performance that can be achieved using commonsense and general knowledge (see the Appendix C.5 for the description of human evaluation protocol). Mean episode return Self-attention + GloVe, 1 distractor Self-attention + random, 1 distractor Self-attention + GloVe, 8 distractors  As seen in Figure 2, agents trained with GloVe embeddings generalize much better to tasks with recipes that were not part of any training task. This margin of improvement persists in harder task settings, even as the test success rate falls.
WordCraft: An Environment for Benchmarking Commonsense Agents
We benchmark variants of our full agent with the link prediction module and compare zero-shot performance under different combinations of link prediction scores used to modulate the self-attention weights: both combinesWith and componentOf scores, or each score in isolation. In these experiments, L is trained on the same train set of recipes used to generate the training tasks.
WordCraft: An Environment for Benchmarking Commonsense Agents
Our results show that access to a full KG model drastically improves the agent's zero-shot performance (see Figure 3). However, while a partial-KG-model agent reaches an equivalent zero-shot success rate as an agent without any KG model in fewer training steps, they ultimately reach comparable levels of test performance as training progresses (see Appendix for a more detailed discussion).
WordCraft: An Environment for Benchmarking Commonsense Agents
There are multiple avenues that we plan to further explore. Extending WordCraft to the longer horizon setting of the original Little Alchemy 2, in which the user must discover as many entities as possible, could be an interesting setting to study commonsense-driven exploration. We also plan to introduce additional modalities such as images and control into WordCraft. Furthermore, we believe the ideas in this work could benefit more complex RL tasks associated with large corpora of task-specific knowledge, such as NLE. This path of research entails further investigation of methods for automatically constructing knowledge graphs from available corpora as well as agents that retrieve and directly condition on natural language texts in such corpora. 
WordCraft: An Environment for Benchmarking Commonsense Agents
Knowledge Graphs Knowledge graphs naturally represent the ontology of concepts underlying commonsense knowledge about the world. In a knowledge graph, nodes represent concepts, and edges between two nodes represent a relationship between the corresponding concepts. Let E and R denote a set of entities and a set of relation types, respectively. Formally, a KG G ⊆ E × R × E is a set of subject-predicate-object triples s, p, o , with s, o ∈ E and p ∈ R, where each triple encodes a relationship of type p between the subject s and the object o. There are several examples of large-scale KGs encoding commonsense knowledge, both from academia (such as YAGO (Suchanek et al., 2007), DBpedia (Auer et al., 2007)) and industry (such as Microsoft's Bing Knowledge Graph and the Google Knowledge Graph (Noy et al., 2019)). Such large-scale knowledge graphs hold an enormous amount of useful prior knowledge about the world, that should be useful for guiding an RL agent in any environment with real-world semantics. Especially relevant to this work are ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), two KGs encoding common-sense knowledge.
WordCraft: An Environment for Benchmarking Commonsense Agents
Representation Learning of KGs An effective way of enabling statistical learning on KGs consists of learning continuous, distributed representations, also referred to as embeddings, for all entities in a KG. We refer to Nickel et al. 
WordCraft: An Environment for Benchmarking Commonsense Agents
where ·, ·, · denotes the tri-linear dot product, e s , e p , e o ∈ C k denote the complex-valued k-dimensional representations of s, p, and o, x denote the complex conjugate of x ∈ C k , Re (x) is the real part of x. In order to learn the embedding representations Θ for all entities and relation types in a KG G, we follow Lacroix et al. (2018) and minimise the following objective via gradient-based optimisation: 
WordCraft: An Environment for Benchmarking Commonsense Agents
The link-prediction scores between goal and table entities for the componentOf relation and between each selected entity and table entities for the combinesWith relation are combined component-wise with the self-attention weights α i using mixing coefficients to yield final policy logits β:
WordCraft: An Environment for Benchmarking Commonsense Agents
The mixing coefficients λ are computed by passing the selfattention-weighted table encoding through a fully-connected layer:
WordCraft: An Environment for Benchmarking Commonsense Agents
C. Experiments C.1. Self-Attention Actor-Critic Architecture  
WordCraft: An Environment for Benchmarking Commonsense Agents
Self-attention key size 300 Self-attention value size 300 ComplEx embedding size 128 Simple-MLP hidden size 300
WordCraft: An Environment for Benchmarking Commonsense Agents
In addition to initial benchmarks conducted in 5.1, we also compare the performance of the self-attention actor-critic agent described in 4.1 to a simple, single-layer MLP that predicts the policy logits and value estimates from the concatenation of all GloVe word embeddings of goal, selection, and table entities at each time step. These results are presented in Fig. 5.
WordCraft: An Environment for Benchmarking Commonsense Agents
We further benchmark the performance of the self-attention actor-critic agent with and without access to an instance of ComplEx trained on a subgraph of the full recipe graph. We compare agents trained with access to ComplEx trained on a partial graph and those trained on the full graph. The partial KG models are trained on only the relations present in the set of training recipes.
WordCraft: An Environment for Benchmarking Commonsense Agents
As the method for incorporating ComplEx's predicted relation scores into the policy prediction performs a weighted sum of each score with the self-attention weights, we investigate the effect of only using different subsets of the scores. As seen in Fig. 6 and Fig. 7, the subset of relation scores that are helpful to policy learning varies across task depth and distractor settings.
WordCraft: An Environment for Benchmarking Commonsense Agents
We generally find that access to a full KG model drastically improves the agent's zero-shot success rate, as might be expected. In fact, one might expect that having access Mean episode return Self-attention + GloVe, 1 distractor Self-attention + random, 1 distractor Self-attention + GloVe, 2 distractors Self-attention + random, 2 distractors MLP AC + GloVe, 1 distractor MLP AC + learned embed, 1 distractor MLP AC + GloVe, 2 distractors MLP AC + learned embed, 2 distractors Oracle Human, both 1 and 2 distractors to the ground-truth recipe graph as encoded by ComplEx should enable the agent to achieve 100% zero-shot success rates across all task settings. In order to test the extent to which the full-KG ComplEx scores are predictive of the best actions, we also assessed the performance of agents whose final policy logit mixture only assigned positive weights to different subsets of the relation scores. Such full-KG-only agents that choose items only based on the combinesWith relation score will still often choose the same entity twice, believing the entity combines with itself. This is likely because the recipe graph includes several entities that combine with themselves, while the set of all self-combination triplets are not included in the training set. Full-KG-only agents that choose items only based on the componentOf relation score will tend to choose the same item twice, as this approach will always assign the highest action probability to selecting the entity on the table that ComplEx predicts to be most likely to be a component of the goal entity.
WordCraft: An Environment for Benchmarking Commonsense Agents
Interestingly, partial-KG-agents do not always do much better than a KG-free agent. This implies that as the KGfree agent sees more training tasks, it becomes comparable to ComplEx in predicting unseen relations among entities.
WordCraft: An Environment for Benchmarking Commonsense Agents
Early on in training, the relation scores predicted by Com-plEx seem to help the agent achieve higher zero-shot success rates in fewer training steps compared to KG-free agents (see Figure 7).
WordCraft: An Environment for Benchmarking Commonsense Agents
We estimate the generalization ability of humans players on depth-1 tasks with 1 and 8 distractors. Each difficulty setting is tested on a different individual, and we test one individual per setting. The selected individuals were not familiar with the game and got to train on 40 randomly sampled training tasks before being evaluated on 40 testing tasks. The train and test split was the same as in the RL experiments. Surprisingly, we found that changing the number of distractors from 1 to 8 did not significantly influence human performance. We plan to conduct a larger-scale human performance evaluation. 
Grounded Language Learning in a Simulated 3D World
Endowing machines with the ability to relate language to the physical world is a longstanding challenge for the development of Artificial Intelligence. As situated intelligent technology becomes ubiquitous, the development of computational approaches to understanding grounded language has become critical to human-AI interaction. Beginning with Winograd (1972), early attempts to ground language understanding in a physical world were constrained by their reliance on the laborious hard-coding of linguistic and physical rules. Modern devices with voice control may appear more competent but suffer from the same limitation in that their language understanding components are mostly rule-based and do not generalise or scale beyond their programmed domains.
Grounded Language Learning in a Simulated 3D World
This work presents a novel paradigm for simulating language learning and understanding. The approach differs from conventional computational language learning in that the learning and understanding take place with respect to a continuous, situated environment. Simultaneously, we go beyond rule-based approaches to situated language understanding as our paradigm requires agents to learn end-to-end the grounding for linguistic expressions in the context of using language to complete tasks given only pixel-level visual input.
Grounded Language Learning in a Simulated 3D World
The initial experiments presented in this paper take place in an extended version of the DeepMind Lab (Beattie et al., 2016) environment, where agents are tasked with finding and picking up objects based on a textual description of each task. While the paradigm outlined gives rise to a large number of possible learning tasks, even the simple setup of object retrieval presents challenges for conventional machine learning approaches. Critically, we find that language learning is contingent on a combination of reinforcement (rewardbased) and unsupervised learning. By combining these techniques, our agents learn to connect words and phrases with emergent representations of the visible surroundings and embodied experience. We show that the semantic knowledge acquired during this process generalises both with respect to new situations and new language. Our agents exhibit zeroshot comprehension of novel instructions, and the speed at which they acquire new words accelerates as their semantic knowledge grows. Further, by employing a curriculum training regime, we train a single agent to execute phrasal instructions pertaining to multiple tasks requiring distinct action policies as well as lexical semantic and object knowledge. 1
Grounded Language Learning in a Simulated 3D World
Learning semantic grounding without prior knowledge is notoriously difficult, given the limitless possible referents for each linguistic expression (Quine, 1960). A learner must discover correlations in a stream of low level inputs, relate these correlations to both its own actions and to linguistic expressions and retain these relationships in memory. Perhaps unsurprisingly, the few systems that attempt to learn language grounding in artificial agents do so with respect to environments that are far simpler than the continuous, noisy sensory experience encountered by humans (Steels, 2008;Roy and Pentland, 2002;Krening et al., 2016;Yu et al., 2017).
Grounded Language Learning in a Simulated 3D World
The idea of programming computers to understand how to relate language to a simulated physical environment was pioneered in the seminal work of Winograd (1972). His SHRDLU system was programmed to understand user generated language containing a small number of words and predicates, to execute corresponding actions or to ask questions requesting more information. While initially impressive, this system required that all of the syntax and semantics (in terms of the physical world) of each word be hard coded a priori, and thus it was unable to learn new concepts or actions. Such rule-based approaches to language understanding have come to be considered too brittle to scale to the full complexities of natural language. Since this early work, research on language grounding has taken place across a number of disciplines, primarily in robotics, computer vision and computational linguistics. Research in both natural language processing and computer vision has pointed to the importance of cross modal approaches to grounded concept learning. For instance, it was shown that learnt concept representation spaces more faithfully reflect human semantic intuitions if induced from information about the perceptible properties of objects as well as from raw text (Silberer and Lapata, 2012).
Grounded Language Learning in a Simulated 3D World
Semantic parsing, as pursued the field of natural language processing, has predominantly focussed on building a compositional mapping from natural language to formal semantic representations that are then grounded in a database or knowledge graph (Zettlemoyer and Collins, 2005;Berant et al., 2013). The focus of this direction of work is on the compositional mapping between the two abstract modalities, natural language and logical form, where the grounding is usually discrete and high level. This is in contrast to the work presented in this paper where we focus on learning to ground language in low level perception and actions. Siskind (1995) represents an early attempt to ground language in perception by seeking to link objects and events in stick-figure animations to language. Broadly this can be seen as a precursor to more recent work on mapping language to actions in video and similar modalities (Siskind, 2001;Chen and Mooney, 2008;Yu and Siskind, 2013). In a similar vein, the work of Roy and Pentland (2002) applies machine learning to aspects of grounded language learning, connecting speech or text input with images, videos or even robotic controllers. These systems consisted of modular pipelines in which machine learning was used to optimise individual components while complementing hard-coded representations of the input data. Within robotics, there has been interest in using language to facilitate human-robot communication, as part of which it is necessary to devise mechanisms for grounding a perceptible environment with language . In general, the amount of actual learning in these prior works is heavily constrained, either through the extensive use of hand-written grammars and mechanisms to support the grounding, or through simplification in terms of the setup and environment.
Grounded Language Learning in a Simulated 3D World
Other related work focuses on language grounding from the perspective of humanmachine communication (Thomason et al., 2015;Wang et al., 2016;Arumugam et al., 2017).
Grounded Language Learning in a Simulated 3D World
The key difference between these approaches and our work is that here again language is grounded to highly structured environments, as opposed to the continuous perceptible input our learning environment provides.
Grounded Language Learning in a Simulated 3D World
In the field of computer vision, image classification (Krizhevsky et al., 2012) can be interpreted as aligning visual data and semantic or lexical concepts. Moreover, neural networks can effectively map image or video representations from these classification networks to human-written image captions. These mappings can also yield plausible descriptions of visual scenes that were not observed during training (Xu et al., 2015;Vendrov et al., 2015). However, unlike our approach, these captioning models typically learn visual and linguistic processing and representation from fixed datasets as part of two separate, independent optimisations. Moreover, they do not model the grounding of linguistic symbols in actions or a visual stimuli that constantly change based on the exploration policy of the agent.
Grounded Language Learning in a Simulated 3D World
The idea that reinforcement-style learning could play a role in language learning has been considered for decades (Chomsky, 1959). Recently, however, RL agents controlled by deep neural nets have been trained to solve tasks in both 2D (Mnih et al., 2015) and 3D  environments. Our language learning agents build on these approaches and algorithms, but with an agent architecture and auxiliary unsupervised objectives that are specific to our multi-modal learning task. Other recently-proposed frameworks for interactive language learning involve unimodal (text-only) settings (Narasimhan et al., 2015;Mikolov et al., 2015).
Grounded Language Learning in a Simulated 3D World
To conduct our language learning experiments we integrated a language channel into a 3D simulated world (DeepMind Lab, Beattie et al. (2016)). In this environment, an agent perceives its surroundings via a constant stream of continuous visual input and a textual instruction. It perceives the world actively, controlling what it sees via movement of its visual field and exploration of its surroundings. One can specify the general configuration of layouts and possible objects in this environment together with the form of language instructions that describe how the agent can obtain rewards. While the high-level configuration of these simulations is customisable, the precise world experienced by the agent is chosen at random from billions of possibilities, corresponding to different instantiations of objects, their colours, surface patterns, relative positions and the overall layout of the 3D world.
Grounded Language Learning in a Simulated 3D World
To illustrate this setup, consider a very simple environment comprising two connected rooms, each containing two objects. To train the agent to understand simple referring expressions, the environment could be configured to issue an instruction of the form pick the X in each episode. During training, the agent experiences multiple episodes with the shape, colour and pattern of the objects themselves differing in accordance with the instruction. Thus, when the instruction is pick the pink striped ladder, the environment might contain, in random positions, a pink striped ladder (with positive reward), an entirely pink ladder, a pink striped chair and a blue striped hairbrush (all with negative reward).
Grounded Language Learning in a Simulated 3D World
It is important to emphasise the complexity of the learning challenge faced by the agent, even for a simple reference task such as this. To obtain positive rewards across multiple training episodes, the agent must learn to efficiently explore the environment and inspect candidate objects (requiring the execution of hundreds of inter-dependent actions) while simultaneously learning the (compositional) meanings of multi-word expressions and how they pertain to visual features of different objects ( Figure 1)
Grounded Language Learning in a Simulated 3D World
We also construct more complex tasks pertaining to other characteristics of human language understanding, such as the generalisation of linguistic predicates to novel objects, the productive composition of words and short phrases to interpret unfamiliar instructions and the grounding of language in relations and actions as well as concrete objects.
Grounded Language Learning in a Simulated 3D World
Our agent consists of four inter-connected modules optimised as a single neural network. At each time step t, the visual input v t is encoded by the convolutional vision module V and a recurrent (LSTM, Hochreiter and Schmidhuber (1997)) language module L encodes the instruction string l t . A mixing module M determines how these signals are combined before they are passed to a two-layer LSTM action module A. The hidden state s t of the upper LSTM in A is fed to a policy function, which computes a probability distribution over possible motor actions π(a t |s t ), and a state-value function approximator V al(s t ), which computes a scalar estimate of the agent value function for optimisation. To learn from the scalar rewards that can be issued by the environment, the agent employs an actor-critic algorithm .
Grounded Language Learning in a Simulated 3D World
The policy π is a distribution over a discrete set of actions. The baseline function V al estimates the expected discounted future return following the state the agent is currently in. In Figure 1: In this example, the agent begins in position 1 and immediately receives the instruction pick the red object next to the green object. It explores the two-room layout, viewing objects and their relative positions before retrieving the object that best conforms to the instruction. This exploration and selection behaviour emerges entirely from the reward-driven learning and is not preprogrammed. When training on a task such as this, there are billions of possible episodes that the agent can experience, containing different objects in different positions across different room layouts. 
Grounded Language Learning in a Simulated 3D World
where S t is the state of the environment at time t when following policy π and r t is the reward received following the action performed at time t. λ ∈ [0, 1] is a discount parameter.
Grounded Language Learning in a Simulated 3D World
The agent's primary objective is is to find a policy which maximizes the expected discounted return E π [ ∞ t=0 λ t r t ]. We apply the Advantage Actor Critic algorithm  to optimize the policy π-a Softmax multinomial distribution parametrized by the agent's network-towards higher discounted returns.
Grounded Language Learning in a Simulated 3D World
Parameters are updated according to the RMSProp update rule (Tieleman and Hinton, 2012). We share a single parameter vector across 32 asynchronous threads. This configuration offers a suitable trade-off between increased speed and loss of accuracy due to the asynchronous updates . Importantly, early simulation results revealed that this initial design does not learn to solve even comparably simple tasks in our setup. As described thus far, the agent can learn only from comparatively infrequent object selection rewards, without exploiting the stream of potentially useful perceptual feedback available at each time step when exploring the environment. We address this by endowing the agent with ways to learn in an unsupervised manner from its immediate surroundings, by means of auto-regressive objectives that are applied concurrently with the reward-based learning and involve predicting or modelling aspects of the agent's surroundings (Jaderberg et al., 2016).
Grounded Language Learning in a Simulated 3D World
Temporal autoencoding The temporal autoencoder auxiliary task tAE is designed to illicit intuitions in our agent about how the perceptible world will change as a consequence of its actions. The objective is to predict the visual environment v t+1 conditioned on the prior visual input v t and the action a t (Oh et al., 2015). Our implementation reuses the standard visual module V and combines the representation of v t with an embedded representation of a t . The combined representation is passed to a deconvolutional network to predict v t+1 . As well as providing a means to fine-tune the visual system V, the tAE auxiliary task results in additional training of the action-policy network, since the action representations can be shared between tAE and the policy network π.
Grounded Language Learning in a Simulated 3D World
Language prediction To strengthen the ability of the agent to reconcile visual and linguistic modalities we design a word prediction objective LP that estimates instruction words l t given the visual observation v t , using model parameters shared with both V and L. The LP network can also serve to make the behaviour of trained agents more interpretable, as the agent emits words that it considers to best describe what it is currently observing.
Grounded Language Learning in a Simulated 3D World
The tAE and LP auxiliary networks were optimised with mini-batch gradient descent based on the mean-squared error and negative-log-likelihood respectively. We also experimented with reward prediction (RP) and value replay (VR) as additional auxiliary tasks to stabilise reinforcement based training (Jaderberg et al., 2016). Figure 2 gives a schematic organisation of the agent with all the above auxiliary learning objectives. Precise implementation details of the agent are given in Appendix A.
Grounded Language Learning in a Simulated 3D World
In evaluating the agent, we constructed tasks designed to test its capacity to cope with various challenges inherent in language learning and understanding. We first test its ability to efficiently acquire a varied vocabulary of words pertaining to physically observable aspects of the environment. We then examine whether the agent can combine this lexical knowledge to interpret both familiar and unfamiliar word combinations (phrases). This analysis includes phrases whose meaning is dependent of word order, and cases in which the agent must induce and re-use lexical knowledge directly from (potentially ambiguous) phrases. Finally, we test the agent's ability to learn less concrete aspects of language, including instructions referring to relational concepts (Doumas et al., 2008) and phrases referring to actions and behaviours.
Grounded Language Learning in a Simulated 3D World
Our first experiment explored the effect of the auxiliary objectives on the ability of the agent to acquire a vocabulary of different concrete words (and associated lexical concepts). Training consisted of multiple episodes in a single room containing two objects. For each episode, at time t = 0, the agent was spawned in a position equidistant from the two objects, and received a single-word instruction that unambiguously referred to one of the two objects. It received a reward of 1 if it walked over to and selected the correct referent object and −1 if it picked the incorrect object. A new episode began immediately after an object was selected, or if the agent had not selected either object after 300 steps. Objects and instructions were sampled at random from the full set of factors available in the simulation environment. the standard settings and random hyperparameters sampled uniformly from the standard ranges. 3 Figure 3: Unsupervised learning via auxiliary prediction objectives facilitates word learning. Learning curves for a vocabulary acquisition task. The agent is situated in a single room faced with two objects and must select the object that correctly matches the textual instruction. A total of 59 different words were used as instructions during training, referring to either the shape, colours, relative size (larger, smaller), relative shade (lighter, darker) or surface pattern (striped, spotted, etc.) of the target object. RP: reward prediction, VR: value replay, LP: language prediction, tAE: temporal autoencoder. Data show mean and confidence bands (CB) across best five of 16 hyperparameter settings sampled at random from ranges specified in the appendix. Training episodes counts individual levels seen during training.
Grounded Language Learning in a Simulated 3D World
As shown in Figure 3, when relying on reinforcement learning alone, the agent exhibited no learning even after millions of training episodes. The fastest learning was exhibited by an agent applying both temporal auto-encoding and language prediction in conjunction with value replay and reward prediction. These results demonstrate that auto-regressive objectives can extract information that is critical for language learning from the perceptible environment, even when explicit reinforcement is not available. Figure 4: Word learning is much faster once some words are already known The rate at which agents learned a vocabulary of 20 shape words was measured in agents in three conditions. In one condition, the agent had prior knowledge of 20 shapes and their names outside of the training data used here. In the second condition, the agent had prior knowledge of two shape words outside of the target vocabulary (same number of pre-training steps). In the third condition, the agent was trained from scratch. All agents used RP, VR, LP and tAE auxiliary objectives. Data show mean and confidence bands across best five of 16 hyperparameter settings in each condition, sampled at random from ranges specified in Appendix C.
Grounded Language Learning in a Simulated 3D World
Before it can exhibit any lexical knowledge, the agent must learn various skills and capacities that are independent of the specifics of any particular language instruction. These include an awareness of objects as distinct from floors or walls; some capacity to sense ways in which those objects differ; and the ability to both look and move in the same direction. In addition, the agent must infer that solving the solution to tasks is always contingent on both visual and linguistic input, without any prior programming or explicit teaching of the importance of inter-modal interaction. Given the complexity of this learning challenge, it is perhaps unsurprising that the agent requires thousands of training episodes before evidence of word learning emerges.
Grounded Language Learning in a Simulated 3D World
To establish the importance of this 'pre-linguistic' learning, we compared the speed of vocabulary acquisition in agents with different degrees of prior knowledge. The training set consisted of instructions (and corresponding environments) from the twenty shape terms banana, cherries, cow, flower, fork, fridge, hammer, jug, knife, pig, pincer, plant, saxophone, shoe, spoon, tennis-racket, tomato, tree, wine-glass and zebra. The agent with most prior knowledge was trained in advance (in a single room setting with two objects) on the remaining twenty shapes from the full environment. The agent with minimal prior knowledge was trained only on the two terms ball and tv. Both regimes of advanced training were stopped once the agent reached an average reward of 9.5/10 across 1,000 episodes. The agent with no prior knowledge began learning directly on the training set.
Grounded Language Learning in a Simulated 3D World
The comparison presented in Figure 4 demonstrates that much of the initial learning in an agent trained from scratch involves acquiring visual and motor, rather than expressly linguistic, capabilities. An agent already knowing two words (and therefore exhibiting rudimentary motor and visual skills) learned new words at a notably faster rate than an agent trained from scratch. Moreover, the speed of word learning appeared to accelerate as more words were learned. This shows that the acquisition of new words is supported not only by general-purpose motor-skills and perception, but also existing lexical or semantic knowledge. In other words, the agent is able to bootstrap its existing semantic knowledge to enable the acquisition of new semantic knowledge.
Grounded Language Learning in a Simulated 3D World
Two important facets of natural language understanding are the ability to compose the meanings of known words to interpret otherwise unfamiliar phrases, and the ability to generalise linguistic knowledge learned in one setting to make sense of new situations. To examine these capacities in our agent, we trained it in settings where its (linguistic or visual) experience was constrained to a training set, and simultaneously as it learned from the training set, tested the performance of the agent on situations outside of this set ( Figure 5).
Grounded Language Learning in a Simulated 3D World
In the colour-shape composition experiment, the training instructions were either unigrams or bigrams. Possible unigrams were the 40 shape and the 13 colour terms listed in Appendix B. The possible bigrams were any colour-shape combination except those containing the shapes ice lolly, ladder, mug, pencil, suitcase or the colours red, magenta, grey, purple (subsets selected randomly). The test instructions consisted of all possible bigrams excluded from the training set. In each training episode, the target object was 3. See Appendix C for details.
Grounded Language Learning in a Simulated 3D World
rendered to match the instruction (in colour, shape or both) and the confounding object did not correspond to any of the bigrams in the test set. Similarly, in each test episode, both the target object and the confounding object corresponded to bigrams in the test instructions. These constraints ensured that the agent could not interpret test instructions by excluding other objects or terms that it had seen in the training set.
Grounded Language Learning in a Simulated 3D World
The colour-shape decomposition / composition experiment is similar in design to the colour-shape composition experiment. The test tasks were identical, but the possible training instructions consisted only of the bigram instructions from the colour-shape composition training set. To achieve above chance performance on the test set, the agent must therefore isolate aspects of the world that correspond to each of the constituent words in the bigram instructions (decomposition), and then build an interpretation of novel bigrams using these constituent concepts.
Grounded Language Learning in a Simulated 3D World
The relative size and relative shade experiments were designed to test the generality of agents' representation of relational concepts (in this case larger, smaller, larger and darker. Training and testing episodes again took place in a single room with two objects. The relative size experiment involved the 16 shapes in our environment whose size could be varied while preserving their shape. The possible instructions in both training and test episodes were simply the unigrams larger and smaller. The agent was required to choose between two objects of the same shape but different size (and possibly different colour) according to the instruction. All training episodes involved target and confounding objects whose shape was either a tv, ball, balloon, cake, can, cassette, chair, guitar, hairbrush or hat. All test episodes involved objects whose shape was either an ice lolly, ladder, mug, pencil or toothbrush.
Grounded Language Learning in a Simulated 3D World
The relative shade experiment followed the same design, but the agent was presented with two objects of possibly differing shape that differed only in the shade of their colouring (e.g. one light blue and one dark blue). The training colours were green, blue, cyan, yellow, pink, brown and orange. The test colours were red, magenta, grey and purple.
Grounded Language Learning in a Simulated 3D World
When trained on colour and shape unigrams together with a limited number of colourshape bigrams, the agent naturally understood additional colour-shape bigrams if it is familiar with both constituent words. Moreover, this ability to productively compose known words to interpret novel phrases was not contingent on explicit training of those words in isolation. When exposed only to bigram phrases during training, the agent inferred the constituent lexical concepts and reapplied these concepts to novel combinations at test time. Indeed, in this condition (the decomposition/composition case), the agent learned to generalise after fewer training instances than in the apparently simpler composition case. This can be explained by by the fact that episodes involving bigram instructions convey greater information content, such that the latter condition avails the agent of more information per training episode. Critically, the agent's ability to decompose phrases into constituent (emergent) lexical concepts reflects an ability that may be essential for humanlike language learning in naturalistic environments, since linguistic stimuli rarely contain words in isolation.
Grounded Language Learning in a Simulated 3D World
Another key requirement for linguistic generalisation is the ability to extend category terms beyond the specific exemplars from which those concepts were learned (Quinn et al., 1993;Rogers and McClelland, 2004). This capacity was also observed in our agent; when trained on the relational concepts larger and smaller in the context of particular shapes Figure 5: Semantic knowledge generalises to unfamiliar language and objects. Composition (A): training covered all shape and colour unigrams and ∼ 90% of possible colour-shape bigrams, such as blue ladder. Agents were periodically tested on the remaining 10% of bigrams without updating parameters. Decomposition-composition (B): the same regime as in A, but without any training on unigram descriptors. Lighter / darker (C): agents were trained to interpret the terms lighter and darker applied to a set of colours, and tested on the terms in the context of a set of different colours. Relative size (D): agents were trained to interpret the terms larger and smaller applied to a set of shapes, and tested on the terms in the context of a set of different shapes. Data show mean and CB across best five of 16 randomly sampled hyperparameter settings in each condition. See Appendix B for hyperparameter ranges and exact train/test stimuli. it naturally applied them to novel shapes with almost perfect accuracy. In contrast, the ability to generalise lighter and darker to unfamiliar colours was significantly above chance but less than perfect. This may be because it is particularly difficult to infer the mapping corresponding to lighter and darker (as understood by humans) in an RGB colour space from the small number of examples observed during training.
Grounded Language Learning in a Simulated 3D World
Taken together, these instances of generalisation demonstrate that our agent does not simply ground language in hard coded features of the environment such as pixel activations or specific action sequences, but rather learns to ground meaning in more abstract semantic representations. More practically, these results also suggest how artificial agents that are necessarily exposed to finite training regimes may ultimately come to exhibit the productivity characteristic of human language understanding.
Grounded Language Learning in a Simulated 3D World
A consequence of the agent's facility for re-using its acquired knowledge for further learning is the potential to train the agent on more complex language and tasks via exposure to a curriculum of levels. Figure 6 shows an example for the successful application of such a curriculum, here applied to the task of selecting an object based on the floor colour of the room it is located in.
Grounded Language Learning in a Simulated 3D World
We also applied a curriculum to train an agent on a range of multi-word referring instructions of the form pick the X, where X represents a string consisting of either a single noun (shape term, such as chair ) an adjective and a noun (a colour term, pattern term or shade term, followed by a shape term, such as striped ladder ) or two adjectives and a noun (a shade term or a pattern term, followed by a colour term, followed by a shape term, such as dark purple toothbrush). The latter two cases were also possible with the generic term 'object' in place of a shape term. In each case, the training episode involved one object that coincided with the instruction and some number of distractors that did not. Learning curves for this 'referring expression agent' are illustrated in Figure 7.
Grounded Language Learning in a Simulated 3D World
Language is typically used to refer to actions and behaviours as much as to objects and entities. To test the ability of our agents to ground such words in corresponding procedures, we trained a single agent to follow instructions pertaining to three dissociable tasks. We constructed these tasks using a two-room world with both floor colourings and object properties sampled at random.
Grounded Language Learning in a Simulated 3D World
In this environment, the Selection task involved instructions of the form pick the X object or pick all X, where X denotes a colour term. The Next to task involved instructions of the form pick the X object next to the Y object, where X and Y refer to objects. Finally, the In room task involved instructions of the form pick the X in the Y room, where Y referred to the colour of the floor in the target room. Both the Next to and the In room task employed large degrees of ambiguity, i.e. a given Next to level may contain several objects X and Y , but in a constellation that only one X would be located next to a Y .
Grounded Language Learning in a Simulated 3D World
The agent was exposed to instances of each task with equal probability during training. The possible values for variables X and Y in these instructions were red, blue, green, yellow, Figure 6: Curriculum learning is necessary for solving more complex tasks. For the agent to learn to retrieve an object in a particular room as instructed, a four-lesson training curriculum was required. Each lesson involved a more complex layout or a wider selection of objects and words, and was only solved by an agent that had successfully solved the previous lesson. The schematic layout and vocabulary scope for each lesson is shown above the training curves for that lesson. The initial (spawn) position of this agent varies randomly during training among the locations marked x, as do the position of the four possible objects among the positions marked with a white diamond. Data show mean and CB across best five of 16 randomly sampled hyperparameter settings in each condition. Figure 7: Learning curve for the referring expression agent. The trained agent is able to select the correct object in a two-object setup when described using a compositional expression. This ability transfers to more complex environments with a larger number of confounding objects. Figure 8: Multi-task learning via an efficient curriculum of two steps. A single agent can learn to solve a number of different tasks following a two-lesson training curriculum. The different tasks cannot be distinguished based on visual information alone, but require the agent to use the language input to identify the task in question.
Grounded Language Learning in a Simulated 3D World
cyan and magenta. The shape of all objects in the environment was selected randomly from 40 possibilities.
Grounded Language Learning in a Simulated 3D World
As previously, a curriculum was required to achieve the best possible agent performance on these tasks (see Figure 8). When trained from scratch, the agent learned to solve all three types of task in a single room where the colour of the floor was used as a proxy for a different room. However, it was unable to achieve the same learning in a larger layout with two distinct rooms separated by a corridor. When the agent trained in a single room was transferred to the larger environment, it continued learning and eventually was able to solve the more difficult task. 4 By learning these tasks, this agent demonstrates an ability to ground language referring not only to single (concrete) objects, but also to (more abstract) sequences of actions, plans and inter-entity relationships. Moreover, in mastering the Next to and In room tasks, the agent exhibits sensitivity to a critical facet of many natural languages, namely the dependence of utterance meaning on word order. The ability to solve more complex tasks by curriculum training emphasises the generality of the emergent semantic representations acquired by the agent, allowing it to transfer learning from one scenario to a related but more complex environment.
Grounded Language Learning in a Simulated 3D World
An artificial agent capable of relating natural languages to the physical world would transform everyday interactions between humans and technology. We have taken an important step towards this goal by describing an agent that learns to execute a large number of multi-word instructions in a simulated three-dimensional world, with no pre-programming or hard-coded knowledge. The agent learns simple language by making predictions about the world in which that language occurs, and by discovering which combinations of words, perceptual cues and action decisions result in positive outcomes. Its knowledge is distributed across language, vision and policy networks, and pertains to modifiers, relational concepts and actions, as well as concrete objects. Its semantic representations enable the agent to productively interpret novel word combinations, to apply known relations and modifiers to unfamiliar objects and to re-use knowledge pertinent to the concepts it already has in the process of acquiring new concepts.
Grounded Language Learning in a Simulated 3D World
While our simulations focus on language, the outcomes are relevant to machine learning in a more general sense. In particular, the agent exhibits active, multi-modal concept induction, the ability to transfer its learning and apply its knowledge representations in unfamiliar settings, a facility for learning multiple, distinct tasks, and the effective synthesis of unsupervised and reinforcement learning. At the same time, learning in the agent reflects various effects that are characteristic of human development, such as rapidly accelerating rates of vocabulary growth, the ability to learn from both rewarded interactions and predictions about the world, a natural tendency to generalise and re-use semantic knowledge, and improved outcomes when learning is moderated by curricula (Vosniadou and Brewer, 1992;Smith et al., 1996;Pinker, 1987Pinker, , 2009. Taken together, these contributions open many avenues for future investigations of language learning, and learning more generally, in both humans and artificial agents. activation function to yield a probability distribution over words in the vocabulary, and the negative log likelihood of the instruction word l t is computed as the loss. Note that this objective requires a single meaningful word to be extracted from the instruction as the target.
Grounded Language Learning in a Simulated 3D World
Value Description train steps 640m Theoretical maximum number of time steps (across all episodes) for which the agent will be trained. env steps per core step 4 Number of time steps between each action decision (action smoothing) num workers 32 Number of independent workers running replicas of the environment with asynchronous updating. unroll length 50 Number of time steps through which error is backpropagated in the core LSTM action module auxiliary networks vr batch size 1 Aggregated time steps processed by value replay auxiliary for each weight update. rp batch size 10 Aggregated time steps processed by reward prediction auxiliary for each weight update. lp batch size 10 Aggregated time steps processed by language prediction auxiliary for each weight update. tae batch size 10 Aggregated time steps processed by temporal AE auxiliary for each weight update.
Grounded Language Learning in a Simulated 3D World
language encoder encoder type BOW Whether the language encoder uses an additive bag-of-words (BOW) or an LSTM architecture.  Table 1: Agent hyperparameters that are fixed throughout our experimentation but otherwise not specified in the text.
Grounded Language Learning in a Simulated 3D World
Value Description auxiliary networks vr weight uniform(0.1, 1) Scalar weighting of value replay auxiliary loss relative to the core (A3C) objective. rp weight uniform(0.1, 1) Scalar weighting of reward prediction auxiliary loss. lp weight uniform(0.1, 1) Scalar weighting of language prediction auxiliary loss. tae weight uniform(0.1, 1) Scalar weighting of temporal autoencoder prediction auxiliary.
Grounded Language Learning in a Simulated 3D World
language encoder embed init uniform(0.5, 1) Standard deviation of normal distribution (mean = 0) for sampling initial values of word-embedding weights in L.
Grounded Language Learning in a Simulated 3D World
optimisation entropy cost uniform(0.0005, 0.005) Strength of the (additive) entropy regularisation term in the A3C cost function. learning rate start loguniform(0.0001, 0.002) Learning rate at the beginning of training annealed linearly to reach learning rate finish at the end of train steps. 
Interactive Fiction Games: A Colossal Adventure
Interactive fiction (IF) games are software environments in which players observe textual descriptions of the simulated world, issue text actions, and receive score as they progress through the story. As the excerpt of an IF game in Figure 1 shows, humans bring competencies in natural language understanding, commonsense reasoning, and deduction to bear in order to infer the context and objectives of a game. We believe that IF environments provide a good testbed for studying the development of these capabilities in artificial agents. Beyond games, real-world applications such as voice-activated personal assistants can also benefit from advances in these capabilities at the intersection of natural language understanding, natural language generation, and sequential decision making. These real world applications require the ability to reason with ungrounded natural language (unlike multimodal environments that provide visual grounding for language) and IF games provide an excellent suite of environments to tackle these challenges.
Interactive Fiction Games: A Colossal Adventure
From a machine learning perspective, Interactive Fiction games exist at the intersection of natural language processing and sequential decision making. Like many NLP tasks, they require natural language understanding, but unlike most NLP tasks, IF games are sequential decision making problems in which actions change the subsequent world states of the game and choices made early in a game may have long term effects on the eventual endings. Reinforcement Learning [Sutton and Barto, 1998] studies sequential decision making problems and has shown promise in visionbased [Jaderberg et al., 2016] and control-based [OpenAI et al., 2018] environments, but has less commonly been applied in the context of language-based tasks.
Interactive Fiction Games: A Colossal Adventure
The contributions of this paper are as follows: First, we introduce Jericho, a learning environment for human-made IF games. Second, we introduce a template-based action space that we argue is appropriate for language generation. Third, we conduct an empirical evaluation of learning agents across a large set of human-made games.
Interactive Fiction Games: A Colossal Adventure
Observation: This bedroom is extremely spare, with dirty laundry scattered haphazardly all over the floor. Cleaner clothing can be found in the dresser. A bathroom lies to the south, while a door to the east leads to the living room. On the end table are a telephone, a wallet and some keys. The phone rings.
Interactive Fiction Games: A Colossal Adventure
Observation: You pick up the phone. "Hadley!" a shrill voice cries. "Hadley, haven't you even left yet?? You knew that our presentation was at nine o'clock sharp! First the thing with the printers, now this -there won't even be enough left of you for Bowman to fire once he's done with you. Now get the hell down here!!" Action: Examine me Observation: You're covered with mud and dried sweat. It was an exhausting nightno wonder you overslept! Even making it to the bed before conking out was a heroic accomplishment. 
Interactive Fiction Games: A Colossal Adventure
From the perspective of reinforcement learning, IF games can be modeled as partially observable Markov decision processes (POMDPs) defined by (S, T, A, O, R). Observations o ∈ O correspond to the game's text responses, while latent states s ∈ S correspond to player and item locations, inventory contents, monsters, etc. Text-based actions a ∈ A change the game state according to an mostly-deterministic latent transition function T (s |s, a), and the agent receives rewards r from an unknown reward function R(s, a). To succeed in these environments, agents must generate natural language actions, reason about entities and affordances, and represent their knowledge about the game world. We present these challenges in greater detail:
Interactive Fiction Games: A Colossal Adventure
Combinatorial Action Space Reinforcement learning has studied agents that operate in discrete or continuous action space environments. However, IF games require the agent to operate in the combinatorial action space of natural language. Combinatorial spaces pose extremely difficult exploration problems for existing agents. For example, an agent generating a four-word sentence from a modest vocabulary of size 700, is effectively exploring a space of |700 4 | = 240 billion possible actions. Further complicating this challenge, natural language commands are interpreted by the game's parser which recognizes only a subset of possible commands. For example, out of the 240 billion possible actions there may be 100 million that are grammatical and successfully parseable, and out of these there may be only a hundred valid actions -contextually relevant actions that generate a change in world state.
Interactive Fiction Games: A Colossal Adventure
Commonsense Reasoning Due to the lack of graphics, IF games rely on the player's commonsense knowledge as a prior for how to interact with the game world. For example, a human player encountering a locked chest intuitively understands that the chest needs to be unlocked with some type of key, and once unlocked, the chest can be opened and will probably contain useful items. They may make a mental note to return to the chest if a key is found in the future. They may even mark the location of the chest on a map to easily find their way back. These inferences are possible for humans who have years of embodied experience interacting with chests, cabinets, safes, and all variety of objects. Artificial agents lack the commonsense knowledge gained through years of grounded language acquisition and have no reason to prefer opening a chest to eating it. Also known as affordance extraction [Gibson, 1977, Fulda et al., 2017, the problem of choosing which actions or verbs to pair with a particular noun is central to IF game playing. However, the problem of commonsense reasoning extends much further than affordance extraction: Games require planning which items to carry in a limited inventory space, strategically deciding whether to fight or flee from a monster, and spatial reasoning such as stacking garbage cans against the wall of a building to access a second-floor window.
Interactive Fiction Games: A Colossal Adventure
Knowledge Representation IF games span many distinct locations, each with unique descriptions, objects, and characters. Players move between locations by issuing navigational commands like go west. Due to the large number of locations in many games, humans often create maps to navigate efficiently and avoid getting lost. This gives rise to the Textual-SLAM problem, a textual variant of Simultaneous localization and mapping (SLAM) [Thrun et al., 2005] problem of constructing a map while navigating a new environment. In particular, because connectivity between locations is not necessarily Euclidean, agents need to detect when a navigational action has succeeded or failed and whether the location reached was previously seen or new. Beyond location connectivity, it's also helpful to keep track of the objects present at each location, with the understanding that objects can be nested inside of other objects, such as food in a refrigerator or a sword in a chest.
Interactive Fiction Games: A Colossal Adventure
Jericho is an open-source 1 Python-based IF environment, which provides an OpenAI-Gym-like interface [Brockman et al., 2016] for learning agents to connect with IF games. Jericho is intended for reinforcement learning agents, but also supports the ability to load and save game states, enabling planning algorithms Monte-Carlo Tree Search [Coulom, 2007] as well as reinforcement learning approaches that rely on the ability to restore state such as Backplay [Resnick et al., 2018] and GoExplore [Ecoffet et al., 2019]. Jericho additionally provides the option to seed the game's random number generator for replicability. 2 Jericho supports a set of human-made IF games that cover a variety of genres: dungeon crawl, Sci-Fi, mystery, comedy, and horror. Games were selected from classic Infocom titles such as Zork and Hitchhiker's Guide to the Galaxy, as well as newer, community-created titles like Anchorhead and Afflicted. Supported games use a point-based scoring system, which serves as the agent's reward. Beyond the set of supported games, unsupported games may be played through Jericho, without the support of score detection, move counts, or world-change detection.
Interactive Fiction Games: A Colossal Adventure
Template-Based Action Generation We introduce a novel template-based action space in which the agent first chooses an action template (e.g. put _ in _) and then fills in the blanks using words from the parser's vocabulary. Notationally, we employ u ⇐ w 1 , w 2 to denote the filling of template u with vocabulary words w 1 , w 2 . Jericho provides the capability to extract game-specific vocabulary and action templates. These templates contain up to two blanks, so a typical game with 200 templates and a 700 word vocabulary yields an action space of O(T V 2 ) ≈ 98 million, three orders of magnitude smaller than the 240-billion space of 4-word actions using vocabularly alone.
Interactive Fiction Games: A Colossal Adventure
World Object Tree The world object tree 3 is a semi-interpretable latent representation of game state used to codify the relationship between the objects and locations that populate the game world. Each object in the tree has a parent, child, and sibling. Relationships between objects are used to encode posession: a location object contains children corresponding to the items present at that location. Similarly, the player object has the player's current location as a parent and inventory items as children. Possible applications of the object tree include ground-truth identification of player location, ground-truth detection of the objects present at the player's location, and world-change-detection.
Interactive Fiction Games: A Colossal Adventure
Identifying Valid Actions Valid actions are actions recognized by the game's parser that cause changes in the game state. When playing new games, identifying valid actions is one of the primary difficulties encountered by humans and agents alike. Jericho has the facility to detect valid actions by executing a candidate action and looking for resulting changes to the world-object-tree. However, since some changes in game state are reflected only in global variables, it's rare but possible to experience false negatives. In order to identify all the valid actions in a given state, Jericho uses the following procedure:
Interactive Fiction Games: A Colossal Adventure
Algorithm 1 Procedure for Identifying Valid Actions 1: E ← Jericho environment 2: T ← Set of action templates 3: o ← Textual observation 4: P ← {p 1 . . . p n } Interactive objects identified with noun-phrase extraction or world object tree. 5: Y ← ∅ List of valid actions 6: s ← E.save() -Save current game state 7: for template u ∈ T do 8:
Interactive Fiction Games: A Colossal Adventure
for all combinations p 1 , p 2 ∈ P do 9:
Interactive Fiction Games: A Colossal Adventure
E.load(s) -Restore saved game state return Y Handicaps In summary, to ease the difficulty of IF games, Jericho optionally provides the following handicaps: 1) Fixed random seed to enforce determinism. 2) Use of Load, Save functionality. 3) Use of game-specific templates U and vocabulary V. 4) Use of world object tree as an auxillary state representation or method for detecting player location and objects. 5) Use of world-change-detection to identify valid actions. For reproducibilty, we report the handicaps used by all algorithms in this paper and encourage future work to do the same. 
Interactive Fiction Games: A Colossal Adventure
In this section we present three agents: a choice-based single-game agent (DRRN), a parser-based single-game agent (TDQN), and a parser-based general-game agent (NAIL). Single-game agents are trained and evaluated on the same game, while general game playing agents are designed to play unseen games.
Interactive Fiction Games: A Colossal Adventure
The input encoder φ o converts observations into vectors using the following process: Text observations are tokenized by a SentencePiece model [Kudo and Richardson, 2018] using an 8000-large vocabulary trained on strings extracted from sessions of humans playing a variety of different IF games 4 . Tokenized observations are processed by separate GRU encoders for the narrative (i.e., the game's response to the last action), description of current location, contents of inventory, and previous text command. The outputs of these encoders are concatenated into a vector ν o . DRRN and Template-DQN build on this common input representation.
Interactive Fiction Games: A Colossal Adventure
DRRN The Deep Reinforcement Relevance Network (DRRN) He et al. [2016] is an algorithm for choice-based games, which present a set of valid actions A valid (s) at every game state. We implement DRRN using a GRU φ act (a) to encode each valid action into a vector ν a , which is concatenated with the encoded observation vector ν o . Using this combined vector, DRRN then computes a Q-Value Q(o, a) estimating the total discounted reward expected if action a is taken and π DRRN is followed thereafter. This procedure is repeated for each valid action a i ∈ A valid (s). Action selection is performed by sampling from a softmax distribution over Q(o, a i ). The network is updated by sampling a minibatch of transitions (o, a, r, o , A valid (s )) ∼ D from a prioritized replay memory  and minimizing the temporal difference error δ = r + γ * max a Q(o , a ) − Q(o, a). Rather than performing a separate forward pass for each valid action, we batch valid-actions and perform a single forward pass computing Q-Values for all valid actions.
Interactive Fiction Games: A Colossal Adventure
DRRN uses Jericho's world-change-detection handicap to identify valid actions following Algorithm 1. Additionally, it uses Jericho's Load, Save handicap to acquire additional observations without changing the game state: an inventory command is issued for the inventory observation o inv and a look command is issued for the location description o desc .
Interactive Fiction Games: A Colossal Adventure
Template- DQN Narasimhan et al. [2015] introduced LSTM-DQN, an agent for parser-based games that handles the combinatorial action space by generating verb-object actions from a pre-defined set verbs and objects. Specifically, LSTM-DQN uses two output layers to estimate Q-Values over possible verbs and objects. Actions are selected by pairing the maximally valued verb with the maximally valued noun.
Interactive Fiction Games: A Colossal Adventure
We introduce Template-DQN (TDQN) which extends LSTM-DQN by incorporating template-based action generation. This is accomplished using three output heads: one for estimating Q-Values over templates Q(o, u); u ∈ T and two for estimating Q-Values Q(o, p 1 ), Q(o, p 2 ); p ∈ V over vocabulary to fill in the blanks of the template. Even in template-action spaces, exploration remains an issue; The largest action space considered in the original LSTM-DQN paper was 222 actions (6 verbs and 22 objects). In contrast, Zork1 has 237 templates with a 697 word vocabulary, yielding an action space of 115 million. Computationally, this space is too large to naively explore as the vast majority of actions will be un-grammatical or contextually irrelevant. To help guide the agent towards valid actions, we introduce a supervised binary-cross entropy loss with targets for each of the templates and vocabulary words appearing in the list of valid actions. The idea behind this loss is to nudge the agent towards the valid templates and words. This loss is evenly mixed with the standard temporal difference error during each update. Due to the supervised loss, TDQN uses the same set of handicaps as DRRN.
Interactive Fiction Games: A Colossal Adventure
NAIL NAIL [Hausknecht et al., 2019b] is the state-of-the-art agent for general interactive fiction game playing [Atkinson et al., 2018]. Rather than being trained and evaluated on a single game, NAIL is designed to play unseen IF games where it attempts to accumulate as much score as possible in a single episode of interaction. Operating with no handicaps, NAIL employs a set of manually-created heuristics to build a map of objects and locations, reason about which actions were valid or invalid, and uses a web-based language model to decide how to interact with objects. NAIL serves as a point of reference for future work in general IF game playing.
Interactive Fiction Games: A Colossal Adventure
Implementations of DRRN and TDQN are available at https://github.com/microsoft/tdqn. NAIL is available at https://github.com/microsoft/nail_agent.
Interactive Fiction Games: A Colossal Adventure
We evaluate the agents across a set of thirty-two Jericho-supported games with the aims of 1) showing the feasibility of reinforcement learning on a variety of different IF games, 2) creating a reproducible benchmark for future work, 3) investigating the difference between choice-based and template-based agents, and 4) comparing performance of the general IF game playing agent (NAIL), single-game agents (DRRN and TDQN), and a random agent (RAND) which uniformly sample commands from a set of canonical actions: {north, south, east, west, up, down, look, inventory, take all, drop, yes}.
Interactive Fiction Games: A Colossal Adventure
Five separate DRRN and TDQN agents were trained for each game. No environment seeds were set so environments remained stochastic throughout. We compute the score for each agent by averaging return over the last hundred episodes of learning. Hyperparameters for TDQN and DRRN were optimized on Zork1 then held fixed across games. Results in Table 1, supported by learning curves in Figure 3 show that reinforcement learning is a viable across many of the games.
Interactive Fiction Games: A Colossal Adventure
In order to quantify overall progress towards story completion, we normalize agent score by maximum possible game score and average across all games. The resulting progress scores are as follows: RANDOM 1.8%, NAIL 4.9%, TDQN 6.1%, and DRRN 10.7% completion. Comparing the different agents, the random agent shows that more than simple navigation and take actions are needed to succeed at the vast majority of games. Comparing DRRN to TDQN highlights the utility of choice-based game playing agents who need only estimate Q-Values over pre-identified valid-actions. In contrast, TDQN needs to estimate Q-Values over the full space of templates and vocabulary words. As a result, we observed that TDQN was more prone to over-estimating Q-Values due to the Q-Learning update computing a max over a much larger number of possible actions.
Interactive Fiction Games: A Colossal Adventure
Comparing NAIL to single-game agents: NAIL performs surprisingly well considering it uses no handicaps, no training period, and plays the game for only a single episode. It should be noted that NAIL was developed on many of the games used in this evaluation, but contains no game-specific information. The fact that the reinforcement learning agents outperform NAIL serves to highlight the difficulty of engineering a general-purpose IF agent as well as the promise of learning policies from data rather than hand-coding them.
Interactive Fiction Games: A Colossal Adventure
All algorithms have a ways to go before they are solving games of even average difficulty. None of the agents were able to get any score on five of the games. Games like Anchorhead are highly complex and others pose difficult exploration problems like 9:05 which features only a single terminal reward indicating success or failure at the end of the episode. Additional experiment details and hyperparameters are located in the supplementary material. 
Interactive Fiction Games: A Colossal Adventure
Jericho supports a variety of games, covering a diverse set of structures and genres. These games provide different challenges from the perspective of reinforcement learning based agents. Table 2 categorizes the games into three difficulty tiers:
Interactive Fiction Games: A Colossal Adventure
Possible Games are games that learning agents have a credible chance to solve in the near future. These games fall on the lower end on the difficulty spectrum and serve as good initial testbeds for developing new agents because they feature frequent rewards, are solvable by basic navigation and interaction actions, and are largely devoid of the complexities like dialog and inventory limits.
Interactive Fiction Games: A Colossal Adventure
Detective is one of the easiest games and existing agents can finish the game albeit without collecting all possible points. Acorncourt also serves as a sanity check, albeit a more difficult one-with the DRRN outperforming all other agents. This game, although short, requires a high proportion of complex actions, which makes action generation-as in the case of TDQN-more difficult.
Interactive Fiction Games: A Colossal Adventure
Omniquest is a dungeon-crawler style game where TDQN outperforms the rest of the agents due to the relatively smaller number of valid templates as compared to valid actions-i.e. many valid actions come from the same template-TDQN has a smaller effective search space than DRRN.
Interactive Fiction Games: A Colossal Adventure
Difficult Games feature sparser rewards, more complex puzzles and interactions, and require more steps to solve. Current learning agents are able to collect some score on these games, but we expect  to solve in the short term. Difficult games (middle): current agents can make progress on, but will require significant advances to solve. Extreme games (bottom): highly challenging for humans and inconceivable for current RL agents to solve. Action space is the size of the template-action space = |T V 2 | in millions of actions. Solution Length is the number of steps needed for the walkthrough to complete the game. Average Steps per Reward is the average number of steps the agent needs to take between getting rewards. Stochastic indicates if the game's solution changes with the random seed. Dialog indicates that it's necessary to speak with or issue commands to another character such as "john, turn off light." Darkness is a common trope in IF games that limits explorable areas until a light source such as a lantern or torch is found. Nonstandard Actions are actions that wouldn't be found in an English dictionary -such as casting a Frotz spell. Inventory Limit restricts the number of items carried by the player and requires decisions about which items to discard and keep. that new insights and algorithms will be necessary to solve these games. Games like Zork1 typify this category: existing agents can effectively explore the opening areas, but quickly plateau when going underground due to challenges like darkness, mazes, and fighting. For example, darkness requires the player to have a light source such as the lantern equipped and activated in order to move. This challenge is further complicated by limited inventory space, which prevents the player from picking up all the items they encounter and instead forces them to reason about which items are essential and which can be discarded.
Interactive Fiction Games: A Colossal Adventure
Extreme Games are those featuring extensive puzzles often involving high-level, creative problem solving that challenges even skilled human players. These games are the most complex in terms of the length of required solutions, and difficulty of puzzles, and non-standard nature of actions. A prime example is Anchorhead, a Lovecraftian horror complex enough that it has been the subject of prior work on cognition in script learning and drama management [Giannatos et al., 2011]. Its puzzles features physical reasoning such as arranging trash cans in an alleyway in order to access a second floor window, as well as long-range dependencies between when hints are given and when they're needed. Our agents are yet to accumulate any score on this game. Another category of extreme games is the Enchanter Trilogy (Enchanter, Sorcerer, and Spellbreaker) which was developed by Infocom as a followup to the Zork triology. In addition to larger vocabularies, these games require the player to cast spells like Frotz and Rezrov, the effects of which must be learned by playing the game. We expect that such non-standard actions will require agents adaptively learn the names and effects of these spells and reason how to use them on the fly.
Interactive Fiction Games: A Colossal Adventure
Unsupervised learning: DRRN and TDQN agents were trained and evaluated on individual games. While sufficient for a proof-of-concept, this evaluation falls short of demonstrating truly general IF game playing. To this end, it's valuable to evaluate agents on a separate set of games which they have not been trained on. In the Jericho framework, we propose to use the set of Jericho supported games as a test set and the larger set of unsupported games available at https://github.com/ BYU-PCCL/z-machine-games as the training set. In this paradigm, it's necessary have a strong unsupervised learning component to guide the agent's exploration and learning since unsupported games do not provide rewards, and in fact many IF games do not have scores. We hypothesize that surrogate reward functions, like novelty-based rewards [Bellemare et al., 2016, Pathak et al., 2017, will be useful for discovering locations, successful interactions, and objects.
Interactive Fiction Games: A Colossal Adventure
Better Template-based Agents: There are several directions for creating better template-based agents by improving on the limitations of TDQN: When generating actions, TDQN assumes independence between templates and vocabulary words. To understand the problem with this assumption consider the templates go _ and take _ and the vocabulary words north, apple. Independently, take and north may have the highest Q-Values but together yield the invalid action take north. Conditional generation of words based on the chosen template may improve the quality of TDQN's actions.
Interactive Fiction Games: A Colossal Adventure
Second, recent work on transformer-based neural architectures has yielded impressive gains in many NLP tasks , including text-adventure game dialogues [Urbanek et al., 2019]. We expect these advances may be applicable to human-made IF games, but will need to be adapted from a supervised training regime into reinforcement learning.
Interactive Fiction Games: A Colossal Adventure
Interactive Fiction games are rich narrative adventures that challenge even skilled human players. In contrast to other video game environments, IF games stress natural language understanding and commonsense reasoning, and feature combinatorial action spaces. To aid in the study of these environment, we introduce Jericho, an experimental platform with the key of feature of extracting game-specific action templates and vocabulary. Using these features, we proposed a novel templatebased action space which serves to reduce the complexity of full scale language generation. Using this space, we introduced the Template-DQN agent TDQN, which generates actions first by selecting a template then filling in the blanks with words from the vocabulary.
Interactive Fiction Games: A Colossal Adventure
We evaluated TDQN against DRRN, a choice-based agent, and NAIL, a general IF agent. The fact that DRRN outperformed TDQN illustrates the difficulty of language generation, even with templates. However, TDQN showed encouraging progress, outperforming both NAIL and the random agent. However, in many ways these agents represent very different training paradigms, sets of assumptions, and levels of handicap. Rather than comparing agents against each other we aim to provide these scores as benchmark results for future work in all three categories of IF game playing. We believe Jericho can help the community propel research on language understanding agents and expect these environments can serve the community as benchmarks for years to come. 
Interactive Fiction Games: A Colossal Adventure
Episodes are terminated after 100 valid steps or game over/victory. Since agents that frequently decode invalid actions would never make it more than few steps past the start of the game, we only count valid-actions against the hundred step limit. DRRN and TDQN are trained individually on each game. DRRN was trained for 100,000 updates, using data collected from 16 environments in parallel. This corresponds to 1.6 million environment interactions. TDQN was trained for 1 million environment steps using a single environment. Hyperparameters for DRRN and TDQN were tuned on the game of Zork1 and held constant across all other games. Both the random agent and NAIL are given 1000 total environment steps. Final reported scores are an average over 5 runs of each algorithm. 
Interactive Fiction Games: A Colossal Adventure
DRRN playing Zork1