How has the game Dungeons and Dragons been used to study problems in Natural Language Processing?
Dungeons and Dragons as an NLP Challenge.Several studies have used Dungeons and Dragonsto study various problems in NLP such as characterunderstanding (Louis and Sutton, 2018), controlleddialogue generation (Si et al., 2021; Callison-Burchet al., 2022), and description generation (Newmanand Liu, 2022).  Reinforcement learning has alsobeen  applied  to  study  the  goal-driven  aspect  ofD&D (Martin et al., 2018)
Telling Stories through Multi-User Dialogue by Modeling Character Relations
This paper explores character-driven story continuation, in which the story emerges through characters’ first- and second-person narration as well as dialogue—requiring models to select language that is consistent with a character’s persona and their relationships with other characters while following and advancing the story. We hypothesize that a multi-task model that trains on character dialogue plus character relationship information improves transformer-based story continuation. To this end, we extend the Critical Role Dungeons and Dragons Dataset (Rameshkumar and Bailey, 2020)—consisting of dialogue transcripts of people collaboratively telling a story while playing the role-playing game Dungeons and Dragons—with automatically extracted relationships between each pair of interacting characters as well as their personas. A series of ablations lend evidence to our hypothesis, showing that our multi-task model using character relationships improves story continuation accuracy over strong baselines.
[{"first":"Wai Man","last":"Si","middle":[]},{"first":"Prithviraj","last":"Ammanabrolu","middle":[]},{"first":"Mark","last":"Riedl","middle":[]}]
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
An essential aspect to understanding narratives is to grasp the interaction between characters in a story and the actions they take. We examine whether computational models can capture this interaction, when both character attributes and actions are expressed as complex natural language descriptions. We propose role-playing games as a testbed for this problem, and introduce a large corpus of game transcripts collected from online discussion forums. Using neural language models which combine character and action descriptions from these stories, we show that we can learn the latent ties. Action sequences are better predicted when the character performing the action is also taken into account, and vice versa for character attributes.
[{"first":"Annie","last":"Louis","middle":[]},{"first":"Charles","last":"Sutton","middle":[]}]
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
AI researchers have posited Dungeons and Dragons (D&D) as a challenge problem to test systems on various language-related capabilities. In this paper, we frame D&D specifically as a dialogue system challenge, where the tasks are to both generate the next conversational turn in the game and predict the state of the game given the dialogue history. We create a gameplay dataset consisting of nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns, 500,000 dice rolls, and 58 million words. We automatically annotate the data with partial state information about the game play. We train a large language model (LM) to generate the next game turn, conditioning it on different information. The LM can respond as a particular character or as the player who runs the game—i.e., the Dungeon Master (DM). It is trained to produce dialogue that is either in-character (roleplaying in the fictional world) or out-of-character (discussing rules or strategy). We perform a human evaluation to determine what factors make the generated output plausible and interesting. We further perform an automatic evaluation to determine how well the model can predict the game state given the history and examine how well tracking the game state improves its ability to produce plausible conversational output.
[{"first":"Chris","last":"Callison-Burch","middle":[]},{"first":"Gaurav Singh","last":"Tomar","middle":[]},{"first":"Lara","last":"Martin","middle":[]},{"first":"Daphne","last":"Ippolito","middle":[]},{"first":"Suma","last":"Bailis","middle":[]},{"first":"David","last":"Reitter","middle":[]}]
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
We examine the task of generating unique content for the spell system of the tabletop roleplaying game Dungeons and Dragons Fifth Edition using several generative language models. Due to the descriptive nature of the game Dungeons and Dragons Fifth Edition, it presents a number of interesting avenues for generation and analysis of text. In particular, the “spell” system of the game has interesting and unique characteristics as it is primarily made up of high level and descriptive text but has many of the game’s main rules embedded with that text. Thus, we examine the capabilities of several models on the task of generating new content for this game, evaluating the performance through the use of both score-based methods and a survey on the best performing model to determine how the generated content conforms to the rules of the game and how well they might be used in the game.
[{"first":"Pax","last":"Newman","middle":[]},{"first":"Yudong","last":"Liu","middle":[]}]
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Automated storytelling can be thought of as creative, long-from text generation and understanding-requiring explicit long-term memory, consistency, and creativity among other pre-requisites. Most modern (neural) automated storytellers are plot-driven and frame the task in terms of sequentially generating plot points that narrate the story in third-person (Kiros et al., 2015;Mostafazadeh et al., 2016;Fan et al., 2018). This approach does not generally place much weight on individual characters or their interactions-information known to be critical for creating stories (Riedl and Young, 2010).
Telling Stories through Multi-User Dialogue by Modeling Character Relations
We are inspired by the idea of characterdriven and emergent storytelling wherein narrative emerges through characters' interactions as seen Summary They wake up in the morning, preparing for the coming battle. Scanlan turns them all into Ravenites with light clothing. The sleet storm is starting. ...
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Bundle up!
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Okay. How will we know when it's time for me to release? We have to wait for Tooma to go report.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Is Vorugal back? He's back. Scanlan:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
I assume. Vexahlia:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Do we see Larkin around? DM:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
No, you do not see Larkin around.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Vax , do you want to go look? Vaxildan:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
For Larkin?
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Scanlan:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
No Larkin. I attempt to see see if Tooma is coming. I don't want to release this thing before Tooma is there reporting to Vorugal. Vaxildan:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
(Grog voice) Six. It said six. in Table 1. In addition to the challenges faced by automated storytellers, a character-driven storytelling system must produce language while simultaneously: (1) keeping each character's personas consistent while acting; (2) keeping track of relationships between characters that will affect their interactions; and (3) follow and logically advance the plot of the story.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
To better explore how to give automated systems these two abilities, we focus on the task of story continuation solely through dialogue-i.e. picking the next character response that best continues a story. The task and data are seen in Table 1. We build off the Critical Role Dungeons and Drag-ons Dataset or CRD3 (Rameshkumar and Bailey, 2020), a unique dataset that contains dialogue transcripts of a small group of around six players roleplaying various characters while playing the table top role-playing game Dungeons and Dragonstheir adventures and interactions forming a narrative that stretches hundreds of chapters, with each chapter forming a subplot. The original dataset was intended to be used for abstractive summarization and contains ground-truth summaries for each chapter. To better suit our purpose of studying character-driven storytelling, we automatically augment the dataset with information regarding character persona as well as relationship types between pairs of characters (friends, enemies, etc.) by clustering crowdsourced descriptions of character interactions from the Critical Role Wiki. 1 This extended dataset lets us break down the problem of character-driven story continuation into two sub-tasks corresponding to the three challenges mentioned earlier in terms of interacting within the confines of a story while staying consistent with respect to character personas and relationships. We show that training a system to optimize for both of these sub-tasks significantly improves story continuation accuracy.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Our work's two primary contributions are thus: (1) the extension to CRD3 enabling a study of character-driven storytelling and the corresponding methodology used; and (2) a multi-task learning system that leverages character relation and persona information to better complete stories.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Storytelling. Storytelling systems that use symbolic planning (Lebowitz, 1987;Gervás et al., 2005;Porteous and Cavazza, 2009;Riedl and Young, 2010;Ware and Young, 2011) focused on ensuring coherence and consistency of plot through explicitly listed rules in the form of pre-and postconditions, often requiring extensive knowledge engineering. Modern neural language-model based approaches generally attempt to learn to tell plotdriven stories from a corpus of stories via learning objectives that optimize reconstructing the story itself (Kiros et al., 2015;Roemmele and Gordon, 2018;Khalifa et al., 2017;Fan et al., 2018). In particular, a two-step process in which the high level plot is first generated, followed by filling in rest of the story constrained to the plot has emerged (Martin et al., 2017Ammanabrolu et al., 2020;Tambwekar et al., 2019;Yao et al., 2019;Ippolito et al., 2019). Ammanabrolu et al. (2021) look at plot generation from a character-driven perspective using commonsense knowledge, though do not model character interactions at all. Closely related to the spirit of our task is the Story Cloze test (Mostafazadeh et al., 2016), which measures the ability of a model to correctly predict the end of a story. Like the other works mentioned here, however, this task does not require dialogue or other forms of character interactions.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Dialogue. Contemporary neural dialogue retrieval systems, both chit-chat and goal-oriented, more explicitly model agent interactions than most storytelling systems (Henderson et al., 2014;El Asri et al., 2017). Particularly relevant to our work are dialogue systems that attempt to model and stay consistent with an agent's persona, such as Persona Chat (Zhang et al., 2018), or using further contextual information such as setting in addition to character personas using a crowd-sourced fantasy text-game such as LIGHT (Urbanek et al., 2019). None of these works, however, have any notion of story or plot, often using significantly less long-term context than most storytelling systems.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
This section first describes the automated extensions to the CRD3 dataset, specifically information on character relationships, followed by the multitask learning setup and transformer architecture that leverage the new data for story continuation.  
Telling Stories through Multi-User Dialogue by Modeling Character Relations
CRD3, as originally seen in Rameshkumar and Bailey (2020), contains two seasons of 159 transcribed Critical Role episodes, consisting of 398,682 turns in total. It further contains 34,243 ground truth human-written summary dialogue chunks that abstractively summarize dialogue chunks. The chunks themselves consist of a sequence of dialogue and first-and second-person narration turns that form a semantically cohesive unit-with the end of a chunk signifying the completion of a subplot or change in location. Table 2 provides statistics for the number of chunks in the train, as well as the average number of character turns and number of characters within a chunk.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
To enable a more effective study of characterdriven storytelling using this dataset, we automatically extend CRD3 by adding descriptions of character relations from the Critical Role Wiki. These descriptions are free form text and often summarize character emotions during their interactions with another character. To condense them down, we cluster the character relation descriptions in an unsupervised fashion by calculating the vectorized TF-IDF representation of the description and applying the K-means algorithm. Varying the number of clusters changes the qualitative information conveyed by the cluster. For example, if we set the number of clusters to three, we can then also use the popular sentiment analysis tool VADER (Hutto and Gilbert, 2014) to provide human interperable relationship labels for each of the three clusterspositive, negative, or neutral as seen in Table 1. We specifically focus on incorporating these 3 relation types into our models. These relationship labels are attached to every dialogue chunk based on the characters appearing in that chunk. Further information regarding clusters is found in Appendix A.3.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Based on the hypothesis that modeling character interaction information is critical for our overall task of character-driven storytelling, our system optimizes for two sub-tasks: next character prediction and story continuation. The next character prediction task can be summarized as: given current context, predict the next character who will act or speak-providing a proxy for judging who is most likely to respond to the current character in a multi-character setting. Similarly, the story continuation task refers to predicting the next character response that continues the story given the same context. The context itself contains information regarding: (1) a summary of the story so far using the dialogue summary chunks provided in CRD3 and described in Section 3.1, (2) pairwise relationship cluster labels between all characters within the dialogue chunk, and (3) the last n-turns of character interactions.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Our model's architecture is shown in Figure 1.  It is inspired by the bi-encoder featured in Urbanek et al. (2019). In this model, two separate transformers are used to produce vector representations for the input context and each candidate utterance for the response retrieval task. All candidates are scored by via dot product between their vector representations and the context representation and trained using a ranking loss L rank . For the task of next character prediction, we use the same vector representation for the context and pass it through an additional linear layer with softmax layer to predict the correct character from the list of all possible characters. This sub-task uses a cross entropy loss L cls . The entire system is trained jointly by optimizing L = λ 1 L rank + λ 2 L cls for some hyperparameters λ i . By virtue of the architecture, network parameters are shared between the tasks.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
We conduct two ablations studies that analyze:
Telling Stories through Multi-User Dialogue by Modeling Character Relations
(1) the complexity of performing character-driven story continuation on the dataset; and (2) the effectiveness of imbuing the model with relation information via input context and multi-task learning. Our base transformer model that we build off of in each of these is the bi-encoder ranker described in Section 3.2. The transformer encoder is a similar architecture as BERT (Devlin et al., 2019), with   256 million total parameters, and is pre-trained using the Reddit dataset extracted and made available on pushshift.io (Baumgartner et al., 2020) seen in Roller et al. (2020). This dataset has been shown to result in an improved understanding of conversational natural language (Yang et al., 2018;Mazaré et al., 2018). Further hyperparameter and training details are shown in Appendix A.2. For story continuation report standard retrieval metrics of Hits@N , where we measure the ability of the model to output the gold standard dialogue candidate in the top-N of the given candidates. For character prediction, we report F1 weighted by the number of instances of each character type.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
The first set of ablations measures performance on each of the two sub-tasks as a function of historical context required in an attempt to assess the complexity of the CRD3 extended dataset and its suitability for exploring character-driven storytelling. Recall that the CRD3 dataset provides summaries for each separate dialogue chunk. In Table 4, we vary the number of prior chunks of such summaries used as input context to the model and measure performance on each of the sub-tasks after training the model jointly on both sub-tasks.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
The trends shown in Table 4 are quite clearindicating that, overall, the CRD3 dataset requires very long contexts to ensure effective performance. On average, across both evaluation tasks performance gain between using a single historical context chunk and using two is greater than the corresponding differences when using even more chunks. Additionally, performance continues to rise with added historical context up to the maximum context length we tested of 10. We note that this is a significantly greater amount of context than generally required for state-of-the-art chit-chat dialogue datasets (Roller et al., 2020) as well as prior story completion datasets such as ROC Stories (Mostafazadeh et al., 2016), reinforcing our hypothesis that the CRD3 dataset is well suited to enabling character-driven storytelling by focusing on interactions requiring long-term memory.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
These ablations focus on analyzing the effects of our methods to imbue the agent with relationship and character information, specifically including the relationship cluster labels in the input and multitask training. Table 3 outlines these results when evaluated on both the character prediction and story continuation sub-tasks with different: (1) inputs types-with base referring to only character interactions and additional information as seen in Figure 1;and (2) training methods-single referring to training on only the evaluation task and multi to jointly training on both tasks.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
We would first like to note that we use the same relationship labels for characters through the entire story-i.e. across all the dialogue chunks. Our approach intuitively averages the relationship type between characters through time-e.g. characters that are friends at first and then become enemies will have a neutral label throughout all the story. While more fine grained relationship labels that do not perform such averaging might perform better, they would also require extensive additional human annotations to track relationships through time.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
For character prediction, the Base+Summary multi-task and Base+Relations single-task models perform best though are closely comparable to the Base+Summary+Relations multi-task model. For story continuation, the Base+Summary+Relations multi-task model outperforms all others. In all story continuation experiments, multi-task trained models outperform their counterpart single-task trained model. Through these results, we can infer that imbuing character relationship information through both input relationship cluster information as well as next character prediction helps models continue stories-while staying consistent with a particular character's persona-more accurately.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
We hypothesized that injecting models with information on relationships between characters would improve their ability to complete character-driven stories. A series of ablation studies support this, with a key insight being that a particularly efficient way of giving story continuation models this information is by multi-task training them on both character dialogue and relationship information automatically extracted from online sources.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Our work on character-driven storytelling has potential implications extending to the creation of learning agents that communicate using natural language, especially those requiring an agent to stay consistent with a character or persona throughout an interaction. As our system is trained entirely using a dataset collected from character interactions of a set of players role-playing in a fantasy Dungeons and Dragons world, we are prone to echoing biases found in the data. Some of these biases are necessary for effective story continuation, enabling a reader to identify the genre and conveying thematic information. Others may potentially involve non-normative language usage-acceptable in a fantasy world but inappropriate in the real world. Restricting our system to story continuation through a retrieval mechanism as opposed to generating text mitigates, though do not eliminate some of these biases. We urge future researchers and application developers that use automated storytelling techniques to similarly clarify the origins and methodology behind the creation of delivered story content.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Relations <Scanlan, neutral, Vexahlia>, <Grog, neutral, Scanlan>, ...
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Scanlan deceives the clasp leader with a blue gem that can grant one wish if they say the password while holding the gem. He gives the leader the gem and promises to give him the password if they can visit riskel. The leader reveals the clasp helped riskel prepare for his escape. ... Keyleth: okay !
Telling Stories through Multi-User Dialogue by Modeling Character Relations
He looks over at the gentleman who inspected it earlier and nods his head. "accepted." and they continue walking forward. Grog: Lucky fucking druid. DM: It is the piece you put in the actual-
Telling Stories through Multi-User Dialogue by Modeling Character Relations
It's a blue shard that we found in-long, long ago-it's real crystal and it's real magic. DM: Yes. I know what that is. Scanlan: Because I don't.
Telling Stories through Multi-User Dialogue by Modeling Character Relations
DM: Well, it was sufficient upon inspection for this. Scanlan: Okay. Vexahlia: Whoa, I think it opens a portal to another plane. Scanlan: I don't know what it is, but it's magic. 
Telling Stories through Multi-User Dialogue by Modeling Character Relations
Rejoining the party, Vex wonders aloud why desmond is still in the cell. Percy responds that it was originally for his own protection, but that since the problem has been taken care of, it is a precaution that is no longer needed. ... Vexahlia: Are there days of the week? what is a weekend? Keyleth: Yeah, There's days of the week . Scanlan: What is this world? How does time work here?
Telling Stories through Multi-User Dialogue by Modeling Character Relations
There are days of the week, I'm not gon na go into the specifics of it because I'm working on it. This question hasn't really arisen before and I probably should figure that out. It 's the equivalent of a thursday. Scanlan: It's always thursday. 
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Imagine a giant, a dwarf, and a fairy in a combat situation. We would expect them to act differently, and conversely, if we are told of even a few actions taken by a character in a story, we naturally start to draw inferences about that character's personality. Communicating narrative is a fundamental task of natural language, and understanding narrative requires modelling the interaction between events and characters. 2 In this paper, we propose that collaborativelytold stories that arise in certain types of games provide a natural test bed for the problem of inferring interactions between characters and actions in narratives. We present a corpus of role-playing game (RPG) transcripts where characters and action sequences are described with complex natural language texts. Table 1 shows an example character Character description Name: Ana Blackclaw; Age: 27; Gender: Female Appearance: Standing at a mighty 6'5, she is a giant among her fellow humans. Her face is light, though paler than the average man or woman's, and is marked by scars. ... Her body is muscular, as it would have to be to carry both her armor and the hammer. Her light grey eyes nearly always keep a bored expression. Her canines seem a tad larger than the normal person's. Preferred Weapon: Hammer. Preferred Armor: Heavy. Gift: Binoculars. Darksign: No. Action description She stopped dead in her tracks as the hissing began. A grumble escaped her as it did so, and she looked over to make sure the other woman was doing fine. Seeing that all was not entirely well, she allowed herself to slide down, her hand gripping the slope side once more to slow herself. Once that was accomplished, she reached out and grabbed the back of the girl's neck, pulling her back to steady herself. The giant remained silent as she did so, and then glanced over to the nearby skeletons. They would be upon them soon. Her grip tightened on the hammer as she glanced from side to side. It would not be a fun fight. description, and an action text for the same character. This example shows how the ties between characters and their actions are subtly present in the text descriptions, and learning the latent ties between them is a difficult task. Based on our corpus, and using neural language models, this work demonstrates an initial success on this problem.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
The ability to understand and generate narratives is a useful skill for natural language systems, for example, to plan a coherent answer to a question, or to generate a summary of a document. Prior work on narrative processing has focused on inducing disjoint sets of character and event types (as topic models), capturing the relationship between characters in the same story, or extracting character-action pairs as low level noun-verb tuples. However, these models do not aim to match or infer characters and actions from each other.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We make two contributions towards closing this gap. We introduce a corpus of thousands of RPG transcripts and demonstrate predictive cues between characters and actions by building neural language models with facility for adding side information. We show that a language model over action text obtains lower perplexity when we also make available a representation of the character who produced each token. Likewise, a language model for character descriptions benefits from information about the actions the character made. Our findings open up new possibilities for making sophisticated inference over narrative texts.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
In work on narratives, both characters and actions have received significant attention, albeit separately. There is work on inducing types of characters (Bamman et al., 2013(Bamman et al., , 2014 or relationships between characters (Chang et al., 2009;Elson et al., 2010;Iyyer et al., 2016). Often these approaches are based on probabilistic topic models or more recently distributed word representations computed by neural networks. Others focus on learning regular and repetitive event sequences in stories (Chambers and Jurafsky, 2009;McIntyre and Lapata, 2009), together with some information about the agent of the actions. These extractions are fairly lowlevel, in the form of noun-verb pairs. There are also models for clustering stories either based on their characters (Frermann and Szarvas, 2017), or sentiment and topic (Elsner, 2012(Elsner, , 2015.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
The above approaches mine types of actions or characters. This work focuses on infering the latent ties between actions and characters, and whether one aspect can help predict the other. Flekova and Gurevych (2015) present recent work related to this latter idea. They classify characters based on their speech and actions into an introvert or extrovert class. In contrast, we focus on attributes of characters and actions beyond such coarse traits, and when these attributes are expressed as complex descriptions.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Traditionally, RPGs are played orally with players seated around a table. But there are also online forums where users play RPGs by posting text descriptions instead.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We collected a corpus of RPG threads from one such website roleplayerguild.com. Here each game play is recorded in two threads. In one of these, each player posts a detailed text description of the role (character) she is going to play in the game, which we call a character description. This description includes the character's physical appearance, personality, family background, as well as special and supernatural powers, and possessions. A second thread consists of the actual game play where each player contributes a post when his turn comes. Each post describes how the character that is assumed by that specific player responds to the game situation. Thus the story develops collaboratively. We call each post in the story thread an action description. An example from our corpus of a character description and an action description is shown in Table 1.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
A noteworthy aspect of these RPGs is that character attributes are determined by writing the descriptions before the game starts. The story thread itself then focuses predominantly on the actions and does not reiterate character attributes. Moreover, we know unambiguously which character is associated with each action post. Such mapped pairs of clean character descriptions and associated actions would be difficult to obtain from novels or other stories without sophisticated analysis.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Our corpus contains 1,544 RPGs spanning a variety of themes-fantasy, apocalyptic, romance, anime, military, horror, and adventure. There are a total of 56,576 posts, comprising of 25.3M tokens. The maximum number of posts in a story is 753, minimum 2, and the average is 26. Note that many stories are in progress and some are long running. There are 9,771 unique characters in the corpus, and their descriptions amount to 8.5M tokens. There is a minimum of 1, average 6, and maximum 24 characters in a single story.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Even though each character or action description focuses on a single character, it nevertheless contains descriptions of background settings of the scene, and interactions of other characters (eg. descriptions of the parents of a character). Hence we preprocess the texts to only retain parts most related to the character in focus. To this end, in character descriptions, we only keep those sentences which mention the character's name or the personal pronouns 'he' or 'she'. The use of pronouns reflects an intuition that since the description is of one key character, the pronoun is most likely to refer to this salient entity. We also take sentences which mention personality describing words such as 'personality', 'skill', 'specialize', 'ability', 'profile', 'talent', etc., even when they do not contain the name. For action descriptions, we only keep sentences which start with the character's name. We do not use pronouns since action text may refer to other salient characters as well.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Finally, we replace the main character (contributor) of a post with an "ENT" (for entity) token. Other proper names in a post are replaced with a "NAME" token and numbers with a "NUM" token. We drop all punctuation and any text with less than 5 tokens. After these preprocessing steps, we have 1,439 stories containing 1.48M tokens for action descriptions and 2.95M for characters.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We examine the feasibility of inferring characteraction interactions from text using neural language models (LM) with side information.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
The story line, that is, the full text of a story is the token sequence X = x 1 ...x K created by concatenating the tokens across all the action descriptions of the story. The posts are taken in time order without any mark for post boundaries. Let C be the set of all characters in a story. For each character j, we denote the character description as the token sequence C j = c j1 . . . c jm .
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We build separate language models for action sequences and for character descriptions. The action sequence model is over the story lines (the sequence of all action descriptions in a story), i.e. X as defined above. The character description model is over individual character descriptions i.e., C j .
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
First we describe the language model P (X) for the story line. We hypothesize in this work that a better model of X can be built by taking into account the character in focus for each individual action description. First, a baseline recurrent neural network (RNN) language model, which we denote ACTION-LM, would be
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Here x i−1 is the embedding of the input token x i−1 , and h i−1 is the hidden state which summarizes the token sequence x 1 . . . x i−2 . LSTM computes the next hidden state using an LSTM cell (Hochreiter and Schmidhuber, 1997). The output layer produces a probability distribution over the LM vocabulary using weight matrix W hv ∈ R |V | * |h| where |h| is the hidden size and |V | is the vocabulary size; b v is the bias vector.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
To take the character descriptions into account when generating actions, we define a second model ACTION-LMS which estimates
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
where z l is a variable indicating which character produced the token x l . For this model, we essentially augment the RNNs with the character descriptions as side information. For each token x l , the side information is the character description indicated by z l , i.e, C z l . We follow the approach by Mikolov and Zweig (2012), and Hoang et al. (2016), where a feature embedding vector e representing side information is input to both the RNN's hidden and output layers, or to one of them. During development, we found that concatenating the feature embedding with the token embedding at the input layer, and with the hidden state at output layer gave the best performance. More formally, ACTION-LMS computes:
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
where e i is a representation of the character which produced the token x i . The hidden state h i−1 now summarizes both the action tokens up to i − 2 and the character information up to i − 1. The output layer weight matrix is W rv ∈ R |V | * (|h|+|e|) where |h| is the size of the RNN hidden unit, and |e| the feature embedding size. In our work, the feature embedding itself comes from a feedforward neural network trained jointly within the LM. This feature network takes as input the average value of pretrained embeddings 3 for the tokens in the character description (we remove stopwords 4 ). This initial vector is passed through hidden layers to yield the feature embedding e (reminiscent of deep averaging networks by Iyyer et al. (2015)).
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
The language models for character descriptions are similar in structure. First, we call the unconditioned model P (C i ) for a character description C i as CHAR-LM; this is again an LSTM language model. Second, we implement CHAR-LMS which estimates P (C i |X C i ), where X C i is the subsequence of X only containing the tokens produced by C i . We obtain this conditional probability based on the same architecture as ACTION-LMS. Here the input to the feature neural network is the average pretrained embeddings of the tokens (without stopwords) in X C i .
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We randomly divide our corpus into 100 stories for testing, 20 for development, and the rest, 1319 for training. We compare the two ACTION language models, based on a vocabulary size of 20,000, and the CHAR models have a vocabulary of 10,000.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
Some posts are long even after our filtering steps, and create a winding story line when concatenated. So we also explore whether limits on description lengths is useful. In ACTION models, a limit of g means that only the first g words of each post are concatenated to form X. For CHAR models, only the first g words of the description C i is used as the sequence for the LM. The same limit g is given to both the models with and without side information. When using side information, we can restrict the conditioning text as well, to a maximum of h words. We tune these limit parameters, as well as the number of hidden layers, hidden unit sizes and dropout probability on a development set.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
For the ACTION models, we set g to 100 words. ACTION-LM uses 2 layers with 256 hidden units each. ACTION-LMS has 1 layer with 256 hidden units for the feature network with h set to 25 words, and 1 layer with 50 units for the RNN part. For the CHAR models, g = 200 words. CHAR-LM has one hidden layer with 100 units. For CHAR-LMS, the best network was the same as ACTION-LMS but with h = 100 (the first 100 words of all the action posts by that character are combined as the side information). We apply a dropout probability of 0.65, clip gradients at 5.0, and use the Adam algorithm (Kingma and Ba, 2015) for optimization. All our models can trained in an hour, ACTION-LM with 14 epochs, CHAR-LM 62, ACTION-LMS 60 and CHAR-LMS 91 epochs. We implemented the models in TensorFlow 5 .   Table 3: Perplexities of our models pling from the models (Table 2). For side information, we use simple words (taken from the descriptions in our test corpus) for closer examination. For ACTION-LMS, we seed the story line with the priming text " bos ENT called", where ENT is the token in our vocabulary referring to the main character of a post. bos is a beginning of sentence marker. Different inputs for the conditioning character description are shown under "Char. context". We then sample from the LM following a greedy approach taking the most likely token at each step until either the end token eos or a maximum of 12 tokens is reached. The sample is shown under "generated continuation". Similarly, we sample from CHAR-LMS where the sequence is first primed with "ENT is a". We find that both models capture interesting ties between character attributes and actions. However, there is much scope for improved models of generation.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
In this work, we have focused on the possibility of capturing the interactions. For that, we compare the impact of side information using perplexity on held-out data (Table 3). For both charac-ter and action LMs, adding side information leads to a significant decrease in perplexity showing that the interdependence between the two aspects can be learned computationally. Again, there is a lot of scope for improving the language models given that the development and test perplexities are much higher than those during training.
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts
We have proposed and demonstrated the feasibility of capturing interactions between characters and their actions in stories. While our neural models show that the data can be better modeled by combining both aspects, one might eventually want to infer a missing modality by sampling or generation from the model. We plan to work on these improvements for future work, and also explore evaluation methods which go beyond language model perplexities, and capture model aspects closer to the task and domain. Tomas Mikolov and Geoffrey Zweig. 2012 
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Artificial Intelligence has a long and rich history of using games as challenge problems that lead to advances in the field. In many cases, AI game-playing systems have gone on to rival human champions of the game. Dungeons and Dragons has been identified as an appropriate challenge for the next stage of artificial intelligence (Ellis and Hendler, 2017;Louis and Sutton, 2018;Martin et al., 2018b). Ellis and Hendler (2017) proposed open-ended creative games like D&D as the next challenge for AI after the human-level successes of AI at Chess and * *Denotes equal contribution Go, which are zero-sum, deterministic, sequential two-player games with perfect information. Louis and Sutton (2018) understood the importance of narrative in natural language processing (NLP) and generation (NLG). In particular, they saw how cooperative story generation between humans already exists in these games and can be used for automated generation. Martin et al. (2018b) outlined some of the specific challenges D&D presents to the NLP community; such as a state of the game world distributed across the Dungeon Master (DM) and other players or dealing with the intrinsic rewards players get from taking certain actions that would not necessarily provide them with points in the game.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
D&D involves multiple players who roleplay characters in a fantasy setting, guided by a Dungeon Master who sets obstacles and adventures and plays as monsters. In roleplaying games like Dungeons and Dragons, the gameplay happens through language rather than moves on a game board, making it an interesting domain for NLP research. To have an AI successfully play D&D , it would require abilities like • Language generation (multi-party dialog, generating descriptions of the world/actions, storytelling) • Language understanding (knowledge acquisition and representation, state tracking, automated reasoning) • Planning / strategic play during battles (similar to chess or go) Appendix A gives an example of D&D gameplay and the AI challenges presented by it.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Is it possible to design an AI system that is capable of playing a game of D&D either as a character in the game or as the Dungeon Master using current AI technology? We argue that now is the perfect time for this challenge, since large scale neural language models like GPT have shown impressive generation results (Brown et al., 2020), and since incorporating neural LMs into a game setting both exercises their strengths and exposes their weaknesses.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In this paper, we introduce a new dataset of "actual play" game transcripts. Each turn is labeled with game state variables like character information and whether the conversational turn was incharacter or out-of-character. Our data is a novel, large scale, real-world conversational dataset. It is unique in that the dialog turns are generated entirely through player collaboration and written interaction in a multi-player game. We propose our dataset as a challenge for dialogue systems for the following reasons:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• It is naturally occurring conversational dialog that covers a spectrum of task oriented and non-task oriented (e.g. chit chat) dialog. • It is strongly history dependent -a substantive criticism of recent dialog datasets is their history independence (Mosig et al., 2020). • It has many participants in the conversation, since there are several players in the game. • It conveys narrative elements including descriptions of events that denote changes in the state of the game. Unlike existing dialog datasets, our data reflects the challenging nature of the D&D game as a multiparty dialogue with creative roleplaying and underlying game states.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We trained a large language model (LLM) to perform two tasks: Next Utterance Prediction and Game State Tracking.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Next Utterance Prediction. We trained our language model on a corpus of human conversations (see Section 3) to predict the next utterance. We varied the conditioning information to examine the effects on the quality of predicted next utterance. In all variations, we included the conversational history as input. Given the conversational input (and other input in the variant models), the LLM must generate the next utterance, such that it is both interesting and a plausible next turn in the D&D game.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Game State Tracking. In this task, rather than producing the next utterance, we had the model predict the game state for a given dialogue turn in the conversation. We have kept the state definition similar to task-oriented dialogue state tracking Number of campaigns  896  Average players per campaign  8  Average turns per campaign  910  Average words per campaign  64,941  Total turns  815,106  Total words  58,187,526  Average dice rolls per campaign  594  Total dice rolls  532,270   Table 1: Statistics for our play-by-post corpus (DST). In DST, the dialogue state is a collection of slot-value pairs. In our case, each slot is a state variable feature related to D&D games. Our target slot values do not need to appear as a word in the dialogue context. We track several game states aspects including some that remain relatively static throughout the game (character attributes like their pronouns, class, fantasy race, and their inventory), some that change periodically (like being in combat or out of combat), and some that change from turn to turn (like what action the player is taking).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
For this paper, we have created a novel dataset for our dialogue-oriented test of AI's ability to play Dungeons & Dragons. We scraped Play-By-Post data from a web forum 1 where people play by taking turns posting on the forum to describe their move. Figure 1 shows an example of part of the gameplay from the play-by-post forums from D&D Beyond. D&D Beyond provides a mechanism in its forum to roll dice using a "roll" tag. Their dice roller allows players to conduct the rolls that are used for D&D ability checks and in combat. Table 1 summarizes the amount of play-by-post data that we collected from the D&D Beyond website (with permission from the company).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We designed a set of rule-based heuristics to extract game state information from the play by post. These were implemented using regular expressions and NLP tools like named entity recognizers (Gardner et al., 2018). Although this heuristically extracted information is not perfect, it provides a reasonable approximation of the game state. It is useful for testing whether large language models can benefit from inclusion of complex state information for next utterance prediction and whether LLMs can be used for state tracking. We designed rules to extract state information relating to character properties, combat and player actions.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• Name: Perform NER on all the player's turns in a campaign. The character's name is assigned to be the player's most frequently mentioned name, on the assumption that they tend to describe their own character's actions. • Class:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Count how many times each D&D class 2 is mentioned by each player. Most frequently mentioned class is their character's class. • Race: On a player's first turn, check whether any of the D&D fantasy races 3 are mentioned. Assign it to character. If not, guess based on the most frequently mentioned race. • Pronouns: Count pronouns mentioned by a player. Assign their character's pronouns to be the most frequent pronouns used by the player. • Inventory: Use regex to match items occurring after character's personal pronouns (e.g. her sword).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• Spells known: Regex that matches cast followed by a spell name The DM is assumed to be the player who has the first post in the game. The DM's entries in the dataset are scrubbed of other character properties, since they play multiple NPCs (non-player characters) and monsters.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• We detect the start of combat when there is a roll for initiative, or when there are attack rolls before initiative (from surprise attacks Whether a player is in-character or out-ofcharacter for a given dialog turn Changes whether the generated text is more like descriptive text found in a novel, or more like a discussion of rules and strategies. Character Name Name of the character being played by the player of a given dialog turn IC descriptions use the character's name.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Character Class D&D classes Character classes perform different actions (e.g. wizards cast spells, thieves pick locks) Character Race D&D fantasy races Different physical characteristics (e.g. halflings are small, dragonborn have scales). Character Pronouns
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
The character's pronouns Uses the correct pronouns when describing the character. Character Actions
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
List of actions taken by the character in the current turn Allows a description to be generated for a given action. The action can be thought of as a goal for the description.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Whether the players are currently engaged in combat or not during a given dialog turn Affects the likelihood of actions (e.g. attacks are more likely during combat and investigations checks are more likely outside of combat) cure, heal, healing, points. Our heuristics resulted in features for around 60% of all conversational turns. We train a convolutional neural network classifier using these conversational turns to predict all of the above control features for each conversational turn in training data. Appendix C estimates the accuracy of the model's prediction on these state features.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In addition to labeling the game states in our Play by Post data, we also labeled the text of each turn as being either spoken in-character (IC) or out-ofcharacter (OOC). To do so, we crawled another Play by Post forum hosted at Giant in the Playground 4 , where play happens on two discussion boards -one in-character and one out-of-characters. For example, here is an IC post:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Kuros pulls the feathered shaft of the arrow back to his cheek winning easily against the resistance of the bowstring. He pulls a lungful of air to keep himself steady, takes aim at the Bandit with the deer, and lets fly.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
And here is its corresponding OOC post:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Surprise round so only 1 standard or move action. Shoot the bow:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
For our large language model, we use a 64B parameter version of Google's LaMDA language model (Thoppilan et al., 2022), which was trained on conversations. LaMDA is similar to other Transformerbased pre-trained language models like GPT-3. As with other pre-trained language models (Howard and Ruder, 2018), LaMDA can be finetuned to different tasks. The two tasks that we finetune LaMDA to perform are game state tracking and response generation. In both cases, the LLM can be thought of as a function that maps inputs onto an output. For instance, game state tracking is a language understanding task where the function takes in inputs like f (current utterance, previous state, history) → new state, and response generation is a language generation task where f (current state, history) → next utterance. The LLM functions are trained via the fine tuning process.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In our experiments we try a variety of different inputs to our LLM functions to see how they enable better learning of the tasks. We train our LLMs on the conversation history (which is typical in dialog modeling) and we also augment the conversations by conditioning other explicit signals. These conditioning signals can be thought of as sophisticated "control features", inspired by the CTRL language model (Keskar et al., 2019). During training, the model learns a relationship between the control features and appropriate responses. In turn, during inference, one can explicitly influence dimensions of the conversation -enabling more compelling dialogue -by setting the values of control features. These control features can be set dynamically, without necessitating finetuning or additional post-processing. Table 2 describes the control features we have proposed and describes how they could steer generation. Note that we use the terms 'control features' and 'state variables' interchangeably when referring to our next utterance prediction models.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
LaMDA is trained on turn-based conversational data. For a conversation of length n, LaMDA takes as input the first n − 1 turns, and the nth turn as the target. For all models, we used the 7 most recent conversational turns as input, and predict turn 8.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Here is an example of the data used in our versions of LaMDA that are finetuned to on our D&D data.  Figure 2: Negative Log Perplexity of our models after pretraining on generic dialogue data for 600k steps, and then finetuning to our data for a further 60k steps. Colors correspond to the models in Table 3.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
LLM-Dialog: We call our baseline model LLM-Dialog. It is a LaMDA dialogue model that does not use not use any D&D data.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
LLM-DND: LLM-Dialog that has been finetuned on Play-by-post D&D gameplay dataset using no control features LLM-DND-ALL-CTRL: LLM-Dialog that has been finetuned on Play-by-post D&D gameplay dataset using control features (state variables) for all dialog turns including the state variables for the current turn the utterance is being predicted for.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
LLM-DND-PREV-CTRL: LLM-Dialog that has been finetuned on Play-by-post D&D gameplay dataset using control features for all previous dialog turns, not including the current turn.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
LLM-DND-CURRENT-CTRL: LLM-Dialog that has been finetuned on Play-by-post D&D gameplay dataset using control features (state variables) for only the current turn the utterance is being predicted for.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Each of our models starts from a pretrained LaMDA model trained for 600K steps and then is finetuned for a further 60K steps. Figure 2 plots the Negative log perplexity on our development set, and Table 3 shows the final perpexity and token accuracies on the dev set. At the end of finetuning, the models with the best perplexity scores and the best token accuracy scores were LLM-DND-CURRENT-CTRL and LLM-DND-ALL-CTRL, which used our control features.  Table 3: Perplexity and token accuracy of our models after finetuning to our data.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
To evaluate the quality of our models for the task of next utterance prediction in D&D , we perform a human evaluation. We recruited professional raters to perform a manual evaluation. They read a version of the content that was provided to the models -the seven turns of conversational history plus a list of players and the names/classes of the characters that they played. Then they were shown several model outputs for the context (or the "gold", which was the actual next turn in the game), The annotators asked to rate each output along the three dimensions, following the evaluation procedure used for the Meena LM (Adiwardana et al., 2020):
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• Does the response make sense? (yes/no) • Is the response specific? (yes/no) • How interesting is the response? (10 point scale) The full annotator instructions and the annotation interface are given in Appendix D.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Because of the specialized nature of the D&D domain, we recruited 6 professional raters rather than crowd workers to perform the task. The raters were selected based on their professed interest in the fantasy genre, and on their background with D&D. All raters were fantasy fans, and 5 of the 6 had played D&D. 3 raters had been the DM in a game before.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Our raters annotated 500 system outputs with 3way redundancy on each output. For the binary sense and specific scores, pairwise annotator agreement was 0.8, with a chance-adjusted Randolph Kappa score of 0.6. For the scalar interestingness scores, the Kendall's Tau correlation was 0.46.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Model Comparison. Table 4 shows the average sense, specific and interestingness scores for the systems, and for the human-written gold response. All of the D&D adapted systems outperform the  Table 4: Average sense, specific and interestingness scores for our systems, and for human-written gold responses vanilla dialogue system. On average, the adapted systems make sense 6.75% more often than the baseline, are specific 4% more often, and are 0.37 points more interesting. However, the added control features do not seem to differ substantially from the LLM that is adapted to the D&D data without any control features. Why then use the control features at all?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Control Features. One reason is that the control features are useful to allow the model to role-play. For example, we can ask the model to play an elf wizard via the control features. Table 5 gives a qualitative example of this. The Table shows outputs from two LLMs without control features, plus 3 outputs from our LLM-DND-CURRENT-CTRL model roleplaying different characters. As input, we gave the models the final seven turns in the game play example from Appendix A -from "With his eagle eyes, Magnus spots two dead horses. . . " to "... What do you do next?", and then generate what each model thinks the next utterance ought to be. The vanilla dialogue system with no D&D data generates an irrelevant reply about Discord. The dialogue system fine-tuned on D&D data without control feature generates a relevant response. The dialgoue system that adds conditioning control features allows the model to generate specific replies for each of the different characters. These replies are relevant to the character classes -the wizard casts a spell, and the Dwarf cleric shouts a battle cry by invoking the name of a Dwarf god.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In-Character Turns Are More Interesting. Among our most impactful control features was the one that allowed systems to generate in-character (IC) versus out-of-character (OOC) turns. Table 6 shows that control models' scores substantially increased on IC turns compared to when their output was generated OOC. The pronounced increase in intersestingness makes sense because IC turns are ones where the players describe their characters in Model Output LLM-Dialog moon_monk you can just join our discord conversation. it's free and easier to deal with LLM-DND I cast Ice Knife (I'm assuming this is not one of the ones I can't do?) LLM-DND-CURRENT-CTRL Responding as Magnus (human fighter, inventory: axe) In combat I turn from the goblin and go to help Merle and Taako with getting this wagon moving again. Magnus can't see them anymore with such a dense forest. They're likely in a battle right now... I have to find them as fast as I can! LLM-DND-CURRENT-CTRL Responding as Taako (elf wizard, inventory: spellbook) In combat If the goblins still don't look like they're getting closer, Taako will yell, "Alright you little monsters! Let's see what ya got!" And he'll cast Chill Touch on one of the goblins that isn't dead/on its doorstep. Attack: 22 Damage: 6 LLM-DND-CURRENT-CTRL Responding as Mearle (dwarf cleric, inventory: holy symbol) In combat Merle steps away from the wagon. He draws his sword and takes aim at the hidden goblin! Attack: 27 Damage: 8 He puts his free hand on the pommel of his sword and shouts his battle cry, "For Clanggeddin!" His eyes glow brightly red as he strikes!   the fictional world often with evocative language, whereas OOC turns usually discuss rules or mechanics. Our control features allowed the system to intentionally generate IC responses, resulting in substantially improved interestingness scores for those in-character turns.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We conducted an experiment to evaluate whether a LLM could be finetuned to perform game state tracking for D&D using our heuristically annotated game state features. We trained a new model LLM-DND-GST (Game State Tracking). It is a LLM-Dialog that has been finetuned on our Play-by-post D&D gameplay dataset. As input, it takes all previous dialog turns and their state variables, plus the text of the current turn, and then it outputs the corresponding state variables for the current turn. We analyzed the accuracy of the LLM-DND-GST model its ability to do slot-filling for each of the individual game states, and compared its performance to a simple baseline that always output the the majority class. The results are shown in Table 7. The average accuracy of the dialogue state tracker is better than the majority class baseline, but likely falls short of being useful when it comes to joint accuracy. The joint accuracy for LLM-DND-GST is 58%. This suggests that accurately tracking the full game state may require additional machinery beyond a finetuned LLM.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We have demonstrated that training on D&D data results in much higher quality outputs than a vanilla dialogue system (as expected), that controlling the model to generate in-character responses results in substantially more interesting output, and that conditioning on game state information qualitatively results in responses that are appropriate to the character class. Our preliminary experiments with using the large language models to perform game state tracking show low performance even after finetuning, suggesting that other models may be required for an AI to play D&D track the full state of the game.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Although our models are unable to play D&D fully autonomously by acting as the Dungeon Master, they could act as an aid for novice DMs. Since our models can generated evocative, in-character text that is appropriate for the context and the game state, DMs could use it as inspiration as they narrate the adventure to the other players.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Here is some model output to inspire your next adventure:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
You get a much closer look than the other two... the sarcophagi have the inscriptions of some sort of magic, probably to keep the dead inside, but you can not read them to save your life.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
What will you do next? Download our dataset 5 to start your new adventure!
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
One limitation of our human evaluation is that it is a static evaluation. The raters are simply reading the outputs of the model, and there is no interactive evaluation wherein they engage in gameplay with the system. An interactive user-study would be required before any claims could be made about how well AI is able to play D&D alongside human players.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Because our state information was created heuristically, it therefore potentially contains errors. It is also incomplete. There are several kinds of state tracking variables that would be useful to include, but were not possible to heuristically extract from our data. To address this problem in the future, we have begun a collaboration with the developer of Avrae, which is a Discord bot for playing 5 https://www.cis.upenn.edu/~ccb/dnd-data.html 9386 D&D online. Avrae contains many state variables that are missing from our current annotations, such as HIT points and slot-filler values for attacks.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We would like to thank Antony Pegg of Fandom for granting us permission to use D&D Beyond's forum data for research purposes.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Thank you to Rich Burlew and forum moderator truemane for granting us permission to crawl the Giant in the Playground forum and to build models using the forum posts.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We would like to thank the many Google Researchers who gave valuable input on this project, especially Dipanjan Das and Suma Bailis.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Chris Callison-Burch is grateful for having the opportunity to be a part-time visiting researcher at Google for two years. It was amazing to spend time among such incredibly smart people, and it was eye opening to see large LMs before they became widely available. Thanks!
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Instead of the game being a series of moves on a game board, RPGs D&D are language-based. Players create characters that have a class (wizard, fighter, thief, etc.) that denotes their abilities, and a fantasy race (elf, dwarf, human, etc.). Players describe what they want their character to do and roll dice to determine if they are successful. The dungeon master (DM) acts as the narrator who shapes the overall story. The DM describes scenarios and locations, and takes on the role of non-player characters (NPCs), and monsters.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
A common element to the game play is an encounter with monsters. Battles are governed by rules, and unfold in a turn-based fashion where the DM controls the monsters and each player controls their character. Each player and monster has a health meter (called their HIT points), an armor class (which indicates the threshold of the dice roll needed to damage them), and a set of possible attack or move actions. Table 8 provides example dialogue from a game of D&D being played between 3 players -Travis (playing a human fighter named Magnus Burnsides), Clint (playing Merle Highchurch, a dwarf cleric), Justin (playing Taako an elf wizard), and DM Griffin. We add comments about each dialogue turn to describe what is happening in the game, and to highlight the challenges that would need to be addressed if an AI system were to play the game either as a player or as the DM.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
The game session is taken from the podcast The Adventure Zone. In this episode, the hosts are playing an adventure module called Lost Mine of Phadelver, an expert of which is given in Appendix B. In the first episode of the podcast 6 , the hosts explain the rules of D&D .
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Here is an excerpt from the adventure book that the Dungeon Master was using in our example game play. The adventure book provides boxed text, which is descriptive text to be read aloud verbatim or to paraphrase. It also gives details about the combat that is about to ensure, and links to relevant game rules (like stealth checks, and statistics about the monsters that the characters will be in combat with).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
The adventure begins as the player characters are escorting a wagon full of provisions and supplies from Neverwinter to Phandalin. The journey takes them south along the High Road to the Triboar Trail, which heads east (as shown on the overland map). When they're a half-day's march from Phandalin, they run into trouble with goblin raiders from the Cragmaw tribe.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Read the boxed text when you're ready to start. If you create a different adventure hook, skip to the second paragraph and adjust the details as necessary, ignoring the information about driving the wagon.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In the city of Neverwinter, a dwarf named Gundren Rockseeker asked you to bring a wagonload of provisions to the rough-and-tumble settlement of Phandalin, a couple of days' travel southeast of the city. Gundren was clearly excited and more than a little secretive about his reasons for the trip, saying only that he and his brothers had found "something big," and that he'd pay you ten gold pieces each for escorting his supplies safely to Barthen's Provisions, a trading post in Phandalin. He then set out ahead of you on horse, along with a warrior escort named Sildar Hallwinter, claiming he needed to arrive early to "take care of business."
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
You've spent the last few days following the High Road south from Neverwinter, and you've just recently veered east along the Triboar Trail. You've encountered no trouble so far, but this territory can be dangerous. Bandits and outlaws have been known to lurk along the trail.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
You've been on the Triboar Trail for about half a day. As you come around a bend, you spot two
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Game Dialogue D&D Game Description and AI challenges Griffin (DM)
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
A dwarf named Gundren Rockseeker has hired you to transport a wagonload of provisions to the rough-and-tumble settlement of Phandalin, which is a couple days' travel to the southeast. A day and a half after leaving, you turn off the high road that connects the major cities on the coast onto a smaller trail that will lead you to Phandalin. This trail is not as well maintained, and bandits and outlaws have been known to lurk along the trail.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
This game is based on the D&D starter adventure called "Lost Mine of Phadelver". The adventure book is a mixture of rules and "boxed text" which is descriptive text for the DM to read aloud or paraphrase. See the appendix for the text that the DM is consulting. AI challenges: Generation of stories and descriptive text Griffin (DM) Roll a perception check for me. Perception is a wisdom skill, so be sure to add your wisdom modifier.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
The previous text was descriptive text. Here the DM is asking the players to perform a game mechanic and referencing a game rule. This is called "out of character" dialogue. AI challenges: Knowledge base population (extraction of rules from a rulebook) Clint (out of character)
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
I got an eight. I stop the wagon and motion silently to get the attention of Merle and Taako, and kinda pull them up towards the front of the wagon.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Travis is describing what he is doing using "in character" language. AI challenges: Personabased chat Griffin (DM)
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
As you warn them that shit has gone south, you notice a few goblins crouching in a part of the shaded woods off to the side of the road. Two of the goblins begin charging your wagon.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
The DM describes the start of a battle with several monsters. AI challenges: State tracking (in combat v. out of combat).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Travis (out of character) How many goblins are there? AI challenges: Question answering, state tracking (how many monsters).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
There are three goblins; two of them are rushing the group, one is pretty heavily obscured by the brush, probably about 40 feet out, sort of between you and the dead horses laying in the middle of the road.  Four goblins are hiding in the woods, two on each side of the road. They wait until someone approaches the bodies and then attack.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
This will likely be the first of many combat encounters in the adventure. Here are the steps you should follow to run it effectively:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• Review the goblin stat block. Since the goblins are hiding, you'll need their Stealth skill modifier: +6. • Check to see who, if anyone, is surprised. The party can't surprise the goblins, but the goblins might surprise some or all of the characters. Make a Dexterity (Stealth) check for the goblins: roll one d20 for all of them, add their Stealth skill modifier (+6) to the roll, and compare the total to the characters' passive Wisdom (Perception) scores. A character whose score is lower than the goblins' check total is surprised and therefore can't do anything on his or her first turn in the combat (see "Surprise" in the Basic Rules). • Use the initiative rules in the Basic Rules to determine who acts first, second, third, and so on. Keep track of everyone's initiative count on a piece of paper. • When the time comes for the goblins to act, two of them rush forward and make melee attacks while two goblins stand 30 feet away from the party and make ranged attacks. The goblins' stat block contains the information you need to resolve these attacks. For more information on what the goblins can do on their turn, see chapter 9, "Combat," in the Basic Rules. When three goblins are defeated, the last goblin attempts to flee, heading for the goblin trail
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In addition to the heuristics that we used to recover state variables for each turn in the game (described in Section 3.1), we used a CNN to fill in state values when our heuristics did not fire. Table 9 estimates gives an estimate of the CNN's performance on filling in the state variables where the rule-based heuristic did not extract a value. The CNN classifier only uses current post text as input (no additional context).
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
In this task, you will see part of a conversation between a few people playing D&D . The players and their characters are listed at the beginning of the conversation. The conversations that are shown as context are real conversations from players. Your job is to read the context and then rate different responses for a player/character given conversational context. Please note that the context you are given represents only a part of the players' past conversations/interactions with one another during the game.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
For each response, you would be asked the following questions.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• Does the response make sense?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
-Use your common sense here. Is the response completely reasonable in terms of the rules of D&D ? -The response "makes sense" if it is cohesive as a standalone statement, consistent with the rules of the game, and the elements/entities mentioned are plausible, given the prior context. -If anything seems off-not fluent, confusing, illogical, out of context, or wrong according to the rules of D&D -then rate it as Does not make sense. If in doubt, choose Does not make sense. • Is the response specific?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
-You may be asked to assess whether the response is specific to a given context. In other words, do you think that the response represents a good thing for the character to do now? -The response is "specific" if it flows logically from the narrative established by the prior context. * Note: It is possible for a response to "make sense" (due to being cohesive, consistent and plausible in and of itself), but be marked "not specific" when it is not a logical next step in the overall game progression. * Note: "Specific" for the purposes of this task does not have to do with how detailed the response is per se; a response can be fairly general in its language, but still qualify as "specific" when it is a logical next step in the overall game progression. • How interesting is the response?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
-You may be asked to score the response for its interestingness on a scale of 10. Choose a high score for "Interesting" if the response would likely catch someone's attention or arouse curiosity in the game; or it is insightful, creative, or witty with respect to the game. If the response is monotonous and predictable, or if you're unsure, then it is Less Inter-  
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
A mock up of the annotation user interface is given in Figure 3.
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
We recruited raters who had a background in role playing games and an understanding of the fantasy genre. We surveyed our raters, asking them the following questions:
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
1. Have you ever played Dungeons and Dragons or another role playing game before?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
2. If so,
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• roughly how many times have you played • were you a player or a game master or both 3. If not,
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
• what kind of exposure do you have to Dungeons and Dragons? (For example, have you seen it referred to in TV or movies)
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
4. Are you a fan of the fantasy genre (like Lord of the Rings)?
Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Our 6 raters responded to the survey as follows: 5 out of the 6 have played D&D or another role playing game before. All 5 of those who have played D&D /other role playing games before have played more than 6 times. Of the 5 who have played D&D /other role playing games before, 3 played as both Game Master and Player. For the one who had not played D&D /other role playing games, they indicated they had not had much exposure to D&D through TV or other channels. All 6 answered that they were fans of the fantasy genre. Figure 3: The user interface that our raters used to evaluate the quality of our model's next utterance prediction.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Dungeons and Dragons (D&D) 1 is a tabletop roleplaying game (RPG) that boasts a large player base across the world and features a wide and detailed set of rules. It is also a game which encourages creativity and offers its players a great deal of freedom in play. In this work we are using the Fifth Edition version of the game, as it's currently the newest and most popular version of the game.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
The gameplay of D&D revolves around the interaction between two groups to simulate a story taking place in a fictional world, where one group attempts to perform actions in this world and the other determines what happens when those actions take place. This back-andforth gameplay is supported by a set of rules that gives participants a guideline how to resolve the actions and events that take place in the game. The two groups consist of a player known as the dungeon master (DM), and the rest of the participants who are referred to as players. The DM holds a position that is equal parts referee and storyteller. They are responsible for describing and simulating the world of the game for the players. Since the game has a freeform nature to it, the DM also has the responsibility of determining how to enforce or use the rules when players attempt to do something outside of the usual purview of the rules. On the other side, the players hold a relatively simple position of taking the role of characters within the world of the game and, as a player's respective character, interacting with the world that the DM describes for them. Since the DM is responsible for what occurs in the game, they also determine what, if any, extra non-official content to allow players to use. As such, a DM will tend to only allow 1 https://dnd.wizards.com/what-is-dnd non-official content into their game if they deem it to be fair for use. In this context, "fair" generally means that it is not so powerful or useful that it makes the game too easy or invalidates other content. In addition, for players whose primary job is to act as their character, some don't feel that the officially provided content offers enough options and will often seek content created by a third party so that the abilities of their character might better match what they envision.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
One facet of D&D is its spell system, which allows players to use magical abilities during gameplay to enact some action on a being or object in the game. Every spell has eight parts, with many of them being highly dependent on others. Table 1 shows an example of spells in D&D. As seen in the table, these parts are the title, level, components, casting time, range, duration, school, and description. For this work we focus on the title, range, duration, and description as they carry the most importance to how a spell functions during gameplay. The descriptions of spells have a high variance in content, function, and length. They also have the most impact on how a spell works and what it does. Some spells have very simple descriptions that describe a single function the spell performs; others have long descriptions detailing a more complex spell that could be used in many ways. Descriptions also feature narrative language of what occurs when a character in the game casts the spell. No matter what a spell is though, every description has information about broader rules of the game embedded within it through the use of keywords and phrases.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Due to the distinctive qualities of these pieces of text which hold both narrative and rules-oriented language At Higher Levels. When you cast this spell using a spell slot of 5th level or higher, the damage increases by 1d10 for each slot level above 4th Table 1: An example of official spells in them, we see this as a unique type of text for the task of natural language generation. Showing that modern language models have the capability to infer rules from text and generate new content while retaining a narrative style of language is an interesting and unique task that could have broader impacts in the field. In addition, the use of natural language generation in the field of tabletop role-playing games could be a powerful tool. Since these games are based primarily around the use of written and spoken language, they could be enhanced through the use of computer generated content used before or during play to generate content for specific scenarios as the need arises. With this work we also hope to bring to light some of the unique facets of TRPGs as a medium that hold potential for future work in content generation and player interaction. (Liapis et al., 2014) presents the many interesting features of video games when it comes to the topic of computational creativity. We believe TRPGs to hold many of the same features as video games in this field, as well as other features unique to this medium.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Our approach to this work involves the following four components: 1) Create a viable dataset; 2) Train several generative models on the data; 3) Compare each model's performance using scoring metrics BLEU and BERTScore and subjective analysis of the quality of text generated; 4) Lastly, take spells from the best performing model and incorporate them, alongside human-made spells, in a survey given to players of D&D to determine player desirability and consistency with the rules. The models we decided to use for generation are an N-Gram model using Markov Chains, a model utilizing LSTM layers, and GPT-2 by OpenAI fine-tuned on our dataset.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
We used a collection of all 554 official D&D spells 4 , as well as 2,598 player-made spells from the website http://dandwiki.com. For the player-made spells we had to create a web scraper to gather the relevant data. We then combined both datasets into a single file and removed all data irrelevant to the work such as a spell's school or components. The text for each spell then contained its title, range, duration, and description. We tagged each spell's attributes using tags such as "<namestart>" and "<nameend>" and concatenated all the attributes of a spell together into a string. We then removed 50 randomly picked spells from the entire dataset to be used later for evaluation. This left us with 3012 spells in our training set. Table 2 shows an example of our training dataset.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Our N-gram model is a simple word-based 6-gram model that was trained with no smoothing. The training data for this model was unique from the others.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
The input data was not tagged but instead we placed "signifiers" such as "title: " and "description: " before each spell attribute. We then concatenated the spell attributes into a single string for each spell and used this as the training data for our model. In addition, each string was padded with tags for the beginning and end of the spell. During generation we truncate everything generated after the end tag if it has been generated.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Based on common practice, the LSTM model we used is a standard character-based model that contains an embedding layer with dimension of 256, two LSTM layers of size 1024 and 512, respectively, dropout layers after each LSTM layer with a dropout-rate of 0.1, and lastly a dense layer the size of the vocabulary. Our vocabulary for this model contains 48 characters in total (all alphanumeric characters, in addition to 12 punctuations and symbols including !, ?, ;, :. ', ", (, ), -, +, <, and >). This model was implemented using the Keras library 5 . We trained it on 200-character long sequences from our dataset for over 100 epochs and with a learning rate of 0.001. The data for this model was slightly altered, as each spell also contained tags to indicate the start and end of the spell. During generation we truncated everything generated after the end tag if it has been generated.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Our GPT-2 model was implemented with Hugging-Face's transformers library 6 . We fine-tuned the pretrained model offered by the library on a text file that contained every spell in our training dataset. This finetuning took place over 3 epochs. Our training data for this model was slightly different as we combined the tagged text for every spell into a single text file which was then given to the model. We did not use tags to signify the start and end of a spell for this model. However, during generation, we did truncate everything generated after the first tag signifying the end of a spell's description was generated.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Our evaluation contained three parts, a baseline examination using BLEU scores, an examination with BERTScore, and a survey with our best model, the GPT-2 based model.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
BLEU scores (Papineni et al., 2002) are generated based on the average number of overlapping one, two, three, and four-grams between a "reference" sentence and a "hypothesis" sentence. For our evaluation, we generate a hypothesis sentence by taking the first 40 tokens of a reference sentence and using that to generate the hypothesis. Since the LSTM based model uses characters rather than word tokens, we use the first 200 characters of a reference sentence to generate the hypothesis. Each hypothesis is then compared to its reference sentence and all of the scores and average to find the final score of the model. We chose this as a baseline scoring metric as it's widely used in quickly measuring the quality of generated text compared to some reference text and is easy to understand and interpret.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
BERTScore (Zhang et al., 2019) is used to measure similarity between two sentences by leveraging BERT's contextual embeddings to calculate the cosine similarity between each token in a reference sentence to each token in a predicted sentence. For this part of the evaluation we use the same reference and hypothesis sentences described above for the BLEU scores. We chose this as a scoring metric as it can more accurately capture the semantic similarity between two sentences than more naive approaches like BLEU, which is relevant for our work as there could be many ways to word a spell and have a similar result. We used the implementation provided by HuggingFace's Datasets library.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
For our final evaluation we created a survey containing 5 GPT-2-made spells that we determined to be good results, and 5 randomly selected player created spells for D&D from https://www.dandwiki. com/wiki/. Without labeling where the spells were from and with random ordering, we asked respondents the following questions for each spell:
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
• "What do you think made this?"
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
• "How well do you think this spell conforms to D&D's rules?"
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
• "Would you play/allow this spell?"
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Each of these questions were multiple choice. The first had options of "Human", "AI", and "I've seen this spell before, and I know it was made by a human.". The second was a choice from 1 to 5 with 1 being "Doesn't fit in with the rules at all", and 5 being "would fit in right alongside official spells". The third was also a choice from 1 to 5 with 1 being "Definitely wouldn't" and 5 being "Definitely would". The survey was posted in several D&D focused Discord servers including one for a D&D club at our institution.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
The BLEU score results in Table 3 show that there's a much higher number of n-gram overlaps between the reference and generated spells for GPT-2 than both other models. This may indicate that GPT-2 more often uses the same words and phrases in the reference spells. Since BLEU penalizes generations that are Model BLEU Score N-Gram 6.6 LSTM 10.7 GPT-2 17.7  Table 4: BERTScore results for each model shorter than their reference, it's worth mentioning GPT-2's tendency to generate long repetitions of text which may have pushed the results partially in its favor. However, since the other two models also occasionally had generations that were very long, we suspect GPT-2's habit of repetition did not skew the results and that the relative scores are still accurate.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
The BERTScore results in Table 4 show that the semantic similarity of spells generated by GPT-2 is higher than those generated by the other models. Since the hypothesis spells were generated with only the first 40 tokens of the reference sentence, they were generated largely based off of the title, range, duration, and a small piece of the description. Due to this, the scores would indicate that the meaning and wording of the generated description is closely related to that of its reference spell's description. It's worth mentioning that unlike the other models, GPT-2's precision is higher than its recall. This indicates that many tokens generated by GPT-2 are closely related to some set of tokens in the reference spell, but not as many tokens in the reference are in that set and thus have few similar counterparts in the generated spell. This could be due to GPT-2 being pre-trained and thus having the capability to generate words that aren't in our dataset. In addition, general observation of the spells generated by each model such as those in Table 5 demonstrates the level of ability of each model. A reading of the example from GPT-2 indicates that spells generated by GPT-2 are capable of a high level of coherence and the descriptions contain similar patterns and wording as seen in official spells. Due to the high level of performance of GPT-2 compared to the other models, we chose only spells generated by GPT-2 for use in the survey.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
A total of 14 people responded to the survey, and all of them identified as people who play D&D. With a sample size as small as 14 responses, it's hard to glean any true conclusions from our survey. However, as you can see the results are extremely close in all categories. D&D players were successfully able to identify an AI Model Spell N-Gram Title: fire whip Duration: concentration, up to 1 hour Range: touch Description: when you cast this spell using a spell slot of 3rd level or higher, 30 seconds are added to the time the target is banished for each slot level above 4th. the creatures must be within 30 feet of each other when you target them.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Title: Projectile Creature Range: 10 feet Duration: Instantaneous Description: A Warlice distortion of your hands in your fanging prowess, as a bonus action on a location within range you entered. Happiness points that are instantaneously runlois pit functions, along with its quantity of the caster or the caster such as metal weight.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Title: Magical Blade Range: 10 feet Duration: Instantaneous Description: You create a magical blade which resembles a blade of magical force. Choose a creature within range. Each creature in a 20-foot-radius sphere centered on that creature must make a Strength saving throw. On a failed save, a creature takes 5d10 force damage and is knocked prone. On a successful save, the creature takes half as much damage.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
At Higher Levels. When you cast this spell using a spell slot of 4th level or higher, the damage increases by 1d10 for each slot level above 3rd. Average Playability 3.47 3.49 Table 6: Survey results generated spell slightly more than the human ones, while the AI spells were reported to conform to the D&D rules slightly less yet be slightly more playable with more respondents saying they would use/allow these spells in their games. It's also worth noting that for one AI generated spell, 2 of the 14 respondents Title Conjure Ray of Force Range Self Duration Instantaneous Description You channel the power of your spirit into the ray of force and create a ray of force that is stronger than the spell's damage. The ray of force deals an extra 1d8 force damage to all targets within range.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Target one creature in range. On a hit, the target takes 1d8 force damage. This additional damage is increased by 1d8 if you cast the spell at the same time every day for the past 24 hours. Table 7: One example of problematic generation from GPT-2 claimed to have seen the spell before and knew for a fact it was written by a human. No other spell got this response, and it's clear the respondents hadn't seen that spell previously. We suspect that due to the simplistic nature and brevity of the spell, these respondents recognized common traits between it and other similar spells they had seen before. Given the small differences between the corresponding survey scores between human-made and GPT-2-made spells, it appears that our generated spells are of similar quality as the spells from https://www.dandwiki.com/ wiki/.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Although the spells were received relatively well by the survey participants, there are a number of problems in the spells generated by GPT-2. The spell in Table 7 is evidence of generation that shows interesting and novel generation in the sentence, "This additional damage is increased by 1d8 if you cast the spell at the same time every day for the past 24 hours". This introduces a mechanic that could be interesting in play. This mechanic however is not well used in this spell, and would be better used in a spell that is unrelated to combat. In this way the model succeeds in utilizing an interesting mechanic, but fails to use it in a way that enhances the spell. As for the spell in Table 8, its description is fine and works well on its own, however it's entirely inconsistent with the spell's range and duration. In the description it states "You create a bolt of thunder that flies from your hand towards a willing target. Make a ranged spell attack against the target", however the range of the spell is "Self ". This sort of contradiction is a common problem with spells generated by this model and commonly occurs with each attribute of the spell. For example in this spell this type of problem occurs with each attribute of the spell and its description.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Step into Darkness Range Self Duration 1 Minute Description You create a bolt of thunder that flies from your hand towards a willing target. Make a ranged spell attack against the target. On a hit, the target takes 3d6 lightning damage. On a miss, the target takes half as much damage. Generating new content in this manner could also be used during gameplay either to supplement gameplay or as a deliberate mechanic by the game's designers. This could lead to an entirely new kind of tabletop game where the content is generated dynamically as the game progresses.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
As this work is still a preliminary exploration in this field, there are several places for improvement upon both our best model and the others. The first place of improvement would be finding more data, and the second place would be pre-processing the data further. Since many of the spells used for training were sourced without regard for any sense of quality, there were likely biases and problematic sections that the models learned. Removing some of these problematic spells would be a step in the right direction for improving results. However, simply obtaining more data could resolve the issues presented by problematic spells, as the model would likely have more good spells to learn from in a larger dataset. Having a wider array of spells in the training set would also be highly beneficial, as each of the models tend to only generate spells that deal damage.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
One potential method to deal with the bias towards damage-oriented spells would be to split the data apart into spells that are primarily damage focused and those that aren't, and train or fine-tune a model using each of these sets separately. This may be a good method to make up for the large differences in the two types of spells.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Using a more powerful model such as larger and more powerful versions of GPT-2 or newer architectures like GPT-3 could also yield significant gains. Since these spells can sometimes have very long descriptions that need to remember key information during the entire generation, such as the title or range, it would be useful to use a more powerful model that has a better ability to retain information.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
Overall there are many places in which this task could be improved to create even better results than the already considerable ones shown here.
Generating Descriptive and Rules-Adhering Spells for Dungeons & Dragons Fifth Edition
In summary, this work shows that modern generative language models can be a potentially powerful tool to aid the design and play of tabletop roleplaying games like D&D and any other games that rely on descriptive text that is embedded with rules.