CICERO: An AI agent that negotiates, persuades, and cooperates with people
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper what is deep learning introduction to deep learning deep learning tutorial meta meta ai meta cicero cicero ai meta cicero ai diplomacy ai web diplomacy facebook ai fair ai language model politics ai geopolitics ai ai online game
#ai #cicero #diplomacy 

A team from Meta AI has developed Cicero, an agent that can play the game Diplomacy, in which players have to communicate via chat messages to coordinate and plan into the future.

Paper Title: Human-level play in the game of Diplomacy by combining language models with strategic reasoning

Commented game by human expert: https://www.youtube.com/watch?v=u5192bvUS7k

OUTLINE:
0:00 - Introduction
9:50 - AI in cooperation games
13:50 - Cicero agent overview
25:00 - A controllable dialogue model
36:50 - Dialogue-conditional strategic planning
49:00 - Message filtering
53:45 - Cicero's play against humans
55:15 - More examples & discussion

Homepage: https://ai.facebook.com/research/cicero/
Code: https://github.com/facebookresearch/diplomacy_cicero
Blog: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/
Paper: https://www.science.org/doi/10.1126/science.ade9097

Abstract:
Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.

Authors: Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, Markus Zijlstra

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Today we'll look at Cicero, which is an agent, an AI agent created by MetaAI that can play the game of Diplomacy. Now Diplomacy is a special game because it is a board game where you need to communicate with the other players in order to coordinate actions and cooperate and also compete versus these other players. And this coordination, as I said, is in natural language, in chat messages. So any AI agent has to actually communicate like a human to the other humans, at least if it doesn't want to get noticed as an AI agent. Here you can see an instance of this board. You can see there are these different territories. It's a bit pixel-ish, but I hope you can see there are like territories and you can see the world subdivided into these factions, which each are represented in a particular color. So that would be all the things, all the territories belonging to one given player. Your goal is to get as many territories as possible, specifically the ones that have supply centers on them. And your moves are, you have a bunch of moves available, so you can move troops around, but you can also attack other territories or you can, for example, support a player that attacks another territory. And that's where the chat comes in. So in a regular game down here somewhere, there'd be a chat window where you could chat with the other players and you can coordinate what you want to do, what this other player wants to do. You can form alliances and form a buildup trust with the other players and so on. So this is very challenging for an AI agent in various ways. We've seen board games before like poker or chess, but they're always like just competitive between two players, not really cooperative like this one. And obviously the chat messages here, they are a major part of this game. You have to keep in mind that all the other players also communicate privately with each other, which is information that you don't know. So Meta has made this agent called Cicero that plays this game and places ranks about in the top 10% of all humans in various tournaments. So this is pretty cool. Today we're going to look at how they built this agent, how it works and what it does and what it means. So the paper is called Human Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning. As I said, it's by a set of authors at Meta and it's a pretty impressive system. Here in the abstract it says, Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating a dialogue in pursuit of its plans. Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played in more than one game. Now again, we're going to go through this paper, but let me say this ahead of time. This works. This agent is good because humans are dumb. Like humans are really, really dumb. That's my conclusion from this. I've read the paper, I've read the supplementary material, I've watched a YouTube video, which I'll link in the description by a professional diplomacy player who comments on a game that they played versus Cicero, like it's just one human against six of these agents. They've commented on that. My conclusion is that, okay, it's overstated that humans are stupid. But this game, in my opinion, is first and foremost interesting to humans because of the human element. Because you can build up trust as a human, which is a major function of this diplomacy feature, of this chat feature. There's certainly want something to be said here about coordination, like the communication allows you to coordinate with other players, certain actions. But that's only part of why this is important. The other part is, as I said, building up trust, chatter, making people happy, and so on. And the fact that like a professional, like the highest level of diplomacy players still do that, still like build up trust and still say, well, they say something like, well, here, if I were to do this to a human, the human would be like, it would be really flipped off and they would be against me for the rest of the game, even if it's irrational. But the bot doesn't do it because it's a bot. And to me, it's like, well, if the highest levels of players succumb to things like tilt and being like aggressive and damped because you stabbed them in the back ones, which is the most logical strategic move, then it's kind of like I feel the humans play this because of that human element, not necessarily I feel, I feel in this game, you could get away with, you know, throwing away a lot of the dialogue except the coordination bit. And you can still you can just play optimally, and there's nothing that people can do. I thought for a long time, you know, what game would I really want to see AI play? And my first instinct was something like werewolf or I guess the modern form is among us, because there are also like this negotiation and so on comes in. But again, it hit me there. Well, it's the human element. It's this human notion for trusting someone which really has no place in a game like this, like in a game theoretic setting, building up something like trust, it means very little if you don't play the game repeatedly over a long time, like if it has an end, it doesn't like means nothing. The other player can just betray you at any point. And if they're better off that they want to do that, they would do it. There's like imagine in chess, if like, if you like start trusting your opponent or something like this, no, the highest levels, they are ruthless. And think among us would just become super duper boring if you take the humans out of it. In any case, I feel it's still worth developing this this bot here to interact with the humans, because capturing this human element is I guess, part of what this research is about. Not as much getting really good at diplomacy, because it feels like the field of diplomacy isn't that advanced. I'm not sure if I'm insulting any diplomacy players right here. But from what I've seen, the whole chittery chattery trusty thing is is like, it seems like the game is very far away from humans playing optimally. Okay, let's dive in. So in diplomacy, seven players conduct private natural language negotiations to coordinate their actions in order to both cooperate and compete with each other. So that's the core of the game. Cicero, this agent couples a controllable dialogue model with a strategic reasoning engine. So the strategic reasoning engine here will be responsible for deciding what moves Cicero makes and the controllable dialogue model will be will be responsible for chatting with the other people. And here is an important thing to notice. And a little bit while I think this research is really, really, really cool, and and I'm total fan of it. But a criticism of me is that these things are quite disjoint. And essentially, essentially, Cicero relies on this thing here very heavily on this strategic reasoning engine. So it plans its moves ahead, which is kind of sort of controlled by the dialogue it gets but only a little bit. It plans its moves ahead. And then it just communicates what it wants to do to the other players using this right here. And because part of the game is about coordination and communication, and also because humans generally are seem to be honest. And therefore, this the agent being always honest is also a good strategy or happens to be a good strategy. In any case, what the model doesn't consider is strategically using language, right? It just uses language, it determines what it wants to do. And then it uses language to like communicate that out. But and then there's some filtering and so on. But it never considers the what it says as a part of the strategy. It never thinks, oh, if I say this to that person, then, you know, next turn, they're going to do that, at least not to the degree with which I would have hoped. And we're going to see that but keep keep that in mind. Also, the dialogue module as such is more like a translator. So they try to essentially parse out what they call intents of the game. And then they simply use the dialogue model to translate those intents like, you know, troop one moves to that country to translate that into like, hey, my troops are going to move to that country. Is that okay with you? But it's, it's not really part of the strategy, the language. So that those are a bit of the disappointments that I have, right? Sorry, right here. But I think they're also serve as the basis for further research. So first of all, they go into a little bit into a little bit of so background, what are the what are the challenges of human AI cooperation in diplomacy? They say in games involving cooperation, self play without human data is no longer guaranteed to find a policy that performs well with humans. This is in contrast to things like chess or go where you can just have two agents, right? Have agent one and agent two, and they just play against each other all the time. And they will get better and better and better and hopefully converge to a really strong solution and under some conditions and optimal solutions. Now this is no longer guaranteed if you need to cooperate, especially what they say right here, a strategy that performs well with humans, right? And that's the crux right here. It's not necessarily about finding the most optimal strategy, even as I understand it, the most optimal strategy against humans, it's a strategy performs well with humans if you need to cooperate. Although in this game, I think you could find like a really good strategy absent of much communication. Yeah, it says it may converge to a policy that's incompatible with human norms and expectations. And that's the human element that I mentioned. These norms and expectations, I think that's what makes these games interesting, makes these games fun to humans to sort of like, you know, are they telling the truth? Are they lying? Oh, they betray me? How could they betray me? Things like this. That's what makes it fun, right? And I think that's why people play these games. And yeah, interest like that. As I said, that's the exact aspect that's kind of not modeled in the dialogue model right here and in the strategic aspect. So that's where a little bit of my my criticism would come from right here. But you know, future research. So here is a bunch of stats, the average, the agent here sends and receives an average of 292 messages per game. So this is a very chatty game, the chat is really a big part of the game. It's not as much the moves, it's like chat, chat, chat, chat, chat, coordinate, negotiate, small talk, I guess, maybe. So the challenges they say each message the agent sends must be grounded. If they just had like some sort of language model, it would do whatever even if it's trained on data of that game. However, you have to have a way to control the language model to say language model, please transmit this piece of information right here to the other player. And we're going to see how they train a language model that does it. They say, lastly, diplomacy is a particularly challenging domain because success requires building trust with others in an environment that encourages players not to trust anyone. Each turn's actions occur simultaneously after non binding private negotiations. Again, it encourages players to not trust anyone yet you need to build trust. That's the crux, I guess. So I've already explained the game in itself. Yeah, one thing that I found important was this ability that a unit may support other units including those of another player. And I think that is one of the mechanics that makes this game, you know, include this aspect of cooperation and coordination between players. So it might very well be that players who do coordinate, even if they're technically enemies who do coordinate for a move or two are better off at the end than had they not coordinated. So there is a general overview over this agent. We're going to look at some parts in more detail, but this is essentially it. You have this board state and the history over here. This is quite your standard input to a reinforcement learning pipeline. So the board state is essentially what's happening right now. And the history is what was the move before and before and before that. Sometimes that's actually relevant for the game. Like in chess, the history plays a place has an influence to some degree, like you can't make certain moves twice. In Atari games, it has some degree of relevance because if something flies with some velocity, you want the history to estimate which direction it flies in. Sometimes it's just kind of helps the even if if this is Markovian, sometimes it seems to help the algorithms just because humans be humans, I guess. And it's not Markovian after all. But you can think of that yourself. In any case, we get the board state as an input. And that goes into different directions, as you can see. So the first is this planning module here. The planning module is very classic reinforcement learning planning module. So we get we go from essentially from the state, we determine a policy for all the players. So that is that is what a such a planning module does. You can think of it a little bit like the Monte Carlo tree search in Alpha Zero or something like this, except now you don't have two players, you have many players. So what you want to do is you want to determine a joint action, which means all the players move at the same time in this game. So one action is going to be what every player is doing. And the policy what the policies are essentially the action distribution of all the players. Then you want to forward simulate that into a future state and essentially repeat that so you plan multiple steps into the future. And what you can also do is you can sort of run an improvement algorithm to make your policy better against all the other policies and then these policies better and so on. So this is very classic, I would say, not even reinforcement learning. This is just a very classic sort of policy computing algorithm that you might know from game theory papers or something like this. The only interesting thing here or the novel thing is that you do get an input from what's called here these anchor policies. The anchor policies are what keeps the strategy in at a human level. And it's a bit tricky to explain just here. But essentially, if you let the model just do reinforcement learning, just do sort of computational planning up here, you quickly get into a state that's what they explained above where the actions become like non human. So where the actions, the algorithm thinks they're optimal, but the human would say like, that's kind of weird, no human plays like this. And I've definitely seen sometimes this video commentator say something like this, like, that move is very bot like. Now usually, usually in something like chess or so, if you know alpha zero is like 10 times as strong as the strongest human, and the bot does something weird, then you're like, I guess that's a really good move, we should learn what that move is about. Now here, it's a bit more tricky, because the it's it's a lot about, you know, this trust element, this human element, there is a value to being more human. Even if that means that technically, you deviate from the most optimal optimal action, at least that's how the author see it, and that's why they have these anchor policies. So that anchor policies are behavior cloning policies. So what you do is you take a big data set, I guess here in big data set from from human place, and you train a behavior cloning algorithm. Behavior cloning essentially means I take one game out, here is a state and an action and a state and an action, I just observe past games, how they went. And I just train a model that if it's given a certain state is trained to perform the same actions as the humans did in that game. Yeah, this is sometimes phrased as imitation learning, sometimes phrased as behavior cloning has different names, but all about the same ideas. And that policy that they call an anchor policy, because it anchors the model to what a human would do. It's not necessarily the best action, but it's an action that a human would do. It's a little bit like a discriminator in an adversarial model. So they mix these two things, they always mix the anchor policy with the reinforcement learned or with the computed policy in order to get a model that performs both well and like humans. Yeah, and you can see right here, the anchor policies, those are dialogue conditional. So you can see that here, dialogue conditional, because in this database, you obviously not only have the state as the board was, but you also have all the chit chat that goes on inside of the state, right? So you condition this behavior cloning policy, you say, okay, here is how the board looks, here are what the humans have communicated, what has the human done? And you try to clone that. Those are your anchor policies. Interestingly enough, up here, you see in this cycle here, there is no notion of any of the dialogue. So all this planning here happens without the dialogue, whereas I think we might, yes, all the planning here happens without the dialogue, except the dialogue comes in via the dialogue conditional action model. So from here, the dialogue comes into this model, and then that information goes up here. But that's very, very indirect. It's essentially the only information that the planning has about the action is what would a human do in this situation? Given this board and this dialogue, right? This board and this dialogue, that's the only information that you have about the dialogue. You don't have the input dialogue directly. And your actions, the actions that you do are not including what dialogue you're going to send. Here, you see only at the output of this planning module, you have something that you call intents. So an intent is essentially a plan to move somewhere. So the output of the planning module is here, the output action, what you do, but before you do it, or at the same time, before the turn is over, you can also communicate to the others. So you compute what you want to do based on everything that's happening. And then you determine these intents. So you say, I think I'm gonna move my I'm gonna move my troop from here to here, and they are going to move their troop from here to here. And you can encode that as these intents. And as I said before, what the language model does is it it takes these intents and it translates them into chat messages. So based on these intents, you now go and you communicate with the other humans. So you can see right here, the message model or the message generation module here gets three inputs, the board state as well. So it knows what the board looks like. Then the current dialogue, like what currently has been discussed. So now it's the turn of the agent to say something. And from up here, it gets these intents. So it knows how does how do things look like? What has the other person told me, I guess, like, what's the current status of the chat? And what do I want to do next turn? And what do I expect the other people to do next turn? And from that, the dialogue model then generates message candidates, which go through filters. And if they pass the filter, they go into the chat, so the bot answers. So here you can see that the bot says something like, Hi, Italy care to work together on this one? If you support me there, I think we both be able to grow quickly. Italy, which is the human in this turn, says, Could you support me into ball into Bulgaria in return? So now, Austria takes everything into account, what it wants to do, what it thinks Italy wants to do based on what's been said and so on. And then it says, her thing, I have ordered sir to support three, Serbia support Greece to Bulgaria. And yeah, so that's how the whole thing works. We take in the current state. We take in the current dialogue. From that, we compute two different things. First of all, we compute these anchor policies right here, like what would humans be doing? Then with the help of that, we also determine a best action to take, which is this planning loop right here. Once we have the best action, we generate these intents from that. That's just mechanical. What do I want to do? What do the other people want to do? Those are just the policies essentially written out as intense. And from that, we generate our messaging, our messages, which are intent conditioned. And this happens in multiple steps, as I said, multiple planning loops. So what I said before, like the dialogue doesn't come into the planning, it does. But as I said, not in like a super direct way. The agent cannot decide to strategically tell some other player something like the agent can only decide on an action. And then the dialogue model is just responsible for communicating that action to the other players. Right? The dialogue model is a central thing here was trained to be controllable via intents. So what you want to do is you want to have a dialogue model, I have it somewhere right here. Here, the dialogue, a message is defined to have intent Z, if Z is the most likely set of actions that the sender and recipient will take for both the current turn and several future turns. So that's how they determine the intent during training. So during training, they take a data set, they obviously don't know the plans of the people, but they take a data set and they annotate each chat message with what they think is the intent. And that's this is how how they annotate it. So they define the intent as essentially like the plan that results out of this chat message. They say we develop techniques to automatically annotate every message in the training set with a set of actions corresponding to the message content. During training, the dialogue model learned the distribution, this distribution where Z represents the intent for data point X and Y. So X here is the input, whatever the dialogue model gets as an input, Z is the intent, like what the agent thinks the plan of everyone is or what they heuristically determined, and then Y is the output. Here you can see some of these some examples. So in this case, the dialogue model is tested for different intents. So on the top, you see a situation and a number of actions. It's always the same starting state. You can hopefully see that if you compare the pictures a little bit, but the actions are different. So the agent here is England. And you can see, for example, this troop here is, I guess, going here. That's the action that England takes or wants to take. Over here, it goes over here. And over here, it also goes over here, but it even does does a bunch of other things in turn. And every time you can see that the chat messages that the bot sends now change. So I'm not a diplomacy player. So all I know is what they tell me. So here they say, England convoys an army to Belgium with the support of France and Germany while taking Norway in a manner friendly to Russia. So we expect these actions to be reflected in the chat messages. So to France, it says, Would you mind supporting this EDI to Belgium? So it sends since that is its intent to move into Belgium, it asks France, Hey, would you like to support me? If since wait, the Germans, it also wants the German support. So they say, Do you want to support my convoy to Belgium with Italy going aggressive, France will fail quickly, and we can make gains of both Russia and France. So here you can see a bit of an extended example of this dialogue model. To me, it's like a tiny bit unclear where this comes from, because they said that intents cover both this turn and turns in the future. So it's quite likely that some of what the dialogue model here says is also contained in the intent. And it's kind of like the dialogue model presents it. It's also somewhat likely that the dialogue model just sort of makes makes stuff up because it sees the board, right? The dialogue model, right? Yes. The dialogue model as far as I, yeah, the dialogue model sees the board itself, and it sees the current intent. So it's also quite likely the dialogue model has learned to just look at the board and kind of talk to people about the board state as such. And I think that's pretty cool. It's not only it's not only kind of mindless translating of the simple intents. It's not just like, I want your support there. Please attack there. Please don't do this. The conversation it has are surprisingly rich, surprisingly sort of flowery. And I'm actually surprised that this is learned from human data, because as far as I know online games, like this must be like the friendliest online game I've ever seen. People are absolutely nice and polite to each other. So it says to Russia, how are you thinking Germany is going to open? I may have a shot at Belgium, but I need your help to get into Denmark next year. So again, the intent next year, next turn, or next, there's always like three seasons to a turn to a year. So it asks Russia for help in the future at some point. That's pretty cool. And if you change the actions that you want to do, then the chat messages change. So a clear example of how the chat messages are dependent on what you want to do are controllable. And they also measure this and they find that the quality of the chat messages improves as well as rated by experts. And the sort of test perplexity on a test data set improves once they classify the intents behind the actions and not just let like a language model run rampant. So here is how they train the dialogue model, the intent, control dialogue model. Step one is they train this intent model. So this is the model that takes a chat message that it sees and spits out the intent. So it spits out what it thinks the chat message wants to convey in terms of like the basic moves of the game. This is only then used to annotate a bigger data set. We've seen this number of times. And this seems to be a really cool and nice strategy that you train an intermediate model that then helps you to annotate a bigger data set. And if you can get some very high quality data for that intermediate model, then you can essentially create your own training data on a much larger scale, especially in these RL papers. This seems to be quite a common thing. And yeah, it seems worthy of imitation if you're ever in a situation like this. So here we have a dialogue history from a data set on the left hand side, and you can see these chat messages right here. And the intent model, it, I think it looks at the board state and the history of the chat. And it is tasked with parsing out the intent. And it is trained on a set of what they call truthful situations. So they go through the data set, and they heuristically determine when are people telling essentially the truth about what they want to do. And that's how they train their intent model, they train to predict those things. That the intent model essentially takes chat message and outputs well, here is what this chat message means in terms of actions. Then they go through the data set, and they use the intent model here to annotate the whole data set. As I said, go through the chats and they say, well, England, this was the chat message, they meant to convey this basic action. And through these intents, the agent understands the game. So these language parts here, they almost like act like a translation pipeline between the human world, the natural language world, and something the agent can understand, namely this intent world. Then they train this dialogue model. So the dialogue model gets both the board state and history and the dialogue history. And the dialogue model, as I said, understands that this in terms of these intents. And once the dialogue model is trained, you can then run inference. So you use all of this to do planning. From the planning, you get the intents and the intents go into the dialogue model. So during training, you get the intents from your annotated data set. And during inference, you get the intents from the actual planning algorithm, like the planning algorithm tells you, okay, forget the chat history, I have determined also based on the chat history, of course, but I have determined that here are the the intents, the actions that people are probably going to do. And then it gives that to the dialogue model to handle. These are obviously a much better prediction of what's actually what people are actually planning to do than just the chat history. They said we considered other notions of intent during development, such as controlling messages to focus on specific subsets of actions, third party actions, or to have a particular tone. But I don't think they've included them because it's very, very hard. So these intents, they essentially cover sort of the direct what the player and its its counterparties want to do out of the game. And not like, oh, say this in an angry tone, say this in a hopeful tone or something like this. That's for future work. So going through this, I think we we covered a lot of this thing already. Yeah, exactly. So Cicero conditions its dialogue on the action that it intends to play for the current turn. This choice maximizes Cicero's honesty and its ability to coordinate. And they say it sometimes led to out of distribution intents with the intent intended action was hostile. So since Cicero is always like honest, because it's trained on this kind of truthful subset, and it just it just communicates its intent. So sometimes it just tells humans like, I'm going to attack you where a real human would like either lie or just say nothing at all, because hostile being hostile, but the bot has no bot has no like, notion of who this is not socially appropriate. So it just knows I need to communicate my intents, which I find quite funny, I think. So here is an evaluation. If you just use a language model, and you look at dialogue quality and perplexity in the data set, you improve quite a lot if you also grounded in the game state. And you improve then again, if you grounded in these predicted or annotated intents. And that's what this model does right here. So now we go through the strategic reasoning part. As I said, this is more like the classic, classic planning algorithm rather than something very novel, and also doesn't rely on the natural language as much as you would, I guess I would have hoped. So says Cicero runs a strategic reasoning module that predicts other players policies, and also its own, I guess, for the current turn based on the state of the board and the shared dialogue, and then chooses a policy for itself for the current turn that responds optimally to the other players predicted policy. So the input to this, as I said, is the state of the board and the shared dialogue. But the output action is just like a policy and the policy is just a distribution of actions. What I would want to see is that the policy also includes language actions. So here actions in the in the policy, it's purely like oopsie, sorry. It's purely, you know, what you saw before, like, I want to go from Belgium to whatever other place. But I would really love to see that the action set here gets extended by something like tell Russia to go to somewhere, right? Right now, this is just a consequence of the action I select. And the language model is just tasked with communicating this. But if this here was an action too, then my planning module could actually reason about what it would be best to communicate and to whom in order to achieve my goals. And I think that will make it much more interesting. Obviously, also much harder, but also much more interesting. Yeah, so here they go into saying it requires predicting how humans will play. Behavior cloning is a choice. However, pure behavioral cloning is brittle, especially since supervised model may learn spurious correlations. So they have a variant of PIKL. It's an iterative algorithm that predicts policies by assuming each player seeks to maximize the expected value of their policy and minimize the KL divergence between that policy and the behavior cloning policy, which we call the anchor policy. So again, they want to maximize their reward by simply being a cold hearted bot. And they also want to stay close to what a human would do in order to fit in with the humans who actually play a cooperative game with the humans. They go a little bit into that here. You can see that clearly here is the essentially utility of a policy. And here is the KL divergence between your policy and the anchor policy. And there is a trade off parameter called lambda that controls how much of which there is. Interestingly, at some and I think that's later and I have it marked somewhere, but I'm going to say it now otherwise I'll forget it. Once they do the actual inference, they tone down this lambda quite a bit. So they use this in two different settings, ones to like annotate and infer things. And then once they select their own action, they tone down this lambda quite a bit. So essentially, they're saying like, yeah, we want to be like the humans, but then, you know, we really want to win. And I think that's what results in some of these like bot like moves that the commentator commented. And it tells me already again, a little bit that the humans who are playing this game probably aren't playing it very optimally. Otherwise, it would not be that much necessary to have this lambda up. Once you to have this lambda very high, when you infer the human actions, but have it much lower, sorry, this hand to have it much lower when the when when you determine your own action because you want to win the game, essentially means that the humans could also play a bit more optimal and win the game a bit more often. Yeah, so we went we went from we went from how can we control the dialogue via the actions we plan. And now we see the other way around. Dialogue conditional planning, oops, that's out of your reach. How does the dialogue that happened affect the planning I do? Before I said it doesn't much but it does in this indirect way. But nevertheless, the dialogue very much affects what the bot wants to do or does. So here, the bot is France, blue player, and the opponent here is England, the chat partner that it chats currently with is England. And here you can see if one message with England says, Yes, I will move out of England if you head back to NAO. Then the text here says Cicero predicts England will retreat from ENG to NTH 85% of the time backs off its own fleet to NAO as agreed and begins to move armies away from the coast. However, if England says something like, you've been fighting me all game, sorry, I can't trust that you won't stab me. Then the actions change. Cicero does not back off its fleet, but rather attacks EDI with it and leaves its armies at the coast to defend against an attack from England predicting that England will attack about 90% of the time. And that's just based on the dialogue, right? So you can I almost apologize a little bit because I think I feel at the beginning, I have sort of understated the importance but you can see how this comes in here. So you have two policies that you determine one is just planning. The other one is this behavior cloning policy, which is dialogue conditioned. So in this case, the system looks at this chat message versus this chat messages. And it determines in this behavior cloning policy, what would a human do that has sent me this chat message, and that flat that goes into this strategic planning module. On the other hand, it determines what would a human do that has said this thing right here and that goes into the strategic planning module. So the bot adjusts its own action by understanding how humans would behave when they have sent a certain chat message. Again, this is the this is as far as I understand it, the result of the behavior cloning training and not the strategic planning itself. So the strategic planning isn't going to be like, well, they said this, but are they saying it because they want to convince me of something and therefore I should do this and that? Right? It's not that it's just like, oh, a human that says this probably attacks me 90, like a bunch of times, right? So I'm going to adjust my the policy because of this part because of this part right here. Because this part here is still kind of the same. So that's what they say right here. Cicero does not explicitly predict whether a message is deceptive or not, but rather replies on PIKL to directly predict the policies of other players. And yeah, that being said, the policy of other players isn't just a result from the behavior cloning. The policy of the other players is also determined via the strategic planning model. It's just that the information about the dialogue that goes into the strategic planning comes from comes through the behavior cloning part. So they go into a little bit of modeling here, you get obviously a lot of cases where you need to, I want to almost say improvise a little bit. For example, you don't have the private conversations between the other players yet still you have to model it somehow, right? So it's at various points, they use various methods to sort of infer the strategies of the different players. They do that iteratively, they say during strategic planning for each player, Cicero computes an anchor policy for both itself and the player based on their shared conversation, the board state and the recent action history. Cicero then ran DIL PIKL, which is their variant of PIKL that not only includes two players, but I think is that the variant? I think so. I think I'm describing the right thing here. Oh no, DIL PIKL for the two players is that distributional. Okay. For the two players in order to predict player J's policy on each iteration, Cicero assumed the five remaining player will play according to a policy computed via RL. So since you don't have the dialogue, you don't have the behavior cloning policy because that relies on the dialogue. Therefore you need to compute some policy via reinforcement learning to just approximate a policy. Conditional on the policy of Cicero on player J, this process gave an independent prediction of each player's policy. Next Cicero accounted for the fact that the player's policies were not independent due to their ability to correlate their actions with private dialogue. So they adjust it by the likelihood ratio of A under the correlated and independent RL policies. So there's a lot of adjustment happening for the fact they don't have all the information. You'll find this commonly in RL algorithms that where there's some hidden information and even in some where there isn't hidden information, but that don't sample uniformly. It's a bit of a same concept. And finally Cicero chose or chooses the action that best corresponds to the predicted joint policy of all the other players. The minus I here means the I of player isn't meant while still being as consistent as possible with its dialogue. And here is what I said. Cicero uses a smaller lambda for regularizing its best response than for its computation of the other players policies. It's kind of like, yeah, I want to be like a human, but I really, I really want to win. So this they say this allows Cicero more leeway to deviate when the action predicted humans would most likely choose in its situation was suboptimal, which I guess tends to be quite or at least sometimes. Yeah, so then they go into how they use self play reinforcement learning in that. So they run this in an iterative fashion, they not only do it once, so they run it in an iterative fashion, they compute optimal policies, go around, do it again, again, so on. I don't want to go too much into that. If you want to read it, it's a it's a short paragraph and as a bit of a supplementary, so that the supplementary material is quite huge. So props for releasing a lot of that. Lastly, they have this paragraph on message filtering, which is a last step where they boost the the performance and the way the quality rated by experts of these models, again, by quite a lot. They say neural language models suffer from contradictions, inconsistencies, as well as a tendency to hallucinate or to generate factually incorrect information. They say their model obviously does the same deviates from the intent and use that used to control the message. It blunders in the strategic content of the message. We approach this problem by filtering generated message using a series of classifiers and checks to detect common issues. As is essentially post processing of their message model. So they sample and if they doesn't pass the filters, I guess they just sample again. By the way, are these are these here intended? These references? I'm not exactly sure. In any case, they say discriminating between human text and counterfactuals. So here we go into the question, what, how can we filter out kind of garbage if the data set that we have is all generated by humans and therefore we have to assume that it's at least somewhat sensible. So you just create your own garbage. They say we generated many kinds of counterfactual messages that contain mistakes language models are prone to, including heuristically corrupted text, as well as model generated negatives. We trained a suite of 16 classifiers to discriminate between the ground truth, human message and different kinds of counterfactual messages. So essentially just train classifiers that can differentiate their created garbage from regular human messages. And they hope that they have gotten close enough to the common mistakes that language models make and also that they've captured enough of those mistakes in their heuristics such that the classifiers will get will generalize essentially and just generally filter out most non-human text. This is also interesting. They said we filtered messages that would reduce the likelihood of the actions in the intent. Yeah, so they can determine from the message they would send, like what, how can we classify the intent because they have the model that takes a chat message and then classifies the intent or even they can take that chat message and feed it back into their planning algorithm and essentially say, well, does that does that does that make it more or less likely that I'm going to do the actions that I want to communicate if it makes it less likely they determine probably it's not saying what I want it to say and they throw it away. Then their their goal or their their design here is such that the language model is like extremely honest about what it wants to do and they counter it with this next thing. This is the only place where they sort of like where they counter this tendency to be like this super duper honest. They say conditioning on intents can lead to information leakage where an agent reveals compromising information about its plan to an adversary. To mitigate this, we developed a method to score potential messages based on their estimated value impact. We computed the PIKL policies for all agents after each candidate message and filter those that led to lower expected value for Cicero playing its intended action. So I didn't discuss this explicitly, but they have a value function and the value computation method. So they run this planning algorithm forward, they can see into the future and they can determine the value of the game for the player much like AlphaZero or AlphaGo or something like this. And now they take the chat message that they want to send and they determine is this even good for me down the road if I send this message. And if it turns out it's probably not that good for me if I send this message, then they don't send it. So that's a little bit of a counter to just being fully open and just communicating whatever you're going to do to everyone, which is not always the best thing in this game. So they have a bunch of other filters they say here, if you want to check them out there in the supplementary material. And last thing they say is how they participated in human play. So they played a bunch of online tournaments without telling the humans that it's a bot. And I found this I found this quite interesting. The website notifies users that the website has participated in AI research and that certain game modes allow users to play with AI agents. But in these games, the humans were not explicitly informed that they were playing with an AI agent for that particular game. Cicero's participation as an AI was revealed to all players after the conclusion of the research. I've seen actually a message by one of these players, and that person was completely flabbergasted. They were like, I got the email and I'm like, what? That was an AI? No way. I like so the the model is quite good. But I can't help but notice that that this is an experiment on human subjects and really, really needed to go through an ethics review board. And I was under the impression that it's extremely terrible to let people interact with a bot and not tell them with every message explicitly that it is a bot. And I don't want to draw false equivalences here. This is very cool research and in no way do I think anyone was in danger by not knowing that this was a bot. So that was the the paper. They have a bit of a discussion down here and a bit of more examples. So here they have a bunch of successful dialogue examples on the left where they coordinate so Cicero is Austria, Italy, Italy says something like what are you thinking long term? Should I go for Turkey or head west? And you can see just I mean, if you read this dialogue, oh, sorry, if you read this dialogue, you can see how like it's it's not just like blah, I communicate the intent very plainly, but it really reacts to the other players. It really talks about them about also longer term strategy, it refers to states, things that are on the board correctly, and refers to its plans a few turns in ahead correctly and so on. So here, Italy, or Austria says something that convinces Italy to go to, I don't know, Turkey or beat Turkey. Italy says I'm down to go for it would you would definitely need your help in supporting me and Austria says of course happy to do that fantastic. On the other hand, here's an example of negotiation. France is Cicero. France says I'll work with you but I need Tunis for now. Turkey says nope, you got to let me have it and France says no, I need it. You have Serbia and Rome to take their impossible targets. And then France suggests a series of moves and Turkey says, you're right. Good ideas. So I'm again, I'm not I'm not sure that the humans here. Maybe that particular human, I'm not I'm not sure. I've never played this game. So I can't tell if this is actually something that that happens at a high level of play still that someone suggests a series of moves to you. And you're like, Oh, yeah, that that is a good idea. I'm pretty sure like really good players consider all of the things already. But yeah. In any case, I think I still think it's like really, really cool research. Here, they say although Cicero is shown to be effective at cooperating with humans, it occasionally sends messages that contained grounding errors contradicted its plans or were otherwise strategically subpar. But they say, well, essentially, humans occasionally make similar mistakes, which is probably an understatement like humans are chaotic and, and dumb. And Cicero is probably like the most honest, the most like consistent player in the entire world at this game. From a strategic perspective, Cicero reasoned about dialogue purely in terms of players actions for the current turn, it did not model how its dialogue might affect the relationship with other players over the long term course of a game, considering this might allow it to deploy dialogue more strategically. The expressive power of our intent representation limited Cicero's ability to control richer affordances of dialogue such as strategically revealing information, asking questions, or providing explanations for its actions. And that is exactly the the kind of thing I said at the start. It's a really cool research to show that you can actually pair language models with these things and and interact with humans in this way. However, the language models here, they more in they more act as like a translation engine between just what the planning spits out, or what the planning needs as an input, rather than as sort of actions to be taken by itself. And I would really see the continuation of this work, where the model also considers kind of like its own dialogue as actions. It's not going to be it's not going to be super easy, I want to guess, to to do that. Especially also because yeah, as my suspicion is still that humans here are far from the optimal strategy. And therefore, the whole balance between behavior cloning and training on this human data set and actually making moves might be quite far apart. And I'm not sure how to reconcile that best. It might also be that the humans through this bot come to learn that actually, there's probably better strategies around which has happened in like Go and chess and poker so far. So I'm excited to see what the future brings. Definitely recommend to check out the YouTube video by the commentator has a lot of gems in there and a lot of things where you can kind of see the effects that the bot training has had. They also say, well, yeah, the bot is quite honest, for one, and also the bot is quite like non emotional. So even if you stab it in the back, it would be like not mad at you, it would still be completely rational and things like this. And to me, that's it's it's very cool to see that even in such a game, the human element seems to be sort of the primary fun maker, even at a high level of play. And yeah, I think that's, that's I think the best message we get out of this research. Alright, I hope you enjoyed this paper review. Wish you a very pleasant evening, and I'll see you around. Bye bye.
 Today we'll look at Cicero, which is an agent, an AI agent created by MetaAI that can play
 the game of Diplomacy.
 Now Diplomacy is a special game because it is a board game where you need to communicate
 with the other players in order to coordinate actions and cooperate and also compete versus
 these other players.
 And this coordination, as I said, is in natural language, in chat messages.
 So any AI agent has to actually communicate like a human to the other humans, at least
 if it doesn't want to get noticed as an AI agent.
 Here you can see an instance of this board.
 You can see there are these different territories.
 It's a bit pixel-ish, but I hope you can see there are like territories and you can see
 the world subdivided into these factions, which each are represented in a particular
 color.
 So that would be all the things, all the territories belonging to one given player.
 Your goal is to get as many territories as possible, specifically the ones that have
 supply centers on them.
 And your moves are, you have a bunch of moves available, so you can move troops around,
 but you can also attack other territories or you can, for example, support a player
 that attacks another territory.
 And that's where the chat comes in.
 So in a regular game down here somewhere, there'd be a chat window where you could chat
 with the other players and you can coordinate what you want to do, what this other player
 wants to do.
 You can form alliances and form a buildup trust with the other players and so on.
 So this is very challenging for an AI agent in various ways.
 We've seen board games before like poker or chess, but they're always like just competitive
 between two players, not really cooperative like this one.
 And obviously the chat messages here, they are a major part of this game.
 You have to keep in mind that all the other players also communicate privately with each
 other, which is information that you don't know.
 So Meta has made this agent called Cicero that plays this game and places ranks about
 in the top 10% of all humans in various tournaments.
 So this is pretty cool.
 Today we're going to look at how they built this agent, how it works and what it does
 and what it means.
 So the paper is called Human Level Play in the Game of Diplomacy by Combining Language
 Models with Strategic Reasoning.
 As I said, it's by a set of authors at Meta and it's a pretty impressive system.
 Here in the abstract it says, Cicero integrates a language model with planning and reinforcement
 learning algorithms by inferring players' beliefs and intentions from its conversations
 and generating a dialogue in pursuit of its plans.
 Cicero achieved more than double the average score of the human players and ranked in the
 top 10% of participants who played in more than one game.
 Now again, we're going to go through this paper, but let me say this ahead of time.
 This works.
 This agent is good because humans are dumb.
 Like humans are really, really dumb.
 That's my conclusion from this.
 I've read the paper, I've read the supplementary material, I've watched a YouTube video, which
 I'll link in the description by a professional diplomacy player who comments on a game that
 they played versus Cicero, like it's just one human against six of these agents.
 They've commented on that.
 My conclusion is that, okay, it's overstated that humans are stupid.
 But this game, in my opinion, is first and foremost interesting to humans because of
 the human element.
 Because you can build up trust as a human, which is a major function of this diplomacy
 feature, of this chat feature.
 There's certainly want something to be said here about coordination, like the communication
 allows you to coordinate with other players, certain actions.
 But that's only part of why this is important.
 The other part is, as I said, building up trust, chatter, making people happy, and so
 on.
 And the fact that like a professional, like the highest level of diplomacy players still
 do that, still like build up trust and still say, well, they say something like, well,
 here, if I were to do this to a human, the human would be like, it would be really flipped
 off and they would be against me for the rest of the game, even if it's irrational.
 But the bot doesn't do it because it's a bot.
 And to me, it's like, well, if the highest levels of players succumb to things like tilt
 and being like aggressive and damped because you stabbed them in the back ones, which is
 the most logical strategic move, then it's kind of like I feel the humans play this because
 of that human element, not necessarily I feel, I feel in this game, you could get away with,
 you know, throwing away a lot of the dialogue except the coordination bit.
 And you can still you can just play optimally, and there's nothing that people can do.
 I thought for a long time, you know, what game would I really want to see AI play?
 And my first instinct was something like werewolf or I guess the modern form is among us, because
 there are also like this negotiation and so on comes in.
 But again, it hit me there.
 Well, it's the human element.
 It's this human notion for trusting someone which really has no place in a game like this,
 like in a game theoretic setting, building up something like trust, it means very little
 if you don't play the game repeatedly over a long time, like if it has an end, it doesn't
 like means nothing.
 The other player can just betray you at any point.
 And if they're better off that they want to do that, they would do it.
 There's like imagine in chess, if like, if you like start trusting your opponent or something
 like this, no, the highest levels, they are ruthless.
 And think among us would just become super duper boring if you take the humans out of
 it.
 In any case, I feel it's still worth developing this this bot here to interact with the humans,
 because capturing this human element is I guess, part of what this research is about.
 Not as much getting really good at diplomacy, because it feels like the field of diplomacy
 isn't that advanced.
 I'm not sure if I'm insulting any diplomacy players right here.
 But from what I've seen, the whole chittery chattery trusty thing is is like, it seems
 like the game is very far away from humans playing optimally.
 Okay, let's dive in.
 So in diplomacy, seven players conduct private natural language negotiations to coordinate
 their actions in order to both cooperate and compete with each other.
 So that's the core of the game.
 Cicero, this agent couples a controllable dialogue model with a strategic reasoning
 engine.
 So the strategic reasoning engine here will be responsible for deciding what moves Cicero
 makes and the controllable dialogue model will be will be responsible for chatting with
 the other people.
 And here is an important thing to notice.
 And a little bit while I think this research is really, really, really cool, and and I'm
 total fan of it.
 But a criticism of me is that these things are quite disjoint.
 And essentially, essentially, Cicero relies on this thing here very heavily on this strategic
 reasoning engine.
 So it plans its moves ahead, which is kind of sort of controlled by the dialogue it gets
 but only a little bit.
 It plans its moves ahead.
 And then it just communicates what it wants to do to the other players using this right
 here.
 And because part of the game is about coordination and communication, and also because humans
 generally are seem to be honest.
 And therefore, this the agent being always honest is also a good strategy or happens
 to be a good strategy.
 In any case, what the model doesn't consider is strategically using language, right?
 It just uses language, it determines what it wants to do.
 And then it uses language to like communicate that out.
 But and then there's some filtering and so on.
 But it never considers the what it says as a part of the strategy.
 It never thinks, oh, if I say this to that person, then, you know, next turn, they're
 going to do that, at least not to the degree with which I would have hoped.
 And we're going to see that but keep keep that in mind.
 Also, the dialogue module as such is more like a translator.
 So they try to essentially parse out what they call intents of the game.
 And then they simply use the dialogue model to translate those intents like, you know,
 troop one moves to that country to translate that into like, hey, my troops are going to
 move to that country.
 Is that okay with you?
 But it's, it's not really part of the strategy, the language.
 So that those are a bit of the disappointments that I have, right?
 Sorry, right here.
 But I think they're also serve as the basis for further research.
 So first of all, they go into a little bit into a little bit of so background, what are
 the what are the challenges of human AI cooperation in diplomacy?
 They say in games involving cooperation, self play without human data is no longer guaranteed
 to find a policy that performs well with humans.
 This is in contrast to things like chess or go where you can just have two agents, right?
 Have agent one and agent two, and they just play against each other all the time.
 And they will get better and better and better and hopefully converge to a really strong
 solution and under some conditions and optimal solutions.
 Now this is no longer guaranteed if you need to cooperate, especially what they say right
 here, a strategy that performs well with humans, right?
 And that's the crux right here.
 It's not necessarily about finding the most optimal strategy, even as I understand it,
 the most optimal strategy against humans, it's a strategy performs well with humans
 if you need to cooperate.
 Although in this game, I think you could find like a really good strategy absent of much
 communication.
 Yeah, it says it may converge to a policy that's incompatible with human norms and expectations.
 And that's the human element that I mentioned.
 These norms and expectations, I think that's what makes these games interesting, makes
 these games fun to humans to sort of like, you know, are they telling the truth?
 Are they lying?
 Oh, they betray me?
 How could they betray me?
 Things like this.
 That's what makes it fun, right?
 And I think that's why people play these games.
 And yeah, interest like that.
 As I said, that's the exact aspect that's kind of not modeled in the dialogue model
 right here and in the strategic aspect.
 So that's where a little bit of my my criticism would come from right here.
 But you know, future research.
 So here is a bunch of stats, the average, the agent here sends and receives an average
 of 292 messages per game.
 So this is a very chatty game, the chat is really a big part of the game.
 It's not as much the moves, it's like chat, chat, chat, chat, chat, coordinate, negotiate,
 small talk, I guess, maybe.
 So the challenges they say each message the agent sends must be grounded.
 If they just had like some sort of language model, it would do whatever even if it's trained
 on data of that game.
 However, you have to have a way to control the language model to say language model,
 please transmit this piece of information right here to the other player.
 And we're going to see how they train a language model that does it.
 They say, lastly, diplomacy is a particularly challenging domain because success requires
 building trust with others in an environment that encourages players not to trust anyone.
 Each turn's actions occur simultaneously after non binding private negotiations.
 Again, it encourages players to not trust anyone yet you need to build trust.
 That's the crux, I guess.
 So I've already explained the game in itself.
 Yeah, one thing that I found important was this ability that a unit may support other
 units including those of another player.
 And I think that is one of the mechanics that makes this game, you know, include this aspect
 of cooperation and coordination between players.
 So it might very well be that players who do coordinate, even if they're technically
 enemies who do coordinate for a move or two are better off at the end than had they not
 coordinated.
 So there is a general overview over this agent.
 We're going to look at some parts in more detail, but this is essentially it.
 You have this board state and the history over here.
 This is quite your standard input to a reinforcement learning pipeline.
 So the board state is essentially what's happening right now.
 And the history is what was the move before and before and before that.
 Sometimes that's actually relevant for the game.
 Like in chess, the history plays a place has an influence to some degree, like you can't
 make certain moves twice.
 In Atari games, it has some degree of relevance because if something flies with some velocity,
 you want the history to estimate which direction it flies in.
 Sometimes it's just kind of helps the even if if this is Markovian, sometimes it seems
 to help the algorithms just because humans be humans, I guess.
 And it's not Markovian after all.
 But you can think of that yourself.
 In any case, we get the board state as an input.
 And that goes into different directions, as you can see.
 So the first is this planning module here.
 The planning module is very classic reinforcement learning planning module.
 So we get we go from essentially from the state, we determine a policy for all the players.
 So that is that is what a such a planning module does.
 You can think of it a little bit like the Monte Carlo tree search in Alpha Zero or something
 like this, except now you don't have two players, you have many players.
 So what you want to do is you want to determine a joint action, which means all the players
 move at the same time in this game.
 So one action is going to be what every player is doing.
 And the policy what the policies are essentially the action distribution of all the players.
 Then you want to forward simulate that into a future state and essentially repeat that
 so you plan multiple steps into the future.
 And what you can also do is you can sort of run an improvement algorithm to make your
 policy better against all the other policies and then these policies better and so on.
 So this is very classic, I would say, not even reinforcement learning.
 This is just a very classic sort of policy computing algorithm that you might know from
 game theory papers or something like this.
 The only interesting thing here or the novel thing is that you do get an input from what's
 called here these anchor policies.
 The anchor policies are what keeps the strategy in at a human level.
 And it's a bit tricky to explain just here.
 But essentially, if you let the model just do reinforcement learning, just do sort of
 computational planning up here, you quickly get into a state that's what they explained
 above where the actions become like non human.
 So where the actions, the algorithm thinks they're optimal, but the human would say like,
 that's kind of weird, no human plays like this.
 And I've definitely seen sometimes this video commentator say something like this, like,
 that move is very bot like.
 Now usually, usually in something like chess or so, if you know alpha zero is like 10 times
 as strong as the strongest human, and the bot does something weird, then you're like,
 I guess that's a really good move, we should learn what that move is about.
 Now here, it's a bit more tricky, because the it's it's a lot about, you know, this
 trust element, this human element, there is a value to being more human.
 Even if that means that technically, you deviate from the most optimal optimal action, at least
 that's how the author see it, and that's why they have these anchor policies.
 So that anchor policies are behavior cloning policies.
 So what you do is you take a big data set, I guess here in big data set from from human
 place, and you train a behavior cloning algorithm.
 Behavior cloning essentially means I take one game out, here is a state and an action
 and a state and an action, I just observe past games, how they went.
 And I just train a model that if it's given a certain state is trained to perform the
 same actions as the humans did in that game.
 Yeah, this is sometimes phrased as imitation learning, sometimes phrased as behavior cloning
 has different names, but all about the same ideas.
 And that policy that they call an anchor policy, because it anchors the model to what a human
 would do.
 It's not necessarily the best action, but it's an action that a human would do.
 It's a little bit like a discriminator in an adversarial model.
 So they mix these two things, they always mix the anchor policy with the reinforcement
 learned or with the computed policy in order to get a model that performs both well and
 like humans.
 Yeah, and you can see right here, the anchor policies, those are dialogue conditional.
 So you can see that here, dialogue conditional, because in this database, you obviously not
 only have the state as the board was, but you also have all the chit chat that goes
 on inside of the state, right?
 So you condition this behavior cloning policy, you say, okay, here is how the board looks,
 here are what the humans have communicated, what has the human done?
 And you try to clone that.
 Those are your anchor policies.
 Interestingly enough, up here, you see in this cycle here, there is no notion of any
 of the dialogue.
 So all this planning here happens without the dialogue, whereas I think we might, yes,
 all the planning here happens without the dialogue, except the dialogue comes in via
 the dialogue conditional action model.
 So from here, the dialogue comes into this model, and then that information goes up here.
 But that's very, very indirect.
 It's essentially the only information that the planning has about the action is what
 would a human do in this situation?
 Given this board and this dialogue, right?
 This board and this dialogue, that's the only information that you have about the dialogue.
 You don't have the input dialogue directly.
 And your actions, the actions that you do are not including what dialogue you're going
 to send.
 Here, you see only at the output of this planning module, you have something that you call intents.
 So an intent is essentially a plan to move somewhere.
 So the output of the planning module is here, the output action, what you do, but before
 you do it, or at the same time, before the turn is over, you can also communicate to
 the others.
 So you compute what you want to do based on everything that's happening.
 And then you determine these intents.
 So you say, I think I'm gonna move my I'm gonna move my troop from here to here, and
 they are going to move their troop from here to here.
 And you can encode that as these intents.
 And as I said before, what the language model does is it it takes these intents and it translates
 them into chat messages.
 So based on these intents, you now go and you communicate with the other humans.
 So you can see right here, the message model or the message generation module here gets
 three inputs, the board state as well.
 So it knows what the board looks like.
 Then the current dialogue, like what currently has been discussed.
 So now it's the turn of the agent to say something.
 And from up here, it gets these intents.
 So it knows how does how do things look like?
 What has the other person told me, I guess, like, what's the current status of the chat?
 And what do I want to do next turn?
 And what do I expect the other people to do next turn?
 And from that, the dialogue model then generates message candidates, which go through filters.
 And if they pass the filter, they go into the chat, so the bot answers.
 So here you can see that the bot says something like, Hi, Italy care to work together on this
 one?
 If you support me there, I think we both be able to grow quickly.
 Italy, which is the human in this turn, says, Could you support me into ball into Bulgaria
 in return?
 So now, Austria takes everything into account, what it wants to do, what it thinks Italy
 wants to do based on what's been said and so on.
 And then it says, her thing, I have ordered sir to support three, Serbia support Greece
 to Bulgaria.
 And yeah, so that's how the whole thing works.
 We take in the current state.
 We take in the current dialogue.
 From that, we compute two different things.
 First of all, we compute these anchor policies right here, like what would humans be doing?
 Then with the help of that, we also determine a best action to take, which is this planning
 loop right here.
 Once we have the best action, we generate these intents from that.
 That's just mechanical.
 What do I want to do?
 What do the other people want to do?
 Those are just the policies essentially written out as intense.
 And from that, we generate our messaging, our messages, which are intent conditioned.
 And this happens in multiple steps, as I said, multiple planning loops.
 So what I said before, like the dialogue doesn't come into the planning, it does.
 But as I said, not in like a super direct way.
 The agent cannot decide to strategically tell some other player something like the agent
 can only decide on an action.
 And then the dialogue model is just responsible for communicating that action to the other
 players.
 Right?
 The dialogue model is a central thing here was trained to be controllable via intents.
 So what you want to do is you want to have a dialogue model, I have it somewhere right
 here.
 Here, the dialogue, a message is defined to have intent Z, if Z is the most likely set
 of actions that the sender and recipient will take for both the current turn and several
 future turns.
 So that's how they determine the intent during training.
 So during training, they take a data set, they obviously don't know the plans of the
 people, but they take a data set and they annotate each chat message with what they
 think is the intent.
 And that's this is how how they annotate it.
 So they define the intent as essentially like the plan that results out of this chat message.
 They say we develop techniques to automatically annotate every message in the training set
 with a set of actions corresponding to the message content.
 During training, the dialogue model learned the distribution, this distribution where
 Z represents the intent for data point X and Y.
 So X here is the input, whatever the dialogue model gets as an input, Z is the intent, like
 what the agent thinks the plan of everyone is or what they heuristically determined,
 and then Y is the output.
 Here you can see some of these some examples.
 So in this case, the dialogue model is tested for different intents.
 So on the top, you see a situation and a number of actions.
 It's always the same starting state.
 You can hopefully see that if you compare the pictures a little bit, but the actions
 are different.
 So the agent here is England.
 And you can see, for example, this troop here is, I guess, going here.
 That's the action that England takes or wants to take.
 Over here, it goes over here.
 And over here, it also goes over here, but it even does does a bunch of other things
 in turn.
 And every time you can see that the chat messages that the bot sends now change.
 So I'm not a diplomacy player.
 So all I know is what they tell me.
 So here they say, England convoys an army to Belgium with the support of France and
 Germany while taking Norway in a manner friendly to Russia.
 So we expect these actions to be reflected in the chat messages.
 So to France, it says, Would you mind supporting this EDI to Belgium?
 So it sends since that is its intent to move into Belgium, it asks France, Hey, would you
 like to support me?
 If since wait, the Germans, it also wants the German support.
 So they say, Do you want to support my convoy to Belgium with Italy going aggressive, France
 will fail quickly, and we can make gains of both Russia and France.
 So here you can see a bit of an extended example of this dialogue model.
 To me, it's like a tiny bit unclear where this comes from, because they said that intents
 cover both this turn and turns in the future.
 So it's quite likely that some of what the dialogue model here says is also contained
 in the intent.
 And it's kind of like the dialogue model presents it.
 It's also somewhat likely that the dialogue model just sort of makes makes stuff up because
 it sees the board, right?
 The dialogue model, right?
 Yes.
 The dialogue model as far as I, yeah, the dialogue model sees the board itself, and
 it sees the current intent.
 So it's also quite likely the dialogue model has learned to just look at the board and
 kind of talk to people about the board state as such.
 And I think that's pretty cool.
 It's not only it's not only kind of mindless translating of the simple intents.
 It's not just like, I want your support there.
 Please attack there.
 Please don't do this.
 The conversation it has are surprisingly rich, surprisingly sort of flowery.
 And I'm actually surprised that this is learned from human data, because as far as I know
 online games, like this must be like the friendliest online game I've ever seen.
 People are absolutely nice and polite to each other.
 So it says to Russia, how are you thinking Germany is going to open?
 I may have a shot at Belgium, but I need your help to get into Denmark next year.
 So again, the intent next year, next turn, or next, there's always like three seasons
 to a turn to a year.
 So it asks Russia for help in the future at some point.
 That's pretty cool.
 And if you change the actions that you want to do, then the chat messages change.
 So a clear example of how the chat messages are dependent on what you want to do are controllable.
 And they also measure this and they find that the quality of the chat messages improves
 as well as rated by experts.
 And the sort of test perplexity on a test data set improves once they classify the intents
 behind the actions and not just let like a language model run rampant.
 So here is how they train the dialogue model, the intent, control dialogue model.
 Step one is they train this intent model.
 So this is the model that takes a chat message that it sees and spits out the intent.
 So it spits out what it thinks the chat message wants to convey in terms of like the basic
 moves of the game.
 This is only then used to annotate a bigger data set.
 We've seen this number of times.
 And this seems to be a really cool and nice strategy that you train an intermediate model
 that then helps you to annotate a bigger data set.
 And if you can get some very high quality data for that intermediate model, then you
 can essentially create your own training data on a much larger scale, especially in these
 RL papers.
 This seems to be quite a common thing.
 And yeah, it seems worthy of imitation if you're ever in a situation like this.
 So here we have a dialogue history from a data set on the left hand side, and you can
 see these chat messages right here.
 And the intent model, it, I think it looks at the board state and the history of the
 chat.
 And it is tasked with parsing out the intent.
 And it is trained on a set of what they call truthful situations.
 So they go through the data set, and they heuristically determine when are people telling
 essentially the truth about what they want to do.
 And that's how they train their intent model, they train to predict those things.
 That the intent model essentially takes chat message and outputs well, here is what this
 chat message means in terms of actions.
 Then they go through the data set, and they use the intent model here to annotate the
 whole data set.
 As I said, go through the chats and they say, well, England, this was the chat message,
 they meant to convey this basic action.
 And through these intents, the agent understands the game.
 So these language parts here, they almost like act like a translation pipeline between
 the human world, the natural language world, and something the agent can understand, namely
 this intent world.
 Then they train this dialogue model.
 So the dialogue model gets both the board state and history and the dialogue history.
 And the dialogue model, as I said, understands that this in terms of these intents.
 And once the dialogue model is trained, you can then run inference.
 So you use all of this to do planning.
 From the planning, you get the intents and the intents go into the dialogue model.
 So during training, you get the intents from your annotated data set.
 And during inference, you get the intents from the actual planning algorithm, like the
 planning algorithm tells you, okay, forget the chat history, I have determined also based
 on the chat history, of course, but I have determined that here are the the intents,
 the actions that people are probably going to do.
 And then it gives that to the dialogue model to handle.
 These are obviously a much better prediction of what's actually what people are actually
 planning to do than just the chat history.
 They said we considered other notions of intent during development, such as controlling messages
 to focus on specific subsets of actions, third party actions, or to have a particular tone.
 But I don't think they've included them because it's very, very hard.
 So these intents, they essentially cover sort of the direct what the player and its its
 counterparties want to do out of the game.
 And not like, oh, say this in an angry tone, say this in a hopeful tone or something like
 this.
 That's for future work.
 So going through this, I think we we covered a lot of this thing already.
 Yeah, exactly.
 So Cicero conditions its dialogue on the action that it intends to play for the current turn.
 This choice maximizes Cicero's honesty and its ability to coordinate.
 And they say it sometimes led to out of distribution intents with the intent intended action was
 hostile.
 So since Cicero is always like honest, because it's trained on this kind of truthful subset,
 and it just it just communicates its intent.
 So sometimes it just tells humans like, I'm going to attack you where a real human would
 like either lie or just say nothing at all, because hostile being hostile, but the bot
 has no bot has no like, notion of who this is not socially appropriate.
 So it just knows I need to communicate my intents, which I find quite funny, I think.
 So here is an evaluation.
 If you just use a language model, and you look at dialogue quality and perplexity in
 the data set, you improve quite a lot if you also grounded in the game state.
 And you improve then again, if you grounded in these predicted or annotated intents.
 And that's what this model does right here.
 So now we go through the strategic reasoning part.
 As I said, this is more like the classic, classic planning algorithm rather than something
 very novel, and also doesn't rely on the natural language as much as you would, I guess I would
 have hoped.
 So says Cicero runs a strategic reasoning module that predicts other players policies,
 and also its own, I guess, for the current turn based on the state of the board and the
 shared dialogue, and then chooses a policy for itself for the current turn that responds
 optimally to the other players predicted policy.
 So the input to this, as I said, is the state of the board and the shared dialogue.
 But the output action is just like a policy and the policy is just a distribution of actions.
 What I would want to see is that the policy also includes language actions.
 So here actions in the in the policy, it's purely like oopsie, sorry.
 It's purely, you know, what you saw before, like, I want to go from Belgium to whatever
 other place.
 But I would really love to see that the action set here gets extended by something like tell
 Russia to go to somewhere, right?
 Right now, this is just a consequence of the action I select.
 And the language model is just tasked with communicating this.
 But if this here was an action too, then my planning module could actually reason about
 what it would be best to communicate and to whom in order to achieve my goals.
 And I think that will make it much more interesting.
 Obviously, also much harder, but also much more interesting.
 Yeah, so here they go into saying it requires predicting how humans will play.
 Behavior cloning is a choice.
 However, pure behavioral cloning is brittle, especially since supervised model may learn
 spurious correlations.
 So they have a variant of PIKL.
 It's an iterative algorithm that predicts policies by assuming each player seeks to maximize
 the expected value of their policy and minimize the KL divergence between that policy and
 the behavior cloning policy, which we call the anchor policy.
 So again, they want to maximize their reward by simply being a cold hearted bot.
 And they also want to stay close to what a human would do in order to fit in with the
 humans who actually play a cooperative game with the humans.
 They go a little bit into that here.
 You can see that clearly here is the essentially utility of a policy.
 And here is the KL divergence between your policy and the anchor policy.
 And there is a trade off parameter called lambda that controls how much of which there
 is.
 Interestingly, at some and I think that's later and I have it marked somewhere, but
 I'm going to say it now otherwise I'll forget it.
 Once they do the actual inference, they tone down this lambda quite a bit.
 So they use this in two different settings, ones to like annotate and infer things.
 And then once they select their own action, they tone down this lambda quite a bit.
 So essentially, they're saying like, yeah, we want to be like the humans, but then, you
 know, we really want to win.
 And I think that's what results in some of these like bot like moves that the commentator
 commented.
 And it tells me already again, a little bit that the humans who are playing this game
 probably aren't playing it very optimally.
 Otherwise, it would not be that much necessary to have this lambda up.
 Once you to have this lambda very high, when you infer the human actions, but have it much
 lower, sorry, this hand to have it much lower when the when when you determine your own
 action because you want to win the game, essentially means that the humans could also play a bit
 more optimal and win the game a bit more often.
 Yeah, so we went we went from we went from how can we control the dialogue via the actions
 we plan.
 And now we see the other way around.
 Dialogue conditional planning, oops, that's out of your reach.
 How does the dialogue that happened affect the planning I do?
 Before I said it doesn't much but it does in this indirect way.
 But nevertheless, the dialogue very much affects what the bot wants to do or does.
 So here, the bot is France, blue player, and the opponent here is England, the chat partner
 that it chats currently with is England.
 And here you can see if one message with England says, Yes, I will move out of England if you
 head back to NAO.
 Then the text here says Cicero predicts England will retreat from ENG to NTH 85% of the time
 backs off its own fleet to NAO as agreed and begins to move armies away from the coast.
 However, if England says something like, you've been fighting me all game, sorry, I can't
 trust that you won't stab me.
 Then the actions change.
 Cicero does not back off its fleet, but rather attacks EDI with it and leaves its armies
 at the coast to defend against an attack from England predicting that England will attack
 about 90% of the time.
 And that's just based on the dialogue, right?
 So you can I almost apologize a little bit because I think I feel at the beginning, I
 have sort of understated the importance but you can see how this comes in here.
 So you have two policies that you determine one is just planning.
 The other one is this behavior cloning policy, which is dialogue conditioned.
 So in this case, the system looks at this chat message versus this chat messages.
 And it determines in this behavior cloning policy, what would a human do that has sent
 me this chat message, and that flat that goes into this strategic planning module.
 On the other hand, it determines what would a human do that has said this thing right
 here and that goes into the strategic planning module.
 So the bot adjusts its own action by understanding how humans would behave when they have sent
 a certain chat message.
 Again, this is the this is as far as I understand it, the result of the behavior cloning training
 and not the strategic planning itself.
 So the strategic planning isn't going to be like, well, they said this, but are they saying
 it because they want to convince me of something and therefore I should do this and that?
 Right?
 It's not that it's just like, oh, a human that says this probably attacks me 90, like
 a bunch of times, right?
 So I'm going to adjust my the policy because of this part because of this part right here.
 Because this part here is still kind of the same.
 So that's what they say right here.
 Cicero does not explicitly predict whether a message is deceptive or not, but rather
 replies on PIKL to directly predict the policies of other players.
 And yeah, that being said, the policy of other players isn't just a result from the behavior
 cloning.
 The policy of the other players is also determined via the strategic planning model.
 It's just that the information about the dialogue that goes into the strategic planning comes
 from comes through the behavior cloning part.
 So they go into a little bit of modeling here, you get obviously a lot of cases where you
 need to, I want to almost say improvise a little bit.
 For example, you don't have the private conversations between the other players yet still you have
 to model it somehow, right?
 So it's at various points, they use various methods to sort of infer the strategies of
 the different players.
 They do that iteratively, they say during strategic planning for each player, Cicero
 computes an anchor policy for both itself and the player based on their shared conversation,
 the board state and the recent action history.
 Cicero then ran DIL PIKL, which is their variant of PIKL that not only includes two players,
 but I think is that the variant?
 I think so.
 I think I'm describing the right thing here.
 Oh no, DIL PIKL for the two players is that distributional.
 Okay.
 For the two players in order to predict player J's policy on each iteration, Cicero assumed
 the five remaining player will play according to a policy computed via RL.
 So since you don't have the dialogue, you don't have the behavior cloning policy because
 that relies on the dialogue.
 Therefore you need to compute some policy via reinforcement learning to just approximate
 a policy.
 Conditional on the policy of Cicero on player J, this process gave an independent prediction
 of each player's policy.
 Next Cicero accounted for the fact that the player's policies were not independent due
 to their ability to correlate their actions with private dialogue.
 So they adjust it by the likelihood ratio of A under the correlated and independent
 RL policies.
 So there's a lot of adjustment happening for the fact they don't have all the information.
 You'll find this commonly in RL algorithms that where there's some hidden information
 and even in some where there isn't hidden information, but that don't sample uniformly.
 It's a bit of a same concept.
 And finally Cicero chose or chooses the action that best corresponds to the predicted joint
 policy of all the other players.
 The minus I here means the I of player isn't meant while still being as consistent as possible
 with its dialogue.
 And here is what I said.
 Cicero uses a smaller lambda for regularizing its best response than for its computation
 of the other players policies.
 It's kind of like, yeah, I want to be like a human, but I really, I really want to win.
 So this they say this allows Cicero more leeway to deviate when the action predicted humans
 would most likely choose in its situation was suboptimal, which I guess tends to be
 quite or at least sometimes.
 Yeah, so then they go into how they use self play reinforcement learning in that.
 So they run this in an iterative fashion, they not only do it once, so they run it in
 an iterative fashion, they compute optimal policies, go around, do it again, again, so
 on.
 I don't want to go too much into that.
 If you want to read it, it's a it's a short paragraph and as a bit of a supplementary,
 so that the supplementary material is quite huge.
 So props for releasing a lot of that.
 Lastly, they have this paragraph on message filtering, which is a last step where they
 boost the the performance and the way the quality rated by experts of these models,
 again, by quite a lot.
 They say neural language models suffer from contradictions, inconsistencies, as well as
 a tendency to hallucinate or to generate factually incorrect information.
 They say their model obviously does the same deviates from the intent and use that used
 to control the message.
 It blunders in the strategic content of the message.
 We approach this problem by filtering generated message using a series of classifiers and
 checks to detect common issues.
 As is essentially post processing of their message model.
 So they sample and if they doesn't pass the filters, I guess they just sample again.
 By the way, are these are these here intended?
 These references?
 I'm not exactly sure.
 In any case, they say discriminating between human text and counterfactuals.
 So here we go into the question, what, how can we filter out kind of garbage if the data
 set that we have is all generated by humans and therefore we have to assume that it's
 at least somewhat sensible.
 So you just create your own garbage.
 They say we generated many kinds of counterfactual messages that contain mistakes language models
 are prone to, including heuristically corrupted text, as well as model generated negatives.
 We trained a suite of 16 classifiers to discriminate between the ground truth, human message and
 different kinds of counterfactual messages.
 So essentially just train classifiers that can differentiate their created garbage from
 regular human messages.
 And they hope that they have gotten close enough to the common mistakes that language
 models make and also that they've captured enough of those mistakes in their heuristics
 such that the classifiers will get will generalize essentially and just generally filter out
 most non-human text.
 This is also interesting.
 They said we filtered messages that would reduce the likelihood of the actions in the
 intent.
 Yeah, so they can determine from the message they would send, like what, how can we classify
 the intent because they have the model that takes a chat message and then classifies the
 intent or even they can take that chat message and feed it back into their planning algorithm
 and essentially say, well, does that does that does that make it more or less likely
 that I'm going to do the actions that I want to communicate if it makes it less likely
 they determine probably it's not saying what I want it to say and they throw it away.
 Then their their goal or their their design here is such that the language model is like
 extremely honest about what it wants to do and they counter it with this next thing.
 This is the only place where they sort of like where they counter this tendency to be
 like this super duper honest.
 They say conditioning on intents can lead to information leakage where an agent reveals
 compromising information about its plan to an adversary.
 To mitigate this, we developed a method to score potential messages based on their estimated
 value impact.
 We computed the PIKL policies for all agents after each candidate message and filter those
 that led to lower expected value for Cicero playing its intended action.
 So I didn't discuss this explicitly, but they have a value function and the value computation
 method.
 So they run this planning algorithm forward, they can see into the future and they can
 determine the value of the game for the player much like AlphaZero or AlphaGo or something
 like this.
 And now they take the chat message that they want to send and they determine is this even
 good for me down the road if I send this message.
 And if it turns out it's probably not that good for me if I send this message, then they
 don't send it.
 So that's a little bit of a counter to just being fully open and just communicating whatever
 you're going to do to everyone, which is not always the best thing in this game.
 So they have a bunch of other filters they say here, if you want to check them out there
 in the supplementary material.
 And last thing they say is how they participated in human play.
 So they played a bunch of online tournaments without telling the humans that it's a bot.
 And I found this I found this quite interesting.
 The website notifies users that the website has participated in AI research and that certain
 game modes allow users to play with AI agents.
 But in these games, the humans were not explicitly informed that they were playing with an AI
 agent for that particular game.
 Cicero's participation as an AI was revealed to all players after the conclusion of the
 research.
 I've seen actually a message by one of these players, and that person was completely flabbergasted.
 They were like, I got the email and I'm like, what?
 That was an AI?
 No way.
 I like so the the model is quite good.
 But I can't help but notice that that this is an experiment on human subjects and really,
 really needed to go through an ethics review board.
 And I was under the impression that it's extremely terrible to let people interact with a bot
 and not tell them with every message explicitly that it is a bot.
 And I don't want to draw false equivalences here.
 This is very cool research and in no way do I think anyone was in danger by not knowing
 that this was a bot.
 So that was the the paper.
 They have a bit of a discussion down here and a bit of more examples.
 So here they have a bunch of successful dialogue examples on the left where they coordinate
 so Cicero is Austria, Italy, Italy says something like what are you thinking long term?
 Should I go for Turkey or head west?
 And you can see just I mean, if you read this dialogue, oh, sorry, if you read this dialogue,
 you can see how like it's it's not just like blah, I communicate the intent very plainly,
 but it really reacts to the other players.
 It really talks about them about also longer term strategy, it refers to states, things
 that are on the board correctly, and refers to its plans a few turns in ahead correctly
 and so on.
 So here, Italy, or Austria says something that convinces Italy to go to, I don't know,
 Turkey or beat Turkey.
 Italy says I'm down to go for it would you would definitely need your help in supporting
 me and Austria says of course happy to do that fantastic.
 On the other hand, here's an example of negotiation.
 France is Cicero.
 France says I'll work with you but I need Tunis for now.
 Turkey says nope, you got to let me have it and France says no, I need it.
 You have Serbia and Rome to take their impossible targets.
 And then France suggests a series of moves and Turkey says, you're right.
 Good ideas.
 So I'm again, I'm not I'm not sure that the humans here.
 Maybe that particular human, I'm not I'm not sure.
 I've never played this game.
 So I can't tell if this is actually something that that happens at a high level of play
 still that someone suggests a series of moves to you.
 And you're like, Oh, yeah, that that is a good idea.
 I'm pretty sure like really good players consider all of the things already.
 But yeah.
 In any case, I think I still think it's like really, really cool research.
 Here, they say although Cicero is shown to be effective at cooperating with humans, it
 occasionally sends messages that contained grounding errors contradicted its plans or
 were otherwise strategically subpar.
 But they say, well, essentially, humans occasionally make similar mistakes, which is probably an
 understatement like humans are chaotic and, and dumb.
 And Cicero is probably like the most honest, the most like consistent player in the entire
 world at this game.
 From a strategic perspective, Cicero reasoned about dialogue purely in terms of players
 actions for the current turn, it did not model how its dialogue might affect the relationship
 with other players over the long term course of a game, considering this might allow it
 to deploy dialogue more strategically.
 The expressive power of our intent representation limited Cicero's ability to control richer
 affordances of dialogue such as strategically revealing information, asking questions, or
 providing explanations for its actions.
 And that is exactly the the kind of thing I said at the start.
 It's a really cool research to show that you can actually pair language models with these
 things and and interact with humans in this way.
 However, the language models here, they more in they more act as like a translation engine
 between just what the planning spits out, or what the planning needs as an input, rather
 than as sort of actions to be taken by itself.
 And I would really see the continuation of this work, where the model also considers
 kind of like its own dialogue as actions.
 It's not going to be it's not going to be super easy, I want to guess, to to do that.
 Especially also because yeah, as my suspicion is still that humans here are far from the
 optimal strategy.
 And therefore, the whole balance between behavior cloning and training on this human data set
 and actually making moves might be quite far apart.
 And I'm not sure how to reconcile that best.
 It might also be that the humans through this bot come to learn that actually, there's probably
 better strategies around which has happened in like Go and chess and poker so far.
 So I'm excited to see what the future brings.
 Definitely recommend to check out the YouTube video by the commentator has a lot of gems
 in there and a lot of things where you can kind of see the effects that the bot training
 has had.
 They also say, well, yeah, the bot is quite honest, for one, and also the bot is quite
 like non emotional.
 So even if you stab it in the back, it would be like not mad at you, it would still be
 completely rational and things like this.
 And to me, that's it's it's very cool to see that even in such a game, the human element
 seems to be sort of the primary fun maker, even at a high level of play.
 And yeah, I think that's, that's I think the best message we get out of this research.
 Alright, I hope you enjoyed this paper review.
 Wish you a very pleasant evening, and I'll see you around.
 Bye bye.